xiangfu0 opened a new pull request, #19300:
URL: https://github.com/apache/pinot/pull/19300
Part 4/5 of the split of #19287 (Fix FULL-upsert vector candidate
generation).
This is the core fix.
## Summary
`FilterPlanNode` constructed and executed `VECTOR_SIMILARITY` before adding
`SegmentContext.getDocIdsSnapshot()` as an outer AND. Obsolete physical
versions of
upserted rows could therefore occupy per-segment ANN top-K slots and be
removed only
afterward, producing fewer than K rows or omitting nearer current rows.
- Treat the upsert snapshot as a mandatory candidate-generation scope
(`VectorCandidateScope`) wired into `VectorSimilarityFilterOperator` and
`ExactVectorScanFilterOperator`, while retaining the final outer bitmap
AND as defense
in depth.
- Keep required upsert scopes separate from optimizer-selected metadata
filters; when
both exist their private copies are intersected before candidate
generation.
- Use filtered ANN when the reader supports it; otherwise scan exact
distances only over
allowed document IDs via the shared `computeExactMatches` helper, and fail
clearly when
no safe path exists. Skip candidate generation entirely when the effective
allowed set
is empty.
- Preserve adaptive metadata behavior for ordinary non-upsert queries and
avoid vector
bitmap allocation for non-vector plans.
- Add explain attributes for candidate-filter cardinality, execution mode,
skipped
generation, and fallback reason.
## Performance and compatibility
ANN approximation semantics are unchanged. Readers that cannot honor the
mandatory
bitmap use a correctness-first exact scan whose cost is proportional to the
allowed-document count times vector dimension; in particular, consuming
(mutable)
segments of upsert tables take this path today because `MutableVectorIndex`
is not
filter-aware. Making the mutable index filter-aware would retire that
fallback and can
be optimized separately.
## Validation
All commands ran with JDK 25.
- Core vector/filter suite: 102 tests passed
(`FilterPlanNodeTest`, `VectorSimilarityFilterOperatorTest`,
`FilterAwareVectorSearchTest`, `ExactVectorScanFilterOperatorTest`,
`VectorRadiusFilterOperatorTest`, `VectorSearchStrategyTest`).
- Spotless, Checkstyle, and license checks passed.
> Stacked on #19299 — this PR shows the earlier stack commits until
predecessors merge; review only the last commit here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
