TheR1sing3un opened a new pull request, #9999: URL: https://github.com/apache/paimon/pull/9999
### Purpose Prevent candidate-only scalar index answers from admitting non-matching rows into vector top-k. For example, with a BTree name index and an IVF vector index, `name LIKE '%zeta%'` can currently return the nearer `alpha` row instead of `beta zeta`. Filtering after top-k cannot recover the displaced match. Carry scalar-result exactness through offsets, unions, intersections, coverage padding and conjunctions that drop an unsupported child. BTree prefix/substring answers are candidates; exact BTree equality/range results and exact bitmap string matches retain their existing path without a data recheck. Tracking the result also handles partially evaluable conditions on the same field, where contributing field IDs alone are insufficient. Use the `global-index.filter.refine-from-data` table option introduced by #9953, with the same default of `false`. Inexact index candidates are excluded with a warning by default. When enabled, read the candidate filter columns at the pinned snapshot to build an exact include bitmap **before** ANN top-k. This read is streamed, applies deletion handling, and does not project candidate vectors. Completely unsupported scalar evaluations can also use this opt-in verification. Local/Ray and single/batch vector execution share this path. Related to #9883. This change covers Python data-evolution vector search; full-text filtering remains unchanged. ### Tests - New native-index and resource-lifetime tests: 37 passed, including all local/Ray, single/batch and full/fast combinations of the refinement option. - The same 37 new tests also passed on Ray 2.44 / NumPy 1.24 / Arrow 18 (primary environment: Ray 2.54 / NumPy 2.4 / Arrow 19). - Existing scalar-index, sorted-index, full-text/vector filtering and Ray single/batch search/refinement regressions: 246 passed. - Tests cover BTree prefix/substring candidates, exact BTree and bitmap fast paths, unsupported predicates on the same field, result-wrapper propagation, historical snapshots/tags and deletions, a commit before candidate verification, projected columns and stream cleanup. - Flake8, Python 3.6 syntax parsing and `git diff --check` passed for changed files. Native index tests used paimon-vindex 0.4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
