JingsongLi commented on code in PR #10771:
URL: https://github.com/apache/paimon/pull/10771#discussion_r4374056639
##########
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexQuery.java:
##########
@@ -330,8 +330,9 @@ private static GlobalIndexQuery createForIndexedField(
}
List<IndexGroup> selectedGroups = new ArrayList<>();
for (IndexGroup group : fieldGroups) {
- // Scalar lookups use independent single-field index definitions.
- if (group.extraFields.isEmpty()) {
+ // Composite-key indexes require their own planner, not
independent field lookups.
+ if (group.extraFields.isEmpty()
+ ||
GlobalIndexerFactoryUtils.supportsGroupedIndexScalarQueries(group.type)) {
Review Comment:
[P1] Refine conservative ES candidates before search ranking
Enabling grouped ES here also admits predicates for which the existing
reader returns an unlabeled superset. For `category > 'c1'` on a keyword field,
`canEvaluateFilter` returns false and `dispatchFilter` returns
`conservativeAllRows()`. That nonempty Optional is credited with full evaluated
coverage and the category field ID, so `FilteredRowIdReader.isExact` treats the
single GreaterThan as exact and skips refinement before Top-K.
I reproduced this with this PR's 12-row ES table fixture and real
`(embedding,category,price)` / `(note,category,price)` archives, using FULL
mode and refine-from-data=true. The data oracle matches IDs {2,8,11}; a
vector(5), limit=1 query returns ID 5 (null category) instead of ID 2.
Full-text `keep` should return {2,8}, but returns {0,2,4,6,8,10}. Using the
parent ES factory, which excludes grouped scalar lookups, restores both
expected results.
Please make unsupported reader operations return unsupported, or propagate
candidate/exact status through evaluation and force refinement/exclusion before
ranking. Field IDs and covered row ranges alone do not prove that the complete
predicate was evaluated exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
