JingsongLi commented on code in PR #10771:
URL: https://github.com/apache/paimon/pull/10771#discussion_r4491624083
##########
paimon-eslib/src/main/java/org/apache/paimon/eslib/index/ESIndexGlobalIndexerFactory.java:
##########
@@ -43,6 +43,11 @@ public boolean supportsFullTextSearch() {
return true;
}
+ @Override
+ public boolean supportsGroupedIndexScalarQueries() {
+ return true;
Review Comment:
[P2] Initialize distributed Spark readers for the grouped definition too
This opt-in makes filtered grouped queries retain ANN splits, but Spark's
distributed worker still creates its indexer from
`Collections.singletonList(vectorColumn)` in `SparkDataEvolutionVectorRead`,
bypassing the repaired local per-definition path. I reproduced it with 32 real
rows, a grouped `[v, category]` ES index, 2 rows per shard,
`global-index.thread-num=2` and `vector-search.distribute.enabled=true`:
`vector_search(..., 1) WHERE category='keep'` fails in a Spark task with `ES
index field count does not match current schema: indexed=2, current=1`. The
same SQL returns row 0 when only this capability is restored to false (the
parent behavior), and passes when the worker uses the persisted definition.
Once indexed splits reach `2 * parallelism`, these supported filtered queries
become unusable. Apply type/ordered-field-definition reader selection in the
distributed tasks as well, and add a real task regression beyond that threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]