JingsongLi opened a new pull request, #8664:
URL: https://github.com/apache/paimon/pull/8664
## What changed
- Pass the residual data filter from `SparkVectorSearchBuilderImpl` to
`SparkPrimaryKeyVectorRead`.
- Add a predicate-aware Spark primary-key vector reader constructor while
preserving the existing constructor.
- Add a distributed Spark regression test that verifies filtering happens
before Top-K and the next nearest neighbor is recovered.
## Why
The distributed primary-key vector path dropped the residual filter when
constructing its reader. It selected Top-K from unfiltered candidates and only
applied the Spark filter afterward, so filtered candidates could not be
replaced and fewer than K correct neighbors were returned.
## Validation
```text
mvn -pl paimon-spark/paimon-spark-ut -am -Pspark3 \
-DfailIfNoTests=false \
-DwildcardSuites=org.apache.paimon.spark.sql.PrimaryKeyVectorSearchTest \
-Dtest=none test
```
All 10 tests passed, including the new distributed residual-filter
regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
