huwh opened a new issue, #9002: URL: https://github.com/apache/paimon/issues/9002
### Search before asking - [x] I searched existing issues and pull requests and found nothing similar. ### Paimon version Current master at `b35e19f38d055267712b5fc13d57f45e219970d9`. ### Compute Engine Spark 3.5.1. The root cause is in the engine-independent Vortex file reader. ### Minimal reproduce step 1. Create an append-only Paimon table whose main data and dedicated vector data are stored separately, with the vector field using Vortex. 2. Write enough data for a Vortex file scan to be split into multiple internal scan partitions. 3. Read rows or build/search a vector index that joins the main file and dedicated vector file by `FileRecordIterator.returnedPosition()`. 4. When Vortex completes scan partitions out of order, compare the vector values with the main-file rows at the reported positions. ### What doesn't meet your expectations? `VortexRecordsReader` advances `returnedPosition` monotonically, so it assumes that scan results preserve physical row order. Since #8040, the reader uses the Vortex `DataSource` / `Scan` API, while `ScanOptions.ordered()` defaults to `false`. An unordered scan may emit physical rows in partition-completion order while Paimon still reports positions `0, 1, 2, ...`. This can attach dedicated Vortex vector values to the wrong main-file rows and return incorrect vector-search results. The scan should explicitly preserve physical row order whenever Paimon derives row positions this way. ### Anything else? Proposed fix: set `ordered(true)` on the Vortex scan options and add a deterministic unit test for this scan contract. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
