LuciferYang opened a new pull request, #11024: URL: https://github.com/apache/paimon/pull/11024
### Purpose close #11023 With a filter, `with_shard` and `with_slice` return rows that belong to another shard, and a filtered read of `_ROW_ID` on a row-tracking table returns renumbered `_ROW_ID`s. The issue has the repro. Both count the rows the format reader returns, and the format reader has already applied the pushed-down filter. `file_reader_supplier` no longer pushes the filter into the format reader when it reads a shard range or the read assigns `_ROW_ID`, as it already does for deletion vectors and physical row ranges. `RawFileSplitRead` applies the filter to those reads after the rows are selected. Reads without a shard and without `_ROW_ID` still push the filter down. #10723 changes the same condition in `create_reader` for Vortex files; it does not touch shard or `_ROW_ID` reads. ### Tests `read_predicate_positions_test.py`, all failing on master: - For parquet, orc and avro, shard 0 and shard 1 of a two-shard scan with an `IN` filter each return only their own row, `with_slice(0, 5)` returns only the row inside the slice, and a filtered read of `_ROW_ID` on a row-tracking table returns the rows' own ids. - For vortex, a sharded filtered read of a row-tracking table returns the right rows and `_ROW_ID`s in each shard. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
