LuciferYang opened a new pull request, #11024:
URL: https://github.com/apache/paimon/pull/11024

   ### Purpose
   
   close #11023
   
   With a filter, `with_shard` and `with_slice` return rows that belong to 
another shard, and a filtered read of `_ROW_ID` on a row-tracking table returns 
renumbered `_ROW_ID`s. The issue has the repro. Both count the rows the format 
reader returns, and the format reader has already applied the pushed-down 
filter.
   
   `file_reader_supplier` no longer pushes the filter into the format reader 
when it reads a shard range or the read assigns `_ROW_ID`, as it already does 
for deletion vectors and physical row ranges. `RawFileSplitRead` applies the 
filter to those reads after the rows are selected. Reads without a shard and 
without `_ROW_ID` still push the filter down.
   
   #10723 changes the same condition in `create_reader` for Vortex files; it 
does not touch shard or `_ROW_ID` reads.
   
   ### Tests
   
   `read_predicate_positions_test.py`, all failing on master:
   
   - For parquet, orc and avro, shard 0 and shard 1 of a two-shard scan with an 
`IN` filter each return only their own row, `with_slice(0, 5)` returns only the 
row inside the slice, and a filtered read of `_ROW_ID` on a row-tracking table 
returns the rows' own ids.
   - For vortex, a sharded filtered read of a row-tracking table returns the 
right rows and `_ROW_ID`s in each shard.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to