LuciferYang opened a new issue, #11023: URL: https://github.com/apache/paimon/issues/11023
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (b000c3945f) ### Compute Engine Python ### Minimal reproduce step Write rows `a = 0..9` in one commit to an append table (parquet, orc or avro), then read with a filter and a shard, or with a filter and `_ROW_ID` on a row-tracking table: ```python rb = table.new_read_builder().with_filter(pb.is_in('a', [2, 8])) rb.new_read().to_arrow(rb.new_scan().with_shard(0, 2).plan().splits()) # a = [2, 8] rb.new_read().to_arrow(rb.new_scan().with_shard(1, 2).plan().splits()) # a = [] rb.new_read().to_arrow(rb.new_scan().with_slice(0, 5).plan().splits()) # a = [2, 8] # 'row-tracking.enabled' = 'true' rb = table.new_read_builder().with_filter(pb.greater_or_equal('a', 5)).with_projection(['a', '_ROW_ID']) # a = [5, 6, 7, 8, 9], _ROW_ID = [0, 1, 2, 3, 4] ``` ### What doesn't meet your expectations? Shard 0 should return `[2]` and shard 1 `[8]`; the slice `[0, 5)` should return `[2]`; the `_ROW_ID`s should be `[5, 6, 7, 8, 9]`. A shard boundary can fall inside a file. `ShardBatchReader` then keeps the rows of its range by counting the rows the format reader returns, and `_ROW_ID` is assigned by the same count. The format reader has already applied the pushed-down filter, so both count only the rows that passed it: rows end up in the wrong shard, a slice returns rows outside it, and `_ROW_ID`s start again from 0. ### Anything else? Every row still comes back from exactly one shard, so a full sharded scan returns the right set of rows; only the assignment to shards is wrong. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
