LuciferYang opened a new issue, #11023:
URL: https://github.com/apache/paimon/issues/11023

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (b000c3945f)
   
   ### Compute Engine
   
   Python
   
   ### Minimal reproduce step
   
   Write rows `a = 0..9` in one commit to an append table (parquet, orc or 
avro), then read with a filter and a shard, or with a filter and `_ROW_ID` on a 
row-tracking table:
   
   ```python
   rb = table.new_read_builder().with_filter(pb.is_in('a', [2, 8]))
   rb.new_read().to_arrow(rb.new_scan().with_shard(0, 2).plan().splits())  # a 
= [2, 8]
   rb.new_read().to_arrow(rb.new_scan().with_shard(1, 2).plan().splits())  # a 
= []
   rb.new_read().to_arrow(rb.new_scan().with_slice(0, 5).plan().splits())  # a 
= [2, 8]
   
   # 'row-tracking.enabled' = 'true'
   rb = table.new_read_builder().with_filter(pb.greater_or_equal('a', 
5)).with_projection(['a', '_ROW_ID'])
   # a = [5, 6, 7, 8, 9], _ROW_ID = [0, 1, 2, 3, 4]
   ```
   
   ### What doesn't meet your expectations?
   
   Shard 0 should return `[2]` and shard 1 `[8]`; the slice `[0, 5)` should 
return `[2]`; the `_ROW_ID`s should be `[5, 6, 7, 8, 9]`.
   
   A shard boundary can fall inside a file. `ShardBatchReader` then keeps the 
rows of its range by counting the rows the format reader returns, and `_ROW_ID` 
is assigned by the same count. The format reader has already applied the 
pushed-down filter, so both count only the rows that passed it: rows end up in 
the wrong shard, a slice returns rows outside it, and `_ROW_ID`s start again 
from 0.
   
   ### Anything else?
   
   Every row still comes back from exactly one shard, so a full sharded scan 
returns the right set of rows; only the assignment to shards is wrong.
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to