weijietong commented on PR #9705:
URL: https://github.com/apache/paimon/pull/9705#issuecomment-5611671755
The critical RowRange use case is slicing the stream after a filter — e.g.
"give me the 1000th–2000th matching rows for training." Even composed with a
filter, IndexedSplit only ANDs two physical-position bitmaps; it never
renumbers rows into a
post-filter effective sequence, so it cannot express "the N-th row after
filtering."
This is exactly what AbstractDataTableRead.outerWrap encodes:
- RowRange + filter (executeFilter): outerWrap=true — RowRange wraps
outside the filter, slicing the filtered effective stream.
- IndexedSplit + filter: both reduce to physical-position bitmaps ANDed
together — no "post-filter renumbering" step.
The AI training-shard use case (why RowRange exists)
In AI / ML data loading, a common pattern is sharding a dataset by
effective sample position:
- A training job with worldSize=W, rank=R wants its local shard — the rows
at effective positions [R*N, (R+1)*N - 1] of the dataset.
- With filters active (e.g. quality filters, column predicates) or
deletion vectors, the "effective row order" is the filtered/compacted sequence.
Each worker must receive a deterministic, contiguous slice of that sequence,
not of the raw file layout.
- RowRange models exactly this: a single contiguous [start, end] in the
0-based effective-row space of a split. The reader either pushes it down as a
selection bitmap (parquet, full-scan — no extra rows decoded) or wraps a single
RangeSkipReader over
the filtered output (filter/DV/ORC/merge-tree) to skip start and take
count.
IndexedSplit cannot express this:
- It cuts physical row-ids, which have no relation to the post-filter
effective order — worker rank=R wouldn't get a contiguous slice of matching
rows.
- It requires firstRowId (so plain append tables are out) or a pre-built
global index.
- For merge-tree primary-key tables, physical position ≠ output order, so
the shard would be scrambled.
Conclusion
IndexedSplit and RowRange are complementary, not interchangeable:
- IndexedSplit: read a discrete set of physical row-ids (vector search /
global-index hits), AND-composed with filters in the physical-position space,
requires row-ids or a pre-built index, excludes append tables / post-filter
slicing / merge-output
slicing.
- RowRange: read a contiguous slice in effective-row space (AI training
shards / pagination / range queries), applied as skip+limit over the filtered
effective stream, works across append / primary-key / DE tables, no pre-built
index needed.
RowRange was introduced precisely to cover IndexedSplit's blind spots —
append tables, post-filter effective-row slicing, and merge-output slicing —
all validated by tests in this PR (testAppendOrcRowRange...,
testRowRangeWithFilterWrapsOutsideFilter,
testPrimaryKeyRowRangeOverMergeRead). They cover behavior IndexedSplit
cannot express, so RowRange cannot be replaced by it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
