Short version, since my original was too long - thanks to the reviewer on
the PR for saying so.
Problem Spark's vectorized reader asks the position delete index
"is row N deleted?" once per row - 5,000 times per batch -
while those positions are a contiguous ascending range. It
re-resolves the same Roaring container every time.
Change Pass the range once. PositionDeleteIndex gets a
forEachInRange default method; the default implementation is
the current per-row loop, so nothing external has to change.
ColumnarBatchUtil takes it only when there are no equality
deletes.
Effect 2.6-9.3x less delete-check CPU. End to end, 13-19% on narrow
all-integer projections, up to 29% for a single column on a
dedicated instance, and inside my noise floor at 10+ columns.
One rule: it pays off than ~21%
of scan CPU.
Links PR https://github.com/apache/iceberg/pull/18027
Data https://github.com/apache/iceberg/issues/18026
The part I would most like opinions on is the interface addition itself -
whether growing PositionDeleteIndex with another default method is the
right call, or whether the traversre.
One correction to how this is easy to read: more deletes is not more
benefit. The largest gain is just below 6.25%, where the Roaring container
is still a sorted array at its deepest; past that it flips to a bitmap and
gets cheaper on its own.
>
