VaibhaveS commented on PR #12350:
URL: https://github.com/apache/arrow-rs/pull/12350#issuecomment-7066650153
Thanks for the review, @Mola-maker! I've pushed changes, and updated the
description:
1. **Pages.** I reproduced this: with the default properties,
`write_with_selection` wrote 20,480-row pages where filter + write wrote
20,000-row pages. `filter_record_batch` drops a column's validity buffer when
no selected row is null, so `write` stores the levels as a run, and the column
writer then fills pages up to `data_page_row_count_limit`.
`write_with_selection` now also writes the levels of selected rows that are all
valid or all null as a run. The exception is a dictionary whose values have
nulls, because `filter` keeps a dictionary's values, and with them its validity
buffer. `write_with_selection_matches_filter_and_write_pages` covers your case
with the default properties.
One case can still differ, and the docs say so: one call writes rows to
more than one row group, and only some of those row groups have nulls in a
column. `filter` keeps the validity buffer for every part of that column, so
`write` stores the levels of a part without nulls one per row. Matching that
would mean either copying more of `filter`'s null handling, or making `write`
choose how to store levels from the nulls of the rows it writes. The second
would change `write`'s pages for some inputs, so it would be a separate PR if
maintainers want it.
2. **Error index.** The error gives the index of the null in `batch`, which
is the batch the caller has. I changed the docs to "fails in the same cases as
`write` for the filtered batch", and the test now checks the index.
3. **`map` change.** I moved it to #12392, with its own benchmarks. Without
it, `write_with_selection` is slower than filter + write when 99% of rows are
skipped, because it converts the `Decimal128` values of all rows. The
description has the numbers.
4. **CDC.** I filed #12391, with reproductions on `main`. Sliced lists, and
fixed-size lists with null rows, get different pages for the same data, and
list views panic.
5. **Scope.** The async writer and a selection-aware `compute_leaves` can
follow in separate PRs. I used `RowSelection` because it is the crate's
existing type for selecting rows, and it accepts runs as well as a bitmap.
`RowSelection::from_boolean_buffer` keeps the bitmap without copying it, and
`write_with_selection` uses that bitmap as it is. Only a selection made of runs
is converted to a bitmap, once per call. I'm happy to switch to `&BooleanArray`
or `&BooleanBuffer` if maintainers prefer. As for a docs note on the 99% case,
the difference there comes from the conversion that #12392 removes, so I'd
rather fix it than document it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]