VaibhaveS commented on PR #12350:
URL: https://github.com/apache/arrow-rs/pull/12350#issuecomment-7066650153

   Thanks for the review, @Mola-maker! I've pushed changes, and updated the 
description:
   
   1. **Pages.** I reproduced this: with the default properties, 
`write_with_selection` wrote 20,480-row pages where filter + write wrote 
20,000-row pages. `filter_record_batch` drops a column's validity buffer when 
no selected row is null, so `write` stores the levels as a run, and the column 
writer then fills pages up to `data_page_row_count_limit`. 
`write_with_selection` now also writes the levels of selected rows that are all 
valid or all null as a run. The exception is a dictionary whose values have 
nulls, because `filter` keeps a dictionary's values, and with them its validity 
buffer. `write_with_selection_matches_filter_and_write_pages` covers your case 
with the default properties.
   
      One case can still differ, and the docs say so: one call writes rows to 
more than one row group, and only some of those row groups have nulls in a 
column. `filter` keeps the validity buffer for every part of that column, so 
`write` stores the levels of a part without nulls one per row. Matching that 
would mean either copying more of `filter`'s null handling, or making `write` 
choose how to store levels from the nulls of the rows it writes. The second 
would change `write`'s pages for some inputs, so it would be a separate PR if 
maintainers want it.
   
   2. **Error index.** The error gives the index of the null in `batch`, which 
is the batch the caller has. I changed the docs to "fails in the same cases as 
`write` for the filtered batch", and the test now checks the index.
   
   3. **`map` change.** I moved it to #12392, with its own benchmarks. Without 
it, `write_with_selection` is slower than filter + write when 99% of rows are 
skipped, because it converts the `Decimal128` values of all rows. The 
description has the numbers.
   
   4. **CDC.** I filed #12391, with reproductions on `main`. Sliced lists, and 
fixed-size lists with null rows, get different pages for the same data, and 
list views panic.
   
   5. **Scope.** The async writer and a selection-aware `compute_leaves` can 
follow in separate PRs. I used `RowSelection` because it is the crate's 
existing type for selecting rows, and it accepts runs as well as a bitmap. 
`RowSelection::from_boolean_buffer` keeps the bitmap without copying it, and 
`write_with_selection` uses that bitmap as it is. Only a selection made of runs 
is converted to a bitmap, once per call. I'm happy to switch to `&BooleanArray` 
or `&BooleanBuffer` if maintainers prefer. As for a docs note on the 99% case, 
the difference there comes from the conversion that #12392 removes, so I'd 
rather fix it than document it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to