Mola-maker commented on PR #12350:
URL: https://github.com/apache/arrow-rs/pull/12350#issuecomment-7031528903

   Hey,I read through the diff against #12187 and found a few gaps between what 
the issue asks for and what the PR delivers.
   
   **1. Output does not match `filter_record_batch` + `write` under default 
settings**
   
   #12187 asks for the same rows, statistics, page indexes and row counts. The 
doc comment on `write_with_selection` already says nullable columns can 
paginate differently. The tests only show byte equality after setting 
`write_batch_size` equal to `data_page_row_count_limit`, or with batches too 
small to fill a page. With default `WriterProperties` and a 200k-row nullable 
`Int32` column (nulls only in skipped rows, every other row selected), the two 
files differ:
   
   - default props: selection writes pages of 20480 values, filter + write 
writes pages of 20000
   - `data_page_row_count_limit(1000)`: 98 pages vs 100 pages
   
   Skipping the null rows is probably the most common use of this API, so this 
case will come up often. The page boundaries, and therefore the offset index, 
differ from what the issue specifies. Should the selected path compute levels 
as if the selected rows were the whole array, so a leaf with no selected nulls 
is treated as having no nulls? If not, the docs and #12187 should say plainly 
that only rows and statistics are guaranteed to match, and pages are not.
   
   **2. Error indices for required fields**
   
   The docs say this "fails as `Self::write` would for the filtered batch". But 
`reject_null` reports the row index in the original batch, while `write` on the 
filtered batch reports the index in the filtered batch. The test only checks 
for `for required field 'c'`. Either the docs or the index should change.
   
   **3. Unrelated change to `write`**
   
   The `map` change (converting only non-null values, and passing no value 
indices) also changes the existing `write` path for 
`Int8`/`Int16`/unsigned/`Date64`/decimal columns. It looks like an independent 
optimization. Could it go in a separate PR with its own benchmark, so this one 
only adds the new API?
   
   **4. Content-defined chunking difference is documented but not tracked**
   
   The CDC test drops the fixed-size-list, list-view and (with row group 
splitting) list columns. The comment says `write` hashes the wrong values for 
them today. If that is a bug in the existing `write` + CDC path, it should have 
its own issue. As it stands, `write_with_selection` and `filter` + `write` 
produce different chunk boundaries for those types.
   
   **5. Scope left for #12187**
   
   - `AsyncArrowWriter::write_with_selection` is not included (the PR says 
"Part of", which is fine).
   - The issue left the selection type open. `RowSelection` lives in 
`arrow_reader` and describes runs over file rows; `write_with_selection` 
converts it to a bitmap on every call (`into_boolean_buffer`). Accepting a 
`&BooleanArray` like `filter_record_batch` does may fit the writer API better. 
Maintainers should decide this before the API ships.
   - The description's benchmarks show up to 8% more CPU when 99% of rows are 
skipped. A note in the docs about when `filter` + `write` is the better choice 
would help users.
   
   New public API in this PR: `ArrowWriter::write_with_selection`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to