Mola-maker commented on PR #12350: URL: https://github.com/apache/arrow-rs/pull/12350#issuecomment-7031528903
Hey,I read through the diff against #12187 and found a few gaps between what the issue asks for and what the PR delivers. **1. Output does not match `filter_record_batch` + `write` under default settings** #12187 asks for the same rows, statistics, page indexes and row counts. The doc comment on `write_with_selection` already says nullable columns can paginate differently. The tests only show byte equality after setting `write_batch_size` equal to `data_page_row_count_limit`, or with batches too small to fill a page. With default `WriterProperties` and a 200k-row nullable `Int32` column (nulls only in skipped rows, every other row selected), the two files differ: - default props: selection writes pages of 20480 values, filter + write writes pages of 20000 - `data_page_row_count_limit(1000)`: 98 pages vs 100 pages Skipping the null rows is probably the most common use of this API, so this case will come up often. The page boundaries, and therefore the offset index, differ from what the issue specifies. Should the selected path compute levels as if the selected rows were the whole array, so a leaf with no selected nulls is treated as having no nulls? If not, the docs and #12187 should say plainly that only rows and statistics are guaranteed to match, and pages are not. **2. Error indices for required fields** The docs say this "fails as `Self::write` would for the filtered batch". But `reject_null` reports the row index in the original batch, while `write` on the filtered batch reports the index in the filtered batch. The test only checks for `for required field 'c'`. Either the docs or the index should change. **3. Unrelated change to `write`** The `map` change (converting only non-null values, and passing no value indices) also changes the existing `write` path for `Int8`/`Int16`/unsigned/`Date64`/decimal columns. It looks like an independent optimization. Could it go in a separate PR with its own benchmark, so this one only adds the new API? **4. Content-defined chunking difference is documented but not tracked** The CDC test drops the fixed-size-list, list-view and (with row group splitting) list columns. The comment says `write` hashes the wrong values for them today. If that is a bug in the existing `write` + CDC path, it should have its own issue. As it stands, `write_with_selection` and `filter` + `write` produce different chunk boundaries for those types. **5. Scope left for #12187** - `AsyncArrowWriter::write_with_selection` is not included (the PR says "Part of", which is fine). - The issue left the selection type open. `RowSelection` lives in `arrow_reader` and describes runs over file rows; `write_with_selection` converts it to a bitmap on every call (`into_boolean_buffer`). Accepting a `&BooleanArray` like `filter_record_batch` does may fit the writer API better. Maintainers should decide this before the API ships. - The description's benchmarks show up to 8% more CPU when 99% of rows are skipped. A note in the docs about when `filter` + `write` is the better choice would help users. New public API in this PR: `ArrowWriter::write_with_selection`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
