hhhizzz opened a new pull request, #12019: URL: https://github.com/apache/arrow-rs/pull/12019
> [!IMPORTANT] > Fork CI has not run for this branch yet; this draft currently has local validation only. See the [fork CI setup instructions](https://github.com/apache/arrow-rs/blob/main/CONTRIBUTING.md#pull-requests-from-forks). # Which issue does this PR close? - Part of [#7973](https://github.com/apache/arrow-rs/issues/7973). - Part of [#11478](https://github.com/apache/arrow-rs/issues/11478). This is **step 2a** of the [overall solution and PR breakdown](https://github.com/apache/arrow-rs/issues/11478#issuecomment-6333545397). The capacity work is split into top-level and nested columns to keep each change focused. The [original prototype](https://github.com/hhhizzz/arrow-rs/tree/pr/byte-array-mask-windows) contains the broader approach. | Step | Scope | PR / status | | --- | --- | --- | | 1 | Error handling and overflow safety | [#11791](https://github.com/apache/arrow-rs/pull/11791), merged | | 2a | Top-level Utf8/Binary staging and capacity-safe output | This draft | | 2b | Extend capacity handling to nested columns | Planned | | 3 | Bounded Mask decoding windows | Planned | | 4a | Capacity-safe coalescing of small windows and nested View compaction | Planned | | 4b | Predicate cache lifetime and copying improvements | Planned | | 4c | Byte-array payload reservation hints | Planned | # Rationale for this change Reading a top-level `Utf8` or `Binary` column can fail with `index overflow decoding byte array` when one decoded batch exceeds `i32::MAX` payload bytes, even when each individual value fits. This change decodes eligible columns with 64-bit offsets and restores their requested types at the output boundary. Oversized results are split at complete row boundaries, allowing the read to finish with valid output batches. # What changes are included in this PR? - Stage top-level `Utf8`/`Binary` BYTE_ARRAY columns as `LargeUtf8`/`LargeBinary`, while retaining the projected logical schema. - Restore the requested types before returning output or evaluating predicates. Fitting batches share their payload buffers; split batches copy their staged payloads so those payloads do not retain the entire decoded window. Other columns and null buffers may still share storage. - Apply the same staging to sync, async, push-decoder and predicate-cache paths, including rebuilding a push decoder between row groups. - Document that output batches can contain fewer than `batch_size` rows when a top-level byte-array column reaches its capacity. Nested byte arrays and explicitly requested Large, View and Dictionary representations retain their existing behavior. A single row that cannot fit in the requested output type still returns an error. This step does not bound the raw rows decoded by Mask or provide a peak-memory limit. Performance impact has not been remeasured for this draft. # Are these changes tested? Yes. Fifteen focused unit, builder and end-to-end tests cover: - Capacity boundaries, nulls, sliced inputs, payload sharing and compaction. - Projected schemas, field metadata, duplicate field names, virtual columns and empty projections. - PLAIN, RLE_DICTIONARY, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY with Mask and Selectors. - Multiple-column batch alignment, offset/limit, stateful predicates, cache reuse, async reads and push-decoder rebuilding. Small output limits exercise the splitting paths in ordinary tests. The existing large-input regressions now write legal small batches so they reach the reader, then verify successful streaming reads of **700 × 3 MiB** of logical payload. All four encodings were run explicitly in release mode and returned **[682, 18]** rows in two valid batches, with every value checked. These memory-intensive cases remain ignored by default. The local workspace and all-feature Parquet test suites, formatting, spelling and Clippy checks pass. # Are there any significant user-facing changes? There is no new public API. Staged columns use 64-bit offsets in the predicate cache, so offset storage increases from four to eight bytes per entry and can affect cache admission. A `ParquetRecordBatchReader` containing staged columns ends after an error rather than continuing past a failed buffered window. AI assistance: OpenAI Codex assisted with implementation and regression tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
