thswlsqls opened a new pull request, #17057: URL: https://github.com/apache/iceberg/pull/17057
Fixes the same over-allocation described in #11672. ## Summary - `VectorizedArrowReader.allocateVectorBasedOnTypeName()` passed `batchSize * typeWidth` (or `batchSize * AVERAGE_VARIABLE_WIDTH_RECORD_SIZE` for `BINARY`) to Arrow's `setInitialCapacity(int)`. - `setInitialCapacity(int valueCount)` already multiplies by type width internally (verified against arrow-vector 15.0.2, pinned in `gradle/libs.versions.toml`), so the extra multiplication double-scaled the allocation. - Affected: `FIXED_LEN_BYTE_ARRAY` (`fixed(N)`/UUID), `BINARY`, `INT96`. Large `fixed(N)` columns can over-allocate per batch, risking OOM (default `MAX_ALLOCATION_SIZE` is unbounded, so it does not always throw). - Fix: pass `batchSize` alone, matching the sibling branches in the same method and the logical-type paths (UUID, decimal) that already allocate correctly with `allocateNew(batchSize)`. - Scope limited to `allocateVectorBasedOnTypeName()`; the same pattern also exists in `allocateVectorForEnumJsonBsonString()` (STRING/ENUM/JSON/BSON), left for a follow-up. ## Testing done - Added `TestVectorizedReaderAllocation#fixedLenByteArrayVectorCapacityMatchesBatchSize` and `#binaryVectorCapacityMatchesBatchSize`, writing a single row through a real Parquet file and asserting the vector's `getValueCapacity()` stays within a small multiple of the batch size. - Verified against the pre-fix code that both tests fail as expected: a `fixed(1000)` column with `batchSize=16` measured capacity 16775 (bound 64), and a `BINARY` column measured 247 (bound 64). - `./gradlew :iceberg-arrow:check` — 31 tests passed (0 failures), including spotlessCheck and checkstyle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
