huan233usc opened a new issue, #17305: URL: https://github.com/apache/iceberg/issues/17305
### Apache Iceberg version main (development) ### Query engine Spark ### Please describe the improvement `VectorizedArrowReader` initializes variable-width vectors with: ```java vector.setInitialCapacity(batchSize * AVERAGE_VARIABLE_WIDTH_RECORD_SIZE); ``` The single-argument Arrow API accepts a value count, not a byte count. For a batch size of 5,000 and an average-width constant of 10, this reserves offsets for 50,000 values and uses Arrow's default density of 8 bytes/value, rather than reserving 5,000 values at an estimated 10 bytes/value. The reader already receives `ColumnChunkMetaData` in `setRowGroupInfo`, including total compressed/uncompressed sizes and value count. We should investigate using that metadata to estimate a bounded average width and initialize `BaseVariableWidthVector` with the density-aware API: ```java setInitialCapacity(batchSize, estimatedBytesPerValue) ``` The estimate should: - use `batchSize` as the value capacity; - fall back to a conservative default when metadata is missing or unusable; - account for page/dictionary overhead and avoid unbounded over-allocation; - cover physical BINARY and logical variable-width types such as string, JSON, BSON, geometry, and geography; - preserve safe reallocation for unusually large values. Related: #11672 identified the value-count/byte-count mismatch but was closed automatically as stale without a fix. ### Willingness to contribute - [X] I can contribute a fix for this improvement independently - [ ] I would be willing to contribute a fix with guidance from the Iceberg community - [ ] I cannot contribute a fix at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
