andygrove opened a new issue, #6920: URL: https://github.com/apache/datafusion-comet/issues/6920
### What is the problem the feature request solves? Two more per-value costs in the parquet-rs writer show up for nested types in `CometIcebergEndToEndBenchmark` (#6910): - Dictionary interning of string values (`Interner::intern` plus `ahash`): about 5% of all CPU. - Computing levels from Arrow nested arrays (`LevelInfoBuilder::write`, `LevelData::append_run` and the list visitors): about 8%, including reallocation, since the level buffers grow without being pre-sized. ### Describe the potential solution Upstream in arrow-rs: - Pre-size the level buffers from the array lengths. - Build on #9831 and #10037 (compact levels, batched writes when there is no repetition), which parquet 59.3.0 already has, for list-heavy schemas. - Look at a cheaper interner path for short values. Upgrading to arrow-rs 60 also brings #10880, which resolves writer properties once per column instead of on every write call. ### Additional context parquet 59.3.0 (the version Comet uses): `parquet/src/util/interner.rs` and `parquet/src/arrow/arrow_writer/levels.rs`. Part of #6911. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
