hamilton-earthscope opened a new pull request, #595: URL: https://github.com/apache/arrow-go/pull/595
## Overview In https://github.com/apache/iceberg-go/pull/643 a benchmark test suite is introduced to measure write performance to iceberg tables. This PR fixes a few hot spots in the parquet writer that improve write performance to iceberg tables across a variety of scenarios. These improvements should be broadly beneficial to anyone using arrow-go to write parquet. ## Optimizations ### 1. Zstd Encoder Pooling Implemented `sync.Pool` for zstd compression encoders in `arrow-go/parquet/compress/zstd.go`. Previously, every parquet file write allocated a new 96MB encoder, causing a lot of memory bloat and GC churn. Now encoders are pooled and reused across writes. ### 2. Column Path Caching Pre-compute column path strings (e.g., "data.field.subfield") in `arrow-go/parquet/schema/column.go` during Column creation instead of rebuilding via `strings.Join()` on every access. Eliminates CPU overhead, especially effective for nested schemas. ## Performance Results These results are measured using the test suite linked above. No new performance benchmarks are added to arrow-go. The metrics in the table below indicate 1. the write speed measured in records per second and 2. the volume of memory allocated during the test (not the number of allocations which these changes don't really affect) | Benchmark | Before | After | Improvement | |-----------|--------|-------|-------------| | **Simple 100K** | 734K rec/sec, 3.9 GB | 2.5M rec/sec, 155 MB | **3.5x faster, 96% less memory** | | **Simple 2.5M** | 3.9M rec/sec, 9.3 GB | 4.3M rec/sec, 1.9 GB | **1.1x faster, 79% less memory** | | **ListPrimitive 100K** | 496K rec/sec, 4.6 GB | 1.3M rec/sec, 317 MB | **2.7x faster, 93% less memory** | | **ListPrimitive 2.5M** | 1.5M rec/sec, 14.2 GB | 2.0M rec/sec, 3.3 GB | **1.3x faster, 76% less memory** | | **ListStruct 100K** | 377K rec/sec, 5.7 GB | 927K rec/sec, 565 MB | **2.4x faster, 90% less memory** | | **ListStruct 2.5M** | 772K rec/sec, 19.7 GB | 1.1M rec/sec, 5.0 GB | **1.4x faster, 74% less memory** | | **MapPrimitive 100K** | 446K rec/sec, 5.6 GB | 995K rec/sec, 511 MB | **2.2x faster, 91% less memory** | | **MapPrimitive 2.5M** | 1.1M rec/sec, 18.0 GB | 1.3M rec/sec, 4.7 GB | **1.2x faster, 74% less memory** | | **1000 partitions** | 35K rec/sec, 85.9 GB | 341K rec/sec, 576 MB | **9.6x faster, 99% less memory** | **Key improvements:** - Small batches (100K): 2.2-3.5x throughput, 90-96% memory reduction - Large batches (2.5M): 1.1-1.4x throughput, 74-79% memory reduction - High partition counts see dramatic gains (9.6x faster for 1000 partitions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
