hamilton-earthscope commented on PR #573: URL: https://github.com/apache/arrow-go/pull/573#issuecomment-3530197583
Improvement from iceberg writes with list columns: --- ### **ListPrimitive Schema** (`list<string>`) | Records | Throughput Before | Throughput After | **Improvement** | |---------|------------------|------------------|-----------------| | 100K | 366K rec/sec | 451K rec/sec | **+23%** | | 500K | 475K rec/sec | 1.09M rec/sec | **+130%** | | 2.5M | 224K rec/sec | 1.52M rec/sec | **+577%** | **Memory Usage (2.5M records)**: - Before: **342 GB** allocated - After: **14 GB** allocated - **Reduction: -96%** --- ### **ListStruct Schema** (`list<struct<...>>`) | Records | Throughput Before | Throughput After | **Improvement** | |---------|------------------|------------------|-----------------| | 100K | 247K rec/sec | 359K rec/sec | **+45%** | | 500K | 195K rec/sec | 594K rec/sec | **+205%** | | 2.5M | 74K rec/sec | 729K rec/sec | **+883%** | **Memory Usage (2.5M records)**: - Before: **1,100 GB** allocated - After: **19.7 GB** allocated - **Reduction: -98%** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
