joechenrh commented on PR #880: URL: https://github.com/apache/arrow-go/pull/880#issuecomment-4893416152
### Status **Unit tests**: pass locally — full `parquet/...` suite plus the streaming-specific tests. **Perf**: quick in-memory benchmark reading large (8 MiB) data pages via `ReadBatchInPage`, streaming off → on: | value | codec | peak (off → on) | time | |---|---|---|---| | 16 KiB | uncompressed | 32 MiB → **1.0 MiB** | −23% | | 16 KiB | zstd | 32 MiB → **1.0 MiB** | +34% | | 16 KiB | gzip | 32 MiB → **1.0 MiB** | −18% | | 256 KiB | uncompressed | 64 MiB → **1.0 MiB** | −43% | | 256 KiB | zstd | 64 MiB → **1.0 MiB** | +38% | | 256 KiB | gzip | 64 MiB → **1.0 MiB** | −28% | - **Memory** (the goal): peak decode allocation stays flat at ~1 MiB with streaming on, vs page-scale off — **~31–63×** lower, scaling with page size. Same result verified end-to-end through a downstream reader's `ReadBatchInPage` path. - **Speed**: uncompressed/gzip are on par or faster for medium/large values (smaller working set, cache-friendly). zstd is ~+35% slower on large values — structural: streaming decodes incrementally with an extra copy, vs one-shot `DecodeAll` + zero-copy slicing on the non-streaming path. I prototyped a read-ahead buffer to close it and measured ≈0 improvement (the cost is the extra copy, not the read count), so I dropped it. Tiny values (~64 B) are slightly slower for all codecs (per-value overhead). **TODO**: end-to-end validation on the real S3 read path. Decode CPU is a small fraction of I/O-bound S3 reads, so the zstd delta should shrink there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
