joechenrh commented on PR #880:
URL: https://github.com/apache/arrow-go/pull/880#issuecomment-4911576405
Ready for review. Thanks for taking a look!
+ Scope: PLAIN-encoded V1 + V2 pages, ByteArray / FixedLenByteArray,
unencrypted streaming-capable codecs (UNCOMPRESSED/GZIP/BROTLI/ZSTD). Every
other page keeps the existing materialized path. No public interface changes.
+ ~620 lines of production code, ~1270 of tests.
+ Validation: verified end-to-end on real S3 and local disk. On large pages
peak decode memory drops ~100–1000× (e.g. a 256 MiB page: 512 MiB → 5 MiB; a 1
GiB page: ~2 GiB → ~2 MiB), with read time on par with the non-streaming path
at BufferSize ≥ 4 MiB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
