andygrove commented on PR #5869: URL: https://github.com/apache/datafusion-comet/pull/5869#issuecomment-6100134445
@peterxcli, a heads-up on related work. While digging into #5485 I found that most of what remains of the Spark-reader gap is `zstd` decompression, so I opened #6859, which adds `lz4` (Arrow's `LZ4_FRAME`, written with lz4-java) as an opt-in cache codec. A full read through the native scan takes less than half as long as with `zstd`, but the cache takes about 2.8 times the memory, and on sequential `bigint` columns more than Spark's own format, which delta-encodes them: the wide relations in `CometInMemoryCacheBenchmark` take about 74 MiB with `lz4`, against 26 to 27 MiB in Spark's format and with `zstd`. That is the data this PR targets. The delta pass compresses with whatever codec the batch uses, so the two should compose without further changes once both land, and `lz4` with delta-encoded longs looks like the best candidate for a faster default. I'd like to measure that combination after both merge. Both PRs touch `CachedBatchIpc`, the codec docs in `CometConf` and `in-memory-cache.md`, `CometInMemoryCacheSuite` and `CometCachedBatchHelper`, so whichever merges second will need a small rebase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
