andygrove commented on issue #7296: URL: https://github.com/apache/datafusion-comet/issues/7296#issuecomment-6673271110
I prototyped this and measured it against the current path. On Linux it is no faster for any codec, and slower for `lz4` and `none`, so I am parking it rather than opening a PR. The code is on [andygrove/datafusion-comet@`native-cache-decode`](https://github.com/andygrove/datafusion-comet/tree/andygrove/native-cache-decode) (7f920fabd) for reference. ### What the prototype does When a native operator reads the cache, the scan is planned as a `ShuffleScan` with a new `CachedBatchLayout` (the full relation's types plus the projection). On the JVM, a `CometShuffleBlockIterator` subclass copies each `CometCachedBatch` payload into a direct buffer. Native code then decodes it with arrow-rs `read_record_batch`, which skips the columns outside the projection, so only the selected buffers are decompressed. Pruning, the read accumulators and `numOutputRows` all behave as they do on the JVM path. It also covers the AQE table-cache stage. A row-count-only scan, and a shuffle that reads the cache directly, still use the JVM path. Answers match Spark and the JVM decode path under every codec, with AQE on and off, and the existing cache suites pass. ### Results `CometInMemoryCacheBenchmark`, read cases, 5M rows by 6 columns (nested: 1M rows), Spark 4.1, JDK 17, release build. Best time in ms, JVM decode vs native decode. | Read | Linux (7950X3D) JVM | Linux native | macOS (M3 Max) JVM | macOS native | | --- | --: | --: | --: | --: | | zstd, full (6 of 6) | 236 | 240 | 243 | 214 | | zstd, narrow (1 of 6) | 33 | 34 | 34 | 30 | | zstd, nested full | 102 | 98 | 97 | 85 | | lz4, full | 161 | 507 | 103 | 282 | | lz4, narrow | 32 | 94 | 28 | 57 | | none, full | 56 | 68 | 49 | 57 | | none, narrow | 27 | 60 | 26 | 34 | ### Why it does not pay - **zstd:** decompression dominates a wide read, and arrow-rs's zstd is no faster than zstd-jni on Linux. The JVM-side work this issue set out to remove (copying the selected buffers off-heap, `VectorLoader`, wrapping `CometVector`s, the FFI export) is a small share of a read. - **lz4:** arrow-rs decompresses `LZ4_FRAME` with pure-Rust `lz4_flex`, while the JVM path uses JNI lz4-java. On Linux the native lz4 runs also varied widely (average 5.6 s against a 507 ms best), which I did not investigate. - **none, and narrow reads in general:** the payload is copied twice (heap chunks into a direct buffer, then into an arrow-rs `Buffer`), and the whole payload rather than only the selected columns. The JVM path copies only the selected buffers, once. ### What would be needed for it to win 1. A single copy of only the selected buffers: native allocates the destination and the JVM fills it, or the JVM copies only the projected windows. 2. A C lz4 decoder on the native side, which is a new dependency. Even with both, zstd is at parity on Linux, so the best case is "no slower, with less JVM allocation per read". That could still matter for GC pressure on large cached relations, but it is not the throughput win this issue assumed. Moving the write side (statistics and compression) to native code looks unlikely to pay either, for the same reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
