dwsmith1983 commented on PR #5565: URL: https://github.com/apache/datafusion-comet/pull/5565#issuecomment-5559185617
Trimmed to the write path in 27c2033c1: the decoder context, the thread-local, the scan operator changes and the decode bench are gone, and ipc.rs is back to main. The retained-size cap and the level initialization stay, for the reasons sunchao gave. Remaining diff is about 260 production lines. One correction to my earlier reply: the posted table implies 4.1 percent wall and 8.9 percent encode at 10,000 partitions, not 7 and 11. Linux run, since you asked: rust 1.94 on linux/arm64 in Docker, 10,000 partitions, zstd level 3, ten timed iterations, base and head alternated. Round one gives 3.0 percent wall and 6 percent encode with the min/max ranges not overlapping. A second round showed more, but the base run drifted, so I would not read anything into it beyond the direction. That is in line with your per-block estimate. On macOS the same shape over five iterations gives 5.6 percent wall and 6.6 percent encode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
