andygrove opened a new pull request, #7509: URL: https://github.com/apache/datafusion-comet/pull/7509
## Which issue does this PR close? Backport of #7307 to `branch-1.2`. Closes #7305 on this branch. ## Rationale for this change An executor JVM with Native Memory Tracking enabled aborted with `NMT corruption: ... footer canary broken (buffer overflow?)` while materializing Comet's in-memory cache with the default `zstd` codec (#7305). Arrow 18.3.0's `ZstdCompressionCodec.doCompress` points zstd 8 bytes into its destination buffer but still passes the full capacity, so zstd can write up to 8 bytes past the buffer. `branch-1.2` ships the same Arrow version and the same cache codec path, so it has the bug. See #7307 for the full analysis. ## What changes are included in this PR? Cherry-pick of 5252c82a5b (`git cherry-pick -x`) with no conflicts and no adaptations. The diff against the source commit is identical apart from line numbers and blob hashes. - `CometZstdCompressionCodec`, a subclass of Arrow's codec whose `doCompress` passes `compressBound(n)` as the capacity, as apache/arrow-java#1119 does. `CachedBatchIpc.compressionCodec` uses it for `zstd`. - `CometZstdCompressionCodecSuite` is registered in `pr_build_linux.yml` and `pr_build_macos.yml`. - `CometInMemoryCacheSuite` runs the cache through Comet's codec. Other release branches: `branch-1.1` also has `CachedBatchIpc`; whether it needs a backport is not part of this PR. `branch-1.2` is the newest release branch, so there is no newer one to port to. ## How are these changes tested? On this branch, using JDK 17 and a native library built from `branch-1.2`: - `CometZstdCompressionCodecSuite` and `CometInMemoryCacheSuite`: 78 tests, 0 failures. - `dev/ci/check-suites.py` and `dev/ci/check-ci-config.py` pass. I did not rerun the tests without the fix on this branch; the source PR shows that Arrow's codec overruns on the suite's inputs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
