dongjoon-hyun opened a new pull request, #59679: URL: https://github.com/apache/spark/pull/59679
### What changes were proposed in this pull request? This PR aims to validate `spark.io.compression.zstd.bufferSize` via `checkValue` and to document its valid range in `docs/configuration.md`. Like the existing `spark.shuffle.file.buffer` validation, the value must be positive and less than or equal to `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH` (`2147483632`) bytes. Note that this PR doesn't validate `spark.io.compression.zstd.strategy` because zstd may add new strategies in the future. This is consistent with the decision in [SPARK-52924](https://github.com/apache/spark/pull/51630#discussion_r2225184049). ### Why are the changes needed? Currently, `spark.io.compression.zstd.bufferSize` accepts any value. `ZStdCompressionCodec` reads it with `.toInt`, so `0`, `-1`, `2g` (overflows to `-2147483648`), and `4g` (overflows to `0`) all become invalid buffer sizes. An invalid value is not detected until `ZStdCompressionCodec` wraps a stream. At that point, the zstd stream has already allocated its native context, and neither `ZstdOutputStreamNoFinalizer` nor `ZstdInputStreamNoFinalizer` has a finalizer or a `Cleaner`. So, each failed wrap leaves an unreachable zstd stream whose native context is never freed. This happens for shuffle, broadcast, event logs, and the standalone Master recovery store. https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/core/src/main/scala/org/apache/spark/io/CompressionCodec.scala#L240-L262 ```scala // The native context is allocated here. val os = new ZstdOutputStreamNoFinalizer(s, bufferPool).setLevel(level).setWorkers(workers) strategy.foreach(os.setStrategy) // IllegalArgumentException: Buffer size <= 0 new BufferedOutputStream(os, bufferSize) ``` With this PR, Spark fails to create `ZStdCompressionCodec` before allocating any zstd stream. Note that `spark.io.compression.zstd.bufferSize` has been available since [v2.3.0](https://github.com/apache/spark/releases/tag/v2.3.0) (2018-02-22). ### Does this PR introduce _any_ user-facing change? Yes, but only for invalid values. Previously, they were accepted, but every zstd stream creation failed after leaking a native zstd context. Now, Spark rejects them when it creates `ZStdCompressionCodec`. - **Before** ``` java.lang.IllegalArgumentException: Buffer size <= 0 ``` - **After** ``` org.apache.spark.SparkIllegalArgumentException: [INVALID_CONF_VALUE.REQUIREMENT] The value '0b' in the config "spark.io.compression.zstd.bufferSize" is invalid. The buffer size must be positive and less than or equal to 2147483632. SQLSTATE: 22022 ``` ### How was this patch tested? Pass the CIs with the newly added test case in `CompressionCodecSuite`. It checks that `1` and `2147483632` are accepted while `0`, `-1`, `2147483633`, `2g`, and `4g` are rejected. I also verified that the new test case fails without the main code change. ``` $ build/sbt 'core/testOnly *CompressionCodecSuite' ... [info] Tests: succeeded 16, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
