This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 960d2393bb30 [SPARK-57921][CORE] Fix `spark.memory.offHeap.size` error
message to require at least 1MiB
960d2393bb30 is described below
commit 960d2393bb30370d8b69846d742f79aaf3f6e860
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Jul 3 03:15:09 2026 -0700
[SPARK-57921][CORE] Fix `spark.memory.offHeap.size` error message to
require at least 1MiB
### What changes were proposed in this pull request?
This PR aims to fix the error message of `Utils.checkOffHeapEnabled` to say
that `spark.memory.offHeap.size` must be at least 1MiB, instead of `must be >
0`.
### Why are the changes needed?
`Utils.executorOffHeapMemorySizeAsMb` converts `spark.memory.offHeap.size`
to MiB with truncation before validation. So, a positive value less than 1MiB
(e.g. `500000`) is truncated to `0` and rejected with a misleading message.
```
$ bin/spark-shell -c spark.memory.offHeap.enabled=true -c
spark.memory.offHeap.size=500000
...
java.lang.IllegalArgumentException: requirement failed:
spark.memory.offHeap.size must be > 0 when spark.memory.offHeap.enabled ==
true
```
The user set a value greater than 0, but the error still says it `must be >
0`. Since all callers of `checkOffHeapEnabled` pass MiB-based values, the
effective minimum is 1MiB. The check in `MemoryManager` which validates the raw
byte value is untouched because `must be > 0` is accurate there.
### Does this PR introduce _any_ user-facing change?
No, this is an error message fix.
### How was this patch tested?
Pass the CIs with the updated test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #56983 from dongjoon-hyun/SPARK-57921.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 5615098afd961b2fc345f487fbc5aa3f7c1b2cc6)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/util/Utils.scala | 3 ++-
core/src/test/scala/org/apache/spark/util/UtilsSuite.scala | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/util/Utils.scala
b/core/src/main/scala/org/apache/spark/util/Utils.scala
index 11dc885ca86b..2b03a068f132 100644
--- a/core/src/main/scala/org/apache/spark/util/Utils.scala
+++ b/core/src/main/scala/org/apache/spark/util/Utils.scala
@@ -3091,7 +3091,8 @@ private[spark] object Utils
def checkOffHeapEnabled(sparkConf: ReadOnlySparkConf, offHeapSize: Long):
Long = {
if (sparkConf.get(MEMORY_OFFHEAP_ENABLED)) {
require(offHeapSize > 0,
- s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true")
+ s"${MEMORY_OFFHEAP_SIZE.key} must be at least 1MiB when " +
+ s"${MEMORY_OFFHEAP_ENABLED.key} == true")
offHeapSize
} else {
0
diff --git a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
index 96e00db64bd4..e1e49a226735 100644
--- a/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
+++ b/core/src/test/scala/org/apache/spark/util/UtilsSuite.scala
@@ -1603,7 +1603,7 @@ class UtilsSuite extends SparkFunSuite with
ResetSystemProperties {
val sparkConf = new SparkConf()
.set(MEMORY_OFFHEAP_ENABLED, true)
val expected =
- s"${MEMORY_OFFHEAP_SIZE.key} must be > 0 when
${MEMORY_OFFHEAP_ENABLED.key} == true"
+ s"${MEMORY_OFFHEAP_SIZE.key} must be at least 1MiB when
${MEMORY_OFFHEAP_ENABLED.key} == true"
val message = intercept[IllegalArgumentException] {
Utils.executorOffHeapMemorySizeAsMb(sparkConf)
}.getMessage
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
