This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new b301361ae926 [MINOR][PYTHON] Reject empty memory strings with
ValueError
b301361ae926 is described below
commit b301361ae92605473076a690259d31a71adfb727
Author: Aditya Vaish <[email protected]>
AuthorDate: Sat Nov 7 11:53:09 2026 +0800
[MINOR][PYTHON] Reject empty memory strings with ValueError
### What changes were proposed in this pull request?
Reject empty strings in PySpark's `_parse_memory` with the existing
`ValueError("invalid format: " + s)` rather than indexing an empty string.
Move the existing memory parser test into a plain `unittest.TestCase` so it
does not start a SparkContext. Add regression coverage for empty input and
preserve coverage for invalid units, all supported unit suffixes in both cases,
fractional values, and truncation to MiB. Update the helper's docstring to
document invalid-input behavior.
### Why are the changes needed?
`_parse_memory` reads `s[-1]` before checking whether the string is empty.
Consequently an empty memory setting, such as `spark.python.worker.memory=""`,
raises an unrelated `IndexError: string index out of range`. Empty input should
be rejected consistently with other invalid memory formats.
### Does this PR introduce _any_ user-facing change?
Yes. Empty memory strings now raise `ValueError: invalid format: ` instead
of `IndexError: string index out of range`. Valid memory conversions and
existing invalid-unit errors are unchanged. This affects the shared parser used
by Python worker memory settings and executor resource requests.
### How was this patch tested?
Added `ParseMemoryTests`. Before the fix, `test_invalid_memory` fails for
the empty-string subtest with the reported IndexError; after the fix it passes.
Ran the following against the working-tree PySpark sources using Python
3.13.17 with Py4J 0.10.9.9:
```powershell
$env:PYTHONPATH = Join-Path (Get-Location) 'python'
.\.venv\Scripts\python.exe -m unittest
pyspark.tests.test_util.ParseMemoryTests
pyspark.tests.test_util.KeywordOnlyTests
pyspark.tests.test_util.HandleWorkerExceptionTests
```
All 11 tests passed. No JVM was needed for these unit tests; the full
Spark/JVM suite was not run.
Also passed `ruff check` and `ruff format --check` for both modified files
using the repository-pinned Ruff 0.14.8, and `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot App 1.0.102-0.unstable.r40949665530.gab1eef5
(GPT-6.1 Sol).
Closes #60180 from adityavaish/adityavaish-empty-memory-validation.
Lead-authored-by: Aditya Vaish
<[email protected]>
Co-authored-by: Aditya Vaish <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 1d3149fc1b907bd28286b44ef1ecba5a219b5695)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/tests/test_util.py | 29 ++++++++++++++++++++++++-----
python/pyspark/util.py | 4 ++--
2 files changed, 26 insertions(+), 7 deletions(-)
diff --git a/python/pyspark/tests/test_util.py
b/python/pyspark/tests/test_util.py
index 90310a5dafa7..c3347d2cfaf8 100644
--- a/python/pyspark/tests/test_util.py
+++ b/python/pyspark/tests/test_util.py
@@ -144,11 +144,6 @@ class UtilTests(PySparkTestCase):
v4 = LooseVersion("1.2.4")
self.assertTrue(v1 <= v4)
- def test_parse_memory(self):
- self.assertEqual(_parse_memory("1g"), 1024)
- with self.assertRaisesRegex(ValueError, "invalid format"):
- _parse_memory("2gs")
-
def test_disable_gc(self):
# Restore the caller's GC state even if an assertion fails.
self.addCleanup(gc.enable if gc.isenabled() else gc.disable)
@@ -181,6 +176,30 @@ class UtilTests(PySparkTestCase):
print("succeeding immediately")
+class ParseMemoryTests(unittest.TestCase):
+ def test_parse_memory(self):
+ for memory, expected in (
+ ("1g", 1024),
+ ("1G", 1024),
+ ("200m", 200),
+ ("200M", 200),
+ ("2t", 2 << 20),
+ ("2T", 2 << 20),
+ ("1024k", 1),
+ ("1024K", 1),
+ ("1.5g", 1536),
+ ("512k", 0),
+ ):
+ with self.subTest(memory=memory):
+ self.assertEqual(_parse_memory(memory), expected)
+
+ def test_invalid_memory(self):
+ for memory in ("", "2gs"):
+ with self.subTest(memory=memory):
+ with self.assertRaisesRegex(ValueError, "^invalid format: " +
memory + "$"):
+ _parse_memory(memory)
+
+
class HandleWorkerExceptionTests(unittest.TestCase):
exception_bytes = b"ValueError: test_message"
traceback_bytes = b"Traceback (most recent call last):"
diff --git a/python/pyspark/util.py b/python/pyspark/util.py
index 0b7f910a9fbe..7e3dc37b2b32 100644
--- a/python/pyspark/util.py
+++ b/python/pyspark/util.py
@@ -339,7 +339,7 @@
________________________________________________________________________________
def _parse_memory(s: str) -> int:
"""
Parse a memory string in the format supported by Java (e.g. 1g, 200m) and
- return the value in MiB
+ return the value in MiB. Raise ValueError for empty or invalid memory
strings.
Examples
--------
@@ -349,7 +349,7 @@ def _parse_memory(s: str) -> int:
2048
"""
units = {"g": 1024, "m": 1, "t": 1 << 20, "k": 1.0 / 1024}
- if s[-1].lower() not in units:
+ if not s or s[-1].lower() not in units:
raise ValueError("invalid format: " + s)
return int(float(s[:-1]) * units[s[-1].lower()])
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]