adityavaish opened a new pull request, #60180:
URL: https://github.com/apache/spark/pull/60180
### What changes were proposed in this pull request?
Reject empty strings in PySpark's `_parse_memory` with the existing
`ValueError("invalid format: " + s)` rather than indexing an empty string.
Move the existing memory parser test into a plain `unittest.TestCase` so it
does not start a SparkContext. Add regression coverage for empty input and
preserve coverage for invalid units, all supported unit suffixes in both cases,
fractional values, and truncation to MiB. Update the helper's docstring to
document invalid-input behavior.
### Why are the changes needed?
`_parse_memory` reads `s[-1]` before checking whether the string is empty.
Consequently an empty memory setting, such as `spark.python.worker.memory=""`,
raises an unrelated `IndexError: string index out of range`. Empty input should
be rejected consistently with other invalid memory formats.
### Does this PR introduce _any_ user-facing change?
Yes. Empty memory strings now raise `ValueError: invalid format: ` instead
of `IndexError: string index out of range`. Valid memory conversions and
existing invalid-unit errors are unchanged. This affects the shared parser used
by Python worker memory settings and executor resource requests.
### How was this patch tested?
Added `ParseMemoryTests`. Before the fix, `test_invalid_memory` fails for
the empty-string subtest with the reported IndexError; after the fix it passes.
Ran the following against the working-tree PySpark sources using Python
3.13.17 with Py4J 0.10.9.9:
```powershell
$env:PYTHONPATH = Join-Path (Get-Location) 'python'
.\.venv\Scripts\python.exe -m unittest
pyspark.tests.test_util.ParseMemoryTests
pyspark.tests.test_util.KeywordOnlyTests
pyspark.tests.test_util.HandleWorkerExceptionTests
```
All 11 tests passed. No JVM was needed for these unit tests; the full
Spark/JVM suite was not run.
Also passed `ruff check` and `ruff format --check` for both modified files
using the repository-pinned Ruff 0.14.8, and `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot App 1.0.102-0.unstable.r40949665530.gab1eef5
(GPT-6.1 Sol).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
