anxkhn opened a new pull request, #57012:
URL: https://github.com/apache/spark/pull/57012
### What changes were proposed in this pull request?
`DataFrame.dropna` on the classic path validates the `how` argument and, on
an
invalid value, raises `PySparkValueError` with error class
`VALUE_NOT_ANY_OR_ALL`. The `messageParameters` passed a key named
`arg_type`,
but the `VALUE_NOT_ANY_OR_ALL` template interpolates `<arg_value>`:
```
"Value for `<arg_name>` must be 'any' or 'all', got '<arg_value>'."
```
So the template placeholder set is `{arg_name, arg_value}` while the code
passed
`{arg_name, arg_type}`. `ErrorClassesReader.get_error_message`
(`python/pyspark/errors/utils.py`) asserts the two sets are equal, so the
mismatch raised an opaque `AssertionError` from inside the exception
constructor
and masked the intended `PySparkValueError`.
This PR changes that one `messageParameters` key from `arg_type` to
`arg_value`
(the value, `how`, is unchanged) so it matches the template, and adds a
regression test. `VALUE_NOT_ANY_OR_ALL` is used at exactly this one site, so
no
other call site is affected.
### Why are the changes needed?
`df.dropna(how="foo")` is a common public DataFrame API call. Before this
change
it surfaced an internal `AssertionError`:
```python
>>> df.dropna(how="foo")
AssertionError: Undefined error message parameter for error class:
VALUE_NOT_ANY_OR_ALL. Parameters: {'arg_name': 'how', 'arg_type': 'foo'}
```
instead of the intended, user-facing error:
```python
>>> df.dropna(how="foo")
pyspark.errors.exceptions.captured.PySparkValueError:
[VALUE_NOT_ANY_OR_ALL] Value for `how` must be 'any' or 'all', got 'foo'.
```
The invalid-`how` branch was never exercised by a test, so the broken error
contract went unnoticed.
### Does this PR introduce _any_ user-facing change?
Yes. Passing an invalid `how` to `DataFrame.dropna` (equivalently
`DataFrame.na.drop`) now raises the documented `PySparkValueError`
(`VALUE_NOT_ANY_OR_ALL`) with a clear message, instead of an internal
`AssertionError`. Both are error paths, so valid usage is unchanged; only the
exception raised for invalid input differs.
### How was this patch tested?
Added a regression case to `DataFrameStatTestsMixin.test_dropna` in
`python/pyspark/sql/tests/test_stat.py` that asserts
`dropna(how="foo")` raises `PySparkValueError` with condition
`VALUE_NOT_ANY_OR_ALL` and parameters `{arg_name: "how", arg_value: "foo"}`,
using the same `check_error` idiom as the adjacent `NOT_EXPECTED_TYPE` case.
Ran the suite locally against a real Spark session:
```
python/run-tests --testnames "pyspark.sql.tests.test_stat
DataFrameStatTests.test_dropna"
```
It fails on the unpatched code (the masked `AssertionError`) and passes with
the
fix. `dev/lint-python` (ruff) is clean on both changed files.
### Was this patch authored or co-authored using generative AI tooling?
<!-- Anas: fill this per ASF policy before opening. If AI tooling was used in
authoring, this must read e.g. `Generated-by: <tool name and version>`;
otherwise `No`. See CONTEXT below. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
