This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 393b630d9286 [SPARK-57964][PYTHON] Fix DataFrame.dropna
VALUE_NOT_ANY_OR_ALL parameter name mismatch
393b630d9286 is described below
commit 393b630d92860d5f25000c3bc33d48ce6e645076
Author: Marcus Lin <[email protected]>
AuthorDate: Fri Jul 10 14:26:21 2026 +0800
[SPARK-57964][PYTHON] Fix DataFrame.dropna VALUE_NOT_ANY_OR_ALL parameter
name mismatch
### What changes were proposed in this pull request?
Fix the `messageParameters` key in `DataFrame.dropna` validation from
`arg_type` to `arg_value`, matching the `VALUE_NOT_ANY_OR_ALL` error template
which interpolates `<arg_value>`.
Also fix the Spark Connect `dropna` implementation which used the wrong
error class (`CANNOT_BE_EMPTY` instead of `VALUE_NOT_ANY_OR_ALL`) with
mismatched parameters, making both classic and Connect paths consistent.
### Why are the changes needed?
`df.dropna(how="foo")` is meant to raise a clear `PySparkValueError` saying
"Value for `how` must be 'any' or 'all', got 'foo'." Instead, the mismatch
between the template placeholder (`<arg_value>`) and the provided parameter key
(`arg_type`) causes an internal assertion failure, surfacing an opaque
`AssertionError` to the user.
The Connect path had a similar issue — it used `CANNOT_BE_EMPTY` (which
expects `<item>`) but passed `{"arg_name": ..., "arg_value": ...}`, also
resulting in an `AssertionError`.
### Does this PR introduce _any_ user-facing change?
Yes. Users who pass an invalid `how` argument to `DataFrame.dropna` will
now see (in both classic and Connect modes):
```
PySparkValueError: [VALUE_NOT_ANY_OR_ALL] Value for `how` must be 'any' or
'all', got 'foo'.
```
Instead of the previous opaque `AssertionError`.
### How was this patch tested?
Added a regression test in `test_stat.py` within the existing `test_dropna`
method that verifies `dropna(how="foo")` raises `PySparkValueError` with the
correct error class and parameters.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for
implementation guidance.
Closes #57135 from marcuslin123/SPARK-57964-fix-dropna-error.
Lead-authored-by: Marcus Lin
<[email protected]>
Co-authored-by: marcus <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 8e13ddb99951f62734889e419ae68898ebde311b)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/sql/classic/dataframe.py | 2 +-
python/pyspark/sql/connect/dataframe.py | 2 +-
python/pyspark/sql/tests/test_stat.py | 11 +++++++++++
3 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/sql/classic/dataframe.py
b/python/pyspark/sql/classic/dataframe.py
index a4288270a5ad..dfe67a64aa53 100644
--- a/python/pyspark/sql/classic/dataframe.py
+++ b/python/pyspark/sql/classic/dataframe.py
@@ -1356,7 +1356,7 @@ class DataFrame(ParentDataFrame, PandasMapOpsMixin,
PandasConversionMixin):
if how is not None and how not in ["any", "all"]:
raise PySparkValueError(
errorClass="VALUE_NOT_ANY_OR_ALL",
- messageParameters={"arg_name": "how", "arg_type": how},
+ messageParameters={"arg_name": "how", "arg_value": how},
)
if subset is None:
diff --git a/python/pyspark/sql/connect/dataframe.py
b/python/pyspark/sql/connect/dataframe.py
index 7551210036c9..6ee47341639c 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -1394,7 +1394,7 @@ class DataFrame(ParentDataFrame):
min_non_nulls = None
else:
raise PySparkValueError(
- errorClass="CANNOT_BE_EMPTY",
+ errorClass="VALUE_NOT_ANY_OR_ALL",
messageParameters={"arg_name": "how", "arg_value":
str(how)},
)
diff --git a/python/pyspark/sql/tests/test_stat.py
b/python/pyspark/sql/tests/test_stat.py
index e3655e24c968..441198f190f2 100644
--- a/python/pyspark/sql/tests/test_stat.py
+++ b/python/pyspark/sql/tests/test_stat.py
@@ -28,6 +28,7 @@ from pyspark.sql.types import (
from pyspark.errors import (
AnalysisException,
PySparkTypeError,
+ PySparkValueError,
)
from pyspark.testing.sqlutils import ReusedSQLTestCase
@@ -127,6 +128,16 @@ class DataFrameStatTestsMixin:
},
)
+ # Regression test: invalid 'how' should raise PySparkValueError, not
AssertionError
+ with self.assertRaises(PySparkValueError) as pe:
+ self.spark.createDataFrame([("Alice", 50, 80.1)],
schema).dropna(how="foo")
+
+ self.check_error(
+ exception=pe.exception,
+ errorClass="VALUE_NOT_ANY_OR_ALL",
+ messageParameters={"arg_name": "how", "arg_value": "foo"},
+ )
+
def test_fillna(self):
schema = StructType(
[
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
