This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 76c06881aa7e [SPARK-58114][PYTHON] Consolidate VALUE_NOT_ANY_OR_ALL
into VALUE_NOT_ALLOWED
76c06881aa7e is described below
commit 76c06881aa7e5880d64af3a3d4caec027c7fb2ec
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Wed Jul 15 08:58:40 2026 +0800
[SPARK-58114][PYTHON] Consolidate VALUE_NOT_ANY_OR_ALL into
VALUE_NOT_ALLOWED
### What changes were proposed in this pull request?
Consolidate the specialized PySpark `VALUE_NOT_ANY_OR_ALL` error condition
into the existing
`VALUE_NOT_ALLOWED` condition. The classic and Spark Connect
implementations of
`DataFrame.dropna` now report the allowed values through the generic
condition, and the shared
test expectation is updated accordingly.
### Why are the changes needed?
`VALUE_NOT_ANY_OR_ALL` is used only for `DataFrame.dropna(how=...)` and
duplicates the existing
generic condition for arguments restricted to a fixed set of values.
Removing it reduces the
number of narrowly scoped error conditions and aligns this validation with
other PySpark APIs.
### Does this PR introduce _any_ user-facing change?
Yes. Invalid values for `DataFrame.dropna(how=...)` now use the
`VALUE_NOT_ALLOWED` error
condition and its generic message. Valid calls and exception types are
unchanged.
### How was this patch tested?
The focused PySpark test was not run. The JSON error-condition file was
parsed successfully,
`git diff --check` passed, and the existing shared error assertion was
updated for both classic
and Spark Connect execution modes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57236 from zhengruifeng/consolidate-value-not-any-or-all-dev4.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/errors/error-conditions.json | 5 -----
python/pyspark/sql/classic/dataframe.py | 4 ++--
python/pyspark/sql/connect/dataframe.py | 4 ++--
python/pyspark/sql/tests/test_stat.py | 4 ++--
4 files changed, 6 insertions(+), 11 deletions(-)
diff --git a/python/pyspark/errors/error-conditions.json
b/python/pyspark/errors/error-conditions.json
index 38417cbf0188..56a8144b9fae 100644
--- a/python/pyspark/errors/error-conditions.json
+++ b/python/pyspark/errors/error-conditions.json
@@ -1198,11 +1198,6 @@
"Value for `<arg_name>` has to be amongst the following values:
<allowed_values>."
]
},
- "VALUE_NOT_ANY_OR_ALL": {
- "message": [
- "Value for `<arg_name>` must be 'any' or 'all', got '<arg_value>'."
- ]
- },
"VALUE_NOT_BETWEEN": {
"message": [
"Value for `<arg_name>` must be between <min> and <max>."
diff --git a/python/pyspark/sql/classic/dataframe.py
b/python/pyspark/sql/classic/dataframe.py
index dfe67a64aa53..ca4b1f17de2a 100644
--- a/python/pyspark/sql/classic/dataframe.py
+++ b/python/pyspark/sql/classic/dataframe.py
@@ -1355,8 +1355,8 @@ class DataFrame(ParentDataFrame, PandasMapOpsMixin,
PandasConversionMixin):
) -> ParentDataFrame:
if how is not None and how not in ["any", "all"]:
raise PySparkValueError(
- errorClass="VALUE_NOT_ANY_OR_ALL",
- messageParameters={"arg_name": "how", "arg_value": how},
+ errorClass="VALUE_NOT_ALLOWED",
+ messageParameters={"arg_name": "how", "allowed_values":
"['any', 'all']"},
)
if subset is None:
diff --git a/python/pyspark/sql/connect/dataframe.py
b/python/pyspark/sql/connect/dataframe.py
index 6ee47341639c..d0ef3061e58a 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -1394,8 +1394,8 @@ class DataFrame(ParentDataFrame):
min_non_nulls = None
else:
raise PySparkValueError(
- errorClass="VALUE_NOT_ANY_OR_ALL",
- messageParameters={"arg_name": "how", "arg_value":
str(how)},
+ errorClass="VALUE_NOT_ALLOWED",
+ messageParameters={"arg_name": "how", "allowed_values":
"['any', 'all']"},
)
if thresh is not None:
diff --git a/python/pyspark/sql/tests/test_stat.py
b/python/pyspark/sql/tests/test_stat.py
index 441198f190f2..5eee90526ce7 100644
--- a/python/pyspark/sql/tests/test_stat.py
+++ b/python/pyspark/sql/tests/test_stat.py
@@ -134,8 +134,8 @@ class DataFrameStatTestsMixin:
self.check_error(
exception=pe.exception,
- errorClass="VALUE_NOT_ANY_OR_ALL",
- messageParameters={"arg_name": "how", "arg_value": "foo"},
+ errorClass="VALUE_NOT_ALLOWED",
+ messageParameters={"arg_name": "how", "allowed_values": "['any',
'all']"},
)
def test_fillna(self):
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
