Anas Khan created SPARK-57964:
---------------------------------
Summary: DataFrame.dropna raises AssertionError instead of
PySparkValueError on invalid 'how'
Key: SPARK-57964
URL: https://issues.apache.org/jira/browse/SPARK-57964
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 4.1.0
Reporter: Anas Khan
DataFrame.dropna validates the `how` argument and is meant to raise
PySparkValueError with error class VALUE_NOT_ANY_OR_ALL on an invalid value.
The messageParameters passed a key named `arg_type`, but the
VALUE_NOT_ANY_OR_ALL template interpolates <arg_value>:
"Value for `<arg_name>` must be 'any' or 'all', got '<arg_value>'."
ErrorClassesReader.get_error_message (python/pyspark/errors/utils.py) asserts
that the set of template placeholders equals the set of provided message
parameters. Since \{arg_name, arg_value} != \{arg_name, arg_type}, the assertion
fires inside PySparkException.__init__, so df.dropna(how="foo") surfaces an
opaque AssertionError instead of the intended PySparkValueError.
Fix: rename the messageParameters key `arg_type` -> `arg_value` at that single
call site (the value, `how`, is unchanged) and add a regression test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
