marcuslin123 opened a new pull request, #57250: URL: https://github.com/apache/spark/pull/57250
### What changes were proposed in this pull request? Remove the specialized `VALUE_NOT_ANY_OR_ALL` PySpark error condition and replace its single use site (`DataFrame.dropna`'s `how` validation, in both classic and Spark Connect) with the more general `VALUE_NOT_ALLOWED` condition, which already exists and expresses the same thing via an `allowed_values` parameter. Before: ``` VALUE_NOT_ANY_OR_ALL: "Value for `<arg_name>` must be 'any' or 'all', got '<arg_value>'." ``` After (reusing the existing general condition): ``` VALUE_NOT_ALLOWED: "Value for `<arg_name>` has to be amongst the following values: <allowed_values>." ``` ### Why are the changes needed? `VALUE_NOT_ANY_OR_ALL` is a narrow, single-purpose error condition that duplicates what `VALUE_NOT_ALLOWED` already provides generically. Consolidating reduces the number of error conditions to maintain and keeps `dropna`'s error consistent with other "value must be one of a fixed set" validations across PySpark (e.g. `between`, profiler options). ### Does this PR introduce _any_ user-facing change? Yes, a minor change to the error message for `df.dropna(how=<invalid>)`. Previously: "Value for `how` must be 'any' or 'all', got 'foo'." Now: "Value for `how` has to be amongst the following values: ['any', 'all']." The raised exception type (`PySparkValueError`) is unchanged. ### How was this patch tested? Updated the existing regression test in `test_stat.py` to assert the `VALUE_NOT_ALLOWED` condition and its parameters. Test passes: `python/run-tests --testnames "pyspark.sql.tests.test_stat DataFrameStatTests.test_dropna"` ### Was this patch authored or co-authored using generative AI tooling? Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
