jubins opened a new pull request, #59170: URL: https://github.com/apache/spark/pull/59170
### What changes were proposed in this pull request? Fixes [SPARK-59909](https://issues.apache.org/jira/browse/SPARK-59909). Several PySpark call sites raise a `PySparkException` whose error condition is not defined in `python/pyspark/errors/error-conditions.json`, or whose `messageParameters` do not match the condition's template. In both cases constructing the exception itself fails, so the user gets an unrelated internal error instead of the intended message. The exception type can change too, e.g. `ValueError` instead of `TypeError`. **Undefined error conditions** (fail with `ValueError: Cannot find main error class '...'`): - `NOT_STR` in `DataStreamReader.changes()`. SPARK-56179 consolidated `NOT_STR` into `NOT_EXPECTED_TYPE`, while this new usage was added concurrently. - `VARIANT_SIZE_LIMIT_EXCEEDED` in `VariantBuilder` (3 places). - `INVALID_OPERATION_UUID_ID` in the Spark Connect client. - `UNSUPPORTED_FILTER` in the Python data source filter pushdown worker. **Mismatched message parameters** (fail with `AssertionError: Undefined error message parameter ...`): - `DATA_SOURCE_EXTRANEOUS_FILTERS` is raised with `input` and `extraneous` parameters that its template does not contain. - `NO_ACTIVE_SESSION` is raised with an `operator` parameter that its template does not contain. **Brief change log** - `python/pyspark/sql/streaming/readwriter.py`: `DataStreamReader.changes()` raises `NOT_EXPECTED_TYPE` instead of the removed `NOT_STR`. - `python/pyspark/errors/error-conditions.json`: - Defines `INVALID_OPERATION_UUID_ID` (same template as the existing `INVALID_SESSION_UUID_ID`). - Defines `UNSUPPORTED_FILTER` and `VARIANT_SIZE_LIMIT_EXCEEDED`. - Adds the input and extraneous filters to the `DATA_SOURCE_EXTRANEOUS_FILTERS` message. The caller already passed them, and they help users debug their `pushFilters()` implementation. - `python/pyspark/sql/variant_utils.py`: passes the size limit to `VARIANT_SIZE_LIMIT_EXCEEDED`, so the message says what the limit is. - `python/pyspark/sql/connect/dataframe.py`: drops the unused `operator` parameter from `NO_ACTIVE_SESSION`. - `python/pyspark/errors/tests/test_errors.py`: adds a test that scans the non-test PySpark sources for `PySpark*` exceptions raised with a literal `errorClass`. For each one it builds the message the same way it is built at raise time, so it fails on undefined conditions and on literal parameter keys that don't match the template. - Regression tests in `test_readwriter.py`, `test_types.py` and `test_python_datasource.py`. ### Why are the changes needed? The intended error messages never reach users. For example: ```python >>> spark.readStream.changes(123) ValueError: Cannot find main error class 'NOT_STR' ``` Expected: ``` PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `tableName` should be str, got int. ``` A Python data source whose `pushFilters()` returns a filter that was not in its input gets `AssertionError: Undefined error message parameter for error class: DATA_SOURCE_EXTRANEOUS_FILTERS ...` instead of the explanation of what went wrong. The existing `test_extraneous_filter` did not catch this. It only asserts that `DATA_SOURCE_EXTRANEOUS_FILTERS` appears in the exception text, and it also appears in the `AssertionError`. The JVM side guards against undefined conditions in `SparkThrowableSuite`, but PySpark had no equivalent check, which is how these slipped in. The new test closes that gap. `UNSUPPORTED_PIPELINES_DATASET_TYPE` has the same kind of problem, but it is already being fixed in SPARK-57965 (#57005). The new test skips that one call site with a comment pointing at the ticket, and the entry should be removed once that PR lands. ### Does this PR introduce _any_ user-facing change? Yes. Users now get the intended error messages instead of internal errors: - `DataStreamReader.changes()` with a non-str table name raises `PySparkTypeError` (`NOT_EXPECTED_TYPE`) instead of `ValueError`. - Exceeding the variant size limit in `VariantVal.parseJson` raises `PySparkValueError` (`VARIANT_SIZE_LIMIT_EXCEEDED`) with the limit in the message. - An invalid operation ID in the Spark Connect client raises `PySparkValueError` (`INVALID_OPERATION_UUID_ID`). - An unknown pushed-down filter raises `PySparkNotImplementedError` (`UNSUPPORTED_FILTER`). - `DATA_SOURCE_EXTRANEOUS_FILTERS` shows its message, now including the input and extraneous filters. - `NO_ACTIVE_SESSION` shows its message. Most of these have been broken since 4.1.0 (`VARIANT_SIZE_LIMIT_EXCEEDED` since 4.0.0). ### How was this patch tested? New and updated tests: - `ErrorsTest.test_error_classes_used_in_source_are_valid`: run against the sources on current master, it reports exactly the 8 call sites fixed here. It passes with this PR. - `ReadwriterTests.test_streaming_changes_rejects_non_str_table_name` (classic only, since Spark Connect's `changes()` does not validate the argument type). - `DataTypeTests.test_variant_builder_size_limit` - `PythonDataSourceWorkerUtilsTests.test_deserialize_unsupported_filter` - `test_extraneous_filter` now checks the actual message. I confirmed it fails when the template change is reverted. All of these suites pass (310 tests): ``` python -m pytest pyspark/errors/tests/test_errors.py \ pyspark/sql/tests/test_readwriter.py \ pyspark/sql/tests/test_python_datasource.py \ pyspark/sql/tests/test_types.py \ pyspark/sql/tests/connect/test_parity_readwriter.py ``` `ruff check`, `ruff format --check`, `mypy` and `dev/lint-python --custom-pyspark-error` pass. ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - Claude Code was used as a pair-programming assistant. All changes were reviewed and verified by the author. Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
