jubins opened a new pull request, #59458: URL: https://github.com/apache/spark/pull/59458
### What changes were proposed in this pull request? Backport of #59170 to `branch-4.3`. Several PySpark call sites raise a `PySparkException` whose error condition is not defined in `error-conditions.json`, or whose `messageParameters` do not match the template. Building the exception fails, so users get an internal error (`ValueError: Cannot find main error class ...` or `AssertionError: Undefined error message parameter ...`) instead of the intended message. This PR fixes those call sites and adds a test that validates every literal error class raised in PySpark sources. The change is the same as the merged commit, except for two test-only adjustments needed on this branch: - `PythonDataSourceWorkerUtilsTests` does not exist on `branch-4.3`, so the new `test_deserialize_unsupported_filter` is added in a minimal class of that name, without the unrelated profiler test from master. - `DataTypeTests` does not mix in `PySparkErrorTestUtils` on this branch, so `test_variant_builder_size_limit` asserts `getCondition()` and `getMessageParameters()` directly instead of using `check_error`. ### Why are the changes needed? All eight affected call sites exist on `branch-4.3`, so users of 4.3.x get internal errors instead of the intended messages. For example: ```python >>> spark.readStream.changes(123) ValueError: Cannot find main error class 'NOT_STR' ``` ### Does this PR introduce _any_ user-facing change? Yes, the same as #59170: the affected APIs raise their intended error messages instead of internal errors. The variant size error is defined as `VARIANT_SIZE_LIMIT`, matching the JVM condition name. ### How was this patch tested? - The new `ErrorsTest.test_error_classes_used_in_source_are_valid` passes against the `branch-4.3` sources, so no other broken raise sites exist on this branch. - `test_errors.py`, `DataTypeTests` and `PythonDataSourceWorkerUtilsTests` pass locally (22 tests). - The JVM-backed tests (`test_readwriter.py`, `test_python_datasource.py`) run in CI. ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - Claude Code was used as a pair-programming assistant. All changes were reviewed and verified by the author. Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
