anxkhn opened a new pull request, #57005:
URL: https://github.com/apache/spark/pull/57005
### What changes were proposed in this pull request?
`SparkConnectGraphElementRegistry.register_output`
(`python/pyspark/pipelines/spark_connect_graph_element_registry.py`) has two
`else`-branch raise sites that report an unsupported pipeline output type.
Both
raised:
```python
raise PySparkTypeError(
errorClass="UNSUPPORTED_PIPELINES_DATASET_TYPE",
messageParameters={"output_type": type(output).__name__},
)
```
but the `UNSUPPORTED_PIPELINES_DATASET_TYPE` template in
`python/pyspark/errors/error-conditions.json` interpolates `<dataset_type>`:
```
"Unsupported pipelines dataset type: <dataset_type>."
```
`ErrorClassesReader.get_error_message` (`python/pyspark/errors/utils.py`)
asserts
that the set of template placeholders equals the set of provided message
parameters. Since `{"dataset_type"} != {"output_type"}`, the assertion fired
inside `PySparkException.__init__`, so the intended `PySparkTypeError` was
replaced by an opaque `AssertionError`.
This PR renames the message parameter key from `"output_type"` to
`"dataset_type"` at both raise sites (the value, `type(output).__name__`, is
unchanged), so it matches the template placeholder. The error class is
referenced only at these two Python sites plus the template, so the change is
fully local. A regression test is added.
### Why are the changes needed?
Registering a pipeline output of an unsupported type is meant to surface a
clear
`PySparkTypeError` with condition `UNSUPPORTED_PIPELINES_DATASET_TYPE`.
Because of
the parameter-name mismatch it instead raised an unrelated `AssertionError`
from
the error-formatting layer, hiding the real cause from the user:
```
AssertionError: Undefined error message parameter for error class:
UNSUPPORTED_PIPELINES_DATASET_TYPE. Parameters: {'output_type': 'Output'}
```
The intended error is never constructed, so callers cannot catch
`PySparkTypeError` or read the intended message.
### Does this PR introduce _any_ user-facing change?
Yes, an error-behavior change on the failure path only. Registering a
pipeline
output whose type is unsupported now raises the intended, documented error.
Before:
```
AssertionError: Undefined error message parameter for error class:
UNSUPPORTED_PIPELINES_DATASET_TYPE. Parameters: {'output_type': 'Output'}
```
After:
```
pyspark.errors.exceptions.base.PySparkTypeError:
[UNSUPPORTED_PIPELINES_DATASET_TYPE] Unsupported pipelines dataset type:
Output.
```
This is a fix relative to the unreleased `master` behavior (the affected
`messageParameters={"output_type": ...}` code path); no released Spark
version
behavior changes here.
### How was this patch tested?
Added
`SparkConnectPipelinesTest.test_register_output_unsupported_dataset_type`
in `python/pyspark/pipelines/tests/test_spark_connect.py`, which builds a
`SparkConnectGraphElementRegistry`, calls `register_output` with a bare
`Output`
(an unsupported type, since it is not `Table`/`TemporaryView`/`Sink`), and
asserts `PySparkTypeError` with `getCondition() ==
"UNSUPPORTED_PIPELINES_DATASET_TYPE"` and `getMessageParameters() ==
{"dataset_type": "Output"}`.
The test extends `ReusedConnectTestCase`, so CI runs it against a real Spark
Connect server. Verified locally against a live Connect server:
- New test on this change: `Ran 1 test ... OK`.
- Temporarily reverting the fix (restoring the `"output_type"` key) makes the
same test fail with the `AssertionError` above, confirming it is a genuine
RED -> GREEN regression test.
- Full `pyspark.pipelines.tests.test_spark_connect` module: `Ran 5 tests ...
OK`,
no regressions.
Lint (repo-pinned versions): `black==26.3.1` `--check`, `ruff==0.14.8`
`check`,
and `mypy==1.19.1` all pass on the two changed files.
### Was this patch authored or co-authored using generative AI tooling?
<!-- Anas: decide and fill this in before opening the PR. The ASF asks that
if
generative AI tooling was used, you include "Generated-by: <tool> <version>".
See https://www.apache.org/legal/generative-tooling.html . -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
