[
https://issues.apache.org/jira/browse/SPARK-59909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-59909.
---------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59170
[https://github.com/apache/spark/pull/59170]
> PySpark raises internal errors instead of intended messages due to undefined
> error conditions and mismatched message parameters
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59909
> URL: https://issues.apache.org/jira/browse/SPARK-59909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 4.1.0, 4.0.0, 4.2.0
> Reporter: Jubin Soni
> Assignee: Jubin Soni
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Several PySpark call sites raise a PySparkException with an error condition
> that does not exist in python/pyspark/errors/error-conditions.json, or with
> messageParameters that do not match the condition's template. Constructing
> the exception itself fails, so the user gets an unrelated internal error
> instead of the intended message, often with a different exception type (e.g.
> ValueError instead of TypeError).
> h3. Undefined error conditions (fails with "ValueError: Cannot find main
> error class ...")
> * NOT_STR - DataStreamReader.changes() (pyspark/sql/streaming/readwriter.py).
> NOT_STR was consolidated into NOT_EXPECTED_TYPE by SPARK-56179 while this new
> usage was added concurrently.
> * VARIANT_SIZE_LIMIT_EXCEEDED - pyspark/sql/variant_utils.py (3 places)
> * INVALID_OPERATION_UUID_ID - pyspark/sql/connect/client/core.py
> * UNSUPPORTED_FILTER - pyspark/sql/worker/data_source_pushdown_filters.py
> h3. Mismatched message parameters (fails with "AssertionError: Undefined
> error message parameter ...")
> * DATA_SOURCE_EXTRANEOUS_FILTERS -
> pyspark/sql/worker/data_source_pushdown_filters.py passes extra "input" and
> "extraneous" parameters
> * NO_ACTIVE_SESSION - pyspark/sql/connect/dataframe.py passes an extra
> "operator" parameter
> h3. Reproduction
> {code:python}
> >>> spark.readStream.changes(123)
> ValueError: Cannot find main error class 'NOT_STR'
> {code}
> Expected: PySparkTypeError saying that `tableName` should be a str.
> The existing test for DATA_SOURCE_EXTRANEOUS_FILTERS
> (test_python_datasource.py) does not catch this, because it only asserts that
> the condition name appears in the exception text, which is also true of the
> AssertionError.
> The JVM side guards against this in SparkThrowableSuite, but PySpark has no
> equivalent check.
> h3. Proposed fix
> Fix each call site, and add a test that scans PySpark sources for errorClass
> literals and verifies that each one is defined and that its literal
> messageParameters keys match the template.
> UNSUPPORTED_PIPELINES_DATASET_TYPE has the same kind of problem but is
> already tracked in SPARK-57965, so it is out of scope here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
