Yicong Huang created SPARK-58575:
------------------------------------
Summary: Inline redundant error_class parameter in worker.py
verify helpers
Key: SPARK-58575
URL: https://issues.apache.org/jira/browse/SPARK-58575
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: Yicong Huang
Follow-up to SPARK-58529, which consolidated the row-count verification helpers
in python/pyspark/worker.py but left two of them over-parameterized with an
errorClass argument that is always passed the same constant.
- verify_output_row_limit: both call sites (SQL_SCALAR_ARROW_ITER_UDF and
SQL_SCALAR_PANDAS_ITER_UDF) pass error_class="OUTPUT_EXCEEDS_INPUT_ROWS".
- verify_iterator_exhausted: both call sites pass
error_class="INPUT_NOT_FULLY_CONSUMED".
The parameter is never used with a different value. These two error classes are
semantically bound to these two helpers (the two halves of the iterator-UDF
"output rows must equal input rows" contract); no other code path can reuse
them. UDTFs, for example, have no such contract and use a different set of
verifiers.
Drop the error_class parameter from each signature and inline the constant into
the raise. Pure internal simplification: the raised error class and message are
unchanged, so there is no user-facing behavior change, no error-conditions.json
change, and no test assertion change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
