[
https://issues.apache.org/jira/browse/SPARK-58575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yicong Huang updated SPARK-58575:
---------------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> Inline redundant error_class parameter in worker.py verify helpers
> ------------------------------------------------------------------
>
> Key: SPARK-58575
> URL: https://issues.apache.org/jira/browse/SPARK-58575
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.3.0
> Reporter: Yicong Huang
> Priority: Major
>
> Follow-up to SPARK-58529, which consolidated the row-count verification
> helpers in python/pyspark/worker.py but left two of them over-parameterized
> with an errorClass argument that is always passed the same constant.
> - verify_output_row_limit: both call sites (SQL_SCALAR_ARROW_ITER_UDF and
> SQL_SCALAR_PANDAS_ITER_UDF) pass error_class="OUTPUT_EXCEEDS_INPUT_ROWS".
> - verify_iterator_exhausted: both call sites pass
> error_class="INPUT_NOT_FULLY_CONSUMED".
> The parameter is never used with a different value. These two error classes
> are semantically bound to these two helpers (the two halves of the
> iterator-UDF "output rows must equal input rows" contract); no other code
> path can reuse them. UDTFs, for example, have no such contract and use a
> different set of verifiers.
> Drop the error_class parameter from each signature and inline the constant
> into the raise. Pure internal simplification: the raised error class and
> message are unchanged, so there is no user-facing behavior change, no
> error-conditions.json change, and no test assertion change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
