Hyukjin Kwon created SPARK-62041:
------------------------------------
Summary: Warn of a window in turn with the picklings of UDFs, send
the clauses of a merge as they were built, keep the execution info of a failed
toPandas, and pickle subquery Columns and Column._expr, as PySpark
Key: SPARK-62041
URL: https://issues.apache.org/jira/browse/SPARK-62041
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the documented differences of UDFs and pickles against
>PySpark 4.3.0:
- A window of no partitions warns before a pickling error of a UDF that its
plan holds, where PySpark's build pickles the UDF first and raises its error
without the warning.
- The merge command of mergeInto carries the pickles of its clauses made when
the merge runs, where PySpark's carries the ones of the clauses, which its
_build_merge_action makes when each clause is added; a window in a clause
warned twice.
- When toPandas fails after its execution, in the conversion or the copy,
PySpark leaves df.executionInfo as it was (None the first time); the drop-in
has already kept the execution's.
- A pickled Column of a subquery (df.scalar(), df.exists()) loses the plan of
its subquery: its requests fail on the server with INTERNAL_ERROR, where
PySpark's unpickled Column works, also on a Python worker of the server.
- pickle.dumps(c._expr) raises "TypeError: cannot pickle '_ColumnExpression'
object", where PySpark pickles the expression.
- The pickle of a Column of a UDF or of a UDTF of a return type that PySpark
does not convert raises at the pickle, where PySpark pickles it and fails its
request.
- The reasons of the documented differences of the GC of shared functions and
of pickled Columns are incomplete.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]