[
https://issues.apache.org/jira/browse/SPARK-62041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62041:
------------------------------------
Assignee: Hyukjin Kwon
> Warn of a window in turn with the picklings of UDFs, send the clauses of a
> merge as they were built, keep the execution info of a failed toPandas, and
> pickle subquery Columns and Column._expr, as PySpark
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62041
> URL: https://issues.apache.org/jira/browse/SPARK-62041
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> From a review of the documented differences of UDFs and pickles against
> PySpark 4.3.0:
> - A window of no partitions warns before a pickling error of a UDF that its
> plan holds, where PySpark's build pickles the UDF first and raises its error
> without the warning.
> - The merge command of mergeInto carries the pickles of its clauses made when
> the merge runs, where PySpark's carries the ones of the clauses, which its
> _build_merge_action makes when each clause is added; a window in a clause
> warned twice.
> - When toPandas fails after its execution, in the conversion or the copy,
> PySpark leaves df.executionInfo as it was (None the first time); the drop-in
> has already kept the execution's.
> - A pickled Column of a subquery (df.scalar(), df.exists()) loses the plan of
> its subquery: its requests fail on the server with INTERNAL_ERROR, where
> PySpark's unpickled Column works, also on a Python worker of the server.
> - pickle.dumps(c._expr) raises "TypeError: cannot pickle '_ColumnExpression'
> object", where PySpark pickles the expression.
> - The pickle of a Column of a UDF or of a UDTF of a return type that PySpark
> does not convert raises at the pickle, where PySpark pickles it and fails its
> request.
> - The reasons of the documented differences of the GC of shared functions and
> of pickled Columns are incomplete.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]