[ 
https://issues.apache.org/jira/browse/SPARK-62004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62004:
-----------------------------------
    Labels: pull-request-available  (was: )

> Pickle DataFrames of failing plans and table arguments, keep the schema that 
> a copy requests, rebuild same-session deep copies, raise frame-bound errors 
> at the build, and report Column._expr to Python's GC, as PySpark
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62004
>                 URL: https://issues.apache.org/jira/browse/SPARK-62004
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> From a review of the documented differences of DataFrames, and of the GC 
> helpers of SPARK-61971, against PySpark 4.3.0:
> - A DataFrame whose plan fails with an AttributeError (a join on a tuple, or 
> on conditions of a Row) refuses to pickle, where PySpark pickles it and its 
> unpickle raises NOT_ITERABLE.
> - A copy, deep copy or unpickle of a DataFrame requests its schema, as 
> PySpark's does, but does not keep it: PySpark's schema, printSchema, toArrow, 
> the set operations and numeric aggregates of the copy read the schema that 
> the copy requested, where the drop-in's read the cached one, so a view 
> replaced in between gives other results.
> - A TableArg (df.asTable()) does not pickle, where PySpark's pickles and 
> cloudpickles.
> - A deep copy of a DataFrame into the same session builds its relation, so 
> later requests lose the warnings and origins that PySpark gives at each build.
> - The errors of the frame bounds of Window.rowsBetween and rangeBetween 
> (VALUE_NOT_BETWEEN) are raised at the call, where PySpark raises them when 
> the plan is built.
> - Column._expr and TableArg._subquery_expr are not reported to Python's 
> garbage collector, so a reference cycle through a UDF of them leaks, and they 
> do not copy (TypeError: cannot pickle '_ColumnExpression' object).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to