Hyukjin Kwon created SPARK-62004:
------------------------------------

             Summary: Pickle DataFrames of failing plans and table arguments, 
keep the schema that a copy requests, rebuild same-session deep copies, raise 
frame-bound errors at the build, and report Column._expr to Python's GC, as 
PySpark
                 Key: SPARK-62004
                 URL: https://issues.apache.org/jira/browse/SPARK-62004
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the documented differences of DataFrames, and of the GC 
>helpers of SPARK-61971, against PySpark 4.3.0:
- A DataFrame whose plan fails with an AttributeError (a join on a tuple, or on 
conditions of a Row) refuses to pickle, where PySpark pickles it and its 
unpickle raises NOT_ITERABLE.
- A copy, deep copy or unpickle of a DataFrame requests its schema, as 
PySpark's does, but does not keep it: PySpark's schema, printSchema, toArrow, 
the set operations and numeric aggregates of the copy read the schema that the 
copy requested, where the drop-in's read the cached one, so a view replaced in 
between gives other results.
- A TableArg (df.asTable()) does not pickle, where PySpark's pickles and 
cloudpickles.
- A deep copy of a DataFrame into the same session builds its relation, so 
later requests lose the warnings and origins that PySpark gives at each build.
- The errors of the frame bounds of Window.rowsBetween and rangeBetween 
(VALUE_NOT_BETWEEN) are raised at the call, where PySpark raises them when the 
plan is built.
- Column._expr and TableArg._subquery_expr are not reported to Python's garbage 
collector, so a reference cycle through a UDF of them leaks, and they do not 
copy (TypeError: cannot pickle '_ColumnExpression' object).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to