Hyukjin Kwon created SPARK-62004:
------------------------------------
Summary: Pickle DataFrames of failing plans and table arguments,
keep the schema that a copy requests, rebuild same-session deep copies, raise
frame-bound errors at the build, and report Column._expr to Python's GC, as
PySpark
Key: SPARK-62004
URL: https://issues.apache.org/jira/browse/SPARK-62004
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the documented differences of DataFrames, and of the GC
>helpers of SPARK-61971, against PySpark 4.3.0:
- A DataFrame whose plan fails with an AttributeError (a join on a tuple, or on
conditions of a Row) refuses to pickle, where PySpark pickles it and its
unpickle raises NOT_ITERABLE.
- A copy, deep copy or unpickle of a DataFrame requests its schema, as
PySpark's does, but does not keep it: PySpark's schema, printSchema, toArrow,
the set operations and numeric aggregates of the copy read the schema that the
copy requested, where the drop-in's read the cached one, so a view replaced in
between gives other results.
- A TableArg (df.asTable()) does not pickle, where PySpark's pickles and
cloudpickles.
- A deep copy of a DataFrame into the same session builds its relation, so
later requests lose the warnings and origins that PySpark gives at each build.
- The errors of the frame bounds of Window.rowsBetween and rangeBetween
(VALUE_NOT_BETWEEN) are raised at the call, where PySpark raises them when the
plan is built.
- Column._expr and TableArg._subquery_expr are not reported to Python's garbage
collector, so a reference cycle through a UDF of them leaks, and they do not
copy (TypeError: cannot pickle '_ColumnExpression' object).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]