Hyukjin Kwon created SPARK-61875:
------------------------------------

             Summary: Keep the cached schema of deferred-error DataFrames, the 
instance attributes of copies and pickles, PySpark's Python origins, and the 
window warning of no partitions
                 Key: SPARK-61875
                 URL: https://issues.apache.org/jira/browse/SPARK-61875
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Four plan-time gaps against PySpark 4.3.0:
- The DataFrames whose counts or seeds fail when their plan is built 
(SPARK-61766) drop the cached schema of their parent, where PySpark's limit, 
coalesce, repartition, repartitionByRange, repartitionById, sample and 
randomSplit copy _cached_schema.
- Copies and pickles of df.na, df.stat, spark.tvf, spark.dataSource, 
PandasCogroupedOps, StreamingQuery, StreamingQueryManager, RuntimeConf and the 
merge builder drop their instance attributes; the merge actions and 
SparkSession.Builder have no __dict__; and the pickle errors of spark.streams, 
the conf and the merge builder differ from PySpark's.
- The Python origins of groupBy().count(), the numeric aggregates, agg of a 
dict, repartitionByRange of names and the conditions of a join of a list of 
Columns differ from PySpark's, which applies col and asc_nulls_first to the 
columns only, aliases the count, and makes the & of the join's conditions when 
it builds the plan.
- PySpark warns "WARN WindowExpression: No Partition Defined for Window 
operation!" at each plan build of a window of no partitions; the drop-in never 
warns.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to