Hyukjin Kwon created SPARK-62007:
------------------------------------

             Summary: Make patches of pyspark.sql.DataFrame reach native frames 
as PySpark's, take PySpark's Arrow path for pandas data of struct subclasses 
and NUL names, and fail names that UTF-8 cannot encode at the request
                 Key: SPARK-62007
                 URL: https://issues.apache.org/jira/browse/SPARK-62007
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the documented differences of DataFrames against PySpark 
>4.3.0, and the other session's review of the last 100 commits:
- A patch of an attribute of pyspark.sql.DataFrame, such as DataFrame.count = 
wrapper or mock.patch.object(DataFrame, "count"), reaches the drop-in's native 
frames, where PySpark's Connect frames define every public name themselves and 
see a patch of the base only through DataFrame.m(df). A wrapper of the base and 
one of pyspark.sql.connect.dataframe.DataFrame that call their saved originals 
make df.count() raise RecursionError, where PySpark returns the count; a patch 
of __getattr__ reaches the frames too.
- pandas createDataFrame with a StructType or StructField subclass, or a column 
name that holds "\x00", takes the row path even with PyArrow, where PySpark 
takes its Arrow path, so 2**53+1 under a DoubleType field converts instead of 
raising, nanoseconds are truncated silently, and errors are other classes.
- createDataFrame of column names that UTF-8 cannot encode fails at the call, 
where PySpark fails at the first request, through toDF.
- The known differences of the surrogate names, the batch reader's option of 
None, the row paths without PyArrow, the invalid UTF-8 of toArrow and toPandas, 
the depth of plans that the server takes, and StorageLevel pickles are stale or 
missing.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to