Hyukjin Kwon created SPARK-61956:
------------------------------------

             Summary: Convert every pandas DataFrame, of a subclass or of 
third-party arrays too, as PySpark's createDataFrame does in the session time 
zone, and let other threads and signals interrupt the conversion of pandas 
strings
                 Key: SPARK-61956
                 URL: https://issues.apache.org/jira/browse/SPARK-61956
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-61863, from a round-3 review of createDataFrame from 
pandas against pyspark-client 4.3.0:

1. A pandas DataFrame subclass, or a frame with a third-party ExtensionArray 
(GeoPandas, pandera, BigQuery's db_dtypes), declined the Arrow path and took 
the drop-in's native row path. That path converted a naive datetime with the 
host time zone rather than the session's. So such frames got timestamps shifted 
whenever the host and the session time zones differ (7 hours for a UTC host and 
an America/Los_Angeles session), and nanosecond values were truncated silently. 
PySpark sends every isinstance(data, pd.DataFrame) through 
create_arrow_table_from_pandas with the session time zone 
(connect/session.py:556, 634-639; pandas/types.py:758). A probe of 106 cases 
wrapped in a subclass differed from PySpark in about 50:
- dict columns became maps where PySpark makes structs;
- ArrowDtype decimals lost their precision;
- object datetimes became timestamp where PySpark gives timestamp_ntz;
- DST gaps converted silently;
- memoryview, VariantVal and list columns raised TypeError.

2. The string fast path (_pandas_string_ipc) looped over every cell natively 
with the GIL held and no interrupt checkpoint: with 6M rows of two string 
columns, another thread waited up to 1.4 s, and a SIGINT up to 0.5 s.

3. dev/parity/known-differences.md still said that NumPy ndarray input is not 
accepted, which SPARK-61103 made it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to