[ 
https://issues.apache.org/jira/browse/SPARK-61956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61956:
-----------------------------------
    Labels: pull-request-available  (was: )

> Convert every pandas DataFrame, of a subclass or of third-party arrays too, 
> as PySpark's createDataFrame does in the session time zone, and let other 
> threads and signals interrupt the conversion of pandas strings
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61956
>                 URL: https://issues.apache.org/jira/browse/SPARK-61956
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> A follow-up of SPARK-61863, from a round-3 review of createDataFrame from 
> pandas against pyspark-client 4.3.0:
> 1. A pandas DataFrame subclass, or a frame with a third-party ExtensionArray 
> (GeoPandas, pandera, BigQuery's db_dtypes), declined the Arrow path and took 
> the drop-in's native row path. That path converted a naive datetime with the 
> host time zone rather than the session's. So such frames got timestamps 
> shifted whenever the host and the session time zones differ (7 hours for a 
> UTC host and an America/Los_Angeles session), and nanosecond values were 
> truncated silently. PySpark sends every isinstance(data, pd.DataFrame) 
> through create_arrow_table_from_pandas with the session time zone 
> (connect/session.py:556, 634-639; pandas/types.py:758). A probe of 106 cases 
> wrapped in a subclass differed from PySpark in about 50:
> - dict columns became maps where PySpark makes structs;
> - ArrowDtype decimals lost their precision;
> - object datetimes became timestamp where PySpark gives timestamp_ntz;
> - DST gaps converted silently;
> - memoryview, VariantVal and list columns raised TypeError.
> 2. The string fast path (_pandas_string_ipc) looped over every cell natively 
> with the GIL held and no interrupt checkpoint: with 6M rows of two string 
> columns, another thread waited up to 1.4 s, and a SIGINT up to 0.5 s.
> 3. dev/parity/known-differences.md still said that NumPy ndarray input is not 
> accepted, which SPARK-61103 made it.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to