[
https://issues.apache.org/jira/browse/SPARK-61956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61956:
---------------------------------
Fix Version/s: connect-rust-4.4.0
> Convert every pandas DataFrame, of a subclass or of third-party arrays too,
> as PySpark's createDataFrame does in the session time zone, and let other
> threads and signals interrupt the conversion of pandas strings
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61956
> URL: https://issues.apache.org/jira/browse/SPARK-61956
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.4.0
>
>
> A follow-up of SPARK-61863, from a round-3 review of createDataFrame from
> pandas against pyspark-client 4.3.0:
> 1. A pandas DataFrame subclass, or a frame with a third-party ExtensionArray
> (GeoPandas, pandera, BigQuery's db_dtypes), declined the Arrow path and took
> the drop-in's native row path. That path converted a naive datetime with the
> host time zone rather than the session's. So such frames got timestamps
> shifted whenever the host and the session time zones differ (7 hours for a
> UTC host and an America/Los_Angeles session), and nanosecond values were
> truncated silently. PySpark sends every isinstance(data, pd.DataFrame)
> through create_arrow_table_from_pandas with the session time zone
> (connect/session.py:556, 634-639; pandas/types.py:758). A probe of 106 cases
> wrapped in a subclass differed from PySpark in about 50:
> - dict columns became maps where PySpark makes structs;
> - ArrowDtype decimals lost their precision;
> - object datetimes became timestamp where PySpark gives timestamp_ntz;
> - DST gaps converted silently;
> - memoryview, VariantVal and list columns raised TypeError.
> 2. The string fast path (_pandas_string_ipc) looped over every cell natively
> with the GIL held and no interrupt checkpoint: with 6M rows of two string
> columns, another thread waited up to 1.4 s, and a SIGINT up to 0.5 s.
> 3. dev/parity/known-differences.md still said that NumPy ndarray input is not
> accepted, which SPARK-61103 made it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]