Hyukjin Kwon created SPARK-61956:
------------------------------------
Summary: Convert every pandas DataFrame, of a subclass or of
third-party arrays too, as PySpark's createDataFrame does in the session time
zone, and let other threads and signals interrupt the conversion of pandas
strings
Key: SPARK-61956
URL: https://issues.apache.org/jira/browse/SPARK-61956
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61863, from a round-3 review of createDataFrame from
pandas against pyspark-client 4.3.0:
1. A pandas DataFrame subclass, or a frame with a third-party ExtensionArray
(GeoPandas, pandera, BigQuery's db_dtypes), declined the Arrow path and took
the drop-in's native row path. That path converted a naive datetime with the
host time zone rather than the session's. So such frames got timestamps shifted
whenever the host and the session time zones differ (7 hours for a UTC host and
an America/Los_Angeles session), and nanosecond values were truncated silently.
PySpark sends every isinstance(data, pd.DataFrame) through
create_arrow_table_from_pandas with the session time zone
(connect/session.py:556, 634-639; pandas/types.py:758). A probe of 106 cases
wrapped in a subclass differed from PySpark in about 50:
- dict columns became maps where PySpark makes structs;
- ArrowDtype decimals lost their precision;
- object datetimes became timestamp where PySpark gives timestamp_ntz;
- DST gaps converted silently;
- memoryview, VariantVal and list columns raised TypeError.
2. The string fast path (_pandas_string_ipc) looped over every cell natively
with the GIL held and no interrupt checkpoint: with 6M rows of two string
columns, another thread waited up to 1.4 s, and a SIGINT up to 0.5 s.
3. dev/parity/known-differences.md still said that NumPy ndarray input is not
accepted, which SPARK-61103 made it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]