Hyukjin Kwon created SPARK-60954:
------------------------------------
Summary: Type and convert all-null, Decimal and map columns of
pandas input in Spark Connect Rust createDataFrame as PySpark does
Key: SPARK-60954
URL: https://issues.apache.org/jira/browse/SPARK-60954
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect converts a pandas DataFrame through Arrow, which types each
column from its pandas dtype and values. The Rust-backed Python client types
each column from its first non-null value, so these PySpark 4.3.0 results
differ:
* A column whose values are all NaN, NaT or None, such as {{pd.DataFrame({"id":
[1, 2], "s": [np.nan, np.nan]})}}, gives {{double}} (and {{void}} for an object
column of None). The client raises {{CANNOT_DETERMINE_TYPE}}.
* A column of {{Decimal}} values, such as {{Decimal("1.5")}} and
{{Decimal("2.25")}}, gives {{decimal(3,2)}} with the values {{1.50}} and
{{2.25}}. The client gives {{decimal(38,1)}} from the scale of the first value,
and silently truncates {{2.25}} to {{2.2}}.
* A column of dicts with a {{None}} key raises a {{PySparkTypeError}} of the
Arrow conversion. The client panics in Rust ({{PanicException}}), as does the
Rust {{SparkSession::create_dataframe}} with a null map key.
Type and convert these columns as PySpark does, and raise an error for a null
map key.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
