Hyukjin Kwon created SPARK-61104:
------------------------------------
Summary: Convert pandas input without a schema as PySpark's Arrow
conversion does in the Python client
Key: SPARK-61104
URL: https://issues.apache.org/jira/browse/SPARK-61104
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{spark.createDataFrame}} of a pandas DataFrame without a
schema, or with a list of names, converts each column with PySpark's Arrow
conversion ({{create_arrow_table_from_pandas}}):
* Arrow infers the type of each column, except that datetimes are timestamps
and timedeltas are day-time intervals. A naive datetime is a time of the
session time zone.
* The table is sent without a schema, so the server infers its types from
Arrow's.
* The columns are named by their labels, as strings, or by the given names
(with {{_n}} for the rest), with {{toDF}}.
* With {{spark.sql.execution.pandas.inferPandasDictAsMap}}, a column of dicts
is a map, else a struct.
* A column of values that Arrow cannot convert raises the {{PySparkValueError}}
of the conversion ("Exception thrown when converting pandas.Series (object)
with name 'a' to Arrow Array ...").
The Python client instead converted such a frame row by row, typed by the first
value of each column:
* A column of dicts was a map: {{pd.DataFrame({"a": [{"k": 1}]})}} gave {{a
map<string,bigint>}} where PySpark gives {{a struct<k:bigint>}}.
* A naive datetime was a time of the client's time zone. With a client in KST
and a session in UTC, {{2020-01-01 01:02:03}} collected as 01:02:03, where
PySpark collects 10:02:03.
* A column of an int and a str raised {{PySparkRuntimeError}} ("Type mismatch
in row data").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
