[
https://issues.apache.org/jira/browse/SPARK-61104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61104.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 308
[https://github.com/apache/spark-connect-rust/pull/308]
> Convert pandas input without a schema as PySpark's Arrow conversion does in
> the Python client
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-61104
> URL: https://issues.apache.org/jira/browse/SPARK-61104
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{spark.createDataFrame}} of a pandas DataFrame without a
> schema, or with a list of names, converts each column with PySpark's Arrow
> conversion ({{create_arrow_table_from_pandas}}):
> * Arrow infers the type of each column, except that datetimes are timestamps
> and timedeltas are day-time intervals. A naive datetime is a time of the
> session time zone.
> * The table is sent without a schema, so the server infers its types from
> Arrow's.
> * The columns are named by their labels, as strings, or by the given names
> (with {{_n}} for the rest), with {{toDF}}.
> * With {{spark.sql.execution.pandas.inferPandasDictAsMap}}, a column of dicts
> is a map, else a struct.
> * A column of values that Arrow cannot convert raises the
> {{PySparkValueError}} of the conversion ("Exception thrown when converting
> pandas.Series (object) with name 'a' to Arrow Array ...").
> The Python client instead converted such a frame row by row, typed by the
> first value of each column:
> * A column of dicts was a map: {{pd.DataFrame({"a": [{"k": 1}]})}} gave {{a
> map<string,bigint>}} where PySpark gives {{a struct<k:bigint>}}.
> * A naive datetime was a time of the client's time zone. With a client in KST
> and a session in UTC, {{2020-01-01 01:02:03}} collected as 01:02:03, where
> PySpark collects 10:02:03.
> * A column of an int and a str raised {{PySparkRuntimeError}} ("Type mismatch
> in row data").
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
