Hyukjin Kwon created SPARK-61000:
------------------------------------
Summary: Convert the columns of toPandas of the Python client as
PySpark does
Key: SPARK-61000
URL: https://issues.apache.org/jira/browse/SPARK-61000
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{DataFrame.toPandas()}} converts each column of the Arrow
result for pandas by its Spark type ({{_create_converter_to_pandas}}, with the
session time zone, {{spark.sql.execution.pandas.structHandlingMode}} and the
other settings of {{SparkConnectClient.to_pandas}}). The Rust-backed Python
client returns {{pyarrow.Table.to_pandas()}} of the result, so for {{SELECT
TIMESTAMP'2024-01-02 03:04:05' AS ts, map('k', 1) AS m}} and a column of a
user-defined type, such as {{array_to_vector}}'s:
||Column||PySpark 4.3.0||Rust-backed client||
|{{ts}}|{{datetime64[ns]}}, {{Timestamp('2024-01-02 03:04:05')}} in the session
time zone|{{datetime64[us, UTC]}}, {{Timestamp('2024-01-02 03:04:05+0000',
tz='UTC')}}|
|{{m}}|{{{'k': 1}}}|{{[('k', 1)]}}|
|a {{VectorUDT}}|{{DenseVector([1.5, 2.0])}}|{{{'type': 1, 'size': None,
'indices': None, 'values': array([1.5, 2.])}}}|
Convert the columns of {{toPandas()}} as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
