Hyukjin Kwon created SPARK-61127:
------------------------------------
Summary: Make the rows of DataFrame.collect and toLocalIterator
honor spark.sql.execution.pyspark.binaryAsBytes in the Python client as PySpark
does
Key: SPARK-61127
URL: https://issues.apache.org/jira/browse/SPARK-61127
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's Spark Connect DataFrame converts the binary values of collect,
toLocalIterator and foreachPartition per the conf
spark.sql.execution.pyspark.binaryAsBytes (DataFrame._get_binary_as_bytes):
bytes when it is true, the default, and bytearray when it is false. That
includes the binary values nested in arrays, maps and structs, and the rows of
first, head, take and tail, which collect.
The Python client of spark-connect-rust always gives bytes. So with the conf
set to false, collect of a binary column gives bytes, where PySpark 4.3.0 gives
bytearray.
The client should read the conf for these actions, and give bytearray when it
is false, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
