Hyukjin Kwon created SPARK-61392:
------------------------------------
Summary: Report the invalid string, not a binary value, in the
UnicodeDecodeError of a result in the Python client
Key: SPARK-61392
URL: https://issues.apache.org/jira/browse/SPARK-61392
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61264, found in a post-merge review. When a result has a
string that is not valid UTF-8, the drop-in scans every column for the first
invalid value, including BinaryType columns, so SELECT X'FF' AS b, CAST(X'C3'
AS STRING) AS s reports byte 0xff of the binary column instead of byte 0xc3 of
the string, as PySpark does. toArrow and toPandas also fail on the Arrow
decode, where PySpark's toArrow returns the table.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]