[
https://issues.apache.org/jira/browse/SPARK-61392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61392:
------------------------------------
Assignee: Hyukjin Kwon
> Report the invalid string, not a binary value, in the UnicodeDecodeError of a
> result in the Python client
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61392
> URL: https://issues.apache.org/jira/browse/SPARK-61392
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> A follow-up of SPARK-61264, found in a post-merge review. When a result has a
> string that is not valid UTF-8, the drop-in scans every column for the first
> invalid value, including BinaryType columns, so SELECT X'FF' AS b, CAST(X'C3'
> AS STRING) AS s reports byte 0xff of the binary column instead of byte 0xc3
> of the string, as PySpark does. toArrow and toPandas also fail on the Arrow
> decode, where PySpark's toArrow returns the table.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]