[
https://issues.apache.org/jira/browse/SPARK-61264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Meehan updated SPARK-61264:
-------------------------------
Fix Version/s: connect-rust-4.3.0
> Raise UnicodeDecodeError for a string that is not valid UTF-8 in collect, in
> the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61264
> URL: https://issues.apache.org/jira/browse/SPARK-61264
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Tim Meehan
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> collect() of a string that is not valid UTF-8 raises UnicodeDecodeError in
> PySpark and PySparkRuntimeError in the Rust client. Spark can produce such a
> string, for example by casting binary to string.
> {code:python}
> spark.sql("SELECT CAST(X'C3' AS STRING) AS s").collect()
> {code}
> Against an OSS Spark 4.3.0 Connect server on loopback:
> * pyspark-client 4.3.0: UnicodeDecodeError: 'utf-8' codec can't decode byte
> 0xc3 in position 0: unexpected end of data
> * spark-connect-rust master 8768cc4: PySparkRuntimeError: Failed to decode
> Arrow batch: Invalid argument error: Invalid UTF8 sequence at string index 0
> (0..1): incomplete utf-8 byte sequence from index 0
> * SELECT CAST(X'61' AS STRING) returns [Row(s='a')] in both.
> From source: arrow-rs validates UTF-8 when the IPC batch is decoded
> (crates/spark-connect/src/dataframe.rs:2606 and :2669), so the error comes
> from decoding the batch rather than from converting the value.
> PySparkRuntimeError is not a ValueError, so code that catches
> UnicodeDecodeError or ValueError misses it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
