Tim Meehan created SPARK-61264:
----------------------------------
Summary: Raise UnicodeDecodeError for a string that is not valid
UTF-8 in collect, in the Python client as PySpark does
Key: SPARK-61264
URL: https://issues.apache.org/jira/browse/SPARK-61264
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
collect() of a string that is not valid UTF-8 raises UnicodeDecodeError in
PySpark and PySparkRuntimeError in the Rust client. Spark can produce such a
string, for example by casting binary to string.
{code:python}
spark.sql("SELECT CAST(X'C3' AS STRING) AS s").collect()
{code}
Against an OSS Spark 4.3.0 Connect server on loopback:
* pyspark-client 4.3.0: UnicodeDecodeError: 'utf-8' codec can't decode byte
0xc3 in position 0: unexpected end of data
* spark-connect-rust master 8768cc4: PySparkRuntimeError: Failed to decode
Arrow batch: Invalid argument error: Invalid UTF8 sequence at string index 0
(0..1): incomplete utf-8 byte sequence from index 0
* SELECT CAST(X'61' AS STRING) returns [Row(s='a')] in both.
>From source: arrow-rs validates UTF-8 when the IPC batch is decoded
>(crates/spark-connect/src/dataframe.rs:2606 and :2669), so the error comes
>from decoding the batch rather than from converting the value.
>PySparkRuntimeError is not a ValueError, so code that catches
>UnicodeDecodeError or ValueError misses it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
