Hyukjin Kwon created SPARK-61143:
------------------------------------
Summary: Decode the Arrow batches that the server compresses with
zstd or lz4 in the Rust client, as PySpark does
Key: SPARK-61143
URL: https://issues.apache.org/jira/browse/SPARK-61143
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
With spark.sql.execution.arrow.compression.codec set to zstd or lz4, the Spark
Connect server compresses the Arrow batches of the results. PySpark reads them
with PyArrow.
The Rust client, and so the Python client, builds arrow without the
ipc_compression feature, and fails to decode them:
PySparkRuntimeError: Failed to decode Arrow batch: Invalid argument error: zstd
IPC decompression requires the zstd feature
So collect, toArrow and toPandas fail, and so do the results of pandas UDFs and
applyInPandas under that conf. Nine tests of Apache Spark's
test_parity_arrow.py, test_parity_pandas_grouped_map.py and
test_parity_pandas_udf_scalar.py (*_with_compression_codec*) fail on this
client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
