Raúl Cumplido created SPARK-62139:
-------------------------------------
Summary: Fix Spark compatibility with newer PyArrow versions
Key: SPARK-62139
URL: https://issues.apache.org/jira/browse/SPARK-62139
Project: Spark
Issue Type: Bug
Components: Pandas API on Spark
Affects Versions: 4.0.6
Environment: CI with PyArrow and Spark main.
Reporter: Raúl Cumplido
On apache/arrow I have been working on fixing integration test with Spark here:
* [https://github.com/apache/arrow/pull/55692]
I've found that with current Arrow main we get several failures, see the run
here:
[https://github.com/ursacomputing/crossbow/actions/runs/44756260561/job/136356691993]
An example of failing log:
{code:java}
======================================================================
ERROR: test_nullability_narrow
(pyspark.sql.tests.arrow.test_arrow_map.MapInArrowWithOutputArrowBatchSlicingRecordsTests.test_nullability_narrow)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 325, in
test_nullability_narrow
df.mapInArrow(func, "a int").collect()
File "/spark/python/pyspark/sql/classic/dataframe.py", line 531, in collect
sock_info = self._jdf.collectToPython()
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line
1362, in __call__
return_value = get_return_value(
^^^^^^^^^^^^^^^^^
File "/spark/python/pyspark/errors/exceptions/captured.py", line 301, in deco
raise converted from None
File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in
invoke_udf
run_process()
^^^^^^^
File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in
run_process
process(iterator)
^^^^^^^^^^^
File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
serializer.dump_stream(out_iter, outfile)
^^^^^^^^^^^^^^^^^
File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line
94, in dump_stream
for batch in iterator:
^^^^^^^^^^^^^^^^
File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line
152, in _write_stream_start
first = next(batch_iterator, None)
^^^^^^^^^^^^^^^^^
File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line
731, in run
yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
^^^^^^^^^^^^^^^^^
File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320, in
func
yield pa.RecordBatch.from_arrays(
^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
converted_arrays = _sanitize_arrays(arrays, names, schema, metadata,
^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
converted_arrays = _schema_from_arrays(arrays, names, metadata,
^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
c_name = tobytes(names[i])
^^^^^^^^^^^^^^^^^
File "EnumTypeToPy", line 15, in
string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
pyspark.errors.exceptions.captured.PythonException: An exception was thrown
from the Python worker:
Traceback (most recent call last):
File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in
invoke_udf
run_process()
File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in
run_process
process(iterator)
File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
serializer.dump_stream(out_iter, outfile)
File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line
94, in dump_stream
for batch in iterator:
File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line
152, in _write_stream_start
first = next(batch_iterator, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line
731, in run
yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320, in
func
yield pa.RecordBatch.from_arrays(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
File "EnumTypeToPy", line 15, in
string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
TypeError: expected bytes, pyarrow.lib.Field found
{code}
This is related to removing deprecated APIs here:
[https://github.com/apache/arrow/pull/53543]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]