[ 
https://issues.apache.org/jira/browse/SPARK-62139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62139:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fix Spark compatibility with newer PyArrow versions
> ---------------------------------------------------
>
>                 Key: SPARK-62139
>                 URL: https://issues.apache.org/jira/browse/SPARK-62139
>             Project: Spark
>          Issue Type: Bug
>          Components: Pandas API on Spark
>    Affects Versions: 4.0.6
>         Environment: CI with PyArrow and Spark main.
>            Reporter: Raúl Cumplido
>            Priority: Major
>              Labels: pull-request-available
>
> On apache/arrow I have been working on fixing integration test with Spark 
> here:
>  * [https://github.com/apache/arrow/pull/55692]
> I've found that with current Arrow main we get several failures, see the run 
> here:
> [https://github.com/ursacomputing/crossbow/actions/runs/44756260561/job/136356691993]
> An example of failing log:
>  
> {code:java}
> ======================================================================
> ERROR: test_nullability_narrow 
> (pyspark.sql.tests.arrow.test_arrow_map.MapInArrowWithOutputArrowBatchSlicingRecordsTests.test_nullability_narrow)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 325, 
> in test_nullability_narrow
>     df.mapInArrow(func, "a int").collect()
>   File "/spark/python/pyspark/sql/classic/dataframe.py", line 531, in collect
>     sock_info = self._jdf.collectToPython()
>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 
> 1362, in __call__
>     return_value = get_return_value(
>                    ^^^^^^^^^^^^^^^^^
>   File "/spark/python/pyspark/errors/exceptions/captured.py", line 301, in 
> deco
>     raise converted from None
>   File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in 
> invoke_udf
>     run_process()
>            ^^^^^^^
>   File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in 
> run_process
>     process(iterator)
>            ^^^^^^^^^^^
>   File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
>     serializer.dump_stream(out_iter, outfile)
>            ^^^^^^^^^^^^^^^^^
>   File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", 
> line 94, in dump_stream
>     for batch in iterator:
>            ^^^^^^^^^^^^^^^^
>   File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", 
> line 152, in _write_stream_start
>     first = next(batch_iterator, None)
>            ^^^^^^^^^^^^^^^^^
>   File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line 
> 731, in run
>     yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
>            ^^^^^^^^^^^^^^^^^
>   File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320, 
> in func
>     yield pa.RecordBatch.from_arrays(
>     ^^^^^^^^^^^^^^^^^
>   File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
>     converted_arrays = _sanitize_arrays(arrays, names, schema, metadata,
>     ^^^^^^^^^^^^^^^^^
>   File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
>     converted_arrays = _schema_from_arrays(arrays, names, metadata,
>     ^^^^^^^^^^^^^^^^^
>   File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
>     c_name = tobytes(names[i])
>     ^^^^^^^^^^^^^^^^^
>   File "EnumTypeToPy", line 15, in 
> string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
> pyspark.errors.exceptions.captured.PythonException: An exception was thrown 
> from the Python worker:
> Traceback (most recent call last):
>   File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in 
> invoke_udf
>     run_process()
>   File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in 
> run_process
>     process(iterator)
>   File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
>     serializer.dump_stream(out_iter, outfile)
>   File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", 
> line 94, in dump_stream
>     for batch in iterator:
>   File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", 
> line 152, in _write_stream_start
>     first = next(batch_iterator, None)
>             ^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line 
> 731, in run
>     yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
>   File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320, 
> in func
>     yield pa.RecordBatch.from_arrays(
>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
>   File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
>   File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
>   File "EnumTypeToPy", line 15, in 
> string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
> TypeError: expected bytes, pyarrow.lib.Field found
> {code}
> This is related to removing deprecated APIs here:
>  
> [https://github.com/apache/arrow/pull/53543]
>  
>  
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to