Raúl Cumplido created SPARK-62139:
-------------------------------------

             Summary: Fix Spark compatibility with newer PyArrow versions
                 Key: SPARK-62139
                 URL: https://issues.apache.org/jira/browse/SPARK-62139
             Project: Spark
          Issue Type: Bug
          Components: Pandas API on Spark
    Affects Versions: 4.0.6
         Environment: CI with PyArrow and Spark main.
            Reporter: Raúl Cumplido


On apache/arrow I have been working on fixing integration test with Spark here:
 * [https://github.com/apache/arrow/pull/55692]

I've found that with current Arrow main we get several failures, see the run 
here:

[https://github.com/ursacomputing/crossbow/actions/runs/44756260561/job/136356691993]

An example of failing log:

 
{code:java}
======================================================================
ERROR: test_nullability_narrow 
(pyspark.sql.tests.arrow.test_arrow_map.MapInArrowWithOutputArrowBatchSlicingRecordsTests.test_nullability_narrow)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 325, in 
test_nullability_narrow
    df.mapInArrow(func, "a int").collect()
  File "/spark/python/pyspark/sql/classic/dataframe.py", line 531, in collect
    sock_info = self._jdf.collectToPython()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 
1362, in __call__
    return_value = get_return_value(
                   ^^^^^^^^^^^^^^^^^
  File "/spark/python/pyspark/errors/exceptions/captured.py", line 301, in deco
    raise converted from None
  File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in 
invoke_udf
    run_process()
           ^^^^^^^
  File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in 
run_process
    process(iterator)
           ^^^^^^^^^^^
  File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
    serializer.dump_stream(out_iter, outfile)
           ^^^^^^^^^^^^^^^^^
  File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 
94, in dump_stream
    for batch in iterator:
           ^^^^^^^^^^^^^^^^
  File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 
152, in _write_stream_start
    first = next(batch_iterator, None)
           ^^^^^^^^^^^^^^^^^
  File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line 
731, in run
    yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
           ^^^^^^^^^^^^^^^^^
  File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320, in 
func
    yield pa.RecordBatch.from_arrays(
    ^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
    converted_arrays = _sanitize_arrays(arrays, names, schema, metadata,
    ^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
    converted_arrays = _schema_from_arrays(arrays, names, metadata,
    ^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
    c_name = tobytes(names[i])
    ^^^^^^^^^^^^^^^^^
  File "EnumTypeToPy", line 15, in 
string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
pyspark.errors.exceptions.captured.PythonException: An exception was thrown 
from the Python worker:
Traceback (most recent call last):
  File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in 
invoke_udf
    run_process()
  File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in 
run_process
    process(iterator)
  File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
    serializer.dump_stream(out_iter, outfile)
  File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 
94, in dump_stream
    for batch in iterator:
  File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 
152, in _write_stream_start
    first = next(batch_iterator, None)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line 
731, in run
    yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
  File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320, in 
func
    yield pa.RecordBatch.from_arrays(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
  File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
  File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
  File "EnumTypeToPy", line 15, in 
string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
TypeError: expected bytes, pyarrow.lib.Field found
{code}
This is related to removing deprecated APIs here:

 

[https://github.com/apache/arrow/pull/53543]

 

 

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to