[
https://issues.apache.org/jira/browse/SPARK-62139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62139:
-----------------------------------
Labels: pull-request-available (was: )
> Fix Spark compatibility with newer PyArrow versions
> ---------------------------------------------------
>
> Key: SPARK-62139
> URL: https://issues.apache.org/jira/browse/SPARK-62139
> Project: Spark
> Issue Type: Bug
> Components: Pandas API on Spark
> Affects Versions: 4.0.6
> Environment: CI with PyArrow and Spark main.
> Reporter: Raúl Cumplido
> Priority: Major
> Labels: pull-request-available
>
> On apache/arrow I have been working on fixing integration test with Spark
> here:
> * [https://github.com/apache/arrow/pull/55692]
> I've found that with current Arrow main we get several failures, see the run
> here:
> [https://github.com/ursacomputing/crossbow/actions/runs/44756260561/job/136356691993]
> An example of failing log:
>
> {code:java}
> ======================================================================
> ERROR: test_nullability_narrow
> (pyspark.sql.tests.arrow.test_arrow_map.MapInArrowWithOutputArrowBatchSlicingRecordsTests.test_nullability_narrow)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 325,
> in test_nullability_narrow
> df.mapInArrow(func, "a int").collect()
> File "/spark/python/pyspark/sql/classic/dataframe.py", line 531, in collect
> sock_info = self._jdf.collectToPython()
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line
> 1362, in __call__
> return_value = get_return_value(
> ^^^^^^^^^^^^^^^^^
> File "/spark/python/pyspark/errors/exceptions/captured.py", line 301, in
> deco
> raise converted from None
> File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in
> invoke_udf
> run_process()
> ^^^^^^^
> File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in
> run_process
> process(iterator)
> ^^^^^^^^^^^
> File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
> serializer.dump_stream(out_iter, outfile)
> ^^^^^^^^^^^^^^^^^
> File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 94, in dump_stream
> for batch in iterator:
> ^^^^^^^^^^^^^^^^
> File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 152, in _write_stream_start
> first = next(batch_iterator, None)
> ^^^^^^^^^^^^^^^^^
> File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line
> 731, in run
> yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
> ^^^^^^^^^^^^^^^^^
> File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320,
> in func
> yield pa.RecordBatch.from_arrays(
> ^^^^^^^^^^^^^^^^^
> File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
> converted_arrays = _sanitize_arrays(arrays, names, schema, metadata,
> ^^^^^^^^^^^^^^^^^
> File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
> converted_arrays = _schema_from_arrays(arrays, names, metadata,
> ^^^^^^^^^^^^^^^^^
> File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
> c_name = tobytes(names[i])
> ^^^^^^^^^^^^^^^^^
> File "EnumTypeToPy", line 15, in
> string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
> pyspark.errors.exceptions.captured.PythonException: An exception was thrown
> from the Python worker:
> Traceback (most recent call last):
> File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 288, in
> invoke_udf
> run_process()
> File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 267, in
> run_process
> process(iterator)
> File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 253, in process
> serializer.dump_stream(out_iter, outfile)
> File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 94, in dump_stream
> for batch in iterator:
> File "/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py",
> line 152, in _write_stream_start
> first = next(batch_iterator, None)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/spark/python/lib/pyspark.zip/pyspark/eval_handlers/_arrow.py", line
> 731, in run
> yield from map(ArrowBatchTransformer.wrap_struct, verified_iter)
> File "/spark/python/pyspark/sql/tests/arrow/test_arrow_map.py", line 320,
> in func
> yield pa.RecordBatch.from_arrays(
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "pyarrow/table.pxi", line 3506, in pyarrow.lib.RecordBatch.from_arrays
> File "pyarrow/table.pxi", line 1593, in pyarrow.lib._sanitize_arrays
> File "pyarrow/table.pxi", line 1581, in pyarrow.lib._schema_from_arrays
> File "EnumTypeToPy", line 15, in
> string.from_py.__pyx_convert_string_from_py_6libcpp_6string_std__in_string
> TypeError: expected bytes, pyarrow.lib.Field found
> {code}
> This is related to removing deprecated APIs here:
>
> [https://github.com/apache/arrow/pull/53543]
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]