[
https://issues.apache.org/jira/browse/SPARK-60211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60211:
--------------------------------
Affects Version/s: 3.5.8
4.0.4
Description:
PyArrow 26.0.0 made {{RecordBatch.slice}} validate {{offset}}/{{length}} with
{{operator.index}} (previously a float was silently truncated by the Cython
int64 conversion).
In {{ApplyInPandasWithStateSerializer.load_stream}},
{{startOffset}}/{{numRows}} are read from the state struct column converted to
pandas. When an Arrow batch contains null padding rows
({{ApplyInPandasWithStateWriter.EMPTY_STATE_METADATA_ROW}}, i.e. whenever a
group has more than one data row), the int32 fields of the struct are promoted
to float64 by the pandas conversion, so {{data_batch.slice(data_start_offset,
num_data_rows)}} fails with:
{noformat}
TypeError: 'float' object cannot be interpreted as an integer
{noformat}
This breaks {{applyInPandasWithState}} for users on PyArrow 26, and
{{FlatMapGroupsInPandasWithStateSuite}} in the branch-4.0 / branch-4.1
scheduled builds since PyArrow 26.0.0 was released (the JVM build job installs
an unpinned pyarrow).
branch-4.2 and later are not affected: SPARK-54962 converts nullable integers
with {{integer_object_nulls=True}}, so the values stay ints.
> Fix applyInPandasWithState failure with PyArrow 26 due to float slice offsets
> -----------------------------------------------------------------------------
>
> Key: SPARK-60211
> URL: https://issues.apache.org/jira/browse/SPARK-60211
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.5.8, 4.1.3, 4.0.4
> Reporter: L. C. Hsieh
> Priority: Major
>
> PyArrow 26.0.0 made {{RecordBatch.slice}} validate {{offset}}/{{length}} with
> {{operator.index}} (previously a float was silently truncated by the Cython
> int64 conversion).
> In {{ApplyInPandasWithStateSerializer.load_stream}},
> {{startOffset}}/{{numRows}} are read from the state struct column converted
> to pandas. When an Arrow batch contains null padding rows
> ({{ApplyInPandasWithStateWriter.EMPTY_STATE_METADATA_ROW}}, i.e. whenever a
> group has more than one data row), the int32 fields of the struct are
> promoted to float64 by the pandas conversion, so
> {{data_batch.slice(data_start_offset, num_data_rows)}} fails with:
> {noformat}
> TypeError: 'float' object cannot be interpreted as an integer
> {noformat}
> This breaks {{applyInPandasWithState}} for users on PyArrow 26, and
> {{FlatMapGroupsInPandasWithStateSuite}} in the branch-4.0 / branch-4.1
> scheduled builds since PyArrow 26.0.0 was released (the JVM build job
> installs an unpinned pyarrow).
> branch-4.2 and later are not affected: SPARK-54962 converts nullable integers
> with {{integer_object_nulls=True}}, so the values stay ints.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
