L. C. Hsieh created SPARK-60395:
-----------------------------------
Summary: Fix the batch size computation and limit check of
maxBytesPerOutputBatch in sliced Arrow output
Key: SPARK-60395
URL: https://issues.apache.org/jira/browse/SPARK-60395
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
SliceBytesArrowOutputProcessorImpl, used when the internal config
spark.sql.execution.arrow.maxBytesPerOutputBatch is set, has two problems in
how it measures and compares the size of an output batch:
1. getBatchBytes sums FieldVector.getBufferSize of the top-level vectors into a
Long, but for a struct vector the size is already truncated to an int inside
Arrow: NonNullableStructVector.getBufferSize sums its children in a long and
returns (int) buffer, and StructVector adds its validity size as an int. The
outputs of mapInArrow, mapInPandas, applyInPandas and Python UDTFs are wrapped
in one top-level struct vector, so a batch over 2 GiB can be measured as a
negative size. For example, a mapInArrow UDF that yields one batch of three
int64 columns with 2^27 rows (about 3 GiB) is measured as -1,006,632,960 bytes,
so with maxBytesPerOutputBatch=256m the whole batch is output without slicing.
The check on each slice can accept an oversized slice the same way. The size
should be computed by summing the buffer sizes of the leaf vectors (and the
validity buffers of the nested vectors) as Long.
2. The doc of spark.sql.execution.arrow.maxBytesPerOutputBatch says Spark will
try to create batches with the size equal or less than the value, but both
checks in SliceBytesArrowOutputProcessorImpl use <, so a batch whose size
equals the limit exactly is sliced. The records processor uses <=. The checks
or the doc should be aligned.
Found while reviewing SPARK-60367 (https://github.com/apache/spark/pull/59616),
which this builds on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
