Qing Fu created SPARK-60428:
-------------------------------
Summary: ArrowWriter.reset() zeroes Arrow vectors to their full
capacity on every batch
Key: SPARK-60428
URL: https://issues.apache.org/jira/browse/SPARK-60428
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Qing Fu
{{ArrowFieldWriter.reset()}} calls Arrow's {{ValueVector.reset()}}, which
zeroes every buffer of the vector up to its allocated capacity. Arrow vectors
never shrink, so once one batch has grown a vector (for example a batch with a
few very large arrays or strings), every later batch written by the same
{{ArrowWriter}} pays for zeroing that capacity, however few rows it has.
{{ArrayWriter}}, {{StructWriter}} and {{MapWriter}} also reset their children
after {{super.reset()}} has already reset them recursively, so nested data is
zeroed twice.
An {{ArrowWriter}} is reused across batches by the Arrow-based Python runners
(scalar pandas UDFs, Arrow-optimized Python UDFs, mapInPandas/mapInArrow,
Python UDTFs), by {{ArrowConverters}} (toPandas/toArrow and Spark Connect
results) and by {{ArrowCachedBatchSerializer}}. applyInPandas and cogroup
create a writer per group, so there it only affects batches within one large
group. In one production workload we observed about 700 ms per batch spent in
{{reset()}} after a single outlier batch.
The zeroing is not needed. Field writers write each index in [0, count) exactly
once, through {{setValue}} or {{setNull}}, and both set the validity bit, so
only the write cursor ({{lastSet}}) of variable-width and list vectors has to
be reset. The one dependency on the zeroing is {{ArrayWriter.setNull}} and
{{MapWriter.setNull}}, which are empty and rely on a zeroed validity buffer;
they should call {{setNull(count)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
