[
https://issues.apache.org/jira/browse/SPARK-60428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60428:
-----------------------------------
Labels: pull-request-available (was: )
> ArrowWriter.reset() zeroes Arrow vectors to their full capacity on every batch
> ------------------------------------------------------------------------------
>
> Key: SPARK-60428
> URL: https://issues.apache.org/jira/browse/SPARK-60428
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Qing Fu
> Priority: Major
> Labels: pull-request-available
>
> {{ArrowFieldWriter.reset()}} calls Arrow's {{ValueVector.reset()}}, which
> zeroes every buffer of the vector up to its allocated capacity. Arrow vectors
> never shrink, so once one batch has grown a vector (for example a batch with
> a few very large arrays or strings), every later batch written by the same
> {{ArrowWriter}} pays for zeroing that capacity, however few rows it has.
> {{ArrayWriter}}, {{StructWriter}} and {{MapWriter}} also reset their children
> after {{super.reset()}} has already reset them recursively, so nested data is
> zeroed twice.
> An {{ArrowWriter}} is reused across batches by the Arrow-based Python runners
> (scalar pandas UDFs, Arrow-optimized Python UDFs, mapInPandas/mapInArrow,
> Python UDTFs), by {{ArrowConverters}} (toPandas/toArrow and Spark Connect
> results) and by {{ArrowCachedBatchSerializer}}. applyInPandas and cogroup
> create a writer per group, so there it only affects batches within one large
> group. In one production workload we observed about 700 ms per batch spent in
> {{reset()}} after a single outlier batch.
> The zeroing is not needed. Field writers write each index in [0, count)
> exactly once, through {{setValue}} or {{setNull}}, and both set the validity
> bit, so only the write cursor ({{lastSet}}) of variable-width and list
> vectors has to be reset. The one dependency on the zeroing is
> {{ArrayWriter.setNull}} and {{MapWriter.setNull}}, which are empty and rely
> on a zeroed validity buffer; they should call {{setNull(count)}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
