qingfureal opened a new pull request, #59677: URL: https://github.com/apache/spark/pull/59677
### What changes were proposed in this pull request? `ArrowFieldWriter.reset()` no longer calls Arrow's `ValueVector.reset()`, which zeroes every buffer of the vector up to its allocated capacity. It now resets only the write cursor of the vectors that track one: - variable-width vectors (`BaseVariableWidthVector`, `BaseLargeVariableWidthVector`) and `ListVector` (including `MapVector`): `setLastSet(-1)`; - fixed-width and struct vectors: nothing; - any other vector: `reset()`, as before. `ArrayWriter.setNull` and `MapWriter.setNull` now call `valueVector.setNull(count)`. They were empty and relied on `reset()` having zeroed the validity buffer. Because each writer now resets only its own vector, nested vectors are no longer zeroed twice (once by the parent vector's recursive `reset()` and again by the child's writer). ### Why are the changes needed? Arrow vectors never shrink, so once one batch has grown a vector, every later batch written by the same `ArrowWriter` pays for zeroing that capacity, however few rows it has. An `ArrowWriter` is reused across batches by the Arrow-based Python runners (scalar pandas UDFs, Arrow-optimized Python UDFs, mapInPandas/mapInArrow, Python UDTFs), by `ArrowConverters` (toPandas/toArrow and Spark Connect results) and by `ArrowCachedBatchSerializer`. In one production workload, `reset()` took about 700 ms per batch after a single outlier batch. The zeroing is not needed. Field writers write each index in `[0, count)` exactly once, through `setValue` or `setNull`, and both set the validity bit explicitly. Offsets of null slots are filled by Arrow itself (`fillHoles` for variable-width vectors, `startNewValue`/`setNull` for lists). Bytes under null slots and past the value count may keep values from an earlier batch of the same column; the Arrow columnar format leaves them undefined. `ArrowWriterBenchmark` (added here) writes 1000 batches of 100 rows with one reused writer, after a first batch of 100 rows or of 100,000 rows. Best time in ms, measured locally on JDK 17 (AMD EPYC-Genoa): | Column type | Before, 100-row first batch | After | Before, 100,000-row first batch | After | |---|---:|---:|---:|---:| | bigint | 2 | 1 | 47 | 1 | | string | 5 | 2 | 117 | 2 | | array<bigint> | 17 | 13 | 1516 | 14 | | struct<a: bigint, b: string> | 13 | 5 | 326 | 5 | After this change the cost of a batch no longer depends on the size of earlier batches. Small batches get faster even without an outlier, because the default capacity (about 4K slots) was zeroed on every reset as well. ### Does this PR introduce _any_ user-facing change? No. Values read back by any Arrow reader are unchanged. Only the undefined bytes under null slots may differ (left over from an earlier batch instead of zero). ### How was this patch tested? - New test in `ArrowWriterSuite`: one writer is reused for a 500-row batch without nulls at any level, then for smaller batches with top-level and nested nulls and an empty batch, over 24 column types (primitives, decimal, string, binary, datetime and interval types, null, nested arrays, structs and maps), with and without large var types. Each batch must equal the output of a fresh writer, both directly and after `VectorUnloader`/`VectorLoader`, which is what a Python worker receives. The test fails if `ArrayWriter.setNull` and `MapWriter.setNull` are left empty. - Existing `ArrowWriterSuite` tests pass. - New `ArrowWriterBenchmark`; the numbers above were measured locally. I will add the results files generated by the GitHub Actions benchmark workflow. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
