FelixYBW opened a new pull request, #13280:
URL: https://github.com/apache/gluten/pull/13280
## What changes are proposed in this pull request?
Part of #13140 (items G2 + G3). Gluten's Java Arrow batches move from
Gluten's own `ArrowWritableColumnVector` (about 2,300 lines plus a
`WritableColumnVectorShim` per Spark version) to Spark's `ArrowColumnVector`
for reading and Spark's `ArrowWriter` for writing. No Spark change is needed:
`ArrowColumnVector(ValueVector)`, `getValueVector`,
`ArrowWriter.create(VectorSchemaRoot)` and `MutableProjection` have the same
API in Spark 3.4 – 4.2.
- **`RefCountedArrowColumnVector`** (new, 57 lines): a subclass of Spark's
`ArrowColumnVector` that keeps Gluten's reference count. The batch lifecycle is
unchanged: `ColumnarBatches` still retains and releases batches by reference
count, and loads and offloads them in place.
- **Reading** goes through Spark's accessors. They check for nulls on all
supported Spark versions, so the `ArrowColumnarBatch` / `ArrowColumnarRow`
workaround for Spark's `ColumnarRow` before 4.0 is removed.
- **Writing**:
- `ColumnarPartialProjectExec`: Spark's `MutableProjection` +
`ArrowWriter`, replacing `ArrowProjection` / `GenerateArrowProjection` /
`InterpretedArrowProjection`.
- `ColumnarPartialGenerateExec`: `ArrowWriter` over the generated rows.
The per-column size estimation is no longer needed.
- `ColumnarRangeExec`, `ExecUtil` (range partition ids): set the Arrow
vector directly.
- **`ArrowColumnVectors`** (new): allocate a root from Gluten's allocator,
wrap vectors, build a batch from rows with `ArrowWriter`, and adopt (move, or
copy across allocator roots) a vector into Gluten's allocator.
`ColumnarBatches.offload` adopts batches of plain Spark `ArrowColumnVector`s.
- **`ColumnarArrowEvalPythonExec`** wraps the Python output vectors with
`RefCountedArrowColumnVector`. Gluten's runner and its shims are unchanged.
- **`VeloxColumnarWriteFilesExec`** reads the VARBINARY fragments column
with `getBinary`. The old accessor allowed `getUTF8String` on binary columns;
Spark's doesn't.
- **Removed**: `ArrowWritableColumnVector`, `WritableColumnVectorShim`
(spark34 – spark42), `ArrowColumnarBatch`, `ArrowColumnarRow`,
`ArrowColumnVectorUtils`, and `ArrowProjection` with its code generators.
Net: +406 / −4,521 lines.
**Not included:** `backends-bolt` uses the same removed classes and is not
migrated in this PR.
## How was this patch tested?
- `ColumnarBatchTest`: ported to batches built with `ArrowWriter`.
- `ArrowColumnVectorTest`: new cases for the reference count, a decimal
round trip, and offloading a Spark Arrow batch allocated from another allocator
root.
- `ArrowEvalPythonExecSuite`: a new case with two chained Arrow UDFs over
many small batches with nulls, checking the results against vanilla Spark and
the `LoadArrowData` / `OffloadArrowData` transitions around the UDFs.
Locally, with `SPARK_ANSI_SQL_MODE=false`, I ran `UDFPartialProjectSuite`,
`MiscOperatorSuite`, `GlutenHiveUDFSuite`, `VeloxColumnarCacheSuite`,
`FallbackSuite`, `VeloxParquetWriteSuite`, `VeloxRDDScanSuite`,
`ArrowEvalPythonExecSuite`, `ColumnarBatchTest` and `ArrowColumnVectorTest`:
| Spark | Result |
|---|---|
| 3.5 | 191 Scala + 9 JUnit tests pass (the Arrow UDF tests used pyspark
3.5.5) |
| 4.0 | 193 Scala + 9 JUnit tests pass (the Arrow UDF tests used pyspark
4.0.2) |
| 4.2 | 57 Scala + 9 JUnit tests pass, with a smaller suite set (no Parquet
write / RDD scan suites); many tests are ignored on 4.2 in main |
| 3.4, 4.1 | Main and test sources compile |
The write paths now use `ArrowWriter`, which writes one row at a time like
the code it replaces. I haven't benchmarked them yet.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
