malinjawi opened a new pull request, #13595:
URL: https://github.com/apache/gluten/pull/13595
## What changes are proposed in this pull request?
This is the write half of #12263, using the lance-spark path agreed there.
Gluten passes Velox's batches to Lance as Arrow, with no row conversion, and
lance-spark still does the commit. It replaces #13515, which only moved
`lance.version`; that bump is part of the first commit here.
The feature is experimental and off by default. With
`spark.gluten.sql.columnar.backend.velox.lance.enableNativeWrite=true`, writes
to Lance tables run through Velox:
- appends and overwrites (`INSERT INTO`, `INSERT OVERWRITE`,
`mode("append")` and `mode("overwrite")`)
- the nested writes of `CREATE TABLE AS SELECT` and `REPLACE TABLE AS
SELECT` (Spark 3.5 and later)
Two commits, reviewable one at a time:
1. **Offload Lance table writes to Velox.**
- Moves `lance.version` from 0.8.0 to 0.9.0-beta.2, the first lance-spark
release with `LanceBatchWrite.taskCommit` (lance-format/lance-spark#715) and
`getFragmentWriteSpec` (lance-format/lance-spark#892). #13527 tracks moving to
a stable lance-spark release before the next Gluten release.
- Each task exports its Velox batches as an Arrow C stream and calls
`Fragment.create` with the write parameters lance-spark resolves for the table:
write mode, file format version, file size limits and storage options. It
returns `LanceBatchWrite.taskCommit(fragments)`, and lance-spark commits the
fragments on the driver as it does for its own writer.
- Velox may leave any value under a null slot, which Arrow allows, but
Lance counts those values when it picks bit widths and dictionaries. The writer
zeroes them, in a copy, so the files are as compact as lance-spark's.
- If a task fails, its stream fails, and Lance deletes the files it wrote
for that task. lance-spark's writer ends its stream instead, which leaves those
files behind.
- Tasks report the same `recordsWritten` and `bytesWritten` metrics as
lance-spark's writer, plus Gluten's native write metrics.
2. **Write Lance fixed-size list columns natively.** Velox has no fixed-size
list type and exports vector columns as plain lists. The writer moves each
top-level fixed-size list of signed integers or floats into the commit schema's
layout: without a copy when no row is null, with one memory copy per row
otherwise, and element by element only when elements are null. A list with the
wrong number of elements fails the task.
Some writes stay on lance-spark's writer, decided at plan time:
- sharded tables
- blob columns (a follow-up writes them natively)
- large strings and binaries, half floats and unsigned integers
- fixed-size lists inside other columns
- `TimestampNTZ` columns, which Gluten's validation keeps on Spark
- timestamps inside a struct that may be null. Velox's Arrow export converts
the timestamps of null struct rows too, where an expression or a join may leave
any value, and a value out of range fails the export. Velox's own Parquet
writer fails the same way.
The first batch each task exports is also checked against the table's Arrow
schema, so a layout mismatch fails the task instead of writing bad data.
## How was this patch tested?
All Lance suites pass on Spark 3.5, 4.0, 4.1 and 4.2 with both commits, and
on Spark 3.5 and 4.0 with the first commit alone, with no Arrow or Velox memory
leaks. On Spark 3.4 every writer test passes; two scan tests fail there as they
do on main (reading large strings, and the filter/shuffle/sort test from
#13488).
- `VeloxLanceWriteSuite` (25 tests in the first commit, 27 with the second):
- append and overwrite, including `write_mode=overwrite`; several tasks
and batches per task; an empty append; a table with dropped columns
- struct, array, map and vector columns
- every offloaded type with nulls, written from a Parquet scan: values
match, and no non-zero value is left under a null slot on disk
- a session time zone other than UTC, `decimal(38,10)`, and `TimestampNTZ`
- a table pinned to file format 2.1; metrics
- SQL `INSERT INTO`, `INSERT OVERWRITE`, CTAS and RTAS through a Lance
catalog
- a task that fails mid-write leaves the table unchanged and no files
behind; a failure inside Lance fails the task; no Arrow memory leaks
- every fallback listed above
- `LanceNullSlotsSuite` (4) and `LanceFixedSizeListsSuite` (5) unit-test the
null-slot zeroing and the vector conversion.
- `VeloxLanceSuite` and the scan's `LanceFixedSizeListsSuite` still pass.
Exact comparison (laptop, Spark 3.5): native output equals both the Parquet
input and lance-spark's output with `EXCEPT ALL` in both directions, on a
`store_sales`-like table (12M rows, identical file sizes), a nullable
`customer`-like table (20M rows) and 1M 768-float vectors with 1% null vectors.
Cluster benchmark (8 x r6id.4xlarge, Spark 3.5, 5 rotating rounds per setup
and dataset, medians; measured before the null-slot zeroing was added):
| Data | Native | Gluten, lance-spark writer | Vanilla Spark | Native vs
lance-spark writer | Native vs vanilla |
|---|---|---|---|---|---|
| TPC-DS `store_sales`, 1.44B rows | 39.2 s | 72.4 s | 89.4 s | 1.86x, 44%
less CPU | 2.26x, 50% less CPU |
| 60M x 768-float vectors | 26.9 s | 35.6 s | 43.7 s | 1.28x, 31% less CPU |
1.62x |
| 30M rows, vectors and nine strings | 67.2 s | 83.6 s | 125.3 s | 1.22x,
30% less CPU | 1.89x |
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-5-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]