ReemaAlzaid commented on code in PR #13338:
URL: https://github.com/apache/gluten/pull/13338#discussion_r4299929126
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
Review Comment:
Also we can add a diagram example to show them the process of delta, by
showing Delta snapshot/file selection → Gluten planning → Velox and Spark
execution
For example, consider a Delta table partitioned by `sale_date`:
```sql
SELECT SUM(amount)
FROM sales
WHERE sale_date = DATE '2026-10-01'
AND amount > 100;
```
```mermaid
flowchart LR
subgraph Planning["Spark driver"]
Snapshot["1. Spark + Delta snapshot<br/>Schema + active files"]
Select["2. Delta file pruning<br/>Partitions + available stats"]
Plan["3. Spark plan<br/>Gluten offload checks"]
Snapshot --> Select --> Plan
end
subgraph Execution["Spark executors"]
Scan["4. Velox scan<br/>Read selected files"]
Compute["5. Velox operators<br/>Filter + SUM"]
Spark["Spark operators<br/>Retained during planning"]
Scan --> Compute
Compute <-->|"Planned conversions"| Spark
end
subgraph Storage["Delta table"]
Log[("_delta_log<br/>Metadata + file statistics")]
Data[("Parquet files")]
end
Result["6. Query result"]
Log --> Snapshot
Plan -->|"Eligible operators"| Scan
Plan -->|"Retained operators"| Spark
Data --> Scan
Data --> Spark
Compute --> Result
Spark --> Result
```
A query can combine native and Spark operators. Delta responsibility for
table metadata and transactions
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features**
when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector
scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) |
Offload | Experimental, off by default | Gluten suites, copied Delta Lake
suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental,
off by default | Gluten suites, copied Delta Lake suites tagged
`ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload |
Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release
in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from
source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle
compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta
Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1"
on this page refers to that job. Its known failures are listed in
`.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are
two 2-billion-row deletion vector tests, run in two suites, that the job
force-fails. Some entries check a query plan that changes once Gluten offloads
it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a
partitioned native `save(path)`. Other native writes on Spark 4.1 are checked
only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in
[#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build
the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no
Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one
built with `-Pdelta`. Add the jar from the table above. When
`io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs
`Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built
with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or
`--enable_abfs=ON`. Without that connector, scans of those paths fall back and
native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf
spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default
`spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back,
Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job
turns ANSI mode off in the test setup that loads Gluten. That setup and some
Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`.
Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false`
removes that fallback. Velox then runs the offloaded operators with ANSI mode
on. For Delta Lake, only the nightly Delta Lake job tests that combination, in
the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem
(see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description
|
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the
data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back
|
+| Fallback | Runs on Spark or Delta Lake
|
+| Exception | Fails instead of falling back
|
+| ResultMismatch | Can return wrong results or write wrong files
|
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI
checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is
nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on
Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT`
and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog.
Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The
`SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the
`VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
+Set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to
have Velox write the data files of the commands marked Offload in [Write
commands](#write-commands). On Spark 4.x, also apply [ANSI mode on Spark
4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column
types before it runs. Unsupported values make the task fail or write wrong
files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for
example `Execute GlutenDelta UpdateCommand` or `GlutenDelta
AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+````
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+````
+
+The node also shows when Velox writes no data files. This happens for a DELETE
that writes only deletion vectors or only metadata, and for a Spark 4.x write
whose schema Velox cannot write, such as one with a VARIANT column. From code.
For `DataFrameWriter` calls, read the plan from the Spark UI SQL tab or a
`QueryExecutionListener`.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise
every row is at best PartialOffload. In PartialOffload rows, Delta writes the
files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on
3.5. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE,
`save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check
results only. On 4.1, the nightly job shows every Offload row except DELETE
going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or
without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when
`t` does not exist and the mode is not overwrite | Offload | Needs a
DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` |
`writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in
overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion
vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload | Velox writes the rewritten
files. A deletion-vector DELETE and a metadata-only DELETE write no data files.
|
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | | Offload | Bin-packing, by path or by
name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a
liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM
...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE
WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in
append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table,
`writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` |
PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans and the DML target scan can run in
Velox. Joins can too (from code). Native MERGE write is in progress in
[#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`,
batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`,
UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake. Seen on 4.1.
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing
shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both
are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload | Compacts natively written tables (seen on 4.1).
After a native write, the compaction runs in a Gluten transaction (from code).
After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name (from code) |
Offload, matched by physical name (seen on 4.1) |
Review Comment:
```suggestion
| Column mapping, id mode | Offload, matched by physical name (from code).
Known ResultMismatch cases: see [Read problems](#read-problems). | Offload,
matched by physical name (seen on 4.1). Known ResultMismatch cases: see [Read
problems](#read-problems). |
```
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
Review Comment:
Can we add a brief overview before the version matrix? It would help readers
understand that read acceleration and experimental native writing are separate
capabilities
Suggested wording:
> Gluten’s Velox backend can accelerate eligible scans and query operators
over Delta Lake tables. Read acceleration does not require enabling native
writes. Native Delta writes are experimental and disabled by default. This
guide covers Spark/Delta Lake version combinations, setup, feature support, and
known limitations.
##########
docs/get-started/VeloxDelta.md:
##########
@@ -1,62 +1,572 @@
-# Delta Lake Feature Support Status in Apache Gluten (Velox Backend)
+---
+layout: page
+title: Delta Lake Support in Velox Backend
+nav_order: 13
+parent: Getting-Started
+---
-This document summarizes the support status of **Delta Lake table features**
when used with **Apache Gluten (Velox backend)**.
+# Delta Lake Support in Velox Backend
-## Supported Spark / Delta combinations
+## Supported Spark version
-| Spark profile | Spark version | Scala version | Delta Lake version | Status |
+| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by |
+|---|---|---|---|---|---|
+| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector
scans fall back. | None | Gluten suites |
+| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) |
Offload | Experimental, off by default | Gluten suites, copied Delta Lake
suites (Scala 2.12 only) |
+| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental,
off by default | Gluten suites, copied Delta Lake suites tagged
`ExtendedSQLTest` |
+| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload |
Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job |
+| 4.2 | None on main | | None | None | None |
+
+Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release
in the table. The copied Delta Lake suites run with native write on.
+
+Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from
source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle
compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta
Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1"
on this page refers to that job. Its known failures are listed in
`.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are
two 2-billion-row deletion vector tests, run in two suites, that the job
force-fails. Some entries check a query plan that changes once Gluten offloads
it. Read the README in that directory before treating an entry as a Gluten bug.
+
+On Spark 4.1, Gluten's own suites check deletion vector reads and a
partitioned native `save(path)`. Other native writes on Spark 4.1 are checked
only by the nightly job.
+
+Delta Lake support for Spark 4.2 is in progress in
[#13271](https://github.com/apache/gluten/pull/13271).
+
+### Build and run
+
+Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build
the native library first, as described in [Gluten with Velox Backend](Velox.md).
+
+````
+./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests
+./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17
-Pdelta \
+ -DskipTests -Dmaven.compiler.release=17
+````
+
+`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no
Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one
built with `-Pdelta`. Add the jar from the table above. When
`io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs
`Excluding runtime-incompatible component` and leaves its Delta support off.
+
+Native writes to HDFS, S3, GCS or ABFS paths need the native library built
with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or
`--enable_abfs=ON`. Without that connector, scans of those paths fall back and
native writes fail (see [Write problems](#write-problems)). From code.
+
+For example, on Spark 3.5:
+
+````
+spark-shell \
+ --packages io.delta:delta-spark_2.12:3.3.2 \
+ --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \
+ --conf spark.plugins=org.apache.gluten.GlutenPlugin \
+ --conf spark.memory.offHeap.enabled=true \
+ --conf spark.memory.offHeap.size=4g \
+ --conf
spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
+ --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
+ --conf
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
+````
+
+On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf
spark.sql.ansi.enabled=false`.
+
+### ANSI mode on Spark 4.x
+
+Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default
`spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back,
Delta scans and native writes included. From code.
+
+````
+spark.sql.ansi.enabled=false
+````
+
+Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job
turns ANSI mode off in the test setup that loads Gluten. That setup and some
Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`.
Spark 3.x behaves the same when ANSI mode is turned on. From code.
+
+Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false`
removes that fallback. Velox then runs the offloaded operators with ANSI mode
on. For Delta Lake, only the nightly Delta Lake job tests that combination, in
the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem
(see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)).
+
+## Support Status
+The following values show the Delta Lake support status:
+
+| Value | Description
|
+|----------------|----------------------------------------------------------------------------|
+| Offload | Runs in the Velox backend. For writes, Velox writes the
data files. |
+| PartialOffload | Some operators run in Velox and the rest fall back
|
+| Fallback | Runs on Spark or Delta Lake
|
+| Exception | Fails instead of falling back
|
+| ResultMismatch | Can return wrong results or write wrong files
|
+
+Not tested: no test covers the case.<br>
+From code: the status comes from reading the source. No test that runs in CI
checks it.<br>
+Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br>
+
+## Adding catalogs
+Fallback
+
+Catalog operations run on the driver.
+
+## Creating a table
+Fallback
+
+````
+CREATE TABLE t (id INT, name STRING) USING delta;
+````
+
+A table created without `AS SELECT` is a metadata-only commit. There is
nothing to offload.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+CREATE TABLE t2 USING delta AS SELECT * FROM t;
+````
+
+With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on
Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT`
and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog.
Seen on 4.1. On 3.5 and 4.0 this is from code.
+
+## Writing
+PartialOffload
+
+````
+INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1;
+````
+
+Delta's writer writes the files of `INSERT`, with or without native write. The
`SELECT` can run in Velox. From code.
+
+Fallback
+
+````
+INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c');
+````
+
+With the default `spark.gluten.sql.columnar.localTableScan=false`, the
`VALUES` scan runs on Spark. Delta's writer writes the files. From code.
+
+Offload
+
+````
+SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true;
+UPDATE t SET name = 'b' WHERE id = 2;
+````
+
+Set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to
have Velox write the data files of the commands marked Offload in [Write
commands](#write-commands). On Spark 4.x, also apply [ANSI mode on Spark
4.x](#ansi-mode-on-spark-4x).
+
+Native write does not check the codec, the path scheme or the partition column
types before it runs. Unsupported values make the task fail or write wrong
files. Read [Write problems](#write-problems) before turning it on.
+
+### Checking the plan
+
+A command that Gluten wraps for native write shows a `GlutenDelta` node, for
example `Execute GlutenDelta UpdateCommand` or `GlutenDelta
AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files.
+
+````
+val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2")
+df.queryExecution.executedPlan.treeString.contains("GlutenDelta ")
+````
+
+The node also shows when Velox writes no data files. This happens for a DELETE
that writes only deletion vectors or only metadata, and for a Spark 4.x write
whose schema Velox cannot write, such as one with a VARIANT column. From code.
For `DataFrameWriter` calls, read the plan from the Spark UI SQL tab or a
`QueryExecutionListener`.
+
+### Write commands
+
+The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise
every row is at best PartialOffload. In PartialOffload rows, Delta writes the
files and the query that feeds the write can run in Velox.
+
+Gluten's CI checks a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on
3.5. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE,
`save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check
results only. On 4.1, the nightly job shows every Offload row except DELETE
going through Velox's writer. The other Offload cases are from code.
+
+| SQL | DataFrame API | Status | Notes |
+|---|---|---|---|
+| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or
without `partitionBy`. |
+| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when
`t` does not exist and the mode is not overwrite | Offload | Needs a
DeltaCatalog target. |
+| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` |
`writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in
overwrite mode | Offload | Needs a DeltaCatalog target. |
+| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion
vectors, Velox writes only the updated rows. |
+| `DELETE` | `DeltaTable.delete()` | Offload | Velox writes the rewritten
files. A deletion-vector DELETE and a metadata-only DELETE write no data files.
|
+| `OPTIMIZE`, `OPTIMIZE ... WHERE` | | Offload | Bin-packing, by path or by
name. |
+| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a
liquid-clustered table | | PartialOffload | From code. |
+| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM
...)` | | PartialOffload | From code. |
+| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE
WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in
append mode. |
+| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table,
`writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` |
PartialOffload | From code. |
+| `MERGE INTO` | | PartialOffload | Scans and the DML target scan can run in
Velox. Joins can too (from code). Native MERGE write is in progress in
[#13324](https://github.com/apache/gluten/pull/13324). |
+| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`,
batch writes follow the rows above. From code. |
+
+With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`,
UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake. Seen on 4.1.
+
+### Write features
+
+| Feature | Status | Notes |
+|---|---|---|
+| Optimized write | Offload | In the commands marked Offload, the rebalancing
shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both
are from code. `OPTIMIZE` never uses it. |
+| Auto compaction | Offload | Compacts natively written tables (seen on 4.1).
After a native write, the compaction runs in a Gluten transaction (from code).
After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). |
+
+## Reading
+### Read data
+Offload/Fallback
+
+| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 |
+|---|---|---|
+| Unpartitioned | Offload | Offload |
+| Partitioned | Offload | Offload |
+| Column mapping, name mode | Offload | Offload |
+| Column mapping, id mode | Offload, matched by physical name (from code) |
Offload, matched by physical name (seen on 4.1) |
+| Files with deletion vectors | Fallback | Offload |
+| Change data feed, no deletion vectors in range | Offload. Scans of
`_change_data` files fall back when the table allows deletion vectors (from
code). | Offload |
+| Change data feed, deletion vectors in range | Fallback | Fallback |
+
+````
+SELECT count(1) AS count, name
+FROM t
+GROUP BY name;
+````
+
+Reads need no Gluten option. In `EXPLAIN` output, the native scan shows up as
`FileDeltaScanTransformer`. `EXPLAIN FORMATTED` and the Spark UI plan graph
show `DeltaScanTransformer`. Delta prunes partitions and skips files by
statistics while it plans the query, before the scan runs.
+
+| Read | Status | Notes |
+|---|---|---|
+| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)`
| Offload | Same native scan. |
+| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. |
+| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code.
Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). |
+| Streaming source | Offload | Each micro-batch is a Delta file scan. From
code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). |
+| Dynamic partition pruning | Not tested | |
+
+### Deletion vectors
+Offload
+
+````
+ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);
+DELETE FROM t WHERE id IN (3, 4);
+SELECT * FROM t;
+````
+
+On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs
Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the
default. The DML target scan, which finds the rows for new deletion vectors,
also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML
target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files.
+
+Fallback
+
+````
+SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false;
+SELECT * FROM t;
+````
+
+| Case | Behavior |
+|---|---|
+| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads
Delta's generated `__delta_internal_is_row_deleted` or
`__delta_internal_row_index` column falls back with `fallback Delta scan
requiring generated DV metadata`. Delta adds the first column when any file in
the scanned table version, micro-batch or DML file set has a deletion vector.
It adds the second to DML target scans that write deletion vectors. Other scans
offload. Tested on Spark 3.5 and 4.0. |
+| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta
DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A
read whose selected files have none offloads (from code). |
+| Spark 3.4, DML target scans, streaming micro-batches and change data feed
scans of `_change_data` files | Fall back whenever the table protocol allows
deletion vectors. From code. |
+| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects
reads of a table with deletion vectors with `Cannot work with a non-pinned
table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0,
the table feature alone triggers it (from code). |
+
+The fallback reasons show in [Gluten UI](GlutenUI.md).
+
+### Change data feed
+Offload
+
+````
+CREATE TABLE c (id INT, name STRING) USING delta
+ TBLPROPERTIES ('delta.enableChangeDataFeed' = true);
+INSERT INTO c VALUES (1, 'a'), (2, 'b');
+UPDATE c SET name = 'bb' WHERE id = 2;
+SELECT id, name, _change_type, _commit_version
+FROM table_changes('c', 1)
+WHERE id = 2;
+````
+
+Gluten expands a batch change data feed read into Delta file scans that run in
Velox with the filters pushed down. Delta also adds an empty placeholder scan
to the plan. It falls back when the table has a MAP column (from code). If the
expansion fails, Gluten logs a warning and the read uses Delta's change data
feed scan on Spark. From code.
+
+Fallback
+
+````
+SET
spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false;
+SELECT * FROM table_changes('c', 1);
+````
+
+A read whose version range adds or removes a file with a deletion vector stays
on Delta's change data feed scan on Spark. Insert-only ranges of a table with
deletion vectors enabled still run in Velox. With
`spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`,
every batch change data feed read uses Delta's scan. Streaming change data feed
reads ignore this option. They run as Delta file scans, and a scan whose files
have deletion vectors falls back (from code).
+
+### Read metadata
+Offload
+
+````
+SELECT id, _metadata.file_path, _metadata.row_index
+FROM t
+WHERE _metadata.row_index = 2;
+````
+
+The native scan fills the `_metadata` file fields and `input_file_name()`
(from code), and `_metadata.row_index`. Gluten tests check
`_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake
tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking
tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively
(from code). They return correct values except in the cases in [Read
problems](#read-problems) (seen on 4.1).
+
+Fallback
+
+````
+DESCRIBE HISTORY t;
+````
+
+Delta reads the transaction log on Spark. Every scan under `_delta_log`,
checkpoints included, stays on Spark. From code. The reason is `fallback Delta
_delta_log scan`.
+
+## Table features
+Offload/Fallback
+
+The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the
row names it. The Native write column applies to Spark 3.5 to 4.1 with native
write on.
+
+| Feature | Read | Native write | Notes |
+|---|---|---|---|
+| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | See
[Deletion vectors](#deletion-vectors). |
+| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors
fall back. On Spark 3.4, scans of `_change_data` files also fall back when the
table allows deletion vectors (from code). | Offload | UPDATE and DELETE write
the `_change_data` files with Velox (from code). Change data feed reads of
their changes are correct (seen on 4.1). See [Change data
feed](#change-data-feed). |
+| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch |
Files have no Parquet field IDs. Seen on 4.1. See [Write
problems](#write-problems). |
+| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when
field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception
when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by
physical name. See [Known problems](#known-problems). |
+| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are
correct. The projection that generates identity values runs on Spark (from
code). |
+| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers
generated columns. Velox computes the generated values when it supports the
expression (from code). Partition filters derived from them reach the native
scan. `DATE_FORMAT` results for years above 9999 differ (see [Write
problems](#write-problems)). |
+| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on
the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5
and 4.0. Seen on 4.1. |
+| Row tracking | Offload, Exception for `_metadata.base_row_id` filters |
Offload | Seen on 4.1. |
+| Liquid clustering | Offload | PartialOffload | Native commands write to
clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer.
Both are from code. |
+| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned
| The write status is from code. No test reads a UniForm table. |
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload,
Exception for FLOAT or DOUBLE partition columns | Seen on 4.1. Spark 3.5 and
4.0 are not covered. |
+| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for
some writes (seen on 4.1) | See [DataType](#datatype). |
+| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code.
See [DataType](#datatype). |
+| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). |
+| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. |
+
+## DataType
+
+The native write columns apply to Spark 3.5 to 4.1 with native write on. On
Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT
and ARRAY columns natively. The other native write statuses are seen on 4.1,
unless a row says they are from code.
+
+| Type | Read | Native write | Native write as partition column | Notes |
|---|---|---|---|---|
-| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported |
-| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported |
-
-Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The
difference between
-the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta
3.3 vs Spark 4.0 +
-Delta 4.0), not a native-write capability gap.
-
-## Build and runtime notes
-
-Build Gluten with Delta support by enabling `-Pdelta` together with the Velox
backend profile and a Spark profile.
-
-- Spark 3.5 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests`
-- Spark 4.0 build example:
- - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13
-Pjava-17 -DskipTests`
-
-Native Delta write is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite`
- - Default: `false`
- - Type: experimental
-
-Native change data feed scan offload is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan`
- - Default: `true`
- - Type: experimental
-
-The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:
-
-- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan`
- - Default: `true`
- - Type: experimental
- - When disabled, the DML target scan that produces file paths and row
indexes for
- deletion-vector writes stays on Spark; other scans are unaffected.
-
-| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg
format-version | Feature type | Supported by Gluten (Velox) |
-|---|---:|---:|---:|---|---|
-| Basic functionality | 2 | 1 | 1 | Writer | Yes |
-| CHECK constraints | 3 | 1 | N/A | Writer | No |
-| Change data feed | 4 | 1 | N/A | Writer | Yes |
-| Generated columns | 4 | 1 | N/A | Writer | Partial |
-| Column mapping | 5 | 2 | N/A | Reader and writer | Yes |
-| Identity columns | 6 | 1 | N/A | Writer | Yes |
-| Row tracking | 7 | 1 | 3 | Writer | Partial |
-| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial |
-| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No |
-| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes |
-| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested |
-| Type widening | 7 | 3 | N/A | Reader and writer | Partial |
-| Variant | 7 | 3 | 3 | Reader and writer | Not tested |
-| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested |
-| Collations | 7 | 3 | N/A | Reader and writer | Not tested |
-| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested |
+| BOOLEAN | Offload | Offload | Offload | |
+| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | |
+| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write
problems](#write-problems). |
+| DECIMAL | Offload | Offload | Offload | Precision above 18 included. |
+| STRING | Offload | Offload | Offload | |
+| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the
length on the JVM before the native write. Both statuses are from code. Reads
return correct results (seen on 4.1). |
+| BINARY | Not tested | Not tested | Not tested | Only indirect coverage. |
+| DATE | Offload | Offload | Offload | |
+| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values
(seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The
partition column status is from code. See [Known problems](#known-problems). |
+| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on
4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters,
sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ
column fall back, even when they do not use that column. Scans, aggregates,
shuffles and some projections can run in Velox. These operator rules are from
code. |
+| STRUCT, ARRAY, MAP | Offload | Offload | | |
+| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall
back too. Both statuses are from code. |
+| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall
back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema
with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the
`collations` feature. From code. |
+
+A plan node falls back when its output contains a type Velox does not support,
such as VARIANT. With the default
`spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan
that does not select such a column can still offload. From code. Delta Lake
does not allow interval columns.
+
+## Format
+PartialOffload
+
+Delta data files are Parquet, and Gluten reads them with Velox's Parquet
reader. Velox writes them only for native write commands on Spark 3.5 to 4.1.
Otherwise Delta's Parquet writer writes them.
+
+### Natively written files
+
+| Property | Behavior |
+|---|---|
+| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop
framing), none. From code. Other codecs: see [Write problems](#write-problems).
|
+| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize`
and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are
ignored. From code. |
+| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is
ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From
code. |
+| Parquet field IDs | Not written, for column-mapped and UniForm tables too.
Seen on 4.1 for column mapping, from code for UniForm. See [Write
problems](#write-problems). |
+| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is
not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on
4.1. |
+| Nested field names | Lowercased below the top level when
`spark.sql.caseSensitive=false`. Spark and Gluten match them
case-insensitively. A case-sensitive reader can miss them. From code. |
+| Page index | Always written. From code. |
+| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays
and maps keep the standard layout. From code. |
+| File statistics | numRecords, min, max and nullCount are computed in Velox
(checked on Spark 3.5). Gluten's tests check the values against the rows in
each file on Spark 3.5 to 4.1. Most of Delta's data-skipping tests pass (seen
on 4.1). 22 data-skipping tests that check statistics still fail. They expect
specific rows in each file after `repartition(2)`. |
+| Data file paths | Come from Delta's committer, so names and random prefixes
follow Delta. File names seen on 4.1, random prefixes from code. |
+
+## SQL
+
+| Statement | Status | Notes |
+|---|---|---|
+| `SELECT` | Offload | See [Reading](#reading). |
+| `table_changes()` | Offload | Ranges with deletion vectors fall back. See
[Change data feed](#change-data-feed). |
+| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With
native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1. See
[Creating a table](#creating-a-table). |
+| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write
commands](#write-commands). |
+| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback |
They read and write no data files. From code. |
+| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms
read and write no data files. `ADD CONSTRAINT` and `ALTER COLUMN ... SYNC
IDENTITY` scan the table, and that scan can run in Velox. `DROP FEATURE` and
`SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data
files with Delta's writer. From code. |
+| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files.
From code. |
+| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless
the statement has `NO STATISTICS` or
`spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files
for statistics. That scan can run in Velox. From code. |
+
+## Schema evolution
+PartialOffload
+
+The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to
4.1. Otherwise Delta writes the files.
+
+| Change | Status | Notes |
+|---|---|---|
+| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema.
|
+| `overwriteSchema` with `save(path)` | Offload | Delta applies the change
before the write. |
+| Schema evolution in SQL `INSERT` | PartialOffload | From code. |
+| Schema evolution in `MERGE` | PartialOffload | Results match Delta Lake
except for some STRUCT map key changes. See [MERGE, UPDATE and DELETE
problems](#merge-update-and-delete-problems). The two TIMESTAMP_NTZ widening
tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values.
Seen on 4.1. |
+| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and
nested renames return correct results. After a top-level rename, partition
pruning and file skipping still work. |
+| `DROP COLUMN` with column mapping | Offload | Seen on 4.1. Plan not checked.
|
+| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Seen on 4.1.
See [Table features](#table-features). |
+| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls
back, Delta scans included. From code. |
+
+## Known problems
+
+### Read problems
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A column default stored as `EXISTS_DEFAULT` metadata, for example after
`CONVERT TO DELTA`, reads as NULL for rows in files without that column. |
ResultMismatch | Seen on 4.1. #11494 reports the same bug for plain Parquet on
3.4 to 4.1. | [#11494](https://github.com/apache/gluten/issues/11494) |
+| In column mapping id mode, when field IDs no longer match the Parquet column
names, Gluten returns the column with the matching physical name. Delta Lake
returns NULL or the column the ID points to. | ResultMismatch | 3.4 to 4.1,
seen on 4.1 | [#12241](https://github.com/apache/gluten/issues/12241) |
+| On a TIMESTAMP-partitioned table with six partition values from .1 to
.123456 seconds, `count(DISTINCT ts)` returns 3, as if the values were
truncated to milliseconds. Equality filters on the column still return the
right rows. | ResultMismatch | Seen on 4.1 | |
+| Reading a column widened from DATE to TIMESTAMP_NTZ fails with `Converted
type DATE is not allowed for requested type TIMESTAMP`. | Exception | Seen on
4.1 | |
+| A query that selects only `_metadata.file_name` from a table with a BIGINT
data column named `file_name` reads the data column and fails with `Converted
type BIGINT is not allowed for requested type VARCHAR`. | Exception | Seen on
4.1 | |
+| A filter on `_metadata.base_row_id` fails with `Synthesized column
'base_row_id' failed filter validation`. | Exception | Seen on 4.1 | |
+| On a row-tracking table whose older files lack base row IDs, selecting
`_metadata.row_id` returns rows. Delta raises an error. Delta Lake's test
creates this state with a manual metadata commit. `ALTER TABLE ... SET
TBLPROPERTIES` backfills base row IDs when it enables row tracking. |
ResultMismatch | Seen on 4.1 | |
+| Reading or deleting from a 2-billion-row table with deletion vectors can
grow native memory until the JVM is killed. Delta Lake's two 2-billion-row
tests reproduce it. The nightly job force-fails both tests. | Exception |
Reported on 4.1 | [#12387](https://github.com/apache/gluten/issues/12387) |
+| With `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the
default, selecting `__delta_internal_is_row_deleted` from a relation built by
hand over `DeltaParquetFileFormat` drops the deleted rows and returns 1 for the
rows it keeps. The column is not in the table schema, so SQL and DataFrame
queries over the table cannot select it. | ResultMismatch | Seen on 4.1 | |
+| Scans of a table with many small files can be slower than vanilla Spark on
EBS storage. In the report, two of five TPC-DS SF1000 queries take 2.2 and 3.4
times as long as on vanilla Spark, with 182,300 files in one table. A
maintainer did not see the slowdown on a host with local NVMe storage. | |
Reported on 3.4 | [#10778](https://github.com/apache/gluten/issues/10778) |
+
+### Write problems
+
+These need native write on, unless a row says otherwise.
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| A write partitioned by a FLOAT or DOUBLE column fails with `Unsupported
partition type`. | Exception | 3.5 to 4.1, seen on 4.1 |
[#13138](https://github.com/apache/gluten/issues/13138) |
+| A write to a path scheme the native library cannot write fails at task time.
This covers custom Hadoop file systems, and S3, GCS, HDFS or ABFS in a build
without that connector. | Exception | 3.5 to 4.1, seen on 4.1 (custom file
system), from code (missing connector) |
[#4963](https://github.com/apache/gluten/issues/4963) |
+| Files written for a column mapping table, in name or id mode, have no
Parquet field IDs, which the Delta Lake protocol requires. Delta Lake and
Gluten read name-mode files by physical name. Readers that match by field ID
cannot. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | |
+| Spark's own reader fails on natively written id-mode files with `Spark read
schema expects field Ids, but Parquet file schema doesn't contain any field
Ids`. Spark uses this reader when a scan falls back or Gluten is off. With
`spark.sql.parquet.fieldId.read.ignoreMissing=true`, Spark skips this check and
returns NULL for the data columns of these files (from code). That setting is
not tested with these files. | Exception | 3.5 to 4.1, seen on 4.1 | |
+| Files written for a UniForm table have no Parquet field IDs, so the Iceberg
metadata points at files without IDs. No error is raised. | ResultMismatch |
3.5 to 4.1, from code | [#12957](https://github.com/apache/gluten/pull/12957) |
+| A write to a partitioned UniForm table fails with `The file schema type
should be equal with the input rowvector type`. | Exception | 3.5 to 4.1, from
code | |
+| Writes to a table with the `materializePartitionColumns` feature leave the
partition columns out of the data files. | ResultMismatch | 4.1, from code | |
+| Some writes of a TIMESTAMP_NTZ data column fail with `The file schema type
should be equal with the input rowvector type`. Delta Lake's type widening
tests fail this way when they write `CAST(value AS TIMESTAMP_NTZ)`. Delta
Lake's data skipping tests also write that cast and pass. On Spark 4.x,
`spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=true`
sends TIMESTAMP_NTZ writes to Delta's writer. It is not tested as a workaround.
| Exception | Seen on 4.1 | |
+| A partitioned write fails in Velox's `PartitionIdGenerator` when a TIMESTAMP
or TIMESTAMP_NTZ partition value has sub-millisecond digits. | Exception | 3.5
to 4.1, seen on 4.1 (TIMESTAMP_NTZ), from code (TIMESTAMP) | |
+| A generated partition column such as `DATE_FORMAT(ts, 'yyyy-MM')` stores
`12345-07` for the year 12345. Spark stores `+12345-07`. Not checked with
native write off. | ResultMismatch | Seen on 4.1 | |
+| The `compression` option set to brotli or lzo fails the write. | Exception |
3.5 to 4.1, from code | |
+| The `compression` option set to lz4_raw writes Snappy data into a
`.lz4raw.parquet` file. | ResultMismatch | 3.5 to 4.1, from code | |
+| `maxRecordsPerFile` on an unpartitioned write counts rows but writes whole
batches. When the limit is smaller than a batch, the whole batch goes into one
file and the files rotated for that batch are empty. | ResultMismatch | 3.5 to
4.1, from code | |
+| `spark.sql.maxConcurrentOutputFileWriters` above 0 makes a partitioned write
with unsorted input fail with `Underlying columnar data is inaccessible from
BatchCarrierRow`. | Exception | 3.5 to 4.1, from code | |
+| An unpartitioned native write whose first task gets no rows, for example a
write of an empty DataFrame, leaves a 0-byte file that is not valid Parquet.
Reading the table directory as plain Parquet can then fail with `is not a
Parquet file (length is too low: 0)`. With statistics on, the file stays on
disk until `VACUUM`. With `spark.databricks.delta.stats.collect=false`, Delta
commits it. The last two sentences are from code. | ResultMismatch | 3.5 to
4.1, seen on 4.1 | |
+| A statistics aggregation that Velox cannot offload fails the write with
`ClassCastException`. The TIMESTAMP_NTZ case from the issue no longer fails on
main. | Exception | 3.5 to 4.1, from code |
[#12538](https://github.com/apache/gluten/issues/12538) |
+
+### MERGE, UPDATE and DELETE problems
+
+MERGE does not use native write, so the MERGE rows also apply with native
write off (from code).
+
+| Problem | Status | Versions | Issue |
+|---|---|---|---|
+| UPDATE or DELETE through a temp view that swaps column names, such as
`SELECT value AS key, key AS value FROM t`, changes different rows than Delta
Lake. In Delta Lake's test, a DELETE removes both rows where Delta Lake removes
one. Not checked with native write off. | ResultMismatch | 3.5 to 4.1, seen on
4.1 | |
+| MERGE operation metrics in `DESCRIBE HISTORY` are too high, and a MERGE that
changes nothing records change data rows on a change data feed table. |
ResultMismatch | 3.5 to 4.1, seen on 4.1 |
[#9003](https://github.com/apache/gluten/issues/9003) |
+| With ANSI mode on, `spark.gluten.sql.ansiFallback.enabled=false` and
`spark.databricks.delta.updateAndMergeCastingFollowsAnsiEnabledFlag=true`, a
MERGE that casts `ARRAY<BIGINT>` to `ARRAY<INT>` or `MAP<STRING,BIGINT>` to
`MAP<STRING,INT>` with overflowing values completes without an error. This was
seen in a `WHEN MATCHED THEN UPDATE` clause and in a `WHEN NOT MATCHED THEN
INSERT` clause inside `foreachBatch`. The Delta Lake flag is internal and false
by default. | ResultMismatch | Seen on 4.1 | |
+| MERGE schema evolution that adds or removes fields of a STRUCT map key can
fail with `map key cannot be indeterminate`. | Exception | Seen on 4.1 | |
+
+### Error types
+
+These operations fail as in Delta Lake, with a different exception type or
error class. A failed write commits nothing (from code). Seen on 4.1. Spark 3.5
and 4.0 share the code.
+
+| Failure | Delta Lake raises | Gluten raises |
+|---|---|---|
+| CHECK, NOT NULL or generated-column violation on a partitioned native write.
The NOT NULL case is from code. | `DeltaInvariantViolationException` |
`SparkException` (`TASK_WRITE_FAILED`) wrapping `GlutenException` |
+| `replaceWhere` violation on a partitioned native write |
`DeltaAnalysisException` (`DELTA_REPLACE_WHERE_MISMATCH`) | `SparkException`
(`TASK_WRITE_FAILED`) wrapping `GlutenException` |
+| Store-assignment cast overflow in a MERGE `WHEN MATCHED THEN UPDATE` clause
| `DeltaArithmeticException` (`DELTA_CAST_OVERFLOW_IN_TABLE_WRITE`) |
`GlutenException` |
+| ANSI cast overflow evaluated in Velox | `SparkArithmeticException` |
`GlutenException` (`VeloxUserError`) |
+| Missing data file | `SparkException` (`FAILED_READ_FILE.FILE_NOT_EXIST`) on
Spark 4.x | `SparkException` wrapping `GlutenException` (`FILE_NOT_FOUND`) |
+
+For errors raised on the JVM, the Delta Lake message stays in the
`GlutenException` message. On unpartitioned `save(path)` writes, CHECK and NOT
NULL violations keep Delta Lake's exception types. Tracked in
[#11912](https://github.com/apache/gluten/issues/11912).
+
+## Configuration
+### Gluten Options
+
+| Gluten option | Default | Description |
+| --- | --- | --- |
+| spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite | false |
Experimental. Velox writes the data files of the commands marked Offload in
[Write commands](#write-commands). Spark 3.5 to 4.1. No effect on Spark 3.4. |
+| spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan |
true | Experimental. When false, the DML target scan that reads row indexes for
deletion vectors stays on Spark. Other scans still offload. |
+| spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan |
true | Experimental. When false, batch change data feed reads use Delta Lake's
change data feed scan. |
+| spark.gluten.sql.columnar.filescan | true | When false, every V1 file source
scan (`FileSourceScanExec`) falls back, Delta scans included. No Gluten option
turns off Delta scans alone. |
+| spark.gluten.sql.ansiFallback.enabled | true | With ANSI mode on, every
query falls back. See [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). |
+| spark.gluten.sql.columnar.parquet.write.blockSize | 128MB | Row-group size
for native Delta writes. |
+| spark.gluten.sql.native.parquet.write.blockRows | 100000000 | Row-group row
limit for native Delta writes. |
+| spark.gluten.sql.native.writer.enabled | (unset) | Controls native writes of
non-Delta Parquet files. Unset means on. No effect on Delta writes. |
+| spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation | false
| When true, every operator whose input or output includes TIMESTAMP_NTZ falls
back. On Spark 4.x, native Delta writes of TIMESTAMP_NTZ schemas then use
Delta's writer. From code. |
+| spark.gluten.sql.columnar.backend.velox.scanMapKeyPruningEnabled | false |
When true, Velox skips map keys a query does not look up, if the lookups sit in
filters and projections directly above the scan. Applies to Delta scans without
column mapping, with or without deletion vectors. Off for column-mapped tables.
|
+| spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames | true | Velox
matches Parquet columns by name. Id-mode column mapping tables are read by
physical name. When false, id-mode tables are read by position and name-mode
tables still by physical name. From code. |
+| spark.gluten.sql.fallbackUnexpectedMetadataParquet | false | When true,
Delta scans fall back if a sampled file is encrypted, carries legacy datetime
rebase metadata, uses LZO or BROTLI, or has variant annotations. LZ4_RAW also
counts on Spark 4.0 and 4.1. The variant check runs on Spark 4.1 only. The
sample is the first `spark.gluten.sql.fallbackUnexpectedMetadataParquet.limit`
files (default 10) under the table root. `_delta_log` files count, so the
sample may contain no data files. The check is skipped when
`spark.gluten.sql.scan.fileSchemeValidation.enabled=false`. From code. |
+| spark.gluten.sql.scan.fileSchemeValidation.enabled | true | Scans on a file
scheme Velox cannot read fall back. |
+
+The three `spark.gluten.sql.columnar.backend.velox.delta.*` options are read
per query, so `SET` changes them for the session.
+
+### Catalogs
+
+Gluten does not change DeltaCatalog. `CREATE TABLE ... AS SELECT` and `REPLACE
TABLE ... AS SELECT` are written natively only through a DeltaCatalog (from
code). With native write on, `UPDATE`, `DELETE`, plain `OPTIMIZE` and
`save(path)` are written natively with any catalog (from code).
+
+### SQL Extensions
+Fallback
+
+Supports `spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension`.
Gluten does not offload the command nodes of Delta's own SQL commands, such as
`VACUUM`, `RESTORE` and `DESCRIBE HISTORY`. Their `_delta_log` scans stay on
Spark. From code. `OPTIMIZE` follows [Write commands](#write-commands).
GlutenPlugin appends its own extension.
+
+### Runtime configuration
+The "Gluten Support" column uses these values:
+
+✅ Supported<br>
+❌ Not Supported<br>
+⚠️ Partial Support<br>
+🚫 Not applied or transparent to Gluten<br>
+
+A blank cell means unknown.
+
+### Spark SQL Options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| spark.sql.extensions | (none) | Set to
`io.delta.sql.DeltaSparkSessionExtension`. | ✅ |
+| spark.sql.catalog.spark_catalog | builtin on Spark 4.x, (unset) on 3.x | Set
to `org.apache.spark.sql.delta.catalog.DeltaCatalog`. | ✅ |
+| spark.sql.ansi.enabled | true on Spark 4.x, false on 3.x | When true, every
query falls back unless `spark.gluten.sql.ansiFallback.enabled=false`. | ⚠️ |
+| spark.sql.parquet.compression.codec | snappy | Codec for native writes when
the `compression` option is unset. | ⚠️ |
+| spark.sql.files.maxRecordsPerFile | 0 | Exact on partitioned native writes.
See [Write problems](#write-problems). | ⚠️ |
+| spark.sql.maxConcurrentOutputFileWriters | 0 | Above 0, partitioned native
writes with unsorted input fail. From code. | ❌ |
+| spark.sql.files.ignoreMissingFiles | false | Passed to Velox. Native scans
skip missing files when true. | ✅ |
+| spark.sql.parquet.mergeSchema | false | When true, Delta scans fall back.
From code. | ❌ |
+| spark.sql.parquet.outputTimestampType | INT96 | Ignored by native writes.
From code. See [Natively written files](#natively-written-files). | ❌ |
+| spark.sql.parquet.writeLegacyFormat | false | Native writes apply it to
DECIMAL only. From code. See [Natively written files](#natively-written-files).
| ⚠️ |
+| spark.sql.caseSensitive | false | When false, native writes lowercase nested
struct field names. From code. | ⚠️ |
+| spark.sql.sources.partitionOverwriteMode | STATIC | With native write on,
Velox writes the files of a dynamic overwrite through `save(path)`, and the
results are correct (seen on 4.1). | ✅ |
+| spark.databricks.delta.deletionVectors.useMetadataRowIndex | true | When
false, scans that read Delta's generated deletion vector columns fall back. See
[Deletion vectors](#deletion-vectors). | ⚠️ |
+| spark.databricks.delta.stats.collect | true | Native writes compute file
statistics in Velox. | ✅ |
+| spark.databricks.delta.stats.skipping | true | Delta skips files by
statistics before the scan runs. The scan offloads either way, except in the
deletion-vector case in [Deletion vectors](#deletion-vectors). | 🚫 |
+| spark.databricks.delta.optimizeWrite.enabled | (unset) | Native writes run
the optimized write shuffle in Velox (from code). File counts match Delta Lake
(seen on 4.1). | ✅ |
+| spark.databricks.delta.optimizeWrite.binSize | 512 (MiB) | Used by the
native optimized write shuffle. From code. | ✅ |
+| spark.databricks.delta.autoCompact.enabled | (unset) | Compacts natively
written tables. Seen on 4.1. | ✅ |
+| spark.databricks.delta.schema.autoMerge.enabled | false | Delta applies
schema changes before the write. | 🚫 |
+| spark.databricks.delta.dynamicPartitionOverwrite.enabled | true | Delta
decides which partitions to replace. | ✅ |
+| spark.databricks.delta.merge.repartitionBeforeWrite.enabled | true | MERGE
uses Delta's writer, so this only shapes Delta's write. | 🚫 |
+| spark.databricks.delta.optimize.maxFileSize | 1073741824 (1 GiB) | Delta
plans the bins. With native write on, Velox writes the files of plain
`OPTIMIZE`. | ✅ |
+| spark.databricks.delta.optimize.minFileSize | 1073741824 (1 GiB) | Delta
plans the bins. With native write on, Velox writes the files of plain
`OPTIMIZE`. | ✅ |
+| spark.databricks.delta.write.txnAppId,
spark.databricks.delta.write.txnVersion | (unset) | Idempotent writes. Commit
only. | 🚫 |
+| spark.databricks.delta.commitInfo.userMetadata | (unset) | Commit only. | 🚫 |
+| spark.databricks.delta.properties.defaults.* | (unset) | Default for the
matching `delta.*` table property on new tables. | 🚫 |
+
+`spark.databricks.delta.deletionVectors.useMetadataRowIndex`,
`spark.databricks.delta.optimizeWrite.enabled`,
`spark.databricks.delta.optimizeWrite.binSize` and
`spark.databricks.delta.autoCompact.enabled` do not exist in Delta Lake 2.4.0.
+
+#### Read options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| versionAsOf | (latest) | Version of the table snapshot to read. Seen on 4.1.
| ✅ |
+| timestampAsOf | (latest) | Read the snapshot current at this timestamp. Seen
on 4.1. | ✅ |
+| readChangeFeed | false | Read the change data feed. Ranges with deletion
vectors fall back. | ⚠️ |
+| startingVersion, endingVersion | (none) | Change data feed version range.
`startingVersion` also applies to streaming reads. | ✅ |
+| startingTimestamp, endingTimestamp | (none) | Change data feed timestamp
range. `startingTimestamp` also applies to streaming reads. Seen on 4.1. | ✅ |
+| maxFilesPerTrigger | 1000 | Streaming: files per micro-batch | 🚫 |
+| maxBytesPerTrigger | (unset) | Streaming: bytes per micro-batch | 🚫 |
+| ignoreDeletes, ignoreChanges, skipChangeCommits | false | Streaming: how to
treat commits that remove data | 🚫 |
+| excludeRegex | (none) | Streaming: skip files whose path matches | 🚫 |
+| failOnDataLoss | true | Streaming: fail when Delta log files the stream
needs are missing | 🚫 |
+| allowSourceColumnDrop, allowSourceColumnRename, allowSourceColumnTypeChange
| (unset) | Streaming schema tracking. Delta Lake 4.0.1 and 4.1.0. | 🚫 |
+
+#### Write options
+
+| Option | Default | Description | Gluten Support |
+| --- | --- | --- | --- |
+| compression | spark.sql.parquet.compression.codec | Native writes support
the codecs in [Natively written files](#natively-written-files). | ⚠️ |
+| maxRecordsPerFile | spark.sql.files.maxRecordsPerFile | Exact on partitioned
native writes. See [Write problems](#write-problems). | ⚠️ |
+| partitionOverwriteMode | spark.sql.sources.partitionOverwriteMode |
`dynamic` works through `save(path)`. `INSERT OVERWRITE` and `insertInto` use
Delta's writer (from code). | ✅ |
+| replaceWhere | (none) | A violating partitioned native write fails and
commits nothing, with a different error type (see [Error types](#error-types)).
| ⚠️ |
+| mergeSchema | spark.databricks.delta.schema.autoMerge.enabled | Delta
applies the schema change before the write. | 🚫 |
+| overwriteSchema | false | Delta applies the schema change before the write.
| 🚫 |
+| optimizeWrite | (unset) | Runs the rebalancing shuffle in Velox before the
native write. From code. | ✅ |
+| txnAppId, txnVersion | (unset) | Idempotent writes. Commit only. | 🚫 |
+| userMetadata | (unset) | Stored in `commitInfo.userMetadata`. Commit only. |
🚫 |
+| dataChange | true | Sets `dataChange` on the file actions. Commit only. | 🚫 |
+| parquet.* (writer option or Hadoop setting) | | Ignored by native writes
(from code). Use `spark.gluten.sql.columnar.parquet.write.blockSize` for row
groups. | ❌ |
+| fs.* Hadoop file system options | | A custom file system scheme fails on
native writes (seen on 4.1). Per-write options for supported stores, such as
`fs.s3a.access.key`, are not tested with native writes. | ❌ |
Review Comment:
```suggestion
| File system schemes unsupported by the native library | | Native writes
fail at task time. A custom Hadoop file system scheme failed on Spark 4.1. See
[Write problems](#write-problems). | ❌ |
| fs.* per-write options for supported stores | | Per-write options such as
`fs.s3a.access.key` have not been tested with native writes. | |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
