zhouyuan commented on code in PR #13338: URL: https://github.com/apache/gluten/pull/13338#discussion_r4312195394
########## docs/get-started/VeloxDelta.md: ########## @@ -1,62 +1,572 @@ -# Delta Lake Feature Support Status in Apache Gluten (Velox Backend) +--- +layout: page +title: Delta Lake Support in Velox Backend +nav_order: 13 +parent: Getting-Started +--- -This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**. +# Delta Lake Support in Velox Backend -## Supported Spark / Delta combinations +## Supported Spark version -| Spark profile | Spark version | Scala version | Delta Lake version | Status | +| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by | +|---|---|---|---|---|---| +| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites | +| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) | +| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` | +| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job | +| 4.2 | None on main | | None | None | None | + +Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. The copied Delta Lake suites run with native write on. + +Spark 4.1 is the most tested. A nightly job builds Delta Lake 4.2.0 from source and runs its Spark test suites on Spark 4.1. It puts a Gluten bundle compiled against Delta Lake 4.1.0 on the test classpath. It patches Delta Lake's `DeltaSQLCommandTest` to load Gluten with native write on. "Seen on 4.1" on this page refers to that job. Its known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug. + +On Spark 4.1, Gluten's own suites check deletion vector reads and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job. + +Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271). + +### Build and run + +Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md). + +```` +./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests +./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \ + -DskipTests -Dmaven.compiler.release=17 +```` + +`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add the jar from the table above. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. + +Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code. + +For example, on Spark 3.5: + +```` +spark-shell \ + --packages io.delta:delta-spark_2.12:3.3.2 \ + --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \ + --conf spark.plugins=org.apache.gluten.GlutenPlugin \ + --conf spark.memory.offHeap.enabled=true \ + --conf spark.memory.offHeap.size=4g \ + --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ + --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ + --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog +```` + +On Spark 4.0 or 4.1, use the Delta Lake jar for that version and add `--conf spark.sql.ansi.enabled=false`. + +### ANSI mode on Spark 4.x + +Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. + +```` +spark.sql.ansi.enabled=false +```` + +Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. Spark 3.x behaves the same when ANSI mode is turned on. From code. + +Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Velox then runs the offloaded operators with ANSI mode on. For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). + +## Support Status +The following values show the Delta Lake support status: + +| Value | Description | +|----------------|----------------------------------------------------------------------------| +| Offload | Runs in the Velox backend. For writes, Velox writes the data files. | +| PartialOffload | Some operators run in Velox and the rest fall back | +| Fallback | Runs on Spark or Delta Lake | +| Exception | Fails instead of falling back | +| ResultMismatch | Can return wrong results or write wrong files | + +Not tested: no test covers the case.<br> +From code: the status comes from reading the source. No test that runs in CI checks it.<br> +Seen on 4.1: the nightly Delta Lake job on Spark 4.1 showed the result.<br> + +## Adding catalogs +Fallback + +Catalog operations run on the driver. + +## Creating a table +Fallback + +```` +CREATE TABLE t (id INT, name STRING) USING delta; +```` + +A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. + +Offload + +```` +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +CREATE TABLE t2 USING delta AS SELECT * FROM t; +```` + +With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. + +## Writing +PartialOffload + +```` +INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1; +```` + +Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code. Review Comment: > with or without native write Could you please also add more detail on this? ########## docs/get-started/VeloxDelta.md: ########## @@ -1,62 +1,632 @@ -# Delta Lake Feature Support Status in Apache Gluten (Velox Backend) +--- +layout: page +title: Delta Lake Support in Velox Backend +nav_order: 13 +parent: Getting-Started +--- -This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**. +# Delta Lake Support in Velox Backend -## Supported Spark / Delta combinations +Gluten's Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read +acceleration does not require native writes. Native Delta writes are experimental and disabled by +default. This guide covers the supported Spark and Delta Lake versions, setup, feature support and +known limitations. -| Spark profile | Spark version | Scala version | Delta Lake version | Status | +## How a Delta query runs + +Take a Delta table partitioned by `sale_date`: + +```sql +SELECT SUM(amount) +FROM sales +WHERE sale_date = DATE '2026-10-01' + AND amount > 100; +``` + +```mermaid +flowchart LR + subgraph Planning["Spark driver"] + Snapshot["1. Delta snapshot<br/>Schema + active files"] + Select["2. Delta file pruning<br/>Partitions + file statistics"] + Plan["3. Spark plan<br/>Gluten offload checks"] + Snapshot --> Select --> Plan + end + + subgraph Execution["Spark executors"] + Scan["4. Velox scan<br/>Reads the selected files"] + Compute["5. Velox operators<br/>Filter + SUM"] + Spark["Spark operators<br/>Operators Gluten does not offload"] + Scan --> Compute + Compute <-->|"Row and columnar conversions"| Spark + end + + subgraph Storage["Delta table"] + Log[("_delta_log<br/>Metadata + file statistics")] + Data[("Parquet files")] + end + + Result["6. Query result"] + + Log --> Snapshot + Plan -->|"Offloaded operators"| Scan + Plan -->|"Other operators"| Spark + Data --> Scan + Data --> Spark + Compute --> Result + Spark --> Result +``` + +Delta Lake loads the snapshot and prunes partitions and files while Spark plans the query. Gluten then +offloads the operators Velox supports, and the rest run on Spark, so one query can mix Velox and Spark +operators. Delta Lake keeps the table metadata, the transaction log and commits. + +## Supported Spark and Delta Lake versions + +| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by | +|---|---|---|---|---|---| +| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites | +| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) | +| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` | +| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job | +| 4.2 | None on main | | None | None | None | + +Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. On Spark 4.0 and 4.1, the Read and Native write columns assume ANSI mode off (see [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x)). The copied Delta Lake suites run with native write on and `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`, which is off by default. + +Spark 4.1 has the most tests, because the nightly job runs on it. That job builds Delta Lake 4.2.0 from source against Spark 4.1.0, which is newer than the Delta Lake 4.1.0 in the table, and runs its Spark test suites. It puts a Gluten bundle compiled against Spark 4.1.1 and Delta Lake 4.1.0 on the test classpath. It replaces Delta Lake's `DeltaSQLCommandTest` with Gluten's copy, which loads Gluten with native write on and the non-default `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`. It then runs `.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh`. That script cherry-picks two Delta Lake test changes that are not in 4.2.0, delta-io/delta#7104 and #7105, so Delta Lake's scan checks accept Gluten's scan node. It also sets a row-group row limit in two test fixtures. The same job also runs on pull requests that change Gluten's Delta code. "Seen on 4.1" on this page refers to that job. + +The job's known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug. + +On Spark 4.1, Gluten's own suites check deletion vector, change data feed, column mapping, time travel and partition filter reads, and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job. + +Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271). + +### Build and run + +Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md). + +```bash +./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests +./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \ + -DskipTests -Dmaven.compiler.release=17 +``` + +`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. `dev/release/build-release.sh` passes `-Pdelta` only for Spark 3.4 and 3.5, so its Spark 4.0 and 4.1 bundles have no Delta Lake support either. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add Delta Lake with `--packages` and the coordinate from the table above, which also adds its `delta-storage` dependency. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. From code. + +Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code. + +For example, on Spark 3.5: + +```bash +spark-shell \ + --packages io.delta:delta-spark_2.12:3.3.2 \ + --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \ + --conf spark.plugins=org.apache.gluten.GlutenPlugin \ + --conf spark.memory.offHeap.enabled=true \ + --conf spark.memory.offHeap.size=4g \ + --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ + --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ + --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog +``` + +On Spark 4.0 or 4.1, use the Gluten bundle and the Delta Lake jar for that version, and add `--conf spark.sql.ansi.enabled=false`. + +### ANSI mode on Spark 4.x + +Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. Turning ANSI mode off also changes Spark's own behavior for invalid input. For example, `CAST('a' AS INT)` returns NULL instead of failing. + +```properties +spark.sql.ansi.enabled=false +``` + +Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. On Spark 3.x, turning ANSI mode on makes every query fall back in the same way. From code. + +Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Gluten passes ANSI mode to Velox. Velox does not implement ANSI behavior for every expression, so some invalid input returns a result where Spark fails ([#10134](https://github.com/apache/gluten/issues/10134)). For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). Cast overflow errors raised in Velox have a different exception type (see [Error types](#error-types)). + +## Support Status +The following values show the Delta Lake support status: + +| Value | Description | +|----------------|----------------------------------------------------------------------------| +| Offload | Runs in the Velox backend. For writes, Velox writes the data files. | +| PartialOffload | Some operators run in Velox and the rest fall back. | +| Fallback | Runs on Spark or Delta Lake. | +| Exception | Fails instead of falling back. | +| ResultMismatch | Can return wrong results or write wrong files. | + +Not tested: no test covers the case.<br> +From code: the status comes from reading the source. No test that runs in CI checks it.<br> +Seen on 4.1: the nightly Delta Lake job showed the result, with Delta Lake 4.2.0 on Spark 4.1.0.<br> +Reported on 3.4, Reported on 4.1: the linked issue reports the result on that Spark version. No CI test checks it.<br> +No label: a test in the Gluten repository that runs in CI checks the status on the stated versions.<br> + +## Adding catalogs +Fallback + +Catalog operations run on the driver. From code. + +## Creating a table +Fallback + +```sql +CREATE TABLE t (id INT, name STRING) USING delta; +``` + +A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. From code. + +Offload + +```sql +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +CREATE TABLE t2 USING delta AS SELECT * FROM t; +``` + +With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). + +## Writing +PartialOffload + +```sql +INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1; +``` + +Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code. + +Fallback + +```sql +INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c'); +``` + +With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code. + +Offload + +```sql +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +UPDATE t SET name = 'b' WHERE id = 2; +``` + +On Spark 3.5 to 4.1, set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). Velox writes the files of this UPDATE. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). + +Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on. + +### Checking the plan + +A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files. + +```scala +val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2") +df.queryExecution.executedPlan.treeString.contains("GlutenDelta ") +``` + +The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the command's plan from the Spark UI SQL tab or a `QueryExecutionListener`. Each write also reports a nested `deltaTransactionalWrite` execution for the query that feeds it. That plan never has the prefix, whichever writer ran. From code. + +### Write commands + +The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox. + +Gluten's CI checks that a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5 run as Gluten native write commands. It also checks on 3.5 and 4.0 that a DELETE that rewrites a file runs in Gluten's transaction and that its scan runs in Velox. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code. + +| SQL | DataFrame API | Status | Notes | +|---|---|---|---| +| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. | +| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. | +| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. | +| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. | +| `DELETE` | `DeltaTable.delete()` | Offload, PartialOffload when it writes deletion vectors (from code on 4.1), Fallback when it is metadata-only (from code) | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. | +| `OPTIMIZE`, `OPTIMIZE ... WHERE` | `DeltaTable.optimize().executeCompaction()`, with or without `where(...)` | Offload | Bin-packing, by path or by name. | +| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. | +| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. | +| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. | +| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. | +| `MERGE INTO` | | PartialOffload | Scans, the DML target scan and joins can run in Velox. On Spark 3.5 and 4.0, Gluten's tests check that a deletion-vector MERGE runs a scan of the target table in Velox. The other cases are from code. Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). | +| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. | + +With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake in `DescribeDeltaHistorySuite`. Seen on 4.1. Three unpartitioned tests in `DeleteMetricsSuite` and `UpdateMetricsSuite` expect two added files and get one. The job sets `spark.default.parallelism=1`, which plans the five small input files as one scan partition (from code). + +### Write features + +| Feature | Status | Notes | +|---|---|---| +| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. | +| Auto compaction | Offload after native writes, PartialOffload after other writes | Compacts natively written tables (seen on 4.1). The auto compaction tests that fail in the nightly job fail because its `spark.default.parallelism=1` makes their first write produce one file (from code). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). | + +## Reading +### Read data +Offload/Fallback + +| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 | +|---|---|---| +| Unpartitioned | Offload | Offload | +| Partitioned | Offload | Offload | +| Column mapping, name mode | Offload | Offload | +| Column mapping, id mode | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | +| Files with deletion vectors | Fallback | Offload | +| Change data feed, no deletion vectors in range | Offload. PartialOffload when the table allows deletion vectors and the range has `_change_data` files, because those scans then fall back (from code). | Offload | +| Change data feed, deletion vectors in range | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors and no `_change_data` files (from code) | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors (from code) | + +```sql +SELECT count(1) AS count, name +FROM t +GROUP BY name; +``` + +Reads need no Gluten option. With `spark.sql.adaptive.enabled=false`, `EXPLAIN` shows the native scan as `FileDeltaScanTransformer` and `EXPLAIN FORMATTED` shows `DeltaScanTransformer`. With AQE on, the default, `EXPLAIN` and `EXPLAIN FORMATTED` of a query with a shuffle, such as this one, print the plan from before Gluten offloads it. The Spark UI plan graph of a finished query shows `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs. + +| Read | Status | Notes | +|---|---|---| +| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. | +| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. | +| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). | +| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). | +| Dynamic partition pruning | Not tested | | + +### Deletion vectors +Offload + +```sql +OPTIMIZE t; +ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true); +DELETE FROM t WHERE id IN (3, 4); +SELECT * FROM t; +``` + +On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files. + +Fallback + +```sql +SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false; +SELECT * FROM t; +RESET spark.databricks.delta.deletionVectors.useMetadataRowIndex; +``` + +| Case | Behavior | +|---|---| +| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch, DML file set or `OPTIMIZE` bin has a deletion vector, so `OPTIMIZE` and `REORG TABLE ... APPLY (PURGE)` scans of those files also fall back (from code). It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. | +| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). | +| Spark 3.4, other scans, such as DML target scans, `OPTIMIZE` scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. Change data feed scans of added or removed files fall back only when a file they read has a deletion vector. From code. | +| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). | + +The fallback reasons show in [Gluten UI](GlutenUI.md). + +### Change data feed +Offload + +```sql +CREATE TABLE c (id INT, name STRING) USING delta + TBLPROPERTIES ('delta.enableChangeDataFeed' = true); +INSERT INTO c VALUES (1, 'a'), (2, 'b'); +UPDATE c SET name = 'bb' WHERE id = 2; +SELECT id, name, _change_type, _commit_version +FROM table_changes('c', 1) +WHERE id = 2; +``` + +Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. That scan falls back when the table schema has a MAP at any nesting level (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan. From code. + +PartialOffload + +```sql +SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false; +SELECT * FROM table_changes('c', 1); +``` + +A read whose version range adds or removes a file with a deletion vector uses Delta's change data feed scan. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Delta's scan reads the data files in a query of its own that Gluten also plans, and `EXPLAIN` does not show it (from code). That query's file scans can run in Velox, except scans of added or removed files that have deletion vectors and, on Spark 3.4, scans of `_change_data` files when the table allows deletion vectors (from code). Streaming change data feed reads ignore this option. They run as Delta file scans. A scan whose files have deletion vectors falls back, and on Spark 3.4 so does a scan of `_change_data` files when the table allows deletion vectors (from code). + +### Read metadata +Offload + +```sql +SELECT id, _metadata.file_path, _metadata.row_index +FROM t +WHERE _metadata.row_index = 2; +``` + +The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. On Spark 3.5 to 4.1, Delta Lake exposes `_metadata.row_index` only on tables that allow deletion vectors or have row tracking, so the example needs `t` from [Deletion vectors](#deletion-vectors) (from code). A query that selects only `_metadata.file_name` can fail when the table has a BIGINT data column named `file_name` (seen on 4.1, see [Read problems](#read-problems)). Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). In the nightly job, they return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1). + +Fallback + +```sql +DESCRIBE HISTORY t; +``` + +`DESCRIBE HISTORY` reads each commit file on Spark, through Delta's LogStore in a `mapPartitions` job. When Delta replays the log for a snapshot, every scan under `_delta_log`, checkpoints included, stays on Spark with the reason `fallback Delta _delta_log scan`. From code. + +## Table features +Offload/Fallback + +The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on. + +| Feature | Read | Native write | Notes | +|---|---|---|---| +| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | Gluten's tests check deletion vector DELETE and MERGE results with native write on, on Spark 3.5 and 4.0. Which writer writes each file is from code. See [Deletion vectors](#deletion-vectors). | +| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). | +| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). | +| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). | +| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). | +| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). | +| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. | +| Row tracking | Offload, Exception for `_metadata.base_row_id` filters, ResultMismatch for `_metadata.row_id` when older files lack base row IDs | Offload | Seen on 4.1. See [Read problems](#read-problems). | +| Liquid clustering | Offload | PartialOffload | Reads use the same native scan as other tables. Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. All three are from code. | +| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. | +| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns, for TIMESTAMP_NTZ partition values with sub-millisecond digits and for some writes of TIMESTAMP_NTZ data columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. | +| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). | +| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). | +| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). | +| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. | + +## DataType + +The Read column applies to Spark 3.4 to 4.1. On all of them, Gluten's tests read INT, BIGINT, DECIMAL(12,2), STRING, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRUCT, ARRAY and MAP columns natively. The other read statuses are from code, unless a row says otherwise. The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code. + +| Type | Read | Native write | Native write as partition column | Notes | |---|---|---|---|---| -| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported | -| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported | - -Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between -the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 + -Delta 4.0), not a native-write capability gap. - -## Build and runtime notes - -Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile. - -- Spark 3.5 build example: - - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests` -- Spark 4.0 build example: - - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests` - -Native Delta write is controlled by: - -- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite` - - Default: `false` - - Type: experimental - -Native change data feed scan offload is controlled by: - -- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan` - - Default: `true` - - Type: experimental - -The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by: - -- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan` - - Default: `true` - - Type: experimental - - When disabled, the DML target scan that produces file paths and row indexes for - deletion-vector writes stays on Spark; other scans are unaffected. - -| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) | -|---|---:|---:|---:|---|---| -| Basic functionality | 2 | 1 | 1 | Writer | Yes | -| CHECK constraints | 3 | 1 | N/A | Writer | No | -| Change data feed | 4 | 1 | N/A | Writer | Yes | -| Generated columns | 4 | 1 | N/A | Writer | Partial | -| Column mapping | 5 | 2 | N/A | Reader and writer | Yes | -| Identity columns | 6 | 1 | N/A | Writer | Yes | -| Row tracking | 7 | 1 | 3 | Writer | Partial | -| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial | -| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No | -| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes | -| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested | -| Type widening | 7 | 3 | N/A | Reader and writer | Partial | -| Variant | 7 | 3 | 3 | Reader and writer | Not tested | -| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested | -| Collations | 7 | 3 | N/A | Reader and writer | Not tested | -| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested | +| BOOLEAN | Offload | Offload | Offload | | +| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | | +| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). | +| DECIMAL | Offload | Offload | Offload | Precision above 18 included. | +| STRING | Offload | Offload | Offload | | +| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). | +| BINARY | Offload | Offload | Not tested | The read status is from code. One Delta Lake test writes a table with only a BINARY column, checks its file statistics and runs a DELETE that filters on that column (seen on 4.1). | +| DATE | Offload | Offload | Offload | | +| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). | +| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. A scan whose pushed filter has a TIMESTAMP_NTZ literal falls back (from code). Gluten's tests check on Spark 3.4 to 4.1 that a scan of the column runs in Velox, and that a filter on the column and a sort on another column cause a fallback. The other operator rules are from code. | +| STRUCT, ARRAY, MAP | Offload | Offload | | | +| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. | +| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. | + +A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns. + +## Format +PartialOffload + +Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them. + +### Natively written files + +| Property | Behavior | +|---|---| +| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). | +| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. | +| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. | +| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). | +| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. | +| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. | +| Page index | Always written. From code. | +| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. | +| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most tests in Delta's data-skipping suites pass (seen on 4.1). Of the 34 that fail, 22 check statistics and expect specific rows in each file after `repartition(2)`. The other 12 hit the `_metadata.file_name` failure in [Read problems](#read-problems). | +| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. | + +## SQL + +| Statement | Status | Notes | +|---|---|---| +| `SELECT` | Offload | See [Reading](#reading). | +| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). | +| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1, from code on 3.5 and 4.0. See [Creating a table](#creating-a-table). | +| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). | +| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. | +| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` scans the table, and that scan can run in Velox. On Spark 3.5 to 4.1, `ALTER COLUMN ... SYNC IDENTITY` also scans it, and `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. | +| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. | +| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. | + +## Schema evolution +PartialOffload + +The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files. + +| Change | Status | Notes | +|---|---|---| +| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema (from code). On Spark 3.5 and 4.0, the copied `DeltaSuite` checks the results of a `mergeSchema` overwrite. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). | +| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. From code. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). | +| Schema evolution in SQL `INSERT` | PartialOffload | From code. | +| Schema evolution in `MERGE` | PartialOffload, Exception for some STRUCT map key changes | The merged table contents match Delta Lake. Some STRUCT map key changes fail in Velox (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. | +| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. | +| `DROP COLUMN` with column mapping | Offload | From code. Delta Lake's `DeltaDropColumnSuite` passes (seen on 4.1). | +| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ and for native writes to FLOAT or DOUBLE partition columns | Seen on 4.1. See [Table features](#table-features). | +| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. | Review Comment: it seems mergeScheme is support already in write section ########## docs/get-started/VeloxDelta.md: ########## @@ -1,62 +1,632 @@ -# Delta Lake Feature Support Status in Apache Gluten (Velox Backend) +--- +layout: page +title: Delta Lake Support in Velox Backend +nav_order: 13 +parent: Getting-Started +--- -This document summarizes the support status of **Delta Lake table features** when used with **Apache Gluten (Velox backend)**. +# Delta Lake Support in Velox Backend -## Supported Spark / Delta combinations +Gluten's Velox backend can accelerate eligible scans and query operators over Delta Lake tables. Read +acceleration does not require native writes. Native Delta writes are experimental and disabled by +default. This guide covers the supported Spark and Delta Lake versions, setup, feature support and +known limitations. -| Spark profile | Spark version | Scala version | Delta Lake version | Status | +## How a Delta query runs + +Take a Delta table partitioned by `sale_date`: + +```sql +SELECT SUM(amount) +FROM sales +WHERE sale_date = DATE '2026-10-01' + AND amount > 100; +``` + +```mermaid +flowchart LR + subgraph Planning["Spark driver"] + Snapshot["1. Delta snapshot<br/>Schema + active files"] + Select["2. Delta file pruning<br/>Partitions + file statistics"] + Plan["3. Spark plan<br/>Gluten offload checks"] + Snapshot --> Select --> Plan + end + + subgraph Execution["Spark executors"] + Scan["4. Velox scan<br/>Reads the selected files"] + Compute["5. Velox operators<br/>Filter + SUM"] + Spark["Spark operators<br/>Operators Gluten does not offload"] + Scan --> Compute + Compute <-->|"Row and columnar conversions"| Spark + end + + subgraph Storage["Delta table"] + Log[("_delta_log<br/>Metadata + file statistics")] + Data[("Parquet files")] + end + + Result["6. Query result"] + + Log --> Snapshot + Plan -->|"Offloaded operators"| Scan + Plan -->|"Other operators"| Spark + Data --> Scan + Data --> Spark + Compute --> Result + Spark --> Result +``` + +Delta Lake loads the snapshot and prunes partitions and files while Spark plans the query. Gluten then +offloads the operators Velox supports, and the rest run on Spark, so one query can mix Velox and Spark +operators. Delta Lake keeps the table metadata, the transaction log and commits. + +## Supported Spark and Delta Lake versions + +| Spark | Delta Lake | Delta Lake jar | Read | Native write | Tested by | +|---|---|---|---|---|---| +| 3.4.4 | 2.4.0 | `io.delta:delta-core_2.12:2.4.0` | Offload. Deletion vector scans fall back. | None | Gluten suites | +| 3.5.5 | 3.3.2 | `io.delta:delta-spark_2.12:3.3.2` (`_2.13` for Scala 2.13) | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites (Scala 2.12 only) | +| 4.0.2 | 4.0.1 | `io.delta:delta-spark_2.13:4.0.1` | Offload | Experimental, off by default | Gluten suites, copied Delta Lake suites tagged `ExtendedSQLTest` | +| 4.1.1 | 4.1.0 | `io.delta:delta-spark_4.1_2.13:4.1.0` | Offload | Experimental, off by default | Gluten suites, nightly Delta Lake 4.2.0 job | +| 4.2 | None on main | | None | None | None | + +Each Spark profile from 3.4 to 4.1 builds against the one Delta Lake release in the table. On Spark 4.0 and 4.1, the Read and Native write columns assume ANSI mode off (see [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x)). The copied Delta Lake suites run with native write on and `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`, which is off by default. + +Spark 4.1 has the most tests, because the nightly job runs on it. That job builds Delta Lake 4.2.0 from source against Spark 4.1.0, which is newer than the Delta Lake 4.1.0 in the table, and runs its Spark test suites. It puts a Gluten bundle compiled against Spark 4.1.1 and Delta Lake 4.1.0 on the test classpath. It replaces Delta Lake's `DeltaSQLCommandTest` with Gluten's copy, which loads Gluten with native write on and the non-default `spark.gluten.sql.fallbackUnexpectedMetadataParquet=true`. It then runs `.github/workflows/util/delta-spark-ut/apply-delta-test-patches.sh`. That script cherry-picks two Delta Lake test changes that are not in 4.2.0, delta-io/delta#7104 and #7105, so Delta Lake's scan checks accept Gluten's scan node. It also sets a row-group row limit in two test fixtures. The same job also runs on pull requests that change Gluten's Delta code. "Seen on 4.1" on this page refers to that job. + +The job's known failures are listed in `.github/workflows/util/delta-spark-ut/known-failures.txt`. Four entries are two 2-billion-row deletion vector tests, run in two suites, that the job force-fails. Some entries check a query plan that changes once Gluten offloads it. Read the README in that directory before treating an entry as a Gluten bug. + +On Spark 4.1, Gluten's own suites check deletion vector, change data feed, column mapping, time travel and partition filter reads, and a partitioned native `save(path)`. Other native writes on Spark 4.1 are checked only by the nightly job. + +Delta Lake support for Spark 4.2 is in progress in [#13271](https://github.com/apache/gluten/pull/13271). + +### Build and run + +Delta Lake support is compiled in only with the `-Pdelta` Maven profile. Build the native library first, as described in [Gluten with Velox Backend](Velox.md). + +```bash +./build/mvn clean install -Pbackends-velox -Pspark-3.5 -Pdelta -DskipTests +./build/mvn clean install -Pbackends-velox -Pspark-4.1 -Pscala-2.13 -Pjava-17 -Pdelta \ + -DskipTests -Dmaven.compiler.release=17 +``` + +`dev/buildbundle-veloxbe.sh` does not pass `-Pdelta`, so its bundle has no Delta Lake support. `dev/release/build-release.sh` passes `-Pdelta` only for Spark 3.4 and 3.5, so its Spark 4.0 and 4.1 bundles have no Delta Lake support either. No Gluten bundle contains the Delta Lake jar, including one built with `-Pdelta`. Add Delta Lake with `--packages` and the coordinate from the table above, which also adds its `delta-storage` dependency. When `io.delta.sql.DeltaSparkSessionExtension` is not on the classpath, Gluten logs `Excluding runtime-incompatible component` and leaves its Delta support off. From code. + +Native writes to HDFS, S3, GCS or ABFS paths need the native library built with `--enable_hdfs=ON`, `--enable_s3=ON`, `--enable_gcs=ON` or `--enable_abfs=ON`. Without that connector, scans of those paths fall back and native writes fail (see [Write problems](#write-problems)). From code. + +For example, on Spark 3.5: + +```bash +spark-shell \ + --packages io.delta:delta-spark_2.12:3.3.2 \ + --jars gluten-velox-bundle-spark3.5_2.12-<os>_<arch>-<version>.jar \ + --conf spark.plugins=org.apache.gluten.GlutenPlugin \ + --conf spark.memory.offHeap.enabled=true \ + --conf spark.memory.offHeap.size=4g \ + --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ + --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \ + --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog +``` + +On Spark 4.0 or 4.1, use the Gluten bundle and the Delta Lake jar for that version, and add `--conf spark.sql.ansi.enabled=false`. + +### ANSI mode on Spark 4.x + +Spark 4.0 and 4.1 turn ANSI mode on by default. With Gluten's default `spark.gluten.sql.ansiFallback.enabled=true`, every query then falls back, Delta scans and native writes included. From code. Turning ANSI mode off also changes Spark's own behavior for invalid input. For example, `CAST('a' AS INT)` returns NULL instead of failing. + +```properties +spark.sql.ansi.enabled=false +``` + +Gluten's Delta tests run with ANSI mode off in CI. The nightly Delta Lake job turns ANSI mode off in the test setup that loads Gluten. That setup and some Gluten Delta suites also set `spark.gluten.sql.ansiFallback.enabled=false`. On Spark 3.x, turning ANSI mode on makes every query fall back in the same way. From code. + +Keeping ANSI on and setting `spark.gluten.sql.ansiFallback.enabled=false` removes that fallback. Gluten passes ANSI mode to Velox. Velox does not implement ANSI behavior for every expression, so some invalid input returns a result where Spark fails ([#10134](https://github.com/apache/gluten/issues/10134)). For Delta Lake, only the nightly Delta Lake job tests that combination, in the Delta Lake 4.2.0 suites that turn ANSI on. It has a known MERGE problem (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). Cast overflow errors raised in Velox have a different exception type (see [Error types](#error-types)). + +## Support Status +The following values show the Delta Lake support status: + +| Value | Description | +|----------------|----------------------------------------------------------------------------| +| Offload | Runs in the Velox backend. For writes, Velox writes the data files. | +| PartialOffload | Some operators run in Velox and the rest fall back. | +| Fallback | Runs on Spark or Delta Lake. | +| Exception | Fails instead of falling back. | +| ResultMismatch | Can return wrong results or write wrong files. | + +Not tested: no test covers the case.<br> +From code: the status comes from reading the source. No test that runs in CI checks it.<br> +Seen on 4.1: the nightly Delta Lake job showed the result, with Delta Lake 4.2.0 on Spark 4.1.0.<br> +Reported on 3.4, Reported on 4.1: the linked issue reports the result on that Spark version. No CI test checks it.<br> +No label: a test in the Gluten repository that runs in CI checks the status on the stated versions.<br> + +## Adding catalogs +Fallback + +Catalog operations run on the driver. From code. + +## Creating a table +Fallback + +```sql +CREATE TABLE t (id INT, name STRING) USING delta; +``` + +A table created without `AS SELECT` is a metadata-only commit. There is nothing to offload. From code. + +Offload + +```sql +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +CREATE TABLE t2 USING delta AS SELECT * FROM t; +``` + +With `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` on Spark 3.5 to 4.1, Velox writes the data files of `CREATE TABLE ... AS SELECT` and `REPLACE TABLE ... AS SELECT` when the target catalog is a DeltaCatalog. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). + +## Writing +PartialOffload + +```sql +INSERT INTO t SELECT id, name FROM source WHERE length(name) = 1; +``` + +Delta's writer writes the files of `INSERT`, with or without native write. The `SELECT` can run in Velox. From code. + +Fallback + +```sql +INSERT INTO t VALUES (1, 'a'), (2, 'b'), (3, 'c'); +``` + +With the default `spark.gluten.sql.columnar.localTableScan=false`, the `VALUES` scan runs on Spark. Delta's writer writes the files. From code. + +Offload + +```sql +SET spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true; +UPDATE t SET name = 'b' WHERE id = 2; +``` + +On Spark 3.5 to 4.1, set `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite=true` to have Velox write the data files of the commands marked Offload in [Write commands](#write-commands). Velox writes the files of this UPDATE. Seen on 4.1. On 3.5 and 4.0 this is from code. On Spark 4.x, also apply [ANSI mode on Spark 4.x](#ansi-mode-on-spark-4x). + +Native write does not check the codec, the path scheme or the partition column types before it runs. Unsupported values make the task fail or write wrong files. Read [Write problems](#write-problems) before turning it on. + +### Checking the plan + +A command that Gluten wraps for native write shows a `GlutenDelta` node, for example `Execute GlutenDelta UpdateCommand` or `GlutenDelta AtomicCreateTableAsSelect`. Without the prefix, Delta's writer wrote the files. + +```scala +val df = spark.sql("UPDATE delta.`/data/t` SET name = 'bb' WHERE id = 2") +df.queryExecution.executedPlan.treeString.contains("GlutenDelta ") +``` + +The node also shows when Velox writes no data files. This happens for a DELETE that writes only deletion vectors or only metadata, and for a Spark 4.x write whose schema Velox cannot write, such as one with a VARIANT column. From code. For `DataFrameWriter` calls, read the command's plan from the Spark UI SQL tab or a `QueryExecutionListener`. Each write also reports a nested `deltaTransactionalWrite` execution for the query that feeds it. That plan never has the prefix, whichever writer ran. From code. + +### Write commands + +The Status column assumes Spark 3.5 to 4.1 with native write on. Otherwise every row is at best PartialOffload. In PartialOffload rows, Delta writes the files and the query that feeds the write can run in Velox. + +Gluten's CI checks that a partitioned `save(path)` on 3.5 to 4.1 and OPTIMIZE on 3.5 run as Gluten native write commands. It also checks on 3.5 and 4.0 that a DELETE that rewrites a file runs in Gluten's transaction and that its scan runs in Velox. On 3.5 and 4.0, the copied Delta Lake suites also run UPDATE, DELETE, `save(path)` and `CREATE TABLE ... AS SELECT` with native write on. They check results only. On 4.1, the nightly job shows every Offload row except DELETE going through Velox's writer. The other Offload cases are from code. + +| SQL | DataFrame API | Status | Notes | +|---|---|---|---| +| | `df.write.format("delta").save(path)` | Offload | Every save mode, with or without `partitionBy`. | +| `CREATE TABLE ... AS SELECT` | `writeTo(t).create()`, `saveAsTable(t)` when `t` does not exist and the mode is not overwrite | Offload | Needs a DeltaCatalog target. | +| `REPLACE TABLE ... AS SELECT`, `CREATE OR REPLACE TABLE ... AS SELECT` | `writeTo(t).replace()`, `writeTo(t).createOrReplace()`, `saveAsTable(t)` in overwrite mode | Offload | Needs a DeltaCatalog target. | +| `UPDATE` | `DeltaTable.update()` | Offload | On a table with deletion vectors, Velox writes only the updated rows. | +| `DELETE` | `DeltaTable.delete()` | Offload, PartialOffload when it writes deletion vectors (from code on 4.1), Fallback when it is metadata-only (from code) | Velox writes the rewritten files. A deletion-vector DELETE and a metadata-only DELETE write no data files. | +| `OPTIMIZE`, `OPTIMIZE ... WHERE` | `DeltaTable.optimize().executeCompaction()`, with or without `where(...)` | Offload | Bin-packing, by path or by name. | +| `OPTIMIZE ... ZORDER BY`, `OPTIMIZE ... FULL`, `OPTIMIZE` on a liquid-clustered table | | PartialOffload | From code. | +| `REORG TABLE ... APPLY (PURGE)`, `REORG TABLE ... APPLY (UPGRADE UNIFORM ...)` | | PartialOffload | From code. | +| `INSERT INTO ... SELECT`, `INSERT OVERWRITE`, `INSERT INTO ... REPLACE WHERE` | | PartialOffload | From code. To append natively, use `save(path)` in append mode. | +| | `insertInto(t)`, `saveAsTable(t)` in append mode on an existing table, `writeTo(t).append()`, `overwrite(condition)`, `overwritePartitions()` | PartialOffload | From code. | +| `MERGE INTO` | | PartialOffload | Scans, the DML target scan and joins can run in Velox. On Spark 3.5 and 4.0, Gluten's tests check that a deletion-vector MERGE runs a scan of the target table in Velox. The other cases are from code. Native MERGE write is in progress in [#13324](https://github.com/apache/gluten/pull/13324). | +| | `writeStream.format("delta")` | PartialOffload | Inside `foreachBatch`, batch writes follow the rows above. From code. | + +With native write on, `DESCRIBE HISTORY` operation metrics of `save(path)`, UPDATE, DELETE and `CREATE TABLE ... AS SELECT` match Delta Lake in `DescribeDeltaHistorySuite`. Seen on 4.1. Three unpartitioned tests in `DeleteMetricsSuite` and `UpdateMetricsSuite` expect two added files and get one. The job sets `spark.default.parallelism=1`, which plans the five small input files as one scan partition (from code). + +### Write features + +| Feature | Status | Notes | +|---|---|---| +| Optimized write | Offload | In the commands marked Offload, the rebalancing shuffle runs in Velox. If it fails validation, Delta's shuffle is used. Both are from code. `OPTIMIZE` never uses it. | +| Auto compaction | Offload after native writes, PartialOffload after other writes | Compacts natively written tables (seen on 4.1). The auto compaction tests that fail in the nightly job fail because its `spark.default.parallelism=1` makes their first write produce one file (from code). After a native write, the compaction runs in a Gluten transaction (from code). After INSERT, MERGE or streaming writes, Delta's writer compacts (from code). | + +## Reading +### Read data +Offload/Fallback + +| Table type | Spark 3.4 / Delta Lake 2.4 | Spark 3.5 to 4.1 | +|---|---|---| +| Unpartitioned | Offload | Offload | +| Partitioned | Offload | Offload | +| Column mapping, name mode | Offload | Offload | +| Column mapping, id mode | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | Offload, matched by physical name. Known ResultMismatch cases: see [Read problems](#read-problems). | +| Files with deletion vectors | Fallback | Offload | +| Change data feed, no deletion vectors in range | Offload. PartialOffload when the table allows deletion vectors and the range has `_change_data` files, because those scans then fall back (from code). | Offload | +| Change data feed, deletion vectors in range | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors and no `_change_data` files (from code) | Fallback, PartialOffload when a file scan inside Delta's change data feed scan reads no files with deletion vectors (from code) | + +```sql +SELECT count(1) AS count, name +FROM t +GROUP BY name; +``` + +Reads need no Gluten option. With `spark.sql.adaptive.enabled=false`, `EXPLAIN` shows the native scan as `FileDeltaScanTransformer` and `EXPLAIN FORMATTED` shows `DeltaScanTransformer`. With AQE on, the default, `EXPLAIN` and `EXPLAIN FORMATTED` of a query with a shuffle, such as this one, print the plan from before Gluten offloads it. The Spark UI plan graph of a finished query shows `DeltaScanTransformer`. Delta prunes partitions and skips files by statistics while it plans the query, before the scan runs. + +| Read | Status | Notes | +|---|---|---| +| By table name, ``delta.`/path` `` or `spark.read.format("delta").load(path)` | Offload | Same native scan. | +| `VERSION AS OF` | Offload | Tested on Spark 3.4 to 4.1. | +| `TIMESTAMP AS OF`, `versionAsOf`, `timestampAsOf` | Offload | From code. Delta Lake's `DeltaTimeTravelSuite` passes (seen on 4.1). | +| Streaming source | Offload | Each micro-batch is a Delta file scan. From code. Delta Lake's `DeltaSourceSuite` passes (seen on 4.1). | +| Dynamic partition pruning | Not tested | | + +### Deletion vectors +Offload + +```sql +OPTIMIZE t; +ALTER TABLE t SET TBLPROPERTIES ('delta.enableDeletionVectors' = true); +DELETE FROM t WHERE id IN (3, 4); +SELECT * FROM t; +``` + +On Spark 3.5 to 4.1, Velox drops the deleted rows in the scan. This needs Delta's `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default. The DML target scan, which finds the rows for new deletion vectors, also runs in Velox. Gluten tests check the scan on Spark 3.5 to 4.1 and the DML target scan on Spark 3.5 and 4.0. Delta writes the deletion-vector files. + +Fallback + +```sql +SET spark.databricks.delta.deletionVectors.useMetadataRowIndex=false; +SELECT * FROM t; +RESET spark.databricks.delta.deletionVectors.useMetadataRowIndex; +``` + +| Case | Behavior | +|---|---| +| Spark 3.5 to 4.1 with `useMetadataRowIndex=false` | A scan that reads Delta's generated `__delta_internal_is_row_deleted` or `__delta_internal_row_index` column falls back with `fallback Delta scan requiring generated DV metadata`. Delta adds the first column when any file in the scanned table version, micro-batch, DML file set or `OPTIMIZE` bin has a deletion vector, so `OPTIMIZE` and `REORG TABLE ... APPLY (PURGE)` scans of those files also fall back (from code). It adds the second to DML target scans that write deletion vectors. Other scans offload. Tested on Spark 3.5 and 4.0. | +| Spark 3.4, normal reads | A read falls back with `fallback Spark 3.4 Delta DV scan` when a file it selected has a deletion vector. Tested on Spark 3.4. A read whose selected files have none offloads (from code). | +| Spark 3.4, other scans, such as DML target scans, `OPTIMIZE` scans, streaming micro-batches and change data feed scans of `_change_data` files | Fall back whenever the table protocol allows deletion vectors. Change data feed scans of added or removed files fall back only when a file they read has a deletion vector. From code. | +| `spark.databricks.delta.stats.skipping=false` | Delta Lake itself rejects reads of a table with deletion vectors with `Cannot work with a non-pinned table snapshot of the TahoeFileIndex` (seen on 4.1). Before Delta Lake 4.1.0, the table feature alone triggers it (from code). | + +The fallback reasons show in [Gluten UI](GlutenUI.md). + +### Change data feed +Offload + +```sql +CREATE TABLE c (id INT, name STRING) USING delta + TBLPROPERTIES ('delta.enableChangeDataFeed' = true); +INSERT INTO c VALUES (1, 'a'), (2, 'b'); +UPDATE c SET name = 'bb' WHERE id = 2; +SELECT id, name, _change_type, _commit_version +FROM table_changes('c', 1) +WHERE id = 2; +``` + +Gluten expands a batch change data feed read into Delta file scans that run in Velox with the filters pushed down. Delta also adds an empty placeholder scan to the plan. That scan falls back when the table schema has a MAP at any nesting level (from code). If the expansion fails, Gluten logs a warning and the read uses Delta's change data feed scan. From code. + +PartialOffload + +```sql +SET spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false; +SELECT * FROM table_changes('c', 1); +``` + +A read whose version range adds or removes a file with a deletion vector uses Delta's change data feed scan. Insert-only ranges of a table with deletion vectors enabled still run in Velox. With `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan=false`, every batch change data feed read uses Delta's scan. Delta's scan reads the data files in a query of its own that Gluten also plans, and `EXPLAIN` does not show it (from code). That query's file scans can run in Velox, except scans of added or removed files that have deletion vectors and, on Spark 3.4, scans of `_change_data` files when the table allows deletion vectors (from code). Streaming change data feed reads ignore this option. They run as Delta file scans. A scan whose files have deletion vectors falls back, and on Spark 3.4 so does a scan of `_change_data` files when the table allows deletion vectors (from code). + +### Read metadata +Offload + +```sql +SELECT id, _metadata.file_path, _metadata.row_index +FROM t +WHERE _metadata.row_index = 2; +``` + +The native scan fills the `_metadata` file fields and `input_file_name()` (from code), and `_metadata.row_index`. On Spark 3.5 to 4.1, Delta Lake exposes `_metadata.row_index` only on tables that allow deletion vectors or have row tracking, so the example needs `t` from [Deletion vectors](#deletion-vectors) (from code). A query that selects only `_metadata.file_name` can fail when the table has a BIGINT data column named `file_name` (seen on 4.1, see [Read problems](#read-problems)). Gluten tests check `_metadata.row_index` in the native scan on Spark 3.5 and 4.0, and Delta Lake tests that read it pass (seen on 4.1). Spark 3.4 is not tested. On row-tracking tables, `_metadata.row_id` and `_metadata.row_commit_version` are read natively (from code). In the nightly job, they return correct values except in the cases in [Read problems](#read-problems) (seen on 4.1). + +Fallback + +```sql +DESCRIBE HISTORY t; +``` + +`DESCRIBE HISTORY` reads each commit file on Spark, through Delta's LogStore in a `mapPartitions` job. When Delta replays the log for a snapshot, every scan under `_delta_log`, checkpoints included, stays on Spark with the reason `fallback Delta _delta_log scan`. From code. + +## Table features +Offload/Fallback + +The Read column covers Spark 3.5 to 4.1. It covers Spark 3.4 only where the row names it. The Native write column applies to Spark 3.5 to 4.1 with native write on. + +| Feature | Read | Native write | Notes | +|---|---|---|---| +| Deletion vectors | Offload, Fallback on Spark 3.4 | PartialOffload | Gluten's tests check deletion vector DELETE and MERGE results with native write on, on Spark 3.5 and 4.0. Which writer writes each file is from code. See [Deletion vectors](#deletion-vectors). | +| Change data feed | Offload, also on Spark 3.4. Ranges with deletion vectors fall back. On Spark 3.4, scans of `_change_data` files also fall back when the table allows deletion vectors (from code). | Offload | UPDATE and DELETE write the `_change_data` files with Velox (from code). Change data feed reads of their changes are correct (seen on 4.1). See [Change data feed](#change-data-feed). | +| Column mapping, name mode | Offload, also on Spark 3.4 | ResultMismatch | Files have no Parquet field IDs. Seen on 4.1. See [Write problems](#write-problems). | +| Column mapping, id mode | Offload, also on Spark 3.4. ResultMismatch when field IDs differ from physical names (seen on 4.1). | ResultMismatch, Exception when Spark's reader reads the files (seen on 4.1) | Gluten matches columns by physical name. See [Known problems](#known-problems). | +| Identity columns | Offload | Offload | Seen on 4.1. High-water marks are correct. The projection that generates identity values runs on Spark (from code). | +| Generated columns | Offload | Offload | Seen on 4.1. No Gluten test covers generated columns. Velox computes the generated values when it supports the expression (from code). Partition filters derived from them reach the native scan. `DATE_FORMAT` results for years above 9999 differ (see [Write problems](#write-problems)). | +| CHECK constraints, NOT NULL | Offload | Offload | Delta checks the rows on the JVM before Velox writes them. Gluten's tests cover NOT NULL on Spark 3.5 and 4.0. Seen on 4.1. | +| Row tracking | Offload, Exception for `_metadata.base_row_id` filters, ResultMismatch for `_metadata.row_id` when older files lack base row IDs | Offload | Seen on 4.1. See [Read problems](#read-problems). | +| Liquid clustering | Offload | PartialOffload | Reads use the same native scan as other tables. Native commands write to clustered tables. `OPTIMIZE`, which clusters the data, uses Delta's writer. All three are from code. | +| UniForm (IcebergCompatV1, V2) | | ResultMismatch, Exception when partitioned | The write status is from code. No test reads a UniForm table. | +| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ | Offload, Exception for FLOAT or DOUBLE partition columns, for TIMESTAMP_NTZ partition values with sub-millisecond digits and for some writes of TIMESTAMP_NTZ data columns | Seen on 4.1. Spark 3.5 and 4.0 are not covered. | +| TIMESTAMP_NTZ | PartialOffload, also on Spark 3.4 | Offload, Exception for some writes (seen on 4.1) | See [DataType](#datatype). | +| Variant, including shredding | Fallback | Fallback | Spark 4.x. From code. See [DataType](#datatype). | +| Collations | Fallback | | Spark 4.x. From code. See [DataType](#datatype). | +| materializePartitionColumns | | ResultMismatch | Spark 4.1. From code. | + +## DataType + +The Read column applies to Spark 3.4 to 4.1. On all of them, Gluten's tests read INT, BIGINT, DECIMAL(12,2), STRING, DATE, TIMESTAMP, TIMESTAMP_NTZ, STRUCT, ARRAY and MAP columns natively. The other read statuses are from code, unless a row says otherwise. The native write columns apply to Spark 3.5 to 4.1 with native write on. On Spark 3.5 and 4.0, Gluten's tests write BOOLEAN, INT, BIGINT, STRING, STRUCT and ARRAY columns natively. The other native write statuses are seen on 4.1, unless a row says they are from code. + +| Type | Read | Native write | Native write as partition column | Notes | |---|---|---|---|---| -| `spark-3.5` | Spark 3.5.x | 2.12 | 3.3.x | Supported | -| `spark-4.0` | Spark 4.0.x | 2.13 | 4.0.x | Supported | - -Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between -the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 + -Delta 4.0), not a native-write capability gap. - -## Build and runtime notes - -Build Gluten with Delta support by enabling `-Pdelta` together with the Velox backend profile and a Spark profile. - -- Spark 3.5 build example: - - `mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests` -- Spark 4.0 build example: - - `mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests` - -Native Delta write is controlled by: - -- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite` - - Default: `false` - - Type: experimental - -Native change data feed scan offload is controlled by: - -- `spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan` - - Default: `true` - - Type: experimental - -The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by: - -- `spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan` - - Default: `true` - - Type: experimental - - When disabled, the DML target scan that produces file paths and row indexes for - deletion-vector writes stays on Spark; other scans are unaffected. - -| Feature | Delta minWriterVersion | Delta minReaderVersion | Iceberg format-version | Feature type | Supported by Gluten (Velox) | -|---|---:|---:|---:|---|---| -| Basic functionality | 2 | 1 | 1 | Writer | Yes | -| CHECK constraints | 3 | 1 | N/A | Writer | No | -| Change data feed | 4 | 1 | N/A | Writer | Yes | -| Generated columns | 4 | 1 | N/A | Writer | Partial | -| Column mapping | 5 | 2 | N/A | Reader and writer | Yes | -| Identity columns | 6 | 1 | N/A | Writer | Yes | -| Row tracking | 7 | 1 | 3 | Writer | Partial | -| Deletion vectors | 7 | 3 | 3 | Reader and writer | Partial | -| TimestampNTZ | 7 | 3 | 1 | Reader and writer | No | -| Liquid clustering | 7 | 3 | 1 | Reader and writer | Yes | -| Iceberg readers (UniForm) | 7 | 2 | N/A | Writer | Not tested | -| Type widening | 7 | 3 | N/A | Reader and writer | Partial | -| Variant | 7 | 3 | 3 | Reader and writer | Not tested | -| Variant shredding | 7 | 3 | 3 | Reader and writer | Not tested | -| Collations | 7 | 3 | N/A | Reader and writer | Not tested | -| Protected checkpoints | 7 | 1 | N/A | Writer | Not tested | +| BOOLEAN | Offload | Offload | Offload | | +| TINYINT, SMALLINT, INT, BIGINT | Offload | Offload | Offload | | +| FLOAT, DOUBLE | Offload | Offload | Exception | See [Write problems](#write-problems). | +| DECIMAL | Offload | Offload | Offload | Precision above 18 included. | +| STRING | Offload | Offload | Offload | | +| CHAR, VARCHAR | Offload | Offload | | Stored as STRING. Delta checks the length on the JVM before the native write. Both statuses are from code. Reads return correct results (seen on 4.1). | +| BINARY | Offload | Offload | Not tested | The read status is from code. One Delta Lake test writes a table with only a BINARY column, checks its file statistics and runs a DELETE that filters on that column (seen on 4.1). | +| DATE | Offload | Offload | Offload | | +| TIMESTAMP | Offload, ResultMismatch for sub-millisecond partition values (seen on 4.1) | Offload | Offload, Exception for sub-millisecond values | The partition column status is from code. See [Known problems](#known-problems). | +| TIMESTAMP_NTZ | PartialOffload | Offload, Exception for some writes (seen on 4.1) | Offload, Exception for sub-millisecond values (seen on 4.1) | Filters, sorts, joins and other operators whose input or output includes a TIMESTAMP_NTZ column fall back, even when they do not use that column. Scans, aggregates, shuffles and some projections can run in Velox. A scan whose pushed filter has a TIMESTAMP_NTZ literal falls back (from code). Gluten's tests check on Spark 3.4 to 4.1 that a scan of the column runs in Velox, and that a filter on the column and a sort on another column cause a fallback. The other operator rules are from code. | +| STRUCT, ARRAY, MAP | Offload | Offload | | | +| VARIANT | Fallback | Fallback | | Spark 4.x. Shredded variant reads fall back too. Both statuses are from code. | +| Collated STRING | Fallback | | | Spark 4.x. Scans that read the column fall back. Delta Lake 4.0.1 and 4.1.0 reject a collated column in a table schema with `DELTA_UNSUPPORTED_DATA_TYPES`. They cannot write to tables that have the `collations` feature. From code. | + +A plan node falls back when its output contains a type Velox does not support, such as VARIANT. With the default `spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames=true`, a scan that does not select such a column can still offload. From code. Delta Lake does not allow interval columns. + +## Format +PartialOffload + +Delta data files are Parquet, and Gluten reads them with Velox's Parquet reader. Velox writes them only for native write commands on Spark 3.5 to 4.1. Otherwise Delta's Parquet writer writes them. + +### Natively written files + +| Property | Behavior | +|---|---| +| Compression | snappy (default), gzip, zstd (always level 3), lz4 (Hadoop framing), none. From code. Other codecs: see [Write problems](#write-problems). | +| Row groups | Sized by `spark.gluten.sql.columnar.parquet.write.blockSize` and `spark.gluten.sql.native.parquet.write.blockRows`. `parquet.*` settings are ignored. From code. | +| TIMESTAMP | INT64 microseconds. `spark.sql.parquet.outputTimestampType` is ignored. Delta Lake's writer uses INT96 by default. Spark reads both. From code. | +| Parquet field IDs | Not written, for column-mapped and UniForm tables too. Seen on 4.1 for column mapping, from code for UniForm. See [Write problems](#write-problems). | +| Spark schema footer entry | `org.apache.spark.sql.parquet.row.metadata` is not written ([#13288](https://github.com/apache/gluten/issues/13288)). Seen on 4.1. | +| Nested field names | Lowercased below the top level when `spark.sql.caseSensitive=false`. Spark and Gluten match them case-insensitively. A case-sensitive reader can miss them. From code. | +| Page index | Always written. From code. | +| `spark.sql.parquet.writeLegacyFormat=true` | Applies to DECIMAL only. Arrays and maps keep the standard layout. From code. | +| File statistics | numRecords, min, max and nullCount are computed in Velox (checked on Spark 3.5). Gluten's tests check the values against the rows in each file on Spark 3.5 to 4.1. Most tests in Delta's data-skipping suites pass (seen on 4.1). Of the 34 that fail, 22 check statistics and expect specific rows in each file after `repartition(2)`. The other 12 hit the `_metadata.file_name` failure in [Read problems](#read-problems). | +| Data file paths | Come from Delta's committer, so names and random prefixes follow Delta. File names seen on 4.1, random prefixes from code. | + +## SQL + +| Statement | Status | Notes | +|---|---|---| +| `SELECT` | Offload | See [Reading](#reading). | +| `table_changes()` | Offload | Ranges with deletion vectors fall back. See [Change data feed](#change-data-feed). | +| `CREATE TABLE ... AS SELECT`, `REPLACE TABLE ... AS SELECT` | Offload | With native write on, Spark 3.5 to 4.1, and a DeltaCatalog target. Seen on 4.1, from code on 3.5 and 4.0. See [Creating a table](#creating-a-table). | +| `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `OPTIMIZE`, `REORG` | | See [Write commands](#write-commands). | +| `CREATE TABLE` without `AS SELECT`, `DROP TABLE`, `DESCRIBE` | Fallback | They read and write no data files. From code. | +| `ALTER TABLE` | Fallback, PartialOffload when it reads data | Most forms read and write no data files. `ADD CONSTRAINT` scans the table, and that scan can run in Velox. On Spark 3.5 to 4.1, `ALTER COLUMN ... SYNC IDENTITY` also scans it, and `DROP FEATURE` and `SET TBLPROPERTIES ('delta.columnMapping.mode' = 'none')` can rewrite data files with Delta's writer. From code. | +| `VACUUM`, `RESTORE`, `SHALLOW CLONE` | | They write no Parquet data files. From code. | +| `CONVERT TO DELTA` | PartialOffload | Writes no Parquet data files. Unless the statement has `NO STATISTICS` or `spark.databricks.delta.stats.collect=false` is set, it scans the Parquet files for statistics. That scan can run in Velox. From code. | + +## Schema evolution +PartialOffload + +The `mergeSchema` and `overwriteSchema` rows need native write on Spark 3.5 to 4.1. Otherwise Delta writes the files. + +| Change | Status | Notes | +|---|---|---| +| `mergeSchema` with `save(path)` | Offload | Velox writes the evolved schema (from code). On Spark 3.5 and 4.0, the copied `DeltaSuite` checks the results of a `mergeSchema` overwrite. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). | +| `overwriteSchema` with `save(path)` | Offload | Delta applies the change before the write. From code. Delta Lake's `SchemaEnforcementWithPathSuite` passes (seen on 4.1). | +| Schema evolution in SQL `INSERT` | PartialOffload | From code. | +| Schema evolution in `MERGE` | PartialOffload, Exception for some STRUCT map key changes | The merged table contents match Delta Lake. Some STRUCT map key changes fail in Velox (see [MERGE, UPDATE and DELETE problems](#merge-update-and-delete-problems)). The two TIMESTAMP_NTZ widening tests fail before the MERGE runs, in a setup write of TIMESTAMP_NTZ values. Seen on 4.1. | +| `RENAME COLUMN` with column mapping | Offload | Reads after top-level and nested renames return correct results. After a top-level rename, partition pruning and file skipping still work. | +| `DROP COLUMN` with column mapping | Offload | From code. Delta Lake's `DeltaDropColumnSuite` passes (seen on 4.1). | +| Type widening | Offload, Exception for DATE to TIMESTAMP_NTZ and for native writes to FLOAT or DOUBLE partition columns | Seen on 4.1. See [Table features](#table-features). | +| `spark.sql.parquet.mergeSchema=true` | Fallback | Every Parquet scan falls back, Delta scans included. From code. | + +## Known problems + +### Read problems + +| Problem | Status | Versions | Issue | +|---|---|---|---| +| A column default stored as `EXISTS_DEFAULT` metadata, for example after `CONVERT TO DELTA`, reads as NULL for rows in files without that column. | ResultMismatch | Seen on 4.1. #11494 reports the same bug for plain Parquet on 3.4 to 4.1. | [#11494](https://github.com/apache/gluten/issues/11494) | +| In column mapping id mode, when field IDs no longer match the Parquet column names, Gluten returns the column with the matching physical name. Delta Lake returns NULL when no file column has that ID (seen on 4.1). It returns the column the ID points to when another column has it (from code). | ResultMismatch | 3.4 to 4.1. Seen on 4.1 for the NULL case, from code otherwise. | [#12241](https://github.com/apache/gluten/issues/12241) | +| On a TIMESTAMP-partitioned table with six partition values from .1 to .123456 seconds, `count(DISTINCT ts)` returns 3, as if the values were truncated to milliseconds. Equality filters on the column still return the right rows. | ResultMismatch | Seen on 4.1 | | +| Reading a data column widened from DATE to TIMESTAMP_NTZ fails on files written before the change with `Converted type DATE is not allowed for requested type TIMESTAMP UTC`. A partition column widened the same way reads correctly. | Exception | Seen on 4.1 | | +| A query that selects only `_metadata.file_name` from a table with a BIGINT data column named `file_name` reads the data column and fails with `Converted type BIGINT is not allowed for requested type VARCHAR`. | Exception | Seen on 4.1 | | +| An equality filter on `_metadata.base_row_id` fails with `Synthesized column 'base_row_id' failed filter validation` when a scanned file has a different base row ID. | Exception | Seen on 4.1 | | +| On a row-tracking table whose older files lack base row IDs, selecting `_metadata.row_id` returns rows. Delta raises an error. Delta Lake's test creates this state with a manual metadata commit. `ALTER TABLE ... SET TBLPROPERTIES` backfills base row IDs when it enables row tracking. | ResultMismatch | Seen on 4.1 | | +| Reading or deleting from a 2-billion-row table with deletion vectors can grow native memory until the JVM is killed. Delta Lake's two 2-billion-row tests reproduce it. The nightly job force-fails both tests. | Exception | Reported on 4.1 | [#12387](https://github.com/apache/gluten/issues/12387) | +| With `spark.databricks.delta.deletionVectors.useMetadataRowIndex=true`, the default, selecting `__delta_internal_is_row_deleted` from a relation built by hand over `DeltaParquetFileFormat` drops the deleted rows and returns 1 for the rows it keeps. The column is not in the table schema, so SQL and DataFrame queries over the table cannot select it. | ResultMismatch | Seen on 4.1 | | +| Scans of a table with many small files can be slower than vanilla Spark on EBS storage. In the report, two of five TPC-DS SF1000 queries take 2.2 and 3.4 times as long as on vanilla Spark, with 182,300 files in one table. A maintainer did not see the slowdown on a host with local NVMe storage. | | Reported on 3.4 | [#10778](https://github.com/apache/gluten/issues/10778) | + +### Write problems + +These need native write on, unless a row says otherwise. + +| Problem | Status | Versions | Issue | +|---|---|---|---| +| A write partitioned by a FLOAT or DOUBLE column fails with `Unsupported partition type`. | Exception | 3.5 to 4.1, seen on 4.1 | [#13138](https://github.com/apache/gluten/issues/13138) | +| A write to a path scheme the native library cannot write fails at task time. This covers custom Hadoop file systems, and S3, GCS, HDFS or ABFS in a build without that connector. | Exception | 3.5 to 4.1, seen on 4.1 (custom file system), from code (missing connector) | [#4963](https://github.com/apache/gluten/issues/4963) | +| Files written for a column mapping table, in name or id mode, have no Parquet field IDs, which the Delta Lake protocol requires. Delta Lake and Gluten read name-mode files by physical name. Readers that match by field ID cannot. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | | +| Spark's own reader fails on natively written id-mode files with `Spark read schema expects field Ids, but Parquet file schema doesn't contain any field Ids`. Spark uses this reader when a scan falls back or Gluten is off. With `spark.sql.parquet.fieldId.read.ignoreMissing=true`, Spark skips this check and returns NULL for the data columns of these files (from code). That setting is not tested with these files. | Exception | 3.5 to 4.1, seen on 4.1 | | +| Files written for a UniForm table have no Parquet field IDs, so the Iceberg metadata points at files without IDs. No error is raised. | ResultMismatch | 3.5 to 4.1, from code | [#12957](https://github.com/apache/gluten/pull/12957) | +| A write to a partitioned UniForm table fails with `The file schema type should be equal with the input rowvector type`. | Exception | 3.5 to 4.1, from code | | +| Writes to a table with the `materializePartitionColumns` feature leave the partition columns out of the data files. | ResultMismatch | 4.1, from code | | +| Some writes of a TIMESTAMP_NTZ data column fail with `The file schema type should be equal with the input rowvector type`. Delta Lake's type widening tests fail this way when they write `CAST(value AS TIMESTAMP_NTZ)`. Delta Lake's data skipping tests also write that cast and pass. On Spark 4.x, `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=true` sends TIMESTAMP_NTZ writes to Delta's writer. It is not tested as a workaround. | Exception | Seen on 4.1 | | +| A partitioned write fails in Velox's `PartitionIdGenerator` when a TIMESTAMP or TIMESTAMP_NTZ partition value has sub-millisecond digits. | Exception | 3.5 to 4.1, seen on 4.1 (TIMESTAMP_NTZ), from code (TIMESTAMP) | | +| A generated partition column such as `DATE_FORMAT(ts, 'yyyy-MM')` stores `12345-07` for the year 12345. Spark stores `+12345-07`. Not checked with native write off. | ResultMismatch | Seen on 4.1 | | +| The `compression` option set to brotli or lzo fails the write. So does `spark.sql.parquet.compression.codec` when the option is unset. The write fails even with the Hadoop codec class on the classpath. Without that class, Delta's writer fails too, and Spark does not ship it. | Exception | 3.5 to 4.1, from code | | +| The `compression` option or `spark.sql.parquet.compression.codec` set to lz4_raw writes Snappy data into a `.lz4raw.parquet` file. | ResultMismatch | 3.5 to 4.1, from code | | +| `maxRecordsPerFile` on an unpartitioned write counts rows but writes whole batches. When the limit is smaller than a batch, the whole batch goes into one file. The other files opened for that batch are 0-byte files that are not valid Parquet. | ResultMismatch | 3.5 to 4.1, from code | | +| `spark.sql.maxConcurrentOutputFileWriters` above 0 makes a partitioned write with unsorted input fail with `Underlying columnar data is inaccessible from BatchCarrierRow`. | Exception | 3.5 to 4.1, from code | | +| An unpartitioned native write whose first task gets no rows, for example a write of an empty DataFrame, leaves a 0-byte file that is not valid Parquet. Reading the table directory as plain Parquet can then fail with `is not a Parquet file (length is too low: 0)`. With statistics on, the file stays on disk until `VACUUM`. With `spark.databricks.delta.stats.collect=false`, Delta commits it. The last two sentences are from code. | ResultMismatch | 3.5 to 4.1, seen on 4.1 | | +| A statistics aggregation that Velox cannot offload fails the write with `ClassCastException`. With the default `spark.gluten.sql.columnar.backend.velox.enableTimestampNtzValidation=false`, the TIMESTAMP_NTZ case from the issue no longer fails on main. Tested on Spark 3.5, seen on 4.1, from code on 4.0. On Spark 3.5 with that option set to true, the TIMESTAMP_NTZ case still fails (from code). | Exception | 3.5 to 4.1, from code | [#12538](https://github.com/apache/gluten/issues/12538) | + +### MERGE, UPDATE and DELETE problems + +MERGE does not use native write, so the MERGE rows also apply with native write off (from code). Review Comment: this statement seems conflicting with the previous write support part -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
