malinjawi opened a new pull request, #13392:
URL: https://github.com/apache/gluten/pull/13392
## What changes are proposed in this pull request?
`REORG TABLE ... APPLY (PURGE)` and `ALTER TABLE ... DROP FEATURE
deletionVectors` scan natively today, but they write the rewritten files
through Delta's JVM writer, because `OffloadDeltaCommand` has no case for
either of them. This PR adds both cases.
**REORG PURGE.** Both Delta 3.3 and 4.4.1 plan REORG as
`ExecutedCommandExec(DeltaReorgTableCommand)`. That command builds an
`OptimizeTableCommand` with a reorg context and runs it directly, so the rule
never sees it. The `optimizeContext.reorg` exclusion in `shouldOffloadOptimize`
therefore never fired; REORG simply had no case. The command is now wrapped in
`GlutenDeltaRunnableCommand`, as plain OPTIMIZE is. The bin jobs are
`OptimizeExecutor`'s, the same ones plain OPTIMIZE uses, so `txn.writeFiles`
runs in `GlutenOptimisticTransaction`. On Spark 4.x,
`GlutenDeltaNativeWriteGate` still sends writes Velox cannot match to Delta's
writer. Some cases keep Delta's command path:
- REORG UPGRADE UNIFORM (only PURGE mode is offloaded);
- liquid-clustered tables, whose REORG reclusters the files the way OPTIMIZE
does;
- column-mapped tables. The Velox writer writes no Parquet field IDs. The
check is in the rule, so it also covers Spark 3.5, whichever of this PR and
#13383 lands first.
`REORG TABLE ... WHERE ... APPLY (PURGE)` is covered too.
**DROP FEATURE deletionVectors.** SQL DROP FEATURE plans as Spark's
`AlterTableExec`, and `DeltaCatalog.alterTable` runs the DV purge (a
`DeltaReorgTableCommand`) inside it. An `AlterTableExec` on a `DeltaCatalog`
whose only change is `DropFeature("deletionVectors")` is now wrapped in
`GlutenDeltaLeafV2CommandExec`, as CTAS and RTAS are, with the same clustering
and column-mapping checks. The rule reads `catalog`, `ident` and `changes` from
the node rather than matching its fields, because Spark 4.2 adds a fourth
constructor argument. Other features still use the plain `AlterTableExec`.
Delta 3.3 cannot drop deletion vectors, so this only has an effect on Spark 4.x.
The docs now list REORG PURGE as Offload, and the `ALTER TABLE`, liquid
clustering and catalog notes are updated to match.
## How was this patch tested?
New `org.apache.gluten.delta.DeltaNativePurgeSuite` lives in src-delta33 and
is symlinked into src-delta4, so the Velox CI's `org.apache.gluten` wildcard
runs it on Spark 3.5 to 4.2. Vanilla Delta sets up every table, so the footers
of the files a purge adds show which writer it used.
- REORG PURGE on a table whose files all have DVs is planned as a Gluten
command and removes every DV. Velox writes the new files, and the commit is
`REORG` with the expected `numDeletionVectorsRemoved`. The rows match a twin
table that vanilla Delta purged, with native write off and no Gluten command in
its plan.
- On a partitioned table, `WHERE part = 1` rewrites only that partition
natively and keeps the DVs in the other one. A full REORG then purges the rest.
- On a column-mapped table, REORG stays on Delta: it is not a Gluten
command, parquet-mr writes the files and the rows are correct. A clustered
table and REORG UPGRADE UNIFORM are planned without running and are not Gluten
commands.
- DROP FEATURE deletionVectors (Spark 4.x, skipped on Delta 3.3) is planned
as a Gluten command, purges the DVs with Velox, drops the feature from the
protocol and keeps the rows. DROP FEATURE of another feature stays a plain
`AlterTableExec`.
Main and test code compile with `./build/mvn
-Pbackends-velox,spark-<v>,delta,spark-ut,fast-build -pl
backends-velox,gluten-ut/spark<vv> -am test-compile` on Spark 3.5 and 4.1. The
suite loads the native library, so it runs in CI only.
I also ran a throwaway suite locally without the Gluten plugin, on Delta
3.3.2 and 4.4.1. It applied `OffloadDeltaCommand` to the real physical plans
and ran the setup with vanilla Delta:
- Both versions plan REORG as `Execute DeltaReorgTableCommand` and DROP
FEATURE as `AlterTableExec`.
- The rule turns them into `Execute GlutenDelta DeltaReorgTableCommand` and
`GlutenDelta AlterTable`, and returns the fallback plans unchanged.
- The vanilla behavior the suite relies on holds: DVs in every file, a purge
scoped by partition, the REORG commit metrics, and on 4.4.1 a DROP FEATURE that
runs REORG inside it.
The Delta UT gate runs Delta's own `DeltaReorgSuite`,
`DeltaFastDropFeatureSuite` and `RowTrackingCompactionSuite` with native write
on, so those now exercise this path.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM Bob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
