yyanyy opened a new pull request, #57178:
URL: https://github.com/apache/spark/pull/57178
### What changes were proposed in this pull request?
Add Catalyst tree-pattern identity bits for the DSv2 relation leaves and the
row-level command nodes, and use them to prune the optimizer/analyzer rules
that match those nodes.
- Add `DATA_SOURCE_V2_RELATION` and `DATA_SOURCE_V2_SCAN_RELATION` to
`TreePattern`, and attach them to `DataSourceV2Relation` and
`DataSourceV2ScanRelation` respectively.
- Add `DELETE_FROM_TABLE`, `UPDATE_TABLE`, `MERGE_INTO_TABLE`, `REPLACE_DATA`
and `WRITE_DELTA`. Since every command already carries the shared `COMMAND`
bit (declared `final` on the `Command` trait), `Command` now exposes a
`nodePatternsInternal()` hook, mirroring how `PlanExpression` combines
`PLAN_EXPRESSION` with a per-subquery bit, so a command node contributes
its
own identity bit while `COMMAND` stays guaranteed-present.
`DeleteFromTable`,
`UpdateTable`, `MergeIntoTable`, `ReplaceData` and `WriteDelta` override
it.
- Guard the previously-unguarded rules that key off these nodes with
`containsPattern` / `containsAnyPattern`:
`RewriteDeleteFromTable`, `RewriteUpdateTable`, `RewriteMergeIntoTable`,
`V2ScanRelationPushDown`, `V2ScanPartitioningAndOrdering`,
`GroupBasedRowLevelOperationScanPlanning`,
`OptimizeMetadataOnlyDeleteFromTable`
and `RowLevelOperationRuntimeGroupFiltering`.
- Add tree-pattern contract tests.
### Why are the changes needed?
A tree-pattern bit is an O(1) bitset lookup on a plan's cached
`treePatternBits`.
Rules that today run an unconditional `transform`/`resolveOperators` walk
over
every node, on every invocation, even for plans that contain none of the
target
nodes, can instead cheaply skip whole subtrees that provably don't contain
the
node they rewrite. The DSv2 scan/relation leaves and the row-level command
nodes
previously had no identity pattern (the command nodes carried only the coarse
`COMMAND` bit shared by all commands), so rules matching them either ran
unguarded or could only prune on the broad `COMMAND` bit. This adds the
missing
per-node bits and applies them as pruning guards, matching the existing idiom
used throughout the optimizer (for example `ReplaceExceptWithFilter` on
`EXCEPT`,
`ResolveRowLevelCommandAssignments` on `COMMAND`).
### Does this PR introduce any user-facing change?
No. This only adds internal Catalyst tree-pattern metadata and rule pruning
guards. Query results and plans are unchanged; the guarded rules fire on
exactly
the same nodes as before, just skipping trees that cannot contain them.
### How was this patch tested?
- New unit tests pinning the pattern contract:
`DataSourceV2RelationSuite` (asserts the relation/scan-relation identity
bits)
and a new `V2CommandTreePatternSuite` (asserts each command node carries
both
`COMMAND` and its own bit).
- Regression suites covering the guarded rules, confirming the pruning
guards do
not drop any rule firing:
- Row-level operations (Group/Delta-based Delete/Update/Merge): 347 tests,
all
passing.
- Scan pushdown / partitioning (`DataSourceV2Suite`,
`KeyGroupedPartitioningSuite`, `ProjectedOrderingAndPartitioningSuite`):
174 tests, all passing.
- V2 write analysis (`V2AppendData*`, `V2Overwrite*` ANSI/Strict suites):
388 tests, all passing, confirming the `Seq(COMMAND) ++
nodePatternsInternal()`
composition does not disturb existing `COMMAND`-pruned analysis rules.
- `./build/sbt catalyst/compile sql/compile` and `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
yes, opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
