anew opened a new pull request, #58986:
URL: https://github.com/apache/spark/pull/58986
### What changes were proposed in this pull request?
Adds an `IGNORE NULL UPDATES` clause to the Declarative Pipelines Auto CDC
SQL syntax
(`CREATE FLOW ... AS AUTO CDC INTO ...` and `CREATE STREAMING TABLE ... FLOW
AUTO CDC ...`),
matching the partial-update support already exposed over Spark Connect
(SPARK-59446) and the
Python API (SPARK-59447). When set, null values in an incoming update are
ignored and the
existing target value is preserved.
The clause has three forms, mirroring the existing `COLUMNS` and `TRACK
HISTORY ON` clauses:
- `IGNORE NULL UPDATES` -- ignore nulls on all columns.
- `IGNORE NULL UPDATES ON (col, ...)` -- ignore nulls only on the listed
columns.
- `IGNORE NULL UPDATES ON * EXCEPT (col, ...)` -- ignore nulls on all
columns except those listed.
Details:
- Grammar (`SqlBaseParser.g4` / `SqlBaseLexer.g4`): adds a new non-reserved
keyword `UPDATES`
and an `autoCdcIgnoreNullClause` rule wired into `autoCdcParameters`.
- Catalyst: `AutoCdcParams`, `AutoCdcInto`, and
`CreateStreamingTableAutoCdc` carry
`ignoreNullUpdates: Boolean` plus the include/except column lists;
`AstBuilder` parses the
clause (rejecting a duplicate clause), and both build sites forward the
fields.
- Dataflow graph: `SqlGraphRegistrationContext.buildChangeArgs` maps the
parsed clause onto
`ChangeArgs.ignoreNullSelection`. Unlike the column and track-history
selections, "all columns"
is distinct from "absent": the bare clause becomes an empty
`ExcludeColumns` selection, while an
absent clause leaves ignore-null updates off (`None`).
Execution of ignore-null updates is handled separately (SPARK-59183) and is
out of scope here.
### Why are the changes needed?
Auto CDC ignore-null updates were reachable over Spark Connect and the
Python API but not from
the SQL syntax. This change closes that gap so the feature is available
across all Declarative
Pipelines authoring surfaces.
### Does this PR introduce _any_ user-facing change?
Yes. The Auto CDC SQL syntax gains an `IGNORE NULL UPDATES [ON ...]` clause,
and `UPDATES` becomes
a non-reserved keyword.
### How was this patch tested?
- `AutoCdcParserSuite`: new positive parse tests for all three forms under
both
`CREATE FLOW ... AS AUTO CDC INTO` and `CREATE STREAMING TABLE ... FLOW
AUTO CDC`, plus
clause-ordering, duplicate-clause rejection, missing-parentheses
rejection, and a test that the
new `UPDATES` keyword remains usable as an identifier. The previous
negative test asserting the
clause was unsupported is replaced.
- `SqlPipelineSuite`: new tests asserting the resulting
`ChangeArgs.ignoreNullSelection` for the
all-columns, include-list, and except-list forms, and that an absent
clause leaves it unset.
- `UnsupportedOperationsSuite`: updated for the new plan fields.
- Regenerated the `keywords.sql.out` golden files and updated the JDBC and
Thrift server keyword
expectations for the new `UPDATES` keyword; refreshed the ANSI compliance
keyword table.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
