nsivabalan opened a new pull request, #20426:
URL: https://github.com/apache/hudi/pull/20426
### Describe the issue this Pull Request addresses
Part of #20371, the subset needed for the first Spark Declarative Pipelines
(SDP) milestone in #20401: a single Hudi streaming table run repeatedly with
incremental upserts/inserts. Two `HoodieCatalog` gaps block it:
1. **Tables created through the DSv2 `Column[]` API are not Hudi tables.**
Spark 4.1's `TableCatalog.createTable(Identifier, TableInfo)`, which SDP uses,
defaults to `createTable(Identifier, Column[], ...)`. `HoodieCatalog` only
overrides the `StructType` overload, and `DelegatingCatalogExtension` forwards
the `Column[]` one straight to the session catalog. The table is registered
with `provider=hudi` but without `hoodie.properties`. This applies to any
caller of the `Column[]` overload since Spark 3.4.
2. **Re-running a pipeline fails.** SDP re-sends every table property as
`TableChange.SetProperty` on each run. `HoodieCatalog.alterTable` throws
`UnsupportedOperationException: not supported table change: ...SetProperty`
even when nothing changes, so every run after the first fails.
The rest of #20371 (`MatchError` for other `ColumnChange`s, `_hoodie_*` meta
columns in `Table.columns()`) is only hit by full refresh and materialized
views and is left for a follow-up.
### Summary and Changelog
- `HoodieCatalog` routes the `Column[]`-based `createTable` overload of Hudi
tables to its `StructType`-based `createTable`, the same conversion Spark's
`TableCatalog` default does. Non-Hudi tables still go to the session catalog.
The override lives in a new trait, `HoodieCatalogColumnsSupport`, with a Spark
3.4+ version (`src/main/scala-spark3.4plus`) and an empty Spark 3.3 version
(`src/main/scala-spark3.3`), since the `Column` API does not exist in Spark
3.3. `hudi-spark-common` adds the source directory chosen by the new
`hudi.spark.common.version.source` property (overridden by the `spark3.3`
profile).
- `HoodieCatalog.alterTable` now accepts `SetProperty` / `RemoveProperty`:
- A change that keeps the current value is a no-op, so re-sending all
properties is idempotent. For properties persisted in `hoodie.properties`, the
value is compared against the table config, so SQL aliases (`primaryKey`,
`type`), write configs (`hoodie.datasource.write.recordkey.field`) and table
configs (`hoodie.table.type`) all match.
- Changing or removing a property persisted in `hoodie.properties` (record
key, ordering fields, table type, payload class, merge mode/strategy, any
`hoodie.table.*` key) is rejected with an error naming the table config, since
the catalog would no longer match the table. Validation happens before any
change is applied.
- Other properties and the table comment are applied to the catalog table,
as for any table in the session catalog.
- New helper `HoodieOptionConfig.mapToTableConfig` maps a SQL option, write
config or table config key to the table config it is persisted as.
- Tests in `TestHoodieCatalogDDL`: re-sent properties (incl. aliases and
casing) leave the catalog unchanged; changes/removals of persisted configs are
rejected without partially applying the call; other properties and the comment
are applied; the table stays writable. The `Column[]` overload creates an
initialized, writable Hudi table and still delegates non-Hudi tables (Spark
3.4+, invoked reflectively so the suite compiles on Spark 3.3).
### Impact
- `CREATE` through the DSv2 `Column[]` / `TableInfo` API now creates a Hudi
table instead of a catalog entry without `hoodie.properties`.
- `ALTER TABLE` property changes through the DSv2 catalog API are supported
instead of always failing. Spark SQL `ALTER TABLE ... SET TBLPROPERTIES` on a
V1 Hudi table keeps its current path.
- No storage format or public API changes.
### Risk Level
low. Both paths previously failed or produced an unusable table. Changes to
`hoodie.properties`-backed configs are rejected rather than applied. Key
generator and partition fields set through their write-config keys (e.g.
`hoodie.datasource.write.partitionpath.field`) are not yet treated as persisted
configs and are applied to the catalog only; this is follow-up work under
#20371.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
Verified locally: `TestHoodieCatalogDDL` 6/6 on Spark 4.1 / Scala 2.13;
`hudi-spark-common` compiles on Spark 3.3 / Scala 2.12 with the Spark 3.3
source directory; scalastyle clean.
@danny0405, this is the #20371 subset for step 3 of the plan in #20401.
Could you take it forward from here?
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
