fhan688 opened a new pull request, #20419:
URL: https://github.com/apache/hudi/pull/20419
### Describe the issue this Pull Request addresses
This PR implements the PR6 catalog and metastore composition milestone of
#19779.
The existing Hudi Architect Flink path supports bounded append-only COW,
mutable COW, mutable MOR, and assessment-only existing-table reconciliation.
However, its executable new-table path previously retained direct SQL table
resolution and could not produce validated artifacts when users required Hudi
Catalog resolution or Hive Metastore discovery.
Catalog resolution and external discovery are separate concerns. Treating
them as one boolean can obscure:
- whether Flink resolves the table through direct SQL, DFS Catalog, or HMS
Catalog;
- whether another engine must discover the table through Hive Metastore;
- which component owns metastore synchronization;
- which component owns the underlying table data lifecycle; and
- whether generated SQL is safe for a new or existing table.
Following #20021, #20198, #20281, #20302, and #20391, this PR adds a
bounded catalog composition contract without expanding the existing Flink
table-design schemas.
### Summary and Changelog
This PR adds catalog contract schema 1 as an orthogonal composition layer
over Flink design contract schemas 1, 2, and 3. It does not introduce a design
schema 4.
The implementation supports exactly five profiles:
- `direct_sql_none`
- `direct_sql_hive_metastore`
- `hudi_dfs_catalog_none`
- `hudi_dfs_catalog_hive_metastore`
- `hudi_hms_catalog_hive_metastore`
The main changes are:
- Add explicit and independently validated fields for:
- table resolution mode;
- external discovery mode;
- metastore synchronization owner; and
- data lifecycle owner.
- Preserve the legacy no-`catalog` contract with
`safety.external_catalog=false` as `direct_sql_none`, including byte-for-byte
compatibility of the previously generated SQL.
- Generate deterministic catalog setup SQL in this order:
- runtime `SET` statements;
- optional `CREATE CATALOG`;
- optional `USE CATALOG`;
- optional `CREATE DATABASE`;
- optional `USE` database;
- Hudi `CREATE TABLE`; and
- sink-side `INSERT INTO`.
- Support Hudi DFS Catalog with:
- explicit catalog-owned lifecycle acceptance;
- the exact `${catalog.path}/${database}/${table.name}` path derivation
used by Hudi 1.2.0; and
- table DDL that omits the connector `path` option because the catalog
owns path derivation.
- Support external Hudi HMS Catalog with:
- an explicit desired table path;
- `table.external=true`;
- application-owned data lifecycle; and
- classpath-based or absolute `hive.conf.dir` configuration.
- Support sink-owned Hive Metastore discovery for direct SQL and DFS
Catalog by rendering the bounded `hive_sync.*` settings.
- Avoid duplicating the standard sink-owned Hive Sync settings for HMS
Catalog, where Hudi derives them from catalog context.
- Derive the MOR Hive synchronization strategy from declared query
requirements:
- snapshot only → `RT`;
- read optimized only → `RO`; and
- both → `ALL`.
- Preserve the pinned Hudi 1.2.0 naming defaults for MOR registration,
including `_ro`, `_rt`, and the base snapshot name for `ALL`.
- Record catalog visibility as `post_commit_eventual` without claiming
atomicity with a Hudi commit or a wall-clock visibility SLA.
- Add stable machine-readable output fields:
- `catalog_contract_schema`;
- `catalog_profile`;
- `catalog_setup_sql`;
- `effective_catalog_configuration`; and
- `catalog_deployment_checks`.
- Keep catalog operations for existing tables assessment-only:
- reconciliation schema remains version 1;
- reconciliation and change disposition remain `UNVERIFIED`;
- executable eligibility remains false; and
- no catalog setup SQL or mutation command is produced.
- Fail closed for:
- unsupported resolution or discovery combinations;
- unknown modes or owners;
- gate/profile conflicts;
- unconfirmed DFS lifecycle ownership;
- managed HMS tables;
- AWS Glue execution;
- existing-table catalog mutation;
- multi-writer operation;
- custom MOR registration naming; and
- credential-bearing catalog configuration.
- Reject usernames, passwords, tokens, IAM or keytab material, and
credential-bearing metastore URIs from the catalog contract.
- Upgrade the Flink capability manifest to schema 6.
- Add pinned Hudi 1.2.0 source anchors for the catalog factory, DFS
Catalog, HMS Catalog, Hive synchronization context, write coordinator, and
synchronization defaults.
- Add released-bundle Java fixtures covering:
- DFS and HMS catalog factory selection;
- unsupported catalog mode rejection;
- DFS table-path derivation;
- external HMS table behavior;
- embedded HMS rejection;
- DFS Catalog DDL and insert planning; and
- MOR Hive synchronization defaults.
- Update the Hudi Architect skill, question flow, decision rules,
warnings, output templates, capability reference, README, and runbook.
No code was copied from another project. The implementation is based on
the pinned Apache Hudi 1.2.0 source behavior and validates that behavior
through source anchors and released-bundle fixtures.
### Impact
This change is scoped to the Hudi Architect Flink path under
`hudi-agent-gateway`.
It does not change:
- Hudi storage format;
- Hudi table version;
- Hudi runtime public APIs;
- Spark behavior;
- HoodieStreamer behavior;
- the existing Flink design contract schemas 1, 2, or 3; or
- the assessment-only nature of existing-table reconciliation.
For new tables, users can opt into one of the five validated catalog
profiles. Existing contracts without a `catalog` block retain the previous
direct SQL behavior and generated SQL.
The capability manifest schema changes from 5 to 6, and validator results
now include catalog-specific machine fields. There is no expected runtime
performance impact in Hudi itself. The skill only validates and generates SQL;
it does not execute catalog SQL or probe a live metastore.
Live dependency availability, filesystem permissions, Hive configuration,
HMS connectivity, registration visibility, partition registration, and MOR
table names remain explicit deployment checks.
### Risk Level
Medium.
The change adds executable SQL generation for catalog setup and Hive
Metastore composition. Incorrect ownership, path, or synchronization behavior
could otherwise create a table at an unexpected location or expose inconsistent
external metadata.
The risk is mitigated by:
- requiring an explicit catalog contract for every non-legacy profile;
- supporting only five fixed compositions;
- preserving the legacy SQL output byte-for-byte;
- withholding every SQL artifact when a status-contributing finding
remains;
- keeping existing-table catalog mutation non-executable;
- rejecting conflicting ownership and lifecycle declarations;
- rejecting credentials and secret-bearing URIs;
- verifying behavior against the pinned Hudi 1.2.0 source revision;
- exercising the released Hudi 1.2.0 / Flink 1.20.1 bundle through Java
fixtures; and
- running the complete Gateway test, lint, and type-check matrix on Python
3.11 and 3.12.
### Documentation Update
Updated the in-repository Hudi Architect documentation:
- `SKILL.md`
- `README.md`
- `RUNBOOK.md`
- `references/flink-catalog-composition.md`
- `references/flink-1.20-hudi-1.2.0-capabilities.md`
- `references/flink-config-templates.md`
- `references/flink-decision-overrides.md`
- `references/flink-existing-table-reconciliation.md`
- `references/flink-question-flow.md`
- `references/flink-warnings.md`
No Hudi website update is included because this feature is scoped to the
Hudi Architect skill under `hudi-agent-gateway`, and its user-facing contract
and operational guidance are maintained in the skill and runbook documentation
above.
Validation performed:
- Python 3.11: `403 passed, 4 skipped`
- Python 3.12: `403 passed, 4 skipped`
- The four skipped tests are opt-in live-stack integration tests.
- Ruff checks passed on both Python versions.
- Mypy checks passed on both Python versions.
- Hudi Architect configuration-key validation passed.
- Pinned Hudi 1.2.0 source-anchor verification passed.
- Hudi Architect skill validation passed.
- Hudi 1.2.0 / Flink 1.20.1 Java fixtures: `30 tests passed`.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
