fhan688 opened a new pull request, #20302:
URL: https://github.com/apache/hudi/pull/20302
### Describe the issue this Pull Request addresses
Related to #19779.
This is the fourth incremental PR in the Flink support plan for Hudi
Architect. It builds on the routing and safety gates, the first executable
append-only Flink SQL sink path, and the mutable COW path added by the
preceding PRs.
The existing mutable executable path is intentionally limited to
Copy-on-Write. Users with sustained updates may instead need Merge-on-Read, but
selecting MOR also introduces compaction ownership and read-freshness
requirements that must be resolved before Hudi Architect can safely emit
executable SQL.
This PR adds the first bounded mutable Flink SQL MOR sink path while
preserving the existing append-only COW and mutable COW contracts.
### Summary and Changelog
This PR adds contract schema 3 for a new-table, single-writer,
no-external-catalog mutable MOR sink using Hudi 1.2.0 and Flink 1.20.
The generated MOR SQL uses a fixed, source-verified contract:
- `MERGE_ON_READ` table type;
- `upsert` write operation;
- event-time ordering with `EVENT_TIME_ORDERING`;
- global `FLINK_STATE` index;
- index state TTL disabled;
- index bootstrap enabled;
- changelog generation disabled;
- asynchronous compaction scheduled and executed by the same Flink writer;
- `num_commits` compaction trigger with five delta commits; and
- clustering scheduling and execution disabled.
The mutable MOR path retains the existing mutable safety requirements:
- a stable, non-null record key;
- an explicit event-time ordering field;
- normalized UPSERT source changelog;
- full-row delete payloads when deletes are emitted;
- immutable partition fields;
- a new Hudi table;
- a single writer; and
- no external catalog ownership.
The design flow now considers both update workload and read requirements
when choosing between COW and MOR:
- COW remains the default for low or occasional update volume and simpler
operational requirements.
- MOR can be selected for sustained updates when snapshot reads are
required or read-optimized consumers accept visibility after compaction.
- Strictly current read-optimized requirements are not accepted for the
MOR path.
The validator fails closed when the request falls outside this bounded
contract. In particular, it does not emit executable SQL for:
- append-only MOR;
- standalone or unresolved compaction ownership;
- custom compaction tuning;
- enabled clustering;
- strict read-optimized freshness;
- multiple writers;
- existing-table takeover;
- external catalogs; or
- other unsupported Flink write paths.
The query requirements are recorded in the machine-readable design
contract, but query-side options are not incorrectly rendered into the sink DDL.
Additional changes include:
- bumping the Hudi Architect manifest schema from 3 to 4;
- adding a machine-readable `mutable_mor_executable_contract`;
- extending pinned capability evidence for MOR, compaction, clustering,
and query semantics;
- adding deterministic validation findings for unsupported MOR requests;
- adding canonical mutable MOR JSON and SQL golden fixtures;
- adding pinned Java planner tests for the MOR write and compaction
operator graph;
- verifying that COW plans do not contain compaction operators;
- verifying that append-only MOR is not treated as an executable
compaction path;
- adding the PR4 fixture to Agent Gateway CI; and
- updating the skill, runbook, capability, decision, question-flow,
configuration, and warning documentation.
Contract schemas 1 and 2 and their existing golden outputs remain
unchanged.
No code was copied from another project.
### Impact
The change is scoped to `hudi-agent-gateway` and is additive.
Hudi Architect can now emit validated Flink SQL for one bounded mutable
MOR configuration in addition to the existing append-only COW and mutable COW
paths.
There is no change to the Hudi storage format, Hudi runtime public APIs,
or existing connector configuration defaults. Existing schema-1 and schema-2
inputs retain their current behavior.
The emitted MOR configuration enables asynchronous compaction, so
deployments using this new path should account for the expected compaction I/O
and should monitor pending compactions and read-optimized freshness.
Unsupported designs continue to fail closed instead of receiving a
best-effort configuration.
### Risk Level
Low.
The implementation is isolated to the Hudi Architect skill, its
validators, documentation, fixtures, and Agent Gateway CI. It does not modify
the Hudi Flink connector runtime.
Risk is further limited by:
- preserving the existing schema-1 and schema-2 contracts;
- using a fixed MOR configuration rather than exposing arbitrary tuning;
- requiring same-writer compaction ownership;
- rejecting unresolved or unsupported configurations;
- checking configuration keys and capability evidence against the pinned
Hudi source revision;
- comparing generated SQL with golden fixtures; and
- verifying the generated table plans against the released Hudi 1.2.0
Flink bundle and Flink 1.20.1.
Validation performed:
- Full Agent Gateway suite on Python 3.11: `306 passed, 4 skipped`
- Full Agent Gateway suite on Python 3.12: `306 passed, 4 skipped`
- Hudi Architect focused suite: `220 passed`
- Pinned Java planner suite: `20 passed`
- Ruff: passed
- MyPy: passed
- Flink capability source verification: passed
- Configuration key validation: passed
- Skill package validation: passed
- `git diff --check`: passed
The remaining test warnings are dependency deprecation warnings from
Authlib, LangGraph, and LangChain and are unrelated to this change.
### Documentation Update
Updated the in-repository Hudi Architect documentation:
- `hudi-agent-gateway/skills/hudi-architect/SKILL.md`
- `hudi-agent-gateway/skills/hudi-architect/README.md`
- `hudi-agent-gateway/skills/hudi-architect/RUNBOOK.md`
- Flink capability and source-evidence references
- Flink configuration templates
- Flink decision overrides
- Flink question flow
- Flink warnings and unsupported-scope documentation
No Hudi website configuration reference update is required because this PR
does not add or change a Hudi connector configuration key or its default value.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
