fhan688 opened a new pull request, #20472:
URL: https://github.com/apache/hudi/pull/20472
### Describe the issue this Pull Request addresses
Part of #19779.
The existing Hudi Architect Flink path requires a confirmed single writer
before it can emit executable SQL. When an independent writer, backfill job,
cleanup writer, cross-engine writer, or separately deployed table service
exists, the Architect previously returned a generic multi-writer review finding
without a structured representation of the writer topology.
This PR adds a bounded, assessment-only Writer Contract for multi-writer
Flink designs. It records explicit writer and ownership facts while preserving
the existing safety boundary: the Architect does not select a concurrency mode,
lock provider, index, cleaner policy, or recovery strategy, and it does not
generate executable multi-writer SQL.
### Summary and Changelog
This PR:
- Adds Writer Contract schema 1 alongside the existing Flink design
contract schemas 1, 2, and 3.
- Records the primary writer ID and additional writer identities, roles,
relationships, engines, engine versions, Hudi versions, and canonical write
operations.
- Records explicit schema-evolution ownership and optional
table-configuration ownership.
- Optionally records observed concurrency mode, lock-provider class, and
lock-configuration consistency without treating them as recommendations.
- Returns deterministic Writer Contract results:
- `NOT_APPLICABLE`
- `COMPLETE`
- `INCOMPLETE`
- `CONFLICT`
- Returns bounded multi-writer capability results:
- `NOT_APPLICABLE`
- `UNVERIFIED`
- `UNSUPPORTED`
- Keeps OCC assessment-only and `UNVERIFIED`, because the fixtures do not
run concurrent writers, a lock service, or recovery.
- Reports NBCC as `UNSUPPORTED` for the existing schemas because schemas 1
and 2 use COW, while schema 3 uses MOR with `FLINK_STATE`. The pinned Hudi
1.2.0 source requires MOR with the simple bucket index for this path.
- Preserves the existing single-writer SQL output without modification.
- Extends existing-table reconciliation with optional provenance-bearing
writer facts:
- Active writer IDs
- Schema-evolution owner
- Observed concurrency mode
- Lock-provider class
- Lock-configuration consistency
- Keeps conflicting Writer Contract evidence `UNVERIFIED`, with change
disposition `UNVERIFIED`, rather than inferring a migration or replacing either
fact.
- Upgrades the Flink capability manifest from schema 6 to schema 7.
- Adds source anchors for the pinned Hudi 1.2.0 concurrency modes, write
operations, write configuration constraints, and Flink lock-provider resolution.
- Adds Java source-backed fixtures for multi-writer cleaner policy, NBCC
constraints, and canonical write-operation names.
- Updates the Hudi Architect skill, runbook, capability documentation,
warnings, decision overrides, question flow, templates, and existing-table
reconciliation documentation.
- Includes the new Writer Contract module in Gateway ruff and mypy
validation.
`COMPLETE` means only that the bounded Writer Contract facts were
captured. It is not a declaration that the deployment is supported or that its
live concurrency behavior was verified.
No code was copied from external sources.
### Impact
The change is limited to the Hudi Architect Flink assessment path.
It does not change:
- Hudi storage format or table metadata.
- Hudi runtime write behavior.
- Public Hudi APIs.
- Connector configuration defaults.
- The existing executable design contract schemas.
- Generated SQL for confirmed single-writer designs.
- Spark Architect behavior.
The machine-readable Flink capability manifest is upgraded from schema 6
to schema 7. Validator output now includes Writer Contract result, capability,
normalized facts, notes, and deployment checks.
A confirmed multi-writer design remains non-executable and never returns
multi-writer capability as `SUPPORTED`.
There is no expected runtime performance impact.
### Risk Level
Low.
The primary risk is incorrect classification of multi-writer facts or
accidental expansion of an existing executable route. This is mitigated by:
- Strict schema and enum validation.
- Fail-closed handling for missing, malformed, unknown, or conflicting
facts.
- Exact regression coverage for existing single-writer SQL.
- Pinned Hudi 1.2.0 source hashes.
- Java tests against the pinned Hudi and Flink dependencies.
- Existing-table evidence conflict tests.
- Full Gateway tests on Python 3.11 and 3.12.
- Ruff and mypy validation on both Python versions.
Validation completed:
- Gateway tests on Python 3.11: 438 passed, 4 skipped.
- Gateway tests on Python 3.12: 438 passed, 4 skipped.
- Focused Hudi Architect tests: 352 passed.
- Full Hudi 1.2.0 and Flink 1.20.1 Maven fixture tests passed.
- Ruff passed on Python 3.11 and 3.12.
- Mypy passed on Python 3.11 and 3.12.
- Capability manifest validation passed.
- Pinned source-anchor verification passed.
- Hudi configuration-key validation passed.
The tests intentionally do not claim to validate live concurrent writers,
an external lock service, or recovery behavior. Those remain deployment checks.
### Documentation Update
Updated the in-repository Hudi Architect documentation, including:
- Skill routing and output rules.
- Flink capability boundaries.
- Writer Contract schema and result semantics.
- Question flow.
- Decision overrides.
- Warning definitions.
- Configuration templates.
- Existing-table evidence reconciliation.
- Runbook.
- README.
No Hudi runtime configuration or default was added or changed, so no Hudi
configuration-reference update is required.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
