fhan688 opened a new pull request, #20198:
URL: https://github.com/apache/hudi/pull/20198
### Describe the issue this Pull Request addresses
Related to #19779.
This PR follows #20021, which introduced deterministic Flink routing and
safety gates in Hudi Architect while intentionally deferring executable Flink
output.
It implements the first bounded executable Flink SQL sink path for the
following validated scenario:
- The target is a new Hudi table.
- The table uses Copy-on-Write storage.
- The write is append-only.
- Exactly one writer is confirmed.
- No external catalog is required.
- Streaming checkpointing is enabled with a concrete positive interval.
- The record identity is provided by a stable, non-null key or by explicit
acceptance of Hudi-generated record keys.
- Replays cannot occur or duplicate rows are explicitly acceptable.
- Both the source and target physical schemas are known.
- The source changelog is explicitly declared as insert-only.
All other Flink write scenarios continue to fail closed or remain deferred
for later changes.
### Summary and Changelog
This PR adds a strict, fail-closed design validator that emits executable
Flink SQL only after the complete bounded contract has been validated.
The generated output contains:
- Streaming runtime configuration.
- A concrete checkpoint interval.
- An explicit source table definition.
- An explicit Hudi target table definition.
- An explicit projected `INSERT INTO ... SELECT ...` statement without
`SELECT *`.
- A credential-free target path.
- `table.type = 'COPY_ON_WRITE'`.
- `write.operation = 'insert'`.
- `write.insert.cluster = 'false'`.
The implementation supports two record-identity paths:
- A stable record key is represented by `PRIMARY KEY (...) NOT ENFORCED`.
- Hudi-generated record keys are allowed only after the user explicitly
accepts their replay and deduplication implications.
The validator collects all applicable findings and selects one
deterministic final status using the precedence `BLOCKED` over
`REVIEW_REQUIRED` over `INCOMPLETE`. SQL artifacts are emitted only when the
final status is `CONFIG_VALIDATED`.
The validator also enforces the following Flink acceptance conditions:
- It rejects a configured Hudi record key that is absent from the target
physical schema, including record-key values supplied through connector options.
- It requires `write.insert.cluster = 'false'` for the bounded
Copy-on-Write insert path and rejects conflicting structured or
connector-option overrides.
- It rejects non-insert-only source changelogs and treats an unspecified
source contract as incomplete.
- It rejects nullable stable record-key columns.
- It verifies partition-field and record-key references against the
physical schema.
- It rejects source and target schema mismatches.
- It accepts only the bounded scalar Flink SQL type surface verified by
this PR.
- It rejects unknown or ignored contract fields instead of silently
accepting them.
- It rejects credential-bearing or otherwise unsafe target paths.
- It does not allow arbitrary connector-option pass-through.
The Flink capability manifest is upgraded to record the exact executable
contract and its supporting evidence. The baseline remains pinned to Hudi 1.2.0
source revision `f05c83f2b97732de7a558ff9b26959e1139c05f5`, with the executable
fixture using Flink 1.20.1 and Java 11.
This PR adds:
- Golden JSON and SQL fixtures for stable-key and explicitly accepted
generated-key designs.
- A standalone Java fixture using the released `hudi-flink1.20-bundle`
1.2.0 artifact.
- Planner coverage for the exact generated source, sink, runtime settings,
and insert statements.
- Regression coverage for the append-mode record-key validation gap in the
pinned factory.
- Regression coverage for the effective insert-clustering requirement.
- Regression coverage for the insert-only source changelog contract.
- Python validation, linting, type checking, and pinned-source
verification in the Agent Gateway CI workflow.
- A dedicated Java 11 Maven CI job for the pinned Flink SQL fixtures.
While exercising the complete workflow, this PR also fixes the existing
chart-lint branch so that charts with `values-local-dev.yaml` are linted
directly with those values, as required by the existing workflow comment. This
prevents the Hudi Trino chart from being linted once with intentionally
incomplete default values before its local-development values are applied.
The Architect documentation, question flow, decision tables, warnings,
configuration examples, runbook, and ADR template are updated to describe the
executable boundary and its fail-closed behavior.
The following scenarios remain outside this PR:
- Mutable, upsert, or delete workloads.
- Merge-on-Read tables.
- Existing-table introspection and validation.
- External catalog integration.
- Multiple writers.
- Replays that must collapse to a single logical record.
- DataStream API generation.
- HoodieFlinkStreamer generation.
- Flink source-connector generation.
- Job submission or deployment.
- Live Flink cluster or storage-system verification.
### Impact
This PR changes Hudi Architect behavior only for Flink requests that
satisfy the bounded contract described above. Those requests can now receive
validated Flink SQL instead of stopping at the PR1 executable-path deferral.
Unsupported or incomplete requests continue to return deterministic
findings without executable SQL.
This PR does not change Hudi storage formats, public connector APIs,
connector defaults, or runtime write behavior. It does not submit jobs, create
live tables, access credentials, or modify user data.
The CI workflow gains one standalone Maven job for the pinned Flink
planner and factory fixtures.
### Risk Level
Medium.
This PR enables executable SQL generation for a deliberately narrow Flink
write path. The main risks are producing SQL that differs from the pinned
connector behavior or allowing an unsafe design through the executable boundary.
These risks are mitigated by:
- A fail-closed input contract.
- Deterministic finding aggregation and status precedence.
- Exact golden SQL fixtures.
- Pinned Hudi 1.2.0 and Flink 1.20.1 factory and planner tests.
- Source-revision and source-hash verification.
- Explicit regression tests for the three Flink acceptance conditions.
- Rejection of unverified options, schema types, paths, and source
changelog modes.
- Withholding SQL unless the final status is `CONFIG_VALIDATED`.
Local verification completed successfully:
- The complete Agent Gateway test suite passed on Python 3.11 with `212
passed` and `4 skipped`.
- The complete Agent Gateway test suite passed on Python 3.12 with `212
passed` and `4 skipped`.
- Ruff passed on Python 3.11 and Python 3.12.
- Mypy passed on Python 3.11 and Python 3.12.
- The Hudi configuration-key validator passed.
- Pinned Flink capability and source-hash verification passed.
- Golden SQL generation passed for both supported identity paths.
- All four pinned Java factory and planner tests passed.
- Helm lint and template validation passed for all charts.
- Workflow YAML parsing and `git diff --check` passed.
The Java fixture provides factory and planner validation only. This PR
does not claim end-to-end execution against a live Flink cluster or storage
system.
### Documentation Update
The checked-in Hudi Architect documentation is updated to cover:
- The supported executable Flink SQL boundary.
- Required design inputs and source contracts.
- Stable-key and generated-key behavior.
- Generated runtime, source, sink, and insert SQL.
- Deterministic validation outcomes and finding codes.
- Fail-closed behavior for unsupported scenarios.
- Operational warnings and deferred capabilities.
- Validation and troubleshooting procedures.
No Hudi website update is included because this PR changes the Agent
Gateway skill contract and documents that behavior alongside the skill. It does
not add or change a public Hudi connector configuration.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
