fhan688 opened a new pull request, #20514:
URL: https://github.com/apache/hudi/pull/20514
### Describe the issue this Pull Request addresses
Related to #19779.
The previous incremental changes added bounded Flink support for new-table
sinks, mutable COW and MOR tables, existing-table reconciliation, catalog and
metastore composition, and multi-writer assessment.
Each path is validated independently, but regressions may still occur when
these capabilities are composed. In particular, cross-path changes must not:
- widen a previously bounded executable path;
- render mutating SQL for an existing table;
- treat an observed multi-writer configuration as verified support;
- allow out-of-scope evidence to become authoritative;
- discard independent findings when selecting the final status; or
- expose malformed or sensitive input through an unstructured failure.
This PR adds a bounded cross-path validation matrix and deterministic
regression coverage for those invariants. It does not introduce a new
production path or expand the supported Flink capability baseline.
### Summary and Changelog
This PR adds cross-path validation and evaluation coverage for the
completed Hudi Architect Flink milestone.
The validation remains pinned to the existing `hudi-1.2.0-flink-1.20`
baseline and reuses the validators and contracts introduced by the preceding
incremental changes.
The change includes:
- a declarative matrix containing ten representative cross-path scenarios;
- exact expected status, executable eligibility, process exit code, and
ordered findings for every scenario;
- regression coverage for:
- append-only COW through direct SQL;
- mutable COW through the Hudi DFS catalog and Hive Metastore;
- mutable MOR through direct SQL and Hive Metastore synchronization;
- compatible existing-table reconciliation;
- unverified existing-table catalog mutation;
- multi-writer OCC assessment;
- unsupported multi-writer NBCC composition;
- out-of-scope evidence attached to an existing-table request;
- multiple independent safety gates failing in one request; and
- malformed contracts containing sensitive values;
- matrix self-consistency checks that independently derive:
- the final status from finding precedence;
- the expected return code from validator type and executable
eligibility; and
- the relationship between an empty finding set and `CONFIG_VALIDATED`;
- assertions that new-table paths preserve their previously validated
execution semantics;
- assertions that existing-table paths never emit mutating SQL;
- assertions that multi-writer paths remain assessment-only unless
explicitly supported;
- assertions that out-of-scope evidence retains provenance but does not
contribute authoritative target-table facts;
- assertions that all independent findings are preserved before
final-status precedence is applied;
- deterministic replay checks for all matrix scenarios; and
- structured and redacted failure checks for malformed cross-path input.
The final-status precedence remains:
1. `BLOCKED`
2. `REVIEW_REQUIRED`
3. `INCOMPLETE`
4. `CONFIG_VALIDATED` only when no applicable finding remains
Status precedence selects one summary status but does not remove
lower-precedence findings.
No production implementation, public API, storage format, configuration
default, or supported execution boundary is changed by this PR. No code was
copied from external sources.
#### Validation
The following validation was completed on Python 3.11 and Python 3.12:
- full Hudi Agent Gateway test suite:
- `778 passed`
- `6 skipped`
- Ruff checks for `src` and `tests`;
- Mypy checks for the Gateway source tree; and
- the focused PR8 cross-path test suite.
An additional isolated forward evaluation covered six representative Agent
scenarios with three independent runs per scenario. All 18 runs produced the
expected canonical route, status, executable boundary, finding order, and
artifact classification. These out-of-band evaluations were graded
independently rather than relying on Agent-reported safety conclusions.
### Impact
There is no public API, storage format, configuration, runtime behavior,
or user-facing feature change.
The PR only adds fixtures and automated validation coverage. Its purpose
is to prevent future changes from weakening the bounded behavior established by
the preceding Hudi Architect Flink changes.
The tests are local and deterministic. They do not require network access,
external services, a running Hive Metastore, or a live Flink cluster.
### Risk Level
Low.
The changes are limited to test fixtures and regression tests. The main
risk is that an intentional future contract change may require updating the
matrix expectations.
This risk is reduced by:
- pinning the matrix to the existing capability baseline;
- deriving status and return-code consistency independently from the
stored expectations;
- asserting exact ordered findings rather than only the final status;
- replaying every scenario to verify deterministic output; and
- running the complete Gateway test suite on both supported Python
versions.
### Documentation Update
None.
This PR does not add or change a user-facing feature, configuration
option, default value, or public API.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
