fhan688 opened a new issue, #19779:
URL: https://github.com/apache/hudi/issues/19779
### Feature Description
## Feature Description
**What the feature achieves:**
This proposal adds a deterministic, Flink-native design path to the Hudi
Architect proposed in [PR #19380](https://github.com/apache/hudi/pull/19380).
The feature collects workload facts through the Architect skill,
normalizes them into a structured `WorkloadProfile`, and uses a version-aware
deterministic planner to produce:
- A Hudi architecture decision record.
- Flink SQL DDL.
- Per-writer concurrency and lock configuration.
- Checkpoint and table-service recommendations.
- HMS or AWS Glue catalog-sync configuration.
- Required runtime bundles.
- Blockers, warnings, assumptions, and revisit conditions.
The planner is design-time only. It will not submit Flink jobs, create or
alter Hudi tables, modify table storage, or mutate an external catalog.
### Relationship to PR #19380
This proposal follows the design principles of PR #19380:
- Ask workload-oriented questions instead of asking users to choose Hudi
internals directly.
- Separate durable table decisions from reversible operational decisions.
- Derive configuration from collected facts.
- Explain trade-offs and preserve them in an ADR.
- Validate emitted configuration keys against the Hudi source tree.
The Flink path is not intended to copy Spark recommendations. It
introduces Flink-specific capability and compatibility rules while preserving
the same overall design approach.
### Proposed architecture
The proposed planning process contains the following stages:
1. The Architect skill collects workload facts.
2. The collected facts are normalized into a `WorkloadProfile`.
3. A `VersionedCapabilityProfile` defines the supported behavior for the
target Hudi and Flink versions.
4. A deterministic Flink planner derives table, index, concurrency,
table-service, and catalog decisions.
5. A cross-domain validator checks whether the combined decisions are
valid.
6. An `ArchitecturePlan` is generated.
7. Renderers generate the ADR, Flink SQL, writer configuration, catalog
configuration, required bundles, warnings, and blockers.
The skill owns conversation, clarification, and explanation.
The planner owns deterministic decisions, compatibility checks, and
artifact generation.
The planner should behave as a pure function from normalized workload
input plus a target capability profile to an `ArchitecturePlan`.
**Why this feature is needed:**
Flink design decisions are coupled in ways that cannot safely be handled
as independent configuration recommendations.
Current Hudi master rejects Flink `RECORD_LEVEL_INDEX` and
`GLOBAL_RECORD_LEVEL_INDEX` when multi-writer concurrency is enabled.
The relevant validation is implemented in:
[HoodieTableFactory.java](https://github.com/apache/hudi/blob/master/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableFactory.java#L204-L232)
Therefore, writer inventory must be collected before the index decision is
finalized. Index and concurrency must be solved together.
A planner that first recommends RLI and later adds OCC configuration can
produce an invalid design.
Catalog sync has a similar routing requirement. The planner must first
establish whether consumers need name-based discovery. It must then use storage
and query-engine facts to derive HMS or Glue.
### Goals
1. Add a deterministic Flink planning kernel behind the Architect skill.
2. Support Flink SQL as the first rendering target.
3. Support COW and MOR table design.
4. Model all processes that can commit actions to the Hudi timeline.
5. Jointly derive index and concurrency decisions.
6. Support single-writer, OCC, and eligible NBCC designs.
7. Generate per-writer lock and concurrency requirements.
8. Support no-sync, HMS sync, and AWS Glue sync.
9. Produce machine-testable ADR, SQL, configuration, warning, and blocker
artifacts.
10. Make version-dependent decisions explicit through a capability profile.
### Non-goals
- Executing or submitting Flink jobs.
- Creating or altering Hudi tables.
- Mutating HMS, Glue, or another catalog.
- Runtime lock-contention tuning.
- Automatically migrating existing Hudi tables.
- Full BigQuery or DataHub sync rendering in the first version.
- Apache XTable guidance, configuration, or integration.
- MCP exposure in the first implementation.
### Multi-writer design
#### Writer inventory
The planner asks about other writers early in the design flow.
The inventory includes:
- Regular ingestion jobs.
- Periodic backfill jobs.
- Delete, GDPR, or reprocessing jobs.
- Other Flink or Spark writers.
- Standalone compaction jobs.
- Standalone clustering jobs.
Async table services running inside the same Flink writer process do not
create a second writer.
A separately deployed table-service process is treated as an additional
writer.
An unknown writer count is treated conservatively as multi-writer. The
uncertainty is recorded as an assumption with a revisit condition.
#### Initial index and concurrency rules
The initial rules are:
- **Exactly one writer:** Use `SINGLE_WRITER`. Select the index from
workload semantics.
- **Multi-writer with Flink RLI or global RLI:** Emit
`FLINK_MULTI_WRITER_RLI_CONFLICT`. Do not silently change the index.
- **Multi-writer with MOR, simple or partition bucket index, and no
clustering:** NBCC may be offered. OCC remains available.
- **Other community-confirmed multi-writer and index combinations:**
Default to OCC.
- **Multi-writer with Flink state index:** Do not claim production safety
until the community confirms the supported semantics.
- **INSERT or BULK_INSERT with overlapping or unknown key spaces:** Emit a
duplicate-risk warning.
NBCC must only be considered when MOR and bucket index were independently
selected from workload requirements.
The planner must not change table type or index merely to make NBCC
available.
If RLI is required by the workload but multi-writer is also required, the
planner returns a blocker with the following alternatives:
1. Consolidate writes into one Flink writer.
2. Use a bucket index only if the workload semantics permit it.
3. Retain RLI and use a single writer.
4. Select another version profile only when that combination is explicitly
supported and verified.
For OCC, the generated plan includes the same concurrency mode,
lock-provider identity, lazy failed-write cleaning policy, and required bundle
for every writer.
The planner should prefer an existing organizational lock backend instead
of unconditionally recommending new DynamoDB, ZooKeeper, or HMS infrastructure.
### Catalog-sync design
The catalog section starts with the following outcome-oriented question:
**Does any query engine or BI tool need to discover this table by database
and table name?**
If no consumer needs name-based discovery, the planner emits no
catalog-sync configuration.
If discovery is required, storage is collected before query engines
because storage constrains the valid engine and catalog combinations.
The initial catalog routing rules are:
- **Flink or Spark reading by path:** Do not enable catalog sync.
- **Athena reading a table stored on S3:** Use AWS Glue.
- **Trino, Presto, or Hive reading an S3 table when the organization
already uses Glue:** Use AWS Glue.
- **Trino, Presto, or Hive in an HDFS or on-premises environment:** Use
HMS.
- **An unsupported engine and storage combination:** Return a blocker and
request clarification.
The first Flink-native implementation supports:
- `NONE`
- `HMS`
- `GLUE`
The renderer uses the canonical Flink options where applicable:
- `hive_sync.enabled`
- `hive_sync.mode`
- `hive_sync.db`
- `hive_sync.table`
- `hive_sync.metastore.uris`
- `hive_sync.partition_fields`
- `hive_sync.partition_extractor_class`
- `hive_sync.table.strategy`
For MOR tables, the plan must explicitly document whether consumers use:
- The read-optimized view.
- The real-time view.
- Both views.
BigQuery and DataHub requirements may be recorded in the workload model,
but exact standalone sync rendering is deferred.
Apache XTable is completely out of scope for this proposal.
### Versioning
The planner should target a `VersionedCapabilityProfile` instead of
hard-coding the behavior of one Hudi release.
The capability profile contains:
- Hudi version.
- Flink version.
- Supported index types.
- Valid index and concurrency combinations.
- NBCC eligibility.
- Catalog-sync modes.
- Canonical configuration keys.
- Required bundle coordinates.
Unknown or unverified combinations produce an explicit
`UNVERIFIED_VERSION_COMBINATION` result instead of a production recommendation.
### User Experience
## User Experience
**How users will use this feature:**
Users interact with the feature through workload-oriented questions.
A typical flow is:
1. Identify the target Hudi and Flink versions.
2. Describe the source, update semantics, delete semantics, and latency
requirements.
3. List every process that may write to the table.
4. Describe the record key, ordering field, and partition behavior.
5. Describe storage and query patterns.
6. Confirm whether consumers need to discover the table by name.
7. Identify the storage system and query engines.
8. Identify existing lock or catalog infrastructure.
9. Review derived decisions, blockers, warnings, and assumptions.
10. Export the ADR, Flink SQL, per-writer configuration, and catalog-sync
plan.
Users are not asked to choose RLI, OCC, NBCC, HMS, or Glue directly unless
multiple semantically valid options remain after workload-based derivation.
If collected facts are incompatible, the planner returns a blocker and
alternatives instead of silently weakening correctness requirements.
### Configuration changes
This feature does not introduce new Hudi table or writer configuration
keys.
It selects and renders existing, version-appropriate Hudi and Flink
options.
For a multi-writer OCC design, the planner may render the following
existing keys:
- `hoodie.write.concurrency.mode`
- `hoodie.write.lock.provider`
- `hoodie.clean.failed.writes.policy`
The generated output must state that the concurrency and lock
configuration applies identically to every process writing the table.
For a Flink HMS catalog-sync design, the planner may render:
- `hive_sync.enabled`
- `hive_sync.mode`
- `hive_sync.db`
- `hive_sync.table`
- `hive_sync.metastore.uris`
Exact options are rendered from the selected capability profile.
Defaults that do not change behavior should generally not be emitted.
### API and internal contracts
The proposed internal workload contract contains:
- A target platform definition.
- A writer inventory.
- Catalog requirements.
- Table requirements.
- Operational requirements.
The proposed architecture-plan contract contains:
- A table plan.
- An index plan.
- A concurrency plan.
- A table-service plan.
- A catalog-sync plan.
- Validation results.
`IndexPlan` and `ConcurrencyPlan` remain separate output objects, but they
must be produced by the same joint decision stage.
The first implementation does not expose a new public REST or MCP API.
Public exposure can be proposed separately after the planning contracts
stabilize.
### Generated artifacts
The planner produces:
- A normalized workload profile.
- An architecture decision record.
- Flink SQL DDL.
- Configuration shared by all writers.
- Writer-specific instructions.
- Catalog-sync configuration.
- Required runtime bundles.
- Confirmed facts and assumptions.
- Warnings and hard blockers.
- Revisit conditions.
- A machine-readable decision trace.
### Validation and testing
The implementation should include the following golden and invariant tests:
1. Single writer, COW, Flink state index, and no catalog.
2. Single writer, MOR, and RLI.
3. Multi-writer with RLI, resulting in a blocker.
4. Multi-writer with MOR and bucket index, eligible for NBCC.
5. Multi-writer with clustering, requiring OCC.
6. A standalone compactor being counted as an additional writer.
7. In-process async compaction not being counted as an additional writer.
8. Multi-writer INSERT with overlapping key spaces producing a warning.
9. Path-based Flink or Spark access producing no catalog configuration.
10. S3 plus Athena selecting Glue.
11. HDFS plus Trino selecting HMS.
12. GCS plus Athena producing a contradiction blocker.
13. MOR catalog sync explicitly selecting RO, RT, or ALL.
14. An unverified Hudi and Flink version producing a version warning.
15. Identical normalized input producing identical output.
All emitted Hudi and Flink configuration keys must be validated against
the selected source profile or an explicit reviewed allowlist.
### Proposed PR plan
#### PR 1: Define Flink architect planning contracts
Suggested title:
`feat(agent): define Flink architect planning contracts`
Scope:
- Add workload-profile models.
- Add writer-inventory models.
- Add catalog-requirement models.
- Add architecture-plan models.
- Add serialization and schema tests.
- Do not generate recommendations yet.
#### PR 2: Add the versioned Flink capability registry
Suggested title:
`feat(agent): add versioned Flink capability registry`
Scope:
- Add Hudi and Flink version profiles.
- Model supported indexes.
- Model valid concurrency combinations.
- Model supported catalog-sync modes.
- Model required runtime bundles.
- Validate renderable configuration keys.
#### PR 3: Add deterministic Flink table-design rules
Suggested title:
`feat(agent): add deterministic Flink table design rules`
Scope:
- Add source analysis.
- Add record-key and ordering-field rules.
- Add partitioning rules.
- Add COW and MOR selection.
- Add initial index candidates.
- Add single-writer and no-catalog golden cases.
#### PR 4: Add joint index and multi-writer planning
Suggested title:
`feat(agent): add joint Flink index and multi-writer planning`
Scope:
- Add writer-inventory evaluation.
- Add OCC and NBCC derivation.
- Add lock-provider selection.
- Add RLI and multi-writer blockers.
- Add per-writer configuration.
- Add standalone table-service detection.
- Add operation-specific warnings.
#### PR 5: Model Flink checkpoints and table services
Suggested title:
`feat(agent): model Flink checkpoints and table services`
Scope:
- Add checkpoint planning.
- Add compaction planning.
- Add clustering planning.
- Add cleaning and retention planning.
- Reconcile table-service deployment with writer inventory.
#### PR 6: Add Flink catalog-sync planning
Suggested title:
`feat(agent): add Flink catalog sync planning`
Scope:
- Add the catalog requirement gate.
- Add storage and query-engine routing.
- Add HMS and Glue selection.
- Add MOR catalog-table strategy.
- Defer BigQuery and DataHub rendering.
- Keep Apache XTable out of scope.
#### PR 7: Render Flink architecture artifacts
Suggested title:
`feat(agent): render Flink architecture artifacts`
Scope:
- Render the ADR.
- Render Flink SQL.
- Render per-writer configuration.
- Render catalog configuration.
- Render required bundles.
- Render warnings and blockers.
- Require deterministic output.
#### PR 8: Integrate the planner with Hudi Architect
Suggested title:
`feat(agent): integrate Flink planner with Hudi Architect skill`
Scope:
- Connect the question flow to normalized planner input.
- Ensure that the skill does not bypass planner decisions.
- Add end-to-end conversational fixtures.
- Add user documentation.
### Hudi RFC Requirements
## Hudi RFC Requirements
**RFC PR link:** N/A at the proposal stage.
**Why RFC is or is not needed:**
- Does this change existing public Hudi storage or writer APIs? **No**
- Does this change the Hudi storage format or table layout? **No**
- Does this introduce a new design-time component or internal interface?
**Yes**
- Does this execute writes or modify external systems? **No**
This proposal introduces structured planning contracts and a deterministic
Flink planning component inside the agent gateway.
It extends the Hudi Architect direction proposed in PR #19380, but does
not change the Hudi storage format, timeline semantics, table layout, or
existing writer APIs.
Maintainer guidance is requested on whether this can be implemented as a
follow-up feature under the design established by PR #19380, or whether the
deterministic planner is significant
enough to require a separate Hudi RFC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
