danny0405 opened a new issue, #20753: URL: https://github.com/apache/hudi/issues/20753
### Task Description **What needs to be done:** Complete the Hudi materialized-view target and schema compatibility matrix through the real Spark pipelines CLI / Spark Connect harness. This carries forward the remaining managed/external target, additive-schema, and identity-change acceptance requirements from #20405. **Why this task is needed:** Merged #20494 and #20675 cover repeated refresh of keyed JSON/Hudi views and a keyless aggregate, partitioned/unpartitioned targets, unchanged definitions, removals, and empty output. That scenario does not establish the full managed/external target and schema-evolution matrix requested by #20405. Shared catalog unit tests alone do not establish the end-to-end SDP behavior. ### Acceptance criteria - [ ] Exercise initial materialization and repeated refresh for both managed targets and explicitly located external targets. Verify catalog location/table type and exact business rows, including removals and empty output. - [ ] Retain partitioned/unpartitioned and unchanged-definition cases in the matrix; reuse the merged scenario where it already supplies coverage. - [ ] Exercise supported additive schema changes through a changed pipeline definition. Assert the resulting catalog schema, persisted Hudi schema, and exact recomputed rows after refresh and on a subsequent unchanged run. - [ ] Exercise attempts to change persisted target identity, including record-key and partition settings as applicable. Assert the intended rejection and diagnostic, and inspect target/catalog state so destructive side effects are visible. - [ ] Distinguish supported schema changes from unsupported ones. If an expected case is not supported by the current SDP/catalog contract, record the limitation and required implementation follow-up rather than silently treating it as passing acceptance. - [ ] Run the matrix through the real CLI/Connect harness, retain generated definitions and diagnostics, and keep checks effective under Python optimization. - [ ] Document the supported target/schema matrix and exact commands. State the tested table type and metadata-indexing settings; do not imply coverage of modes that were not run. ### Scope Use the current purge-style full-table refresh contract from #20404 / #20622. This task does not introduce atomic refresh, timeline-preserving truncate, or checkpoint-generation isolation. Those deferred runtime changes remain under #20404 / #20378. Failure injection is tracked separately from this compatibility matrix. ### Task Type Test enhancement ### Related Issues Parent feature issue: #20401 Extracted from: #20405 Existing acceptance: #20494, #20675 Harness: #20402 / #20453 Catalog integration: #20371 / #20492 Current refresh contract/documentation: #20404, #20622 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
