danny0405 opened a new issue, #20753:
URL: https://github.com/apache/hudi/issues/20753

   ### Task Description
   
   **What needs to be done:**
   
   Complete the Hudi materialized-view target and schema compatibility matrix 
through the real Spark pipelines CLI / Spark Connect harness. This carries 
forward the remaining managed/external target, additive-schema, and 
identity-change acceptance requirements from #20405.
   
   **Why this task is needed:**
   
   Merged #20494 and #20675 cover repeated refresh of keyed JSON/Hudi views and 
a keyless aggregate, partitioned/unpartitioned targets, unchanged definitions, 
removals, and empty output. That scenario does not establish the full 
managed/external target and schema-evolution matrix requested by #20405. Shared 
catalog unit tests alone do not establish the end-to-end SDP behavior.
   
   ### Acceptance criteria
   
   - [ ] Exercise initial materialization and repeated refresh for both managed 
targets and explicitly located external targets. Verify catalog location/table 
type and exact business rows, including removals and empty output.
   - [ ] Retain partitioned/unpartitioned and unchanged-definition cases in the 
matrix; reuse the merged scenario where it already supplies coverage.
   - [ ] Exercise supported additive schema changes through a changed pipeline 
definition. Assert the resulting catalog schema, persisted Hudi schema, and 
exact recomputed rows after refresh and on a subsequent unchanged run.
   - [ ] Exercise attempts to change persisted target identity, including 
record-key and partition settings as applicable. Assert the intended rejection 
and diagnostic, and inspect target/catalog state so destructive side effects 
are visible.
   - [ ] Distinguish supported schema changes from unsupported ones. If an 
expected case is not supported by the current SDP/catalog contract, record the 
limitation and required implementation follow-up rather than silently treating 
it as passing acceptance.
   - [ ] Run the matrix through the real CLI/Connect harness, retain generated 
definitions and diagnostics, and keep checks effective under Python 
optimization.
   - [ ] Document the supported target/schema matrix and exact commands. State 
the tested table type and metadata-indexing settings; do not imply coverage of 
modes that were not run.
   
   ### Scope
   
   Use the current purge-style full-table refresh contract from #20404 / 
#20622. This task does not introduce atomic refresh, timeline-preserving 
truncate, or checkpoint-generation isolation. Those deferred runtime changes 
remain under #20404 / #20378. Failure injection is tracked separately from this 
compatibility matrix.
   
   ### Task Type
   
   Test enhancement
   
   ### Related Issues
   
   Parent feature issue: #20401
   Extracted from: #20405
   Existing acceptance: #20494, #20675
   Harness: #20402 / #20453
   Catalog integration: #20371 / #20492
   Current refresh contract/documentation: #20404, #20622
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to