danny0405 opened a new pull request, #20754:
URL: https://github.com/apache/hudi/pull/20754

   ### Describe the issue this Pull Request addresses
   
   Closes #20405. Closes #20752. Closes #20753. Part of #20401.
   
   Adding a column to an SDP materialized view succeeds, but the next unchanged 
refresh purges the timeline and falls back to the original table schema. The 
result write then fails with INSERT_COLUMN_ARITY_MISMATCH. Both managed and 
external target acceptance reproduced this on master.
   
   This completes the remaining MV acceptance in one follow-up to merged #20494 
and #20675. Closure is for the agreed purge-style scope; timeline-preserving 
truncate and checkpoint-generation isolation remain deferred together under 
#20404 / #20378.
   
   ### Summary and Changelog
   
   - Resolve the current schema before full-table purge and use it when 
initializing the empty table. Clear metadata-partition state through the new 
table config so the old CREATE_SCHEMA is not written back.
   - Add COW/MOR × partitioned/unpartitioned SQL regressions: add a column, 
truncate twice including an empty table, verify the persisted schema and purged 
timeline, then insert and read the evolved row.
   - Extend the existing materialized_views.py suite with SQL MV schema 
evolution, unchanged refresh, partition/key rejection, and failures before 
purge, during reconciliation after purge, and during result writing. Assert 
exact rows, schema, identity, location/type and completed timeline state, then 
retry.
   - Add an opt-in --external manual run of the same matrix against a 
pre-created explicitly located target. Retain generated SQL, expected rows, 
timeline/schema evidence and CLI/server diagnostics. No CI jobs are added.
   - Document the destructive reconciliation failure window and retry 
procedure. No external code was copied.
   
   ### Impact
   
   Repairs schema loss for ordinary full-table TRUNCATE and SDP MV refresh. 
Purge still removes data/history/savepoints and resets metadata state; it is 
non-transactional. Partition truncate is unchanged. No new public API, 
configuration or storage format.
   
   The CLI acceptance uses COW with metadata indexing disabled. It does not 
establish MOR/metadata-enabled SDP acceptance, atomic refresh, or 
streaming-table full refresh.
   
   ### Risk Level
   
   medium. Shared SQL truncate now resolves schema before deletion and retains 
that schema in the reinitialized table config.
   
   Validation on JDK 17 / Spark 4.1 / Scala 2.13:
   - Reactor package, Checkstyle, Scalastyle and RAT checks passed.
   - TestTruncateTable: 10 passed, 0 failed, including all four new schema 
regressions.
   - Real Spark Connect 4.1.1 / spark-pipelines CLI runs passed with python -O 
for managed targets and the manual --external variant. Each passed the original 
four keyed/keyless refreshes and the full schema/failure/retry matrix, 
including empty recomputation and subsequent nonempty refresh.
   - Python compilation and git diff --check passed.
   
   Spark 3.5 and remote CI have not been run for this revision. Failures inside 
a partially completed filesystem purge are not injected; no rollback after 
deletion begins is promised.
   
   ### Documentation Update
   
   Updated hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md with schema 
retention, the observed failure boundaries, exact retry expectations and the 
manual external-target command. This repairs the existing truncate contract 
without introducing a new feature or configuration.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to