This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 948706b059e4 docs(spark): document SDP truncate and full-refresh
limitations (#20622)
948706b059e4 is described below
commit 948706b059e41b24e3918e6362dba38f788dd0f8
Author: Danny Chan <[email protected]>
AuthorDate: Wed Nov 18 03:53:41 2026 +0000
docs(spark): document SDP truncate and full-refresh limitations (#20622)
---
hudi-integ-test/spark-pipelines/ACCEPTANCE.md | 46 ++++++++++++++++++++++
.../spark-pipelines/MATERIALIZED_VIEWS.md | 11 ++++--
2 files changed, 53 insertions(+), 4 deletions(-)
diff --git a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
index 8357c1e07e19..cfcb14196414 100644
--- a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
+++ b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
@@ -67,6 +67,52 @@ fan-in, chaining, or commit/checkpoint failure injection.
The generated source, SQL and pipeline specification are retained in the logs
as
runnable examples within the harness's isolated filesystem layout.
+## Truncate and full-refresh limitations
+
+The release scope agreed in
[#20404](https://github.com/apache/hudi/issues/20404)
+retains the existing full-table truncate behavior. Timeline-preserving truncate
+and streaming checkpoint-generation isolation are deferred together; they are
+not prerequisites for this release's SDP integration.
+
+| Operation | Current behavior |
+| --- | --- |
+| `TRUNCATE TABLE t` | Deletes the table directory and re-initializes the
table from its properties. Data, timeline history, savepoints, metadata table,
and stored sink batch IDs are lost. |
+| `TRUNCATE TABLE t PARTITION (...)` | Uses a `delete_partition` replace
commit; it does not purge and re-initialize the whole table. |
+| SDP full refresh of a streaming table | Purges the target and restarts the
flow from a fresh SDP-managed checkpoint. Old sink batch IDs are removed with
the table, so they cannot suppress replay starting at batch 0. |
+| Ordinary streaming restart | Reuses the existing checkpoint and sink
identifier to skip batches already committed to Hudi. |
+
+Full-table truncate is non-transactional. Do not run it while another writer is
+active. Deletion and table re-initialization can fail independently; a failed
+truncate does not guarantee that the previous snapshot survives. SDP's
+truncate–alter–append sequence is also non-atomic: a failure after truncation
+can leave an empty table, and a failure during replay can leave only the
batches
+committed so far. A successful refresh with empty input leaves an empty table.
+Savepoints and retention settings cannot preserve history across this physical
+purge, and the old metadata table is not retained.
+
+Incremental/streaming reader offsets into the old table history become invalid.
+Downstream consumers are **not reset automatically** when their source is
+truncated or recreated. Stop affected consumers before the reset, then refresh
+or rebuild downstream tables and their checkpoints in dependency order along
+with the upstream table. This applies to chained SDP flows and to
HoodieStreamer
+using `HoodieIncrSource`. Do not assume a downstream reader will detect the
reset
+and fail safely; default failure on source recreation is follow-up work.
+
+Keep the checkpoint and sink identifier on ordinary restarts. Deleting a
+checkpoint in place while retaining the target's sink metadata is unsupported:
+batch IDs restart at zero and can be skipped against an old high-water mark.
+Checkpoint relocation requires an explicit rebuild/migration; changing a path
+alone does not isolate a new generation in the current sink. SDP full refresh
+coordinates the target purge with a fresh checkpoint. Replay requires the
source
+data needed to rebuild the target to remain available.
+
+These are operational limitations, not additional acceptance-test results.
+Timeline-preserving truncate (the proposal in
+[#20378](https://github.com/apache/hudi/pull/20378)) must ship together with
+checkpoint-generation isolation and an explicit legacy-identifier migration
+contract if pursued after this release. Atomic refresh and automatic downstream
+recovery are not provided by the current implementation.
+
## Incremental Hudi table chaining
`chaining_acceptance.py` adds a second partitioned Hudi streaming table that
diff --git a/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
b/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
index 03d783c99d38..2fd3f2d0ee8b 100644
--- a/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
+++ b/hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
@@ -39,10 +39,13 @@ Keyless aggregate materialized views remain deferred under
#20405.
Land this scenario after the #20453 harness dependency. Checks remain active
under
`python -O`, and SIGTERM unwinds the harness for server and
temporary-directory cleanup.
-This is Phase 2 coverage on today's purge-style truncate. History retention,
-atomic refresh and timeline-preserving truncate are intentionally outside this
-scenario and remain Phase 4 work. Metadata indexing is disabled. The test does
-not establish streaming-table chaining or full-refresh behavior.
+This is Phase 2 coverage on today's purge-style truncate. See the
+[truncate and full-refresh
limitations](ACCEPTANCE.md#truncate-and-full-refresh-limitations)
+for history loss, the non-atomic refresh window, and downstream rebuild
requirements.
+Timeline-preserving truncate and checkpoint-generation isolation are deferred
+together beyond this release under #20404; atomic refresh is also out of scope.
+Metadata indexing is disabled. The test does not establish streaming-table
+chaining or streaming-table full-refresh behavior.
The harness bounds each client/CLI process and removes its temporary data.
Logs and generated pipeline definitions remain under
`target/mv-acceptance-logs`.