This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch 
gh-readonly-queue/master/pr-20515-46ccedc365cb46fb8d2e419601c747dc07487bdb
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 9a0974b8280c7ee2f15b0b4e400e4944f55069a8
Author: Danny Chan <[email protected]>
AuthorDate: Wed Nov 11 02:25:36 2026 +0000

    test(spark): cover incremental SDP table chaining (#20515)
    
    * test(spark): cover incremental SDP table chaining
    
    * test(spark): strengthen metadata-enabled SDP chaining acceptance
---
 hudi-integ-test/spark-pipelines/ACCEPTANCE.md      |  74 +++++++-
 .../spark-pipelines/chaining_acceptance.py         | 189 +++++++++++++++++++++
 2 files changed, 258 insertions(+), 5 deletions(-)

diff --git a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md 
b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
index 46143eae9bee..8357c1e07e19 100644
--- a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
+++ b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md
@@ -46,9 +46,9 @@ threshold and requires a completed compaction commit on the 
upsert table before
 the unchanged restart. Each invocation uses an isolated server and filesystem.
 
 Use `--metadata enabled` to exercise the default metadata-enabled 
configuration.
-The default `--metadata disabled` isolates the sink assertions from the known
-metadata-writer lifecycle caveat tracked by #20491; a passing disabled run does
-not establish default-configuration coverage. Run both explicitly, for example:
+The Phase 1 script retains `--metadata disabled` as its default; the chaining
+script below defaults to `enabled`. Run both configurations explicitly when
+checking the Phase 1 scenario, for example:
 
 ```sh
 # Append these options to the command above for separate runs:
@@ -60,9 +60,73 @@ not establish default-configuration coverage. Run both 
explicitly, for example:
 Validated with Spark 4.1.1 / Scala 2.13 / JDK 17 and the candidate bundle built
 from master `b0d16f94b3b`: all four COW/MOR and metadata disabled/enabled
 combinations passed. Both MOR runs completed upsert compaction, and all four
-restart timelines remained unchanged. These bounded runs do not resolve #20491.
+restart timelines remained unchanged.
 
 This scenario uses one writer per target. It does not claim coverage of 
materialized views, full refresh,
-metadata-writer lifetime, fan-in, chaining, or commit/checkpoint failure 
injection.
+fan-in, chaining, or commit/checkpoint failure injection.
 The generated source, SQL and pipeline specification are retained in the logs 
as
 runnable examples within the harness's isolated filesystem layout.
+
+## Incremental Hudi table chaining
+
+`chaining_acceptance.py` adds a second partitioned Hudi streaming table that
+reads the upstream keyed table through SQL `FROM STREAM upsert_events`.
+**Metadata is enabled by default**, matching Hudi's default configuration.
+The scenario verifies that both tables initialize their metadata tables and
+produce completed metadata writes. Run COW and MOR separately:
+
+```sh
+PYTHONOPTIMIZE=1 python hudi-integ-test/spark-pipelines/chaining_acceptance.py 
\
+  --harness-dir /path/to/harness-checkout/hudi-integ-test/spark-pipelines \
+  --bundle /path/to/candidate-hudi-spark4.1-bundle.jar \
+  --table-type cow
+# Repeat with --table-type mor. Use --metadata disabled only for comparison.
+```
+
+All checks use `require()` and remain active under Python optimization, 
including
+verification subprocesses when `PYTHONOPTIMIZE=1` is set. SIGTERM unwinds the
+harness to stop owned processes and remove temporary data.
+
+The first input contains a duplicate key 1 in partition `a`, plus unchanged
+keys 2 and 3 in partitions `b` and `a`. The upstream upsert deduplicates key 1.
+The second input updates key 1 and adds key 4 in partition `b`; it leaves keys
+2 and 3 untouched. Exact business rows are ordered by `id` and `ts`. After the
+second run, the upstream has four rows and the downstream insert table has five
+(the old and new versions of key 1 plus keys 2, 3 and 4). Re-reading 
checkpointed
+history would replay an untouched key and create an extra downstream row, even
+if the updated key is collapsed to its latest state. Both tables must expose
+exactly `part=a` and `part=b`.
+
+A quiescent restart checks exact rows and identical completed data-commit
+filenames for both tables. MOR first drains the completed async compaction
+instant: if compaction finishes after AvailableNow captures its end offset,
+catch-up can create an empty downstream commit. The test records that change,
+requires no business-record replay, then verifies the next restart creates no
+commits. Skipping empty sink batches is proposed follow-up work for #20406;
+this scenario preserves the current sink behavior.
+
+Explicit projection of `id, ts, value, part` excludes reserved upstream 
metadata
+columns from the downstream input. **`SELECT * FROM STREAM upstream` is not
+validated by this scenario.** Its metadata-column handling remains acceptance
+work under #20407: add a real CLI case and either establish safe stripping or
+verify a clear rejection. Do not infer that wildcard chaining is supported from
+these passing projection-based runs. `_hoodie_is_deleted` is a separate 
supported
+business input marker, not a reason to drop every `_hoodie_*` field blindly.
+
+This tests the default earliest starting offset and checkpoint-based resume.
+The source emits latest-state incremental records, not every intermediate
+change or hard-delete events. Its AvailableNow fallback has no bounded 
admission
+control; size the input backlog to fit one micro-batch.
+
+This scenario covers the incremental portion and **leaves #20407 open**.
+Reset/recreation, expired-history recovery and reader-option behavior are
+covered separately by #20516. Stop downstream consumers before resetting
+upstream tables; rebuild affected tables and checkpoints in dependency order.
+
+Validated with Spark 4.1.1 / Scala 2.13 / JDK 17, using a freshly built 
candidate
+bundle from this branch (production base `c58c54e068b`) and the #20453 harness.
+Both COW and MOR passed with `PYTHONOPTIMIZE=1` and no `--metadata` override,
+exercising the enabled default. Both tables passed metadata 
initialization/write,
+exact-row, two-partition, and unchanged-restart checks. MOR completed upstream
+compaction; in this run its catch-up needed no additional downstream commit.
+The bundle package build skipped unit-test execution.
diff --git a/hudi-integ-test/spark-pipelines/chaining_acceptance.py 
b/hudi-integ-test/spark-pipelines/chaining_acceptance.py
new file mode 100644
index 000000000000..671015e21be9
--- /dev/null
+++ b/hudi-integ-test/spark-pipelines/chaining_acceptance.py
@@ -0,0 +1,189 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Run RFC-112 incremental chaining acceptance using the #20453 Spark 
CLI/Connect harness."""
+
+import argparse
+import json
+from pathlib import Path
+import signal
+import sys
+import tempfile
+
+
+SOURCE = '''from pyspark import pipelines as dp
+from pyspark.sql import SparkSession
+
+spark = SparkSession.active()
+
[email protected]_view(name="input_events")
+def input_events():
+    return spark.readStream.schema("id INT, ts LONG, value STRING, part 
STRING").json(spark.conf.get("hudi.test.input"))
+'''
+
+# The downstream insert operation makes replay visible as duplicate business 
rows.
+# Explicit projection excludes the upstream table's reserved Hudi metadata 
fields.
+FLOWS = '''SET spark.hoodie.datasource.write.operation=upsert;
+CREATE STREAMING TABLE upsert_events USING hudi
+PARTITIONED BY (part)
+TBLPROPERTIES ('type'='{table_type}', 'primaryKey'='id', 
'preCombineField'='ts')
+AS SELECT id, ts, value, part FROM STREAM input_events;
+
+SET spark.hoodie.datasource.write.operation=insert;
+CREATE STREAMING TABLE downstream_events USING hudi
+PARTITIONED BY (part)
+TBLPROPERTIES ('type'='{table_type}', 'primaryKey'='id', 
'preCombineField'='ts')
+AS SELECT id, ts, value, part FROM STREAM upsert_events;
+'''
+
+
+def require(condition, message):
+    if not condition:
+        raise AssertionError(message)
+
+
+def verify(mode, remote, root):
+    from pyspark.sql import SparkSession
+
+    spark = SparkSession.builder.remote(remote).getOrCreate()
+    try:
+        require(spark.conf.get("spark.hudi.test.run.id") == root.name, 
"Connected to the wrong server")
+        old = (1, 1, "old", "a")
+        new = (1, 2, "new", "a")
+        stable = [(2, 1, "stable-b", "b"), (3, 1, "stable-a", "a")]
+        added = (4, 2, "added", "b")
+        for table, expected in (
+            ("upsert_events", [old] + stable if mode == "first" else [new] + 
stable + [added]),
+            ("downstream_events", [old] + stable if mode == "first" else [old, 
new] + stable + [added]),
+        ):
+            actual = [tuple(row) for row in spark.table(f"sdp_test.{table}")
+                      .select("id", "ts", "value", "part").orderBy("id", 
"ts").collect()]
+            require(actual == expected, (table, actual, expected))
+            partitions = [row[0] for row in spark.sql(f"SHOW PARTITIONS 
sdp_test.{table}").collect()]
+            require(sorted(partitions) == ["part=a", "part=b"], (table, 
partitions))
+            properties = dict(spark.sql(f"SHOW TBLPROPERTIES 
sdp_test.{table}").collect())
+            require(not any("write.operation" in key for key in properties), 
properties)
+        print(f"Verified {mode}: incremental upstream-to-downstream rows 
through Spark Connect")
+    finally:
+        spark.stop()
+
+
+def scenario(harness, table_type, metadata):
+    flows = FLOWS.format(table_type=table_type)
+    input_dir = harness.root / "input"
+    input_dir.mkdir()
+    (harness.root / "source.py").write_text(SOURCE)
+    (harness.root / "flows.sql").write_text(flows)
+    spec = {
+        "name": "hudi_chaining",
+        "storage": (harness.root / "checkpoints").as_uri(),
+        "catalog": "spark_catalog",
+        "database": "sdp_test",
+        "libraries": [{"glob": {"include": "source.py"}}, {"glob": {"include": 
"flows.sql"}}],
+        "configuration": {
+            "hudi.test.input": input_dir.as_uri(),
+            "spark.sql.shuffle.partitions": "2",
+            "spark.sql.pipelines.maxFlowRetryAttempts": "0",
+            "spark.hoodie.metadata.enable": str(metadata).lower(),
+            "spark.hoodie.datasource.compaction.async.enable": str(table_type 
== "mor").lower(),
+            "spark.hoodie.compact.inline.max.delta.commits": "1",
+        },
+    }
+    spec_path = harness.root / "spark-pipeline.yml"
+    spec_path.write_text(json.dumps(spec, indent=2))
+    for name in ("source.py", "flows.sql", "spark-pipeline.yml"):
+        (harness.logs / name).write_text((harness.root / name).read_text())
+    batches = (
+        ("first", [(1, 1, "old", "a"), (1, 1, "old", "a"),
+                   (2, 1, "stable-b", "b"), (3, 1, "stable-a", "a")]),
+        ("second", [(1, 2, "new", "a"), (4, 2, "added", "b")]),
+    )
+    for name, rows in batches:
+        (input_dir / f"{name}.json").write_text("".join(
+            json.dumps(dict(zip(("id", "ts", "value", "part"), row))) + "\n" 
for row in rows))
+        harness.run_pipeline(spec_path, name=f"pipeline-{name}")
+        harness.client(Path(__file__).resolve(), name)
+        if metadata:
+            for table in ("upsert_events", "downstream_events"):
+                metadata_path = harness.root / "warehouse" / "sdp_test.db" / 
table / ".hoodie" / "metadata"
+                require((metadata_path / ".hoodie" / 
"hoodie.properties").is_file(),
+                        f"Metadata table was not initialized for {table}")
+                metadata_timeline = metadata_path / ".hoodie" / "timeline"
+                require(any(p.suffix in (".commit", ".deltacommit") for p in 
metadata_timeline.iterdir()),
+                        f"No completed metadata writes for {table}")
+    def commits():
+        result = {}
+        for table in ("upsert_events", "downstream_events"):
+            timeline = harness.root / "warehouse" / "sdp_test.db" / table / 
".hoodie" / "timeline"
+            result[table] = sorted(p.name for p in timeline.iterdir()
+                                   if p.suffix in (".commit", ".deltacommit", 
".replacecommit"))
+            require(result[table], f"No completed commits found for {table}")
+        return result
+
+    before = commits()
+    if table_type == "mor":
+        # The upsert update creates a log file and must actually exercise 
async compaction.
+        require(any(name.endswith(".commit") for name in 
before["upsert_events"]), before)
+        # AvailableNow may capture its end offset before async compaction 
completes.
+        # Drain that maintenance instant before asserting a quiescent restart.
+        harness.run_pipeline(spec_path, name="pipeline-compaction-catchup")
+        harness.client(Path(__file__).resolve(), "restart")
+        caught_up = commits()
+        require(before["upsert_events"] == caught_up["upsert_events"], 
(before, caught_up))
+        (harness.logs / "compaction-catchup.json").write_text(
+            json.dumps({"before": before, "after": caught_up}, indent=2))
+        before = caught_up
+
+    # Restart with no new input must leave both outputs and the completed 
timeline unchanged.
+    harness.run_pipeline(spec_path, name="pipeline-restart")
+    harness.client(Path(__file__).resolve(), "restart")
+    after = commits()
+    require(before == after, (before, after))
+    (harness.logs / "restart-commits.json").write_text(json.dumps({"before": 
before, "after": after}, indent=2))
+    print("PASS: exact two-stage rows and unchanged restart timelines")
+
+
+def main():
+    if len(sys.argv) == 4 and sys.argv[1] in ("first", "second", "restart"):
+        verify(sys.argv[1], sys.argv[2], Path(sys.argv[3]))
+        return
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--harness-dir", type=Path, 
default=Path(__file__).resolve().parent)
+    parser.add_argument("--bundle", type=Path, required=True)
+    parser.add_argument("--logs", type=Path, 
default=Path("target/chaining-acceptance-logs"))
+    parser.add_argument("--table-type", choices=("cow", "mor"), default="cow")
+    parser.add_argument("--metadata", choices=("enabled", "disabled"), 
default="enabled")
+    args = parser.parse_args()
+    if not (args.harness_dir / "harness.py").is_file():
+        parser.error("The #20453 harness is required; pass its spark-pipelines 
directory with --harness-dir")
+    sys.path.insert(0, str(args.harness_dir.resolve()))
+    from harness import ConnectHarness
+
+    args.logs.mkdir(parents=True, exist_ok=True)
+    logs = Path(tempfile.mkdtemp(prefix="run-", dir=args.logs.resolve()))
+    print(f"Chaining acceptance logs: {logs}", flush=True)
+
+    def interrupted(signum, frame):
+        raise KeyboardInterrupt(f"Received signal {signum}")
+
+    signal.signal(signal.SIGTERM, interrupted)
+    with ConnectHarness(args.bundle, logs) as harness:
+        scenario(harness, args.table_type, args.metadata == "enabled")
+
+
+if __name__ == "__main__":
+    main()

Reply via email to