This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/master/pr-20515-46ccedc365cb46fb8d2e419601c747dc07487bdb in repository https://gitbox.apache.org/repos/asf/hudi.git
commit 9a0974b8280c7ee2f15b0b4e400e4944f55069a8 Author: Danny Chan <[email protected]> AuthorDate: Wed Nov 11 02:25:36 2026 +0000 test(spark): cover incremental SDP table chaining (#20515) * test(spark): cover incremental SDP table chaining * test(spark): strengthen metadata-enabled SDP chaining acceptance --- hudi-integ-test/spark-pipelines/ACCEPTANCE.md | 74 +++++++- .../spark-pipelines/chaining_acceptance.py | 189 +++++++++++++++++++++ 2 files changed, 258 insertions(+), 5 deletions(-) diff --git a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md index 46143eae9bee..8357c1e07e19 100644 --- a/hudi-integ-test/spark-pipelines/ACCEPTANCE.md +++ b/hudi-integ-test/spark-pipelines/ACCEPTANCE.md @@ -46,9 +46,9 @@ threshold and requires a completed compaction commit on the upsert table before the unchanged restart. Each invocation uses an isolated server and filesystem. Use `--metadata enabled` to exercise the default metadata-enabled configuration. -The default `--metadata disabled` isolates the sink assertions from the known -metadata-writer lifecycle caveat tracked by #20491; a passing disabled run does -not establish default-configuration coverage. Run both explicitly, for example: +The Phase 1 script retains `--metadata disabled` as its default; the chaining +script below defaults to `enabled`. Run both configurations explicitly when +checking the Phase 1 scenario, for example: ```sh # Append these options to the command above for separate runs: @@ -60,9 +60,73 @@ not establish default-configuration coverage. Run both explicitly, for example: Validated with Spark 4.1.1 / Scala 2.13 / JDK 17 and the candidate bundle built from master `b0d16f94b3b`: all four COW/MOR and metadata disabled/enabled combinations passed. Both MOR runs completed upsert compaction, and all four -restart timelines remained unchanged. These bounded runs do not resolve #20491. +restart timelines remained unchanged. This scenario uses one writer per target. It does not claim coverage of materialized views, full refresh, -metadata-writer lifetime, fan-in, chaining, or commit/checkpoint failure injection. +fan-in, chaining, or commit/checkpoint failure injection. The generated source, SQL and pipeline specification are retained in the logs as runnable examples within the harness's isolated filesystem layout. + +## Incremental Hudi table chaining + +`chaining_acceptance.py` adds a second partitioned Hudi streaming table that +reads the upstream keyed table through SQL `FROM STREAM upsert_events`. +**Metadata is enabled by default**, matching Hudi's default configuration. +The scenario verifies that both tables initialize their metadata tables and +produce completed metadata writes. Run COW and MOR separately: + +```sh +PYTHONOPTIMIZE=1 python hudi-integ-test/spark-pipelines/chaining_acceptance.py \ + --harness-dir /path/to/harness-checkout/hudi-integ-test/spark-pipelines \ + --bundle /path/to/candidate-hudi-spark4.1-bundle.jar \ + --table-type cow +# Repeat with --table-type mor. Use --metadata disabled only for comparison. +``` + +All checks use `require()` and remain active under Python optimization, including +verification subprocesses when `PYTHONOPTIMIZE=1` is set. SIGTERM unwinds the +harness to stop owned processes and remove temporary data. + +The first input contains a duplicate key 1 in partition `a`, plus unchanged +keys 2 and 3 in partitions `b` and `a`. The upstream upsert deduplicates key 1. +The second input updates key 1 and adds key 4 in partition `b`; it leaves keys +2 and 3 untouched. Exact business rows are ordered by `id` and `ts`. After the +second run, the upstream has four rows and the downstream insert table has five +(the old and new versions of key 1 plus keys 2, 3 and 4). Re-reading checkpointed +history would replay an untouched key and create an extra downstream row, even +if the updated key is collapsed to its latest state. Both tables must expose +exactly `part=a` and `part=b`. + +A quiescent restart checks exact rows and identical completed data-commit +filenames for both tables. MOR first drains the completed async compaction +instant: if compaction finishes after AvailableNow captures its end offset, +catch-up can create an empty downstream commit. The test records that change, +requires no business-record replay, then verifies the next restart creates no +commits. Skipping empty sink batches is proposed follow-up work for #20406; +this scenario preserves the current sink behavior. + +Explicit projection of `id, ts, value, part` excludes reserved upstream metadata +columns from the downstream input. **`SELECT * FROM STREAM upstream` is not +validated by this scenario.** Its metadata-column handling remains acceptance +work under #20407: add a real CLI case and either establish safe stripping or +verify a clear rejection. Do not infer that wildcard chaining is supported from +these passing projection-based runs. `_hoodie_is_deleted` is a separate supported +business input marker, not a reason to drop every `_hoodie_*` field blindly. + +This tests the default earliest starting offset and checkpoint-based resume. +The source emits latest-state incremental records, not every intermediate +change or hard-delete events. Its AvailableNow fallback has no bounded admission +control; size the input backlog to fit one micro-batch. + +This scenario covers the incremental portion and **leaves #20407 open**. +Reset/recreation, expired-history recovery and reader-option behavior are +covered separately by #20516. Stop downstream consumers before resetting +upstream tables; rebuild affected tables and checkpoints in dependency order. + +Validated with Spark 4.1.1 / Scala 2.13 / JDK 17, using a freshly built candidate +bundle from this branch (production base `c58c54e068b`) and the #20453 harness. +Both COW and MOR passed with `PYTHONOPTIMIZE=1` and no `--metadata` override, +exercising the enabled default. Both tables passed metadata initialization/write, +exact-row, two-partition, and unchanged-restart checks. MOR completed upstream +compaction; in this run its catch-up needed no additional downstream commit. +The bundle package build skipped unit-test execution. diff --git a/hudi-integ-test/spark-pipelines/chaining_acceptance.py b/hudi-integ-test/spark-pipelines/chaining_acceptance.py new file mode 100644 index 000000000000..671015e21be9 --- /dev/null +++ b/hudi-integ-test/spark-pipelines/chaining_acceptance.py @@ -0,0 +1,189 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Run RFC-112 incremental chaining acceptance using the #20453 Spark CLI/Connect harness.""" + +import argparse +import json +from pathlib import Path +import signal +import sys +import tempfile + + +SOURCE = '''from pyspark import pipelines as dp +from pyspark.sql import SparkSession + +spark = SparkSession.active() + [email protected]_view(name="input_events") +def input_events(): + return spark.readStream.schema("id INT, ts LONG, value STRING, part STRING").json(spark.conf.get("hudi.test.input")) +''' + +# The downstream insert operation makes replay visible as duplicate business rows. +# Explicit projection excludes the upstream table's reserved Hudi metadata fields. +FLOWS = '''SET spark.hoodie.datasource.write.operation=upsert; +CREATE STREAMING TABLE upsert_events USING hudi +PARTITIONED BY (part) +TBLPROPERTIES ('type'='{table_type}', 'primaryKey'='id', 'preCombineField'='ts') +AS SELECT id, ts, value, part FROM STREAM input_events; + +SET spark.hoodie.datasource.write.operation=insert; +CREATE STREAMING TABLE downstream_events USING hudi +PARTITIONED BY (part) +TBLPROPERTIES ('type'='{table_type}', 'primaryKey'='id', 'preCombineField'='ts') +AS SELECT id, ts, value, part FROM STREAM upsert_events; +''' + + +def require(condition, message): + if not condition: + raise AssertionError(message) + + +def verify(mode, remote, root): + from pyspark.sql import SparkSession + + spark = SparkSession.builder.remote(remote).getOrCreate() + try: + require(spark.conf.get("spark.hudi.test.run.id") == root.name, "Connected to the wrong server") + old = (1, 1, "old", "a") + new = (1, 2, "new", "a") + stable = [(2, 1, "stable-b", "b"), (3, 1, "stable-a", "a")] + added = (4, 2, "added", "b") + for table, expected in ( + ("upsert_events", [old] + stable if mode == "first" else [new] + stable + [added]), + ("downstream_events", [old] + stable if mode == "first" else [old, new] + stable + [added]), + ): + actual = [tuple(row) for row in spark.table(f"sdp_test.{table}") + .select("id", "ts", "value", "part").orderBy("id", "ts").collect()] + require(actual == expected, (table, actual, expected)) + partitions = [row[0] for row in spark.sql(f"SHOW PARTITIONS sdp_test.{table}").collect()] + require(sorted(partitions) == ["part=a", "part=b"], (table, partitions)) + properties = dict(spark.sql(f"SHOW TBLPROPERTIES sdp_test.{table}").collect()) + require(not any("write.operation" in key for key in properties), properties) + print(f"Verified {mode}: incremental upstream-to-downstream rows through Spark Connect") + finally: + spark.stop() + + +def scenario(harness, table_type, metadata): + flows = FLOWS.format(table_type=table_type) + input_dir = harness.root / "input" + input_dir.mkdir() + (harness.root / "source.py").write_text(SOURCE) + (harness.root / "flows.sql").write_text(flows) + spec = { + "name": "hudi_chaining", + "storage": (harness.root / "checkpoints").as_uri(), + "catalog": "spark_catalog", + "database": "sdp_test", + "libraries": [{"glob": {"include": "source.py"}}, {"glob": {"include": "flows.sql"}}], + "configuration": { + "hudi.test.input": input_dir.as_uri(), + "spark.sql.shuffle.partitions": "2", + "spark.sql.pipelines.maxFlowRetryAttempts": "0", + "spark.hoodie.metadata.enable": str(metadata).lower(), + "spark.hoodie.datasource.compaction.async.enable": str(table_type == "mor").lower(), + "spark.hoodie.compact.inline.max.delta.commits": "1", + }, + } + spec_path = harness.root / "spark-pipeline.yml" + spec_path.write_text(json.dumps(spec, indent=2)) + for name in ("source.py", "flows.sql", "spark-pipeline.yml"): + (harness.logs / name).write_text((harness.root / name).read_text()) + batches = ( + ("first", [(1, 1, "old", "a"), (1, 1, "old", "a"), + (2, 1, "stable-b", "b"), (3, 1, "stable-a", "a")]), + ("second", [(1, 2, "new", "a"), (4, 2, "added", "b")]), + ) + for name, rows in batches: + (input_dir / f"{name}.json").write_text("".join( + json.dumps(dict(zip(("id", "ts", "value", "part"), row))) + "\n" for row in rows)) + harness.run_pipeline(spec_path, name=f"pipeline-{name}") + harness.client(Path(__file__).resolve(), name) + if metadata: + for table in ("upsert_events", "downstream_events"): + metadata_path = harness.root / "warehouse" / "sdp_test.db" / table / ".hoodie" / "metadata" + require((metadata_path / ".hoodie" / "hoodie.properties").is_file(), + f"Metadata table was not initialized for {table}") + metadata_timeline = metadata_path / ".hoodie" / "timeline" + require(any(p.suffix in (".commit", ".deltacommit") for p in metadata_timeline.iterdir()), + f"No completed metadata writes for {table}") + def commits(): + result = {} + for table in ("upsert_events", "downstream_events"): + timeline = harness.root / "warehouse" / "sdp_test.db" / table / ".hoodie" / "timeline" + result[table] = sorted(p.name for p in timeline.iterdir() + if p.suffix in (".commit", ".deltacommit", ".replacecommit")) + require(result[table], f"No completed commits found for {table}") + return result + + before = commits() + if table_type == "mor": + # The upsert update creates a log file and must actually exercise async compaction. + require(any(name.endswith(".commit") for name in before["upsert_events"]), before) + # AvailableNow may capture its end offset before async compaction completes. + # Drain that maintenance instant before asserting a quiescent restart. + harness.run_pipeline(spec_path, name="pipeline-compaction-catchup") + harness.client(Path(__file__).resolve(), "restart") + caught_up = commits() + require(before["upsert_events"] == caught_up["upsert_events"], (before, caught_up)) + (harness.logs / "compaction-catchup.json").write_text( + json.dumps({"before": before, "after": caught_up}, indent=2)) + before = caught_up + + # Restart with no new input must leave both outputs and the completed timeline unchanged. + harness.run_pipeline(spec_path, name="pipeline-restart") + harness.client(Path(__file__).resolve(), "restart") + after = commits() + require(before == after, (before, after)) + (harness.logs / "restart-commits.json").write_text(json.dumps({"before": before, "after": after}, indent=2)) + print("PASS: exact two-stage rows and unchanged restart timelines") + + +def main(): + if len(sys.argv) == 4 and sys.argv[1] in ("first", "second", "restart"): + verify(sys.argv[1], sys.argv[2], Path(sys.argv[3])) + return + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--harness-dir", type=Path, default=Path(__file__).resolve().parent) + parser.add_argument("--bundle", type=Path, required=True) + parser.add_argument("--logs", type=Path, default=Path("target/chaining-acceptance-logs")) + parser.add_argument("--table-type", choices=("cow", "mor"), default="cow") + parser.add_argument("--metadata", choices=("enabled", "disabled"), default="enabled") + args = parser.parse_args() + if not (args.harness_dir / "harness.py").is_file(): + parser.error("The #20453 harness is required; pass its spark-pipelines directory with --harness-dir") + sys.path.insert(0, str(args.harness_dir.resolve())) + from harness import ConnectHarness + + args.logs.mkdir(parents=True, exist_ok=True) + logs = Path(tempfile.mkdtemp(prefix="run-", dir=args.logs.resolve())) + print(f"Chaining acceptance logs: {logs}", flush=True) + + def interrupted(signum, frame): + raise KeyboardInterrupt(f"Received signal {signum}") + + signal.signal(signal.SIGTERM, interrupted) + with ConnectHarness(args.bundle, logs) as harness: + scenario(harness, args.table_type, args.metadata == "enabled") + + +if __name__ == "__main__": + main()
