wombatu-kun commented on code in PR #20515: URL: https://github.com/apache/hudi/pull/20515#discussion_r4465594790
########## hudi-integ-test/spark-pipelines/ACCEPTANCE.md: ########## @@ -66,3 +66,40 @@ This scenario uses one writer per target. It does not claim coverage of material metadata-writer lifetime, fan-in, chaining, or commit/checkpoint failure injection. The generated source, SQL and pipeline specification are retained in the logs as runnable examples within the harness's isolated filesystem layout. + +## Incremental Hudi table chaining + +`chaining_acceptance.py` adds a second Hudi streaming table that reads the +upstream keyed table through SQL `FROM STREAM upsert_events`. Run it using +the same `--harness-dir`, `--bundle`, `--table-type` and `--metadata` options +as the Phase 1 scenario: + +```sh +python hudi-integ-test/spark-pipelines/chaining_acceptance.py \ + --harness-dir /path/to/harness-checkout/hudi-integ-test/spark-pipelines \ + --bundle /path/to/candidate-hudi-spark4.1-bundle.jar +``` + +The first input contains duplicate keys that the upstream upsert deduplicates. +The downstream uses insert deliberately: it must contain one old record after +the first run and exactly the old and new versions after the second run. +Replaying an earlier increment would produce extra rows and fail the assertion. +A quiescent restart verifies exact rows and identical completed commit filenames +for both tables. MOR first drains the completed async compaction instant: if +compaction finished after AvailableNow captured its end offset, this catch-up +can create an empty downstream commit. The test checks that it does not replay +business records, records that timeline change separately, and then asserts +that the next restart creates no commits. Explicit business-column projection prevents +upstream Hudi metadata fields from becoming downstream input columns. + +This tests the default earliest starting offset and checkpoint-based resume. +The source emits latest-state incremental records, not every intermediate +change or hard-delete events. Its AvailableNow fallback has no bounded +admission control; size the input backlog to fit one micro-batch. + +This scenario covers only the incremental portion of #20407. It does not Review Comment: The description's "does not close #20407" still contains the keyword `close #20407`, so GitHub has linked #20407 as closed by this PR and will close it on merge. Could it say "leaves #20407 open" instead? ########## hudi-integ-test/spark-pipelines/chaining_acceptance.py: ########## @@ -0,0 +1,165 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Run RFC-112 incremental chaining acceptance using the #20453 Spark CLI/Connect harness.""" + +import argparse +import json +from pathlib import Path +import signal +import sys +import tempfile + + +SOURCE = '''from pyspark import pipelines as dp +from pyspark.sql import SparkSession + +spark = SparkSession.active() + [email protected]_view(name="input_events") +def input_events(): + return spark.readStream.schema("id INT, ts LONG, value STRING").json(spark.conf.get("hudi.test.input")) +''' + +# The downstream insert operation makes replay visible as duplicate business rows. +# Explicit projection excludes the upstream table's reserved Hudi metadata fields. +FLOWS = '''SET spark.hoodie.datasource.write.operation=upsert; +CREATE STREAMING TABLE upsert_events USING hudi +TBLPROPERTIES ('type'='{table_type}', 'primaryKey'='id', 'preCombineField'='ts') +AS SELECT id, ts, value FROM STREAM input_events; + +SET spark.hoodie.datasource.write.operation=insert; +CREATE STREAMING TABLE downstream_events USING hudi +TBLPROPERTIES ('type'='{table_type}', 'primaryKey'='id', 'preCombineField'='ts') +AS SELECT id, ts, value FROM STREAM upsert_events; +''' + + +def verify(mode, remote, root): + from pyspark.sql import SparkSession + + spark = SparkSession.builder.remote(remote).getOrCreate() + try: + assert spark.conf.get("spark.hudi.test.run.id") == root.name + old = (1, 1, "old") + new = (1, 2, "new") + for table, expected in ( + ("upsert_events", [old] if mode == "first" else [new]), + ("downstream_events", [old] if mode == "first" else [old, new]), + ): + actual = [tuple(row) for row in spark.table(f"sdp_test.{table}") + .select("id", "ts", "value").orderBy("ts").collect()] + assert actual == expected, (table, actual, expected) + properties = dict(spark.sql(f"SHOW TBLPROPERTIES sdp_test.{table}").collect()) + assert not any("write.operation" in key for key in properties), properties + print(f"Verified {mode}: incremental upstream-to-downstream rows through Spark Connect") + finally: + spark.stop() + + +def scenario(harness, table_type, metadata): + flows = FLOWS.format(table_type=table_type) + input_dir = harness.root / "input" + input_dir.mkdir() + (harness.root / "source.py").write_text(SOURCE) + (harness.root / "flows.sql").write_text(flows) + spec = { + "name": "hudi_chaining", + "storage": (harness.root / "checkpoints").as_uri(), + "catalog": "spark_catalog", + "database": "sdp_test", + "libraries": [{"glob": {"include": "source.py"}}, {"glob": {"include": "flows.sql"}}], + "configuration": { + "hudi.test.input": input_dir.as_uri(), + "spark.sql.shuffle.partitions": "2", + "spark.sql.pipelines.maxFlowRetryAttempts": "0", + "spark.hoodie.metadata.enable": str(metadata).lower(), + "spark.hoodie.datasource.compaction.async.enable": str(table_type == "mor").lower(), + "spark.hoodie.compact.inline.max.delta.commits": "1", + }, + } + spec_path = harness.root / "spark-pipeline.yml" + spec_path.write_text(json.dumps(spec, indent=2)) + for name in ("source.py", "flows.sql", "spark-pipeline.yml"): + (harness.logs / name).write_text((harness.root / name).read_text()) + for name, timestamp, value in (("first", 1, "old"), ("second", 2, "new")): + row = json.dumps({"id": 1, "ts": timestamp, "value": value}) + "\n" Review Comment: With a single key that the second input overwrites, a read that also re-reads the checkpointed first commit returns the same latest-state row as the real increment, so this run still ends with exactly `[old, new]`. Could the first input also carry a key the second input leaves alone (with the verify ordered by `id` and `ts`), so such an over-read shows up as an extra downstream row? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
