This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch 
gh-readonly-queue/master/pr-19933-e8a076d08d5875c404736941ae88c9bbc7449b82
in repository https://gitbox.apache.org/repos/asf/hudi.git

commit 6069b4aed732b77c1fc4664e077ba4db2a059838
Author: voonhous <[email protected]>
AuthorDate: Thu Nov 19 08:08:59 2026 +0000

    chore(ci): fold hadoop-mr job into utilities-rest (#19933)
    
    * chore(ci): fold hadoop-mr job into utilities-rest
    
    test-hudi-hadoop-mr-and-hudi-java-client ran 8 minutes, most of them
    checkout and build, on the same JDK 11 lane as test-utilities-rest
    (18 minutes). Its test step moves into that job, whose build now also
    covers hudi-hadoop-mr and hudi-client/hudi-java-client. One job and
    one build fewer per run; the merged job stays well under the run's
    longest jobs at about 42 minutes.
    
    The merged job uploads one report under both of its flags,
    utilities,hadoop-mr-java-client, so hadoop-mr-java-client keeps
    getting fresh reports instead of being carried forward. The fork
    coverage bridge accepted only one flag per artifact; it now takes a
    comma-separated list whose flags are all known.
    
    * chore(ci): keep hadoop-mr tests and coverage flags apart
    
    Run the hudi-hadoop-mr and hudi-java-client unit tests whenever the
    build succeeded, so a hudi-utilities test failure no longer skips
    them; they had a job of their own before the fold.
    
    Upload one coverage report per flag: utilities from the hudi-utilities
    exec files, hadoop-mr-java-client from the hudi-hadoop-mr and
    hudi-java-client ones, so each flag keeps the coverage of its own
    tests. generate_merged_coverage_report.sh and
    merge_jacoco_exec_files.sh take optional exec directories for this.
    The fork coverage bridge no longer needs comma-separated flags, so
    that change is reverted.
    
    * chore(ci): scope the hadoop-mr coverage report
    
    The hadoop-mr-java-client report took its classes and sources from the
    whole combined workspace, so hudi-utilities' classes showed up as
    uncovered under that flag. The report step now lists the modules of
    the reactor that job built before the fold (hudi-hadoop-mr,
    hudi-java-client and their upstream modules) and passes them to
    generate_merged_coverage_report.sh through COVERAGE_MODULE_DIRS. The
    utilities reactor is the whole combined workspace, so its report
    keeps the default.
    
    generate_merged_coverage_report.sh recreates the aggregate class and
    source directories on every call. merge_jacoco_exec_files.sh fails
    when it finds no exec files, instead of writing an empty merge, and
    fails after its last retry instead of exiting 0.
    
    * chore(ci): drop the utilities report before the next
    
    If listing the hadoop-mr reactor failed, the report step stopped
    before generate_merged_coverage_report.sh removed the utilities
    report, and the always() upload that follows published it under
    hadoop-mr-java-client. The step now removes jacoco-report.xml before
    the Maven command, so a failed step leaves no report to upload.
---
 .github/workflows/README.md                       |  2 +-
 .github/workflows/bot.yml                         | 98 ++++++++++-------------
 scripts/jacoco/generate_merged_coverage_report.sh | 49 ++++++++++--
 scripts/jacoco/merge_jacoco_exec_files.sh         | 20 ++++-
 4 files changed, 99 insertions(+), 70 deletions(-)

diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 0508b8a7c281..b825123b2151 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -3,7 +3,7 @@
 When updating the pr template, you must consider if updates need to be made to 
scripts/pr_compliance.py
 
 ## What are the files in workflows?
-- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2), 
Flink, bundle validation and integration tests. Runs on PRs, merge-queue groups 
and branch pushes; only push runs are path-filtered. Forks run it on every 
branch push; upstream it skips every job on pushes to branches other than 
master, release-* and branch-0.x, and every job but the aggregate on PRs from 
forks to master (see [Pull requests from forks](#pull-requests-from-forks)). 
Every test job depends only on t [...]
+- bot.yml: Java CI. Spark lanes (JDK 11 on Spark 3.5, JDK 17 on Spark 4.2), 
Flink, bundle validation and integration tests. Runs on PRs, merge-queue groups 
and branch pushes; only push runs are path-filtered. Forks run it on every 
branch push; upstream it skips every job on pushes to branches other than 
master, release-* and branch-0.x, and every job but the aggregate on PRs from 
forks to master (see [Pull requests from forks](#pull-requests-from-forks)). 
Every test job depends only on t [...]
 - fork_ci_status.yml: Fork CI status. Mirrors the fork's bot.yml run for the 
head commit of each open fork PR to master onto the PR as the `CI (fork)` 
check, which also asks for action when the head commit has no fork run although 
the PR changes more than docs and images. Runs when an upstream Java CI run 
completes and every 15 minutes for late runs and reruns. It never checks out PR 
code. Its tests, .github/ci/scripts/test-fork-ci-status.cjs, also cover the 
routing and concurrency of bot.yml
 - fork_coverage_bridge.yml: Fork coverage bridge. Every 15 minutes, uploads to 
Codecov the JaCoCo reports that the fork's bot.yml runs stored as artifacts 
(through .github/actions/upload-coverage) for the head commit of each recently 
updated fork PR to master, and records each uploaded commit as a `Coverage 
(fork)` check. It never checks out or runs PR code; its tests are 
.github/ci/scripts/test-fork-coverage-bridge.cjs
 - validate_source.yml: PR and Source Validation. One job, 
`validate-pr-and-source`, one of the two required status checks with `Java CI 
result`: it checks the PR description with scripts/pr_compliance.py, runs that 
script's own tests and the fork_ci_status.yml and fork_coverage_bridge.yml 
tests, validates the PR title, rejects commits with a Claude co-author trailer, 
and runs the source release checks (RAT, binary files, copyright). Never 
path-filtered, because a required check must repo [...]
diff --git a/.github/workflows/bot.yml b/.github/workflows/bot.yml
index 36ab7f8ea13c..c8867e553e97 100644
--- a/.github/workflows/bot.yml
+++ b/.github/workflows/bot.yml
@@ -283,7 +283,7 @@ jobs:
           flags: utilities
           token: ${{ secrets.CODECOV_TOKEN }}
 
-  test-utilities-rest:
+  test-utilities-rest-hadoop-mr-java-client:
     if: *route
     needs: changes
     runs-on: ubuntu-24.04
@@ -305,12 +305,13 @@ jobs:
           architecture: x64
           cache: maven
       - name: Build Project
+        id: build
         env:
           SCALA_PROFILE: ${{ matrix.scalaProfile }}
           SPARK_PROFILE: ${{ matrix.sparkProfile }}
           FLINK_PROFILE: ${{ matrix.flinkProfile }}
         run:
-          mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source 
$MVN_ARGS -am -pl hudi-utilities
+          mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service -Pthrift-gen-source 
$MVN_ARGS -am -pl hudi-utilities,hudi-hadoop-mr,hudi-client/hudi-java-client
       - name: UT - hudi-utilities (non-TestHoodieDeltaStreamer)
         env:
           SCALA_PROFILE: ${{ matrix.scalaProfile }}
@@ -325,18 +326,50 @@ jobs:
           FLINK_PROFILE: ${{ matrix.flinkProfile }}
         run:
           mvn test -Pfunctional-tests -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -fae -Dsurefire.failIfNoSpecifiedTests=false -pl 
hudi-utilities $MVN_ARGS -Djacoco.skip=false
+      # These modules had a job of their own, so a hudi-utilities test failure 
must not skip them.
+      - name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client
+        if: ${{ !cancelled() && steps.build.outcome == 'success' }}
+        env:
+          SCALA_PROFILE: ${{ matrix.scalaProfile }}
+          SPARK_PROFILE: ${{ matrix.sparkProfile }}
+          FLINK_PROFILE: ${{ matrix.flinkProfile }}
+        run:
+          mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS 
-Djacoco.skip=false
       - name: Test timing summary
         if: always()
         run: python3 scripts/ci/test_timing_summary.py --title "${{ github.job 
}}"
-      - name: Generate merged coverage report
+      # One report per flag, each from its own modules' exec files, so each 
flag keeps the coverage
+      # of its own tests. Each report is uploaded before the next call 
replaces it.
+      - name: Generate hudi-utilities coverage report
         if: always() && github.event_name != 'merge_group'
-        run: ./scripts/jacoco/generate_merged_coverage_report.sh 
$GITHUB_WORKSPACE
-      - name: Upload coverage to Codecov
+        run: ./scripts/jacoco/generate_merged_coverage_report.sh 
$GITHUB_WORKSPACE $GITHUB_WORKSPACE/hudi-utilities
+      - name: Upload hudi-utilities coverage to Codecov
         if: always() && github.event_name != 'merge_group'
         uses: ./.github/actions/upload-coverage
         with:
           flags: utilities
           token: ${{ secrets.CODECOV_TOKEN }}
+      # Classes and sources come from the reactor this flag's job built on its 
own (hudi-hadoop-mr,
+      # hudi-java-client and their upstream modules), so hudi-utilities' 
classes stay out of its report.
+      # exec-maven-plugin is pinned to the version the repo's poms already use.
+      - name: Generate hudi-hadoop-mr and hudi-java-client coverage report
+        if: always() && github.event_name != 'merge_group'
+        env:
+          SCALA_PROFILE: ${{ matrix.scalaProfile }}
+          SPARK_PROFILE: ${{ matrix.sparkProfile }}
+          FLINK_PROFILE: ${{ matrix.flinkProfile }}
+        run: |
+          # Remove the utilities report first: if this step fails before the 
generator runs, the
+          # always() upload below must not publish it under this flag.
+          rm -f jacoco-report.xml
+          module_dirs=$(mvn -q -B -ntp -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -Phudi-platform-service -am -pl 
hudi-hadoop-mr,hudi-client/hudi-java-client 
org.codehaus.mojo:exec-maven-plugin:1.6.0:exec -Dexec.executable=pwd)
+          COVERAGE_MODULE_DIRS="$module_dirs" 
./scripts/jacoco/generate_merged_coverage_report.sh $GITHUB_WORKSPACE 
$GITHUB_WORKSPACE/hudi-hadoop-mr $GITHUB_WORKSPACE/hudi-client/hudi-java-client
+      - name: Upload hudi-hadoop-mr and hudi-java-client coverage to Codecov
+        if: always() && github.event_name != 'merge_group'
+        uses: ./.github/actions/upload-coverage
+        with:
+          flags: hadoop-mr-java-client
+          token: ${{ secrets.CODECOV_TOKEN }}
 
   test-common-and-other-modules:
     if: *route
@@ -746,54 +779,6 @@ jobs:
 #          flags: spark-scala-tests
 #          token: ${{ secrets.CODECOV_TOKEN }}
 
-  test-hudi-hadoop-mr-and-hudi-java-client:
-    if: *route
-    needs: changes
-    runs-on: ubuntu-24.04
-    timeout-minutes: 40
-    strategy:
-      matrix:
-        include:
-          - scalaProfile: "scala-2.12"
-            sparkProfile: "spark3.5"
-            flinkProfile: "flink2.2"
-
-    steps:
-      - uses: actions/checkout@v5
-      - name: Set up JDK 11
-        uses: actions/setup-java@v5
-        with:
-          java-version: '11'
-          distribution: 'temurin'
-          architecture: x64
-          cache: maven
-      - name: Build Project
-        env:
-          SCALA_PROFILE: ${{ matrix.scalaProfile }}
-          SPARK_PROFILE: ${{ matrix.sparkProfile }}
-          FLINK_PROFILE: ${{ matrix.flinkProfile }}
-        run:
-          mvn clean install -T 2 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -DskipTests=true -Phudi-platform-service $MVN_ARGS -am -pl 
hudi-hadoop-mr,hudi-client/hudi-java-client
-      - name: UT - hudi-hadoop-mr and hudi-client/hudi-java-client
-        env:
-          SCALA_PROFILE: ${{ matrix.scalaProfile }}
-          SPARK_PROFILE: ${{ matrix.sparkProfile }}
-          FLINK_PROFILE: ${{ matrix.flinkProfile }}
-        run:
-          mvn test -Punit-tests -fae -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" 
-D"$FLINK_PROFILE" -pl hudi-hadoop-mr,hudi-client/hudi-java-client $MVN_ARGS 
-Djacoco.skip=false
-      - name: Test timing summary
-        if: always()
-        run: python3 scripts/ci/test_timing_summary.py --title "${{ github.job 
}}"
-      - name: Generate merged coverage report
-        if: always() && github.event_name != 'merge_group'
-        run: ./scripts/jacoco/generate_merged_coverage_report.sh 
$GITHUB_WORKSPACE
-      - name: Upload coverage to Codecov
-        if: always() && github.event_name != 'merge_group'
-        uses: ./.github/actions/upload-coverage
-        with:
-          flags: hadoop-mr-java-client
-          token: ${{ secrets.CODECOV_TOKEN }}
-
   # How the Spark JDK 17 jobs below are sharded. The minutes come from master 
run
   # 34314808833 on 2026-09-09, before the two-fork surefire change earlier on 
this branch,
   # so treat them as relative weights rather than the wall time a job takes 
today.
@@ -2093,8 +2078,8 @@ jobs:
   # cancelled one. Adding or resharding a job means adding its id to `needs` 
below and
   # `needs: changes` and `if: *route` to the job, and nothing else. The ASF 
GitHub Actions policy
   # caps the jobs running at once in a workflow at 20 (15 recommended), 
counted across all
-  # matrices. The test jobs total 23 instances; `max-parallel` on 
validate-bundles and
-  # validate-bundle-spark4 holds back 3, so the test jobs can currently run 20 
at once.
+  # matrices. The test jobs total 22 instances; `max-parallel` on 
validate-bundles and
+  # validate-bundle-spark4 holds back 3, so the test jobs can currently run 19 
at once.
   # test-fork-ci-status.cjs checks that number, counting `max-parallel`, and 
that no test job
   # waits on another. GitHub Actions has no workflow-level job cap, so a 
change that pushes the
   # number past 20 must lower `max-parallel` or serialize jobs through 
`needs`, and make that check
@@ -2115,9 +2100,8 @@ jobs:
     needs:
       - test-spark-client-and-hadoop-common
       - test-utilities-streamer
-      - test-utilities-rest
+      - test-utilities-rest-hadoop-mr-java-client
       - test-common-and-other-modules
-      - test-hudi-hadoop-mr-and-hudi-java-client
       - test-spark-java17-ut-functional
       - test-spark-java17-ut-other
       - test-spark-java17-ut-fat-and-ftc
diff --git a/scripts/jacoco/generate_merged_coverage_report.sh 
b/scripts/jacoco/generate_merged_coverage_report.sh
index a2b8b67276ff..882a2f704653 100755
--- a/scripts/jacoco/generate_merged_coverage_report.sh
+++ b/scripts/jacoco/generate_merged_coverage_report.sh
@@ -17,30 +17,63 @@
 # specific language governing permissions and limitations
 # under the License.
 #
-# Generates a merged JaCoCo coverage report from all exec files in the 
workspace.
+# Generates a merged JaCoCo coverage report from the exec files in the 
workspace.
 # Produces jacoco-report.xml suitable for upload to Codecov.
 #
-# Usage: ./scripts/jacoco/generate_merged_coverage_report.sh [workspace_dir]
+# Usage: ./scripts/jacoco/generate_merged_coverage_report.sh [workspace_dir 
[exec_dir ...]]
 #   workspace_dir: root of the source tree (defaults to current directory)
+#   exec_dir: only the exec files under these directories go into the report 
(defaults to
+#     workspace_dir). A job that uploads one report per Codecov flag calls 
this once per flag,
+#     uploading each report before the next.
+# Environment:
+#   COVERAGE_MODULE_DIRS: optional newline-separated module directories. When 
set, only these
+#     modules' classes and sources go into the report; otherwise every module 
in the workspace does.
 
 set -euo pipefail
 
 WORKSPACE_DIR="${1:-.}"
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+EXEC_DIRS=("${@:2}")
+if [ ${#EXEC_DIRS[@]} -eq 0 ]; then
+  EXEC_DIRS=("$WORKSPACE_DIR")
+fi
+
+# A report left by an earlier call must not be uploaded again under another 
flag.
+rm -f "$WORKSPACE_DIR/jacoco-report.xml"
 
 # Skip if no exec files exist (e.g., tests were skipped for this matrix entry)
-if ! find "$WORKSPACE_DIR" -path '*/jacoco-agent/*.exec' -type f | grep -q .; 
then
+if ! find "${EXEC_DIRS[@]}" -path '*/jacoco-agent/*.exec' -type f | grep -q .; 
then
   echo "No JaCoCo exec files found — skipping coverage report"
   exit 0
 fi
 
-"$SCRIPT_DIR/download_jacoco.sh"
-"$SCRIPT_DIR/merge_jacoco_exec_files.sh" jacoco-lib/lib/jacococli.jar 
"$WORKSPACE_DIR"
+# An earlier call already downloaded the CLI, and unzip would stop to ask 
before overwriting it.
+if [ ! -f jacoco-lib/lib/jacococli.jar ]; then
+  "$SCRIPT_DIR/download_jacoco.sh"
+fi
+"$SCRIPT_DIR/merge_jacoco_exec_files.sh" jacoco-lib/lib/jacococli.jar 
"${EXEC_DIRS[@]}"
 mv merged-jacoco.exec jacoco.exec
 
+# Recreated on every call, so a report never picks up classes or sources 
copied for an earlier one.
+rm -rf aggregate-classes aggregate-sources
 mkdir -p aggregate-classes aggregate-sources
-find "$WORKSPACE_DIR" -path '*/target/classes' -type d | while read d; do cp 
-r "$d"/. aggregate-classes/ 2>/dev/null; done
-find "$WORKSPACE_DIR" -path '*/src/main/java' -type d | while read d; do cp -r 
"$d"/. aggregate-sources/ 2>/dev/null; done
-find "$WORKSPACE_DIR" -path '*/src/main/scala' -type d | while read d; do cp 
-r "$d"/. aggregate-sources/ 2>/dev/null; done
+if [ -n "${COVERAGE_MODULE_DIRS:-}" ]; then
+  while read -r d; do
+    # Skips pom-only modules, which have no classes, and any line that is not 
a module directory.
+    if [ ! -d "$d/target/classes" ]; then
+      continue
+    fi
+    cp -r "$d/target/classes"/. aggregate-classes/
+    for src in "$d/src/main/java" "$d/src/main/scala"; do
+      if [ -d "$src" ]; then
+        cp -r "$src"/. aggregate-sources/
+      fi
+    done
+  done <<< "$COVERAGE_MODULE_DIRS"
+else
+  find "$WORKSPACE_DIR" -path '*/target/classes' -type d | while read d; do cp 
-r "$d"/. aggregate-classes/ 2>/dev/null; done
+  find "$WORKSPACE_DIR" -path '*/src/main/java' -type d | while read d; do cp 
-r "$d"/. aggregate-sources/ 2>/dev/null; done
+  find "$WORKSPACE_DIR" -path '*/src/main/scala' -type d | while read d; do cp 
-r "$d"/. aggregate-sources/ 2>/dev/null; done
+fi
 
 "$SCRIPT_DIR/generate_jacoco_coverage_report.sh" jacoco-lib/lib/jacococli.jar 
"$WORKSPACE_DIR"
diff --git a/scripts/jacoco/merge_jacoco_exec_files.sh 
b/scripts/jacoco/merge_jacoco_exec_files.sh
index 4f6b4048b935..a3f7ec086fa6 100755
--- a/scripts/jacoco/merge_jacoco_exec_files.sh
+++ b/scripts/jacoco/merge_jacoco_exec_files.sh
@@ -18,13 +18,23 @@
 # under the License.
 #
 /bin/bash --version
-shopt -s globstar
+# nullglob drops a directory with no exec files instead of passing its pattern 
to merge.
+shopt -s globstar nullglob
 echo "Jacoco CLI jar: $1"
-echo "Hudi source directory: $2"
+echo "Hudi source directories: ${*:2}"
+exec_files=()
+for dir in "${@:2}"; do
+  exec_files+=($dir/**/jacoco-agent/**/*.exec)
+done
+# Without this check, merge with no input files succeeds and writes an empty 
exec file.
+if [[ ${#exec_files[@]} -eq 0 ]]; then
+  echo "No JaCoCo exec files found under: ${*:2}" >&2
+  exit 1
+fi
 retry_count=0
 while [[ $retry_count -lt 3 ]]; do
-  ls -l $2/**/jacoco-agent/**/*.exec
-  java -jar $1 merge $2/**/jacoco-agent/**/*.exec --destfile merged-jacoco.exec
+  ls -l "${exec_files[@]}"
+  java -jar $1 merge "${exec_files[@]}" --destfile merged-jacoco.exec
   exit_status=$?
 
   if [[ $exit_status -eq 0 ]]; then
@@ -35,3 +45,5 @@ while [[ $retry_count -lt 3 ]]; do
   retry_count=$((retry_count + 1))
   sleep 10
 done
+echo "Jacoco merge failed after $retry_count attempts" >&2
+exit 1

Reply via email to