This is an automated email from the ASF dual-hosted git repository.
JingsongLi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/paimon.git
The following commit(s) were added to refs/heads/master by this push:
new 301d7b307f [ci] Run affected test suites through reusable workflows
(#9787)
301d7b307f is described below
commit 301d7b307f39cb40d6a2e4e64c26289f2ba0e570
Author: Jingsong Lee <[email protected]>
AuthorDate: Sun Sep 13 17:55:19 2026 +0800
[ci] Run affected test suites through reusable workflows (#9787)
---
.github/CI.md | 83 +++++++
.github/workflows/{docs-tests.yml => ci-docs.yml} | 15 +-
.github/workflows/ci-java.yml | 73 +++++++
.../{check-licensing.yml => ci-licensing.yml} | 31 ++-
.../{paimon-python-checks.yml => ci-python.yml} | 43 ++--
.github/workflows/ci.yml | 112 ++++++++++
.github/workflows/e2e-tests-flink-1.x.yml | 80 -------
.github/workflows/e2e-tests-flink-2.x-jdk11.yml | 81 -------
.github/workflows/file-size-check.yml | 48 -----
.../{stale-pr.yml => maintenance-stale-pr.yml} | 2 +-
...pshot-jdk17.yml => publish-snapshot-spark4.yml} | 14 +-
.../{publish_snapshot.yml => publish-snapshot.yml} | 14 +-
.github/workflows/release-java.yml | 2 +-
.github/workflows/release-python-publish.yml | 2 +-
.github/workflows/release-python.yml | 2 +-
.github/workflows/utcase-eslib.yml | 73 -------
.github/workflows/utcase-full-text.yml | 61 ------
.github/workflows/utitcase-flink-1.x-common.yml | 75 -------
.github/workflows/utitcase-flink-1.x-others.yml | 75 -------
.github/workflows/utitcase-flink-2.x-jdk11.yml | 76 -------
.github/workflows/utitcase-iceberg-ga.yml | 66 ------
.github/workflows/utitcase-jdk11.yml | 72 -------
.github/workflows/utitcase-spark-3.x.yml | 79 -------
.github/workflows/utitcase-spark-4.x.yml | 76 -------
.github/workflows/utitcase.yml | 78 -------
docs/docs/project/verifying-a-release-candidate.md | 2 +-
tools/ci/plan.py | 180 ++++++++++++++++
tools/ci/run-java-tests.sh | 129 +++++++++++
tools/ci/test_java_commands.py | 112 ++++++++++
tools/ci/test_plan.py | 238 +++++++++++++++++++++
tools/ci/test_result.py | 56 +++++
{.github/workflows => tools/ci}/utils.sh | 0
32 files changed, 1051 insertions(+), 999 deletions(-)
diff --git a/.github/CI.md b/.github/CI.md
new file mode 100644
index 0000000000..d0383aa5ff
--- /dev/null
+++ b/.github/CI.md
@@ -0,0 +1,83 @@
+# Continuous integration
+
+`workflows/ci.yml` is the entry point for pull requests, pushes to any branch
or
+tag, and manual full runs. Contributors can validate changes in their own forks
+before opening an upstream PR, once GitHub Actions is enabled in the fork.
+Both push and PR runs select tests by changed paths. Updating a PR cancels its
+previous CI run; push and publishing runs are not cancelled in progress.
+
+The entry point calls `ci-java.yml`, `ci-python.yml`, `ci-docs.yml` and
+`ci-licensing.yml`. `publish-snapshot*.yml`, `release*.yml` and
+`maintenance-stale-pr.yml` have separate publishing or maintenance triggers.
+
+## Selecting tests
+
+`tools/ci/plan.py` is the single source of path routing and the Java matrix.
+Its selection is shown in the workflow run summary. It uses the PR merge-base
+diff, or the entire before/after push diff. Both sides of a rename are
included.
+Diffs are read from Git without a changed-file API limit. If a complete diff is
+unavailable (including a new branch), CI runs all suites. Manual dispatch also
+runs everything.
+
+PRs check changed files against a 1 MiB size limit. When the PR diff is
unavailable,
+this check is skipped with a warning in the log and run summary so that
existing
+oversized files do not block the full CI run.
+
+| Changed paths | Java tests |
+| --- | --- |
+| `paimon-spark/**` | Spark 3/4, core and integrations, Flink 1/Spark E2E |
+| `paimon-flink/**` | Flink 1/2, core and integrations, both E2E groups,
Iceberg GA |
+| `paimon-e2e-tests/**` | Both E2E groups |
+| `paimon-eslib/**` | ESLib |
+| `paimon-iceberg/**` | Core and integrations, Iceberg GA |
+| `paimon-full-text/**` | Core and integrations, full text |
+| Shared modules, root/build configuration, unclassified source paths | All
Java groups |
+
+Java changes retain the artifact licensing check. Changes to `docs/**` build
the
+website; other Markdown-only changes skip expensive jobs. Changes to
+`paimon-python/**` and Java/Python `JavaPy*` test fixtures run Python checks.
+Root POM changes also run Python to validate its Java build. Mixed changes take
+the union of all required groups. Changes to CI routing or execution scripts
+validate every consumer, including Python and documentation.
+
+The core and integrations jobs deliberately still run for Spark/Flink changes:
+`paimon-spark-common` owns tests outside the Spark connector test jobs,
+`paimon-docs` tests engine configuration documentation, and Hive/Iceberg tests
+depend on Flink. Do not remove these jobs by looking only at the workflow name.
+`SparkE2eTest` runs in the JDK 8 E2E job and is disabled on JDK 11.
+
+## Builds and caches
+
+`tools/ci/run-java-tests.sh` owns Java build/test commands. Each group installs
+its selected modules and their dependencies with `-pl ... -am -DskipTests`,
then
+tests only its modules. The test invocation does not clean away the first build
+or run both `test` and `verify`. Normal Maven checks remain enabled.
+
+Java dependency caches are isolated by suite, JDK and Scala. This prevents a
+small group's cache from permanently starving other groups of their
dependencies.
+Failed Java jobs upload Surefire reports with a seven-day retention period.
+Python builds only the Java modules used by the mixed-language tests and their
+dependencies; its Python version and compatibility coverage is preserved.
+
+## Required checks and maintenance
+
+Configure branch protection to require **CI result** when adopting this layout.
+Remove any requirements referring to the retired individual workflow/job names.
+The result job always runs and requires the planner and every selected group to
+succeed. Unselected groups must be skipped. A planner failure, failed test,
+unexpected skip or cancellation cannot produce a green result.
+
+Every run validates the workflow syntax with pinned actionlint 1.7.11 and runs
+the routing/command tests before starting expensive jobs. Locally:
+
+```sh
+python3 -m unittest discover -s tools/ci -p 'test_*.py'
+actionlint -shellcheck= -pyflakes=
+CI_DRY_RUN=true bash tools/ci/run-java-tests.sh spark3 2.12 8
+```
+
+When adding an engine version, update both the matrix in `plan.py` and the
+module lists in `run-java-tests.sh`, including the core exclusions where
needed.
+Add behavioral tests for any new routing boundary and retain shared-dependency
+fallbacks. Use a manual CI run to validate the complete matrix before adopting
+changes to build profiles or test ownership.
diff --git a/.github/workflows/docs-tests.yml b/.github/workflows/ci-docs.yml
similarity index 84%
rename from .github/workflows/docs-tests.yml
rename to .github/workflows/ci-docs.yml
index d6d1df1987..e9d77e644b 100644
--- a/.github/workflows/docs-tests.yml
+++ b/.github/workflows/ci-docs.yml
@@ -13,16 +13,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-name: docs-tests
+name: CI / Documentation
+
on:
- push:
- pull_request:
- paths:
- - 'docs/**'
+ workflow_call:
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
+permissions:
+ contents: read
jobs:
build:
@@ -31,7 +28,7 @@ jobs:
steps:
- name: Checkout code
- uses: actions/checkout@v4
+ uses: actions/checkout@v6
- name: Setup Node.js
uses: actions/setup-node@v4
diff --git a/.github/workflows/ci-java.yml b/.github/workflows/ci-java.yml
new file mode 100644
index 0000000000..1283bbe19f
--- /dev/null
+++ b/.github/workflows/ci-java.yml
@@ -0,0 +1,73 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: CI / Java
+
+on:
+ workflow_call:
+ inputs:
+ matrix:
+ description: Java suites selected by tools/ci/plan.py
+ required: true
+ type: string
+
+permissions:
+ contents: read
+
+env:
+ MAVEN_OPTS: -Xmx4096m -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
+
+jobs:
+ test:
+ name: ${{ matrix.name }}
+ runs-on: ubuntu-latest
+ timeout-minutes: ${{ matrix.timeout }}
+ strategy:
+ # Keep independent results when one engine fails.
+ fail-fast: false
+ matrix: ${{ fromJSON(inputs.matrix) }}
+ steps:
+ - uses: actions/checkout@v6
+ with:
+ persist-credentials: false
+ - name: Set up JDK ${{ matrix.java }}
+ uses: actions/setup-java@v5
+ with:
+ java-version: ${{ matrix.java }}
+ distribution: temurin
+ - name: Cache Maven dependencies
+ uses: actions/cache@v5
+ with:
+ path: ~/.m2/repository
+ key: ci-maven-${{ runner.os }}-${{ runner.arch }}-${{ matrix.suite
}}-jdk${{ matrix.java }}-scala${{ matrix.scala }}-${{ hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ci-maven-${{ runner.os }}-${{ runner.arch }}-${{ matrix.suite
}}-jdk${{ matrix.java }}-scala${{ matrix.scala }}-
+ - name: Build dependencies and run tests
+ env:
+ CI_SUITE: ${{ matrix.suite }}
+ CI_SCALA: ${{ matrix.scala }}
+ CI_JAVA: ${{ matrix.java }}
+ run: bash tools/ci/run-java-tests.sh "$CI_SUITE" "$CI_SCALA" "$CI_JAVA"
+ - name: Upload failed test reports
+ if: failure()
+ uses: actions/upload-artifact@v6
+ with:
+ name: test-reports-${{ matrix.suite }}-jdk${{ matrix.java
}}-scala${{ matrix.scala }}
+ path: '**/target/surefire-reports/**'
+ if-no-files-found: ignore
+ retention-days: 7
diff --git a/.github/workflows/check-licensing.yml
b/.github/workflows/ci-licensing.yml
similarity index 82%
rename from .github/workflows/check-licensing.yml
rename to .github/workflows/ci-licensing.yml
index 207cdd13fa..3857c2022e 100644
--- a/.github/workflows/check-licensing.yml
+++ b/.github/workflows/ci-licensing.yml
@@ -13,29 +13,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-name: Check licensing
+name: CI / Licensing
on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
+ workflow_call:
+
+permissions:
+ contents: read
env:
MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
jobs:
- build:
+ check:
+ name: Artifact licensing
+ timeout-minutes: 60
runs-on: ubuntu-latest
env:
MVN_COMMON_OPTIONS: -U -B --no-transfer-progress
@@ -50,6 +42,13 @@ jobs:
with:
java-version: 8
distribution: 'temurin'
+ - name: Cache Maven dependencies
+ uses: actions/cache@v5
+ with:
+ path: ~/.m2/repository
+ key: ci-maven-licensing-${{ runner.os }}-${{ runner.arch }}-${{
hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ci-maven-licensing-${{ runner.os }}-${{ runner.arch }}-
- name: Build
run: |
set -o pipefail
diff --git a/.github/workflows/paimon-python-checks.yml
b/.github/workflows/ci-python.yml
old mode 100755
new mode 100644
similarity index 95%
rename from .github/workflows/paimon-python-checks.yml
rename to .github/workflows/ci-python.yml
index ac48f730fc..daffd050ad
--- a/.github/workflows/paimon-python-checks.yml
+++ b/.github/workflows/ci-python.yml
@@ -16,19 +16,13 @@
# limitations under the License.
################################################################################
-name: Python Check Code Style and Test
+name: CI / Python
on:
- push:
- paths:
- - 'paimon-python/**'
- - '!**/*.md'
- - '.github/workflows/paimon-python-checks.yml'
- pull_request:
- paths:
- - 'paimon-python/**'
- - '!**/*.md'
- - '.github/workflows/paimon-python-checks.yml'
+ workflow_call:
+
+permissions:
+ contents: read
env:
JDK_VERSION: 8
@@ -37,13 +31,11 @@ env:
PYPAIMON_RUST_REV: be5456e96b326939570b6ef5d63acddda3311c4f
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
jobs:
# Lint + test on selected versions only (3.6 / 3.7 low end, 3.10 / 3.11 /
3.12 / 3.13 current), not every Python version.
- lint-python:
+ test:
+ name: Tests / Python ${{ matrix.python-version }}
+ timeout-minutes: 90
runs-on: ubuntu-latest
container: "python:${{ matrix.python-version }}-slim"
strategy:
@@ -61,6 +53,14 @@ jobs:
java-version: ${{ env.JDK_VERSION }}
distribution: 'temurin'
+ - name: Cache Maven dependencies
+ uses: actions/cache@v5
+ with:
+ path: ~/.m2/repository
+ key: ci-maven-python-${{ runner.os }}-${{ runner.arch }}-${{
hashFiles('**/pom.xml') }}
+ restore-keys: |
+ ci-maven-python-${{ runner.os }}-${{ runner.arch }}-
+
- name: Set up Maven
uses: stCarolas/setup-maven@v5
with:
@@ -116,7 +116,8 @@ jobs:
- name: Build Java
run: |
echo "Start compiling modules"
- mvn -T 2C -B -ntp clean install -DskipTests
+ mvn -T 2C -B -ntp install -DskipTests \
+ -pl
paimon-core,paimon-lance,paimon-vortex/paimon-vortex-format,paimon-full-text,paimon-lumina,paimon-vector
-am
- name: Install Python dependencies
shell: bash
@@ -184,7 +185,9 @@ jobs:
fi
./paimon-python/dev/lint-python.sh -e pytest_torch
- torch_test:
+ torch:
+ name: PyTorch / Python 3.10
+ timeout-minutes: 60
runs-on: ubuntu-latest
container: "python:3.10-slim"
@@ -219,7 +222,9 @@ jobs:
./paimon-python/dev/lint-python.sh -i pytest_torch
# One job: check dev/requirements.txt on each Python version in sequence,
then Ray version test on 3.10.
- requirement_version_compatible_test:
+ compatibility:
+ name: Requirements and Ray compatibility
+ timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- name: Checkout code
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 0000000000..f250062d4b
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,112 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+name: CI
+
+on:
+ pull_request:
+ push:
+ workflow_dispatch:
+
+permissions:
+ contents: read
+
+concurrency:
+ group: ci-${{ github.event_name }}-${{ github.event.pull_request.number ||
github.ref }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+ plan:
+ name: Select tests and validate CI
+ runs-on: ubuntu-latest
+ timeout-minutes: 10
+ outputs:
+ java: ${{ steps.plan.outputs.java }}
+ matrix: ${{ steps.plan.outputs.matrix }}
+ python: ${{ steps.plan.outputs.python }}
+ docs: ${{ steps.plan.outputs.docs }}
+ licensing: ${{ steps.plan.outputs.licensing }}
+ steps:
+ - uses: actions/checkout@v6
+ with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Test CI selection and Maven commands
+ run: python3 -m unittest discover -s tools/ci -p 'test_*.py'
+ - name: Validate workflow syntax
+ env:
+ GH_TOKEN: ${{ github.token }}
+ run: |
+ gh release download v1.7.11 --repo rhysd/actionlint \
+ --pattern 'actionlint_1.7.11_linux_amd64.tar.gz' --dir
"$RUNNER_TEMP"
+ tar -xzf "$RUNNER_TEMP/actionlint_1.7.11_linux_amd64.tar.gz" -C
"$RUNNER_TEMP" actionlint
+ "$RUNNER_TEMP/actionlint" -shellcheck= -pyflakes=
+ - name: Select affected tests and check changed file sizes
+ id: plan
+ run: python3 tools/ci/plan.py
+
+ java:
+ name: Java
+ needs: plan
+ if: needs.plan.outputs.java == 'true'
+ uses: ./.github/workflows/ci-java.yml
+ with:
+ matrix: ${{ needs.plan.outputs.matrix }}
+
+ python:
+ name: Python
+ needs: plan
+ if: needs.plan.outputs.python == 'true'
+ uses: ./.github/workflows/ci-python.yml
+
+ docs:
+ name: Documentation
+ needs: plan
+ if: needs.plan.outputs.docs == 'true'
+ uses: ./.github/workflows/ci-docs.yml
+
+ licensing:
+ name: Licensing
+ needs: plan
+ if: needs.plan.outputs.licensing == 'true'
+ uses: ./.github/workflows/ci-licensing.yml
+
+ result:
+ # Use this stable check for branch protection, not individual matrix jobs.
+ name: CI result
+ if: always()
+ needs: [plan, java, python, docs, licensing]
+ runs-on: ubuntu-latest
+ timeout-minutes: 5
+ steps:
+ - name: Require all selected jobs to pass
+ env:
+ RESULTS: ${{ toJSON(needs) }}
+ run: |
+ python3 - <<'PYTHON'
+ import json
+ import os
+ results = json.loads(os.environ['RESULTS'])
+ if results['plan']['result'] != 'success':
+ raise SystemExit('CI selection or validation failed')
+ selected = results['plan']['outputs']
+ for name in ('java', 'python', 'docs', 'licensing'):
+ expected = 'success' if selected[name] == 'true' else 'skipped'
+ if results[name]['result'] != expected:
+ raise SystemExit(name + ': ' + results[name]['result'] + ',
expected ' + expected)
+ PYTHON
diff --git a/.github/workflows/e2e-tests-flink-1.x.yml
b/.github/workflows/e2e-tests-flink-1.x.yml
deleted file mode 100644
index 3d7c25b600..0000000000
--- a/.github/workflows/e2e-tests-flink-1.x.yml
+++ /dev/null
@@ -1,80 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: End to End Tests Flink 1.x
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 8
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event.pull_request.number ||
github.ref }}
- cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
- timeout-minutes: 60
- strategy:
- fail-fast: true
- matrix:
- # Only Test Latest Version
- flink_version: [ '1.20' ]
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Flink
- run: mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3 -pl
paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }}
-
- - name: Test Flink
- run: |
- # run tests with random timezone to find out timezone related bugs
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- profile="flink-${{ matrix.flink_version }}"
- if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1]
}}" ]; then
- mvn -T 1C -B -ntp test -Pflink1,spark3 -pl paimon-e2e-tests
-Duser.timezone=$jvm_timezone
- else
- mvn -T 1C -B -ntp test -Pflink1,spark3 -pl paimon-e2e-tests
-Duser.timezone=$jvm_timezone -P${profile}
- fi
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/e2e-tests-flink-2.x-jdk11.yml
b/.github/workflows/e2e-tests-flink-2.x-jdk11.yml
deleted file mode 100644
index 548916d106..0000000000
--- a/.github/workflows/e2e-tests-flink-2.x-jdk11.yml
+++ /dev/null
@@ -1,81 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: End to End Tests Flink 2.x on JDK 11
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 11
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event.pull_request.number ||
github.ref }}
- cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
- timeout-minutes: 60
- strategy:
- fail-fast: true
- matrix:
- # Only Test Latest Version
- flink_version: [ '2.2' ]
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Flink
- run: |
- mvn -T 2C -B -ntp clean install -DskipTests -Pflink2,spark3 -pl
paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11
-
- - name: Test Flink
- run: |
- # run tests with random timezone to find out timezone related bugs
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- profile="flink-${{ matrix.flink_version }}"
- if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1]
}}" ]; then
- mvn -T 1C -B -ntp test -Pflink2,spark3 -pl paimon-e2e-tests
-Duser.timezone=$jvm_timezone -Pjava11
- else
- mvn -T 1C -B -ntp test -Pflink2,spark3 -pl paimon-e2e-tests
-Duser.timezone=$jvm_timezone -P${profile},java11
- fi
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/file-size-check.yml
b/.github/workflows/file-size-check.yml
deleted file mode 100644
index 9b4f073b0a..0000000000
--- a/.github/workflows/file-size-check.yml
+++ /dev/null
@@ -1,48 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: Check File Size
-
-on:
- pull_request:
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event.pull_request.number ||
github.ref }}
- cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
- check-file-size:
- runs-on: ubuntu-latest
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
- with:
- fetch-depth: 0
-
- - name: Check file size
- run: |
- files=$(git diff --name-only ${{ github.event.pull_request.base.sha
}}...${{ github.event.pull_request.head.sha }})
- for file in $files; do
- if [ -f "$file" ]; then
- size=$(ls -l "$file" | awk '{print $5}')
- if [ "$size" -gt 1048576 ]; then
- echo "Error: File $file is larger than 1MB ($size bytes)."
- exit 1
- fi
- fi
- done
diff --git a/.github/workflows/stale-pr.yml
b/.github/workflows/maintenance-stale-pr.yml
similarity index 98%
rename from .github/workflows/stale-pr.yml
rename to .github/workflows/maintenance-stale-pr.yml
index ca5c3dae46..252f8c7237 100644
--- a/.github/workflows/stale-pr.yml
+++ b/.github/workflows/maintenance-stale-pr.yml
@@ -19,7 +19,7 @@
#
# See [email protected] "Stale PR cleanup for Paimon" thread.
-name: Stale PR reminder
+name: Maintenance / Stale PR reminder
on:
schedule:
diff --git a/.github/workflows/publish_snapshot-jdk17.yml
b/.github/workflows/publish-snapshot-spark4.yml
similarity index 89%
rename from .github/workflows/publish_snapshot-jdk17.yml
rename to .github/workflows/publish-snapshot-spark4.yml
index 5835a4d14b..c1b8eabeb0 100644
--- a/.github/workflows/publish_snapshot-jdk17.yml
+++ b/.github/workflows/publish-snapshot-spark4.yml
@@ -16,7 +16,7 @@
# limitations under the License.
################################################################################
-name: Publish Snapshot on JDK 17
+name: Publish / Snapshot / Spark 4
on:
schedule:
@@ -29,12 +29,16 @@ env:
MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
+ group: publish-snapshot-spark4-${{ github.ref }}
+ cancel-in-progress: false
+
+permissions:
+ contents: read
jobs:
publish-snapshot:
if: github.repository == 'apache/paimon'
+ timeout-minutes: 120
runs-on: ubuntu-latest
steps:
- name: Checkout code
@@ -48,9 +52,9 @@ jobs:
uses: actions/cache@v5
with:
path: ~/.m2/repository
- key: snapshot-maven-${{ hashFiles('**/pom.xml') }}
+ key: snapshot-maven-spark4-${{ runner.os }}-${{ runner.arch }}-${{
hashFiles('**/pom.xml') }}
restore-keys: |
- snapshot-maven-
+ snapshot-maven-spark4-${{ runner.os }}-${{ runner.arch }}-
- name: Publish snapshot
env:
ASF_USERNAME: ${{ secrets.NEXUS_USER }}
diff --git a/.github/workflows/publish_snapshot.yml
b/.github/workflows/publish-snapshot.yml
similarity index 89%
rename from .github/workflows/publish_snapshot.yml
rename to .github/workflows/publish-snapshot.yml
index b1a9335dc4..b1f201593c 100644
--- a/.github/workflows/publish_snapshot.yml
+++ b/.github/workflows/publish-snapshot.yml
@@ -16,7 +16,7 @@
# limitations under the License.
################################################################################
-name: Publish Snapshot
+name: Publish / Snapshot / JDK 8
on:
schedule:
@@ -29,12 +29,16 @@ env:
MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
+ group: publish-snapshot-${{ github.ref }}
+ cancel-in-progress: false
+
+permissions:
+ contents: read
jobs:
publish-snapshot:
if: github.repository == 'apache/paimon'
+ timeout-minutes: 120
runs-on: ubuntu-latest
steps:
- name: Checkout code
@@ -50,9 +54,9 @@ jobs:
uses: actions/cache@v5
with:
path: ~/.m2/repository
- key: snapshot-maven-${{ hashFiles('**/pom.xml') }}
+ key: snapshot-maven-jdk8-${{ runner.os }}-${{ runner.arch }}-${{
hashFiles('**/pom.xml') }}
restore-keys: |
- snapshot-maven-
+ snapshot-maven-jdk8-${{ runner.os }}-${{ runner.arch }}-
- name: Publish snapshot
env:
ASF_USERNAME: ${{ secrets.NEXUS_USER }}
diff --git a/.github/workflows/release-java.yml
b/.github/workflows/release-java.yml
index 12160e9808..fe6ed56e0f 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -16,7 +16,7 @@
# limitations under the License.
################################################################################
-name: Release Java
+name: Release / Java packages
on:
workflow_call:
diff --git a/.github/workflows/release-python-publish.yml
b/.github/workflows/release-python-publish.yml
index 9ae869a590..a153ff2d4c 100644
--- a/.github/workflows/release-python-publish.yml
+++ b/.github/workflows/release-python-publish.yml
@@ -16,7 +16,7 @@
# limitations under the License.
################################################################################
-name: Release PyPaimon Publish
+name: Release / Python publish
on:
workflow_call:
diff --git a/.github/workflows/release-python.yml
b/.github/workflows/release-python.yml
index 0dc5112821..c8d3ef32e2 100644
--- a/.github/workflows/release-python.yml
+++ b/.github/workflows/release-python.yml
@@ -19,7 +19,7 @@
# Build the PyPaimon source distribution and universal wheel. Publishing is
# handled by release-python-publish.yml after this package job passes.
-name: Release PyPaimon Package
+name: Release / Python packages
on:
workflow_call:
diff --git a/.github/workflows/utcase-eslib.yml
b/.github/workflows/utcase-eslib.yml
deleted file mode 100644
index a5dc7d9d20..0000000000
--- a/.github/workflows/utcase-eslib.yml
+++ /dev/null
@@ -1,73 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase ESLib on JDK 11
-
-on:
- push:
- paths:
- - 'paimon-api/**'
- - 'paimon-common/**'
- - 'paimon-eslib/**'
- - 'paimon-test-utils/**'
- - 'pom.xml'
- - '.github/workflows/utcase-eslib.yml'
- pull_request:
- paths:
- - 'paimon-api/**'
- - 'paimon-common/**'
- - 'paimon-eslib/**'
- - 'paimon-test-utils/**'
- - 'pom.xml'
- - '.github/workflows/utcase-eslib.yml'
-
-permissions:
- contents: read
-
-env:
- JDK_VERSION: 11
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- eslib_test:
- runs-on: ubuntu-latest
- timeout-minutes: 30
-
- steps:
- - name: Checkout code
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
- with:
- persist-credentials: false
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build ESLib dependencies
- run: mvn -T 2C -B -ntp clean install -DskipTests -Ppaimon-eslib -pl
paimon-eslib -am
-
- - name: Test ESLib
- run: mvn -B -ntp verify -Ppaimon-eslib -pl paimon-eslib
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utcase-full-text.yml
b/.github/workflows/utcase-full-text.yml
deleted file mode 100644
index c9904529cf..0000000000
--- a/.github/workflows/utcase-full-text.yml
+++ /dev/null
@@ -1,61 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase Full Text
-
-on:
- push:
- paths:
- - 'paimon-full-text/**'
- - 'pom.xml'
- - '.github/workflows/utcase-full-text.yml'
- pull_request:
- paths:
- - 'paimon-full-text/**'
- - 'pom.xml'
- - '.github/workflows/utcase-full-text.yml'
-
-env:
- JDK_VERSION: 8
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- full_text_test:
- runs-on: ubuntu-latest
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build and test full-text module
- timeout-minutes: 30
- run: |
- mvn -T 2C -B -ntp clean install -DskipTests
- mvn -B -ntp verify -pl paimon-full-text -Dcheckstyle.skip=true
-Dspotless.check.skip=true
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utitcase-flink-1.x-common.yml
b/.github/workflows/utitcase-flink-1.x-common.yml
deleted file mode 100644
index 59d124804d..0000000000
--- a/.github/workflows/utitcase-flink-1.x-common.yml
+++ /dev/null
@@ -1,75 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-name: UTCase and ITCase Flink 1.x Common
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 8
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
- timeout-minutes: 100
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Flink
- run: |
- COMPILE_MODULE="org.apache.paimon:paimon-flink-common"
- echo "Start compiling modules: $COMPILE_MODULE"
- mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3 -pl
"${COMPILE_MODULE}" -am
-
- - name: Test Flink
- run: |
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- TEST_MODULE="org.apache.paimon:paimon-flink-common"
- echo "Start testing module: $TEST_MODULE"
- mvn -T 2C -B -ntp test verify -Pflink1,spark3 -pl "${TEST_MODULE}"
-Duser.timezone=$jvm_timezone
- echo "All modules tested"
- env:
- MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2
diff --git a/.github/workflows/utitcase-flink-1.x-others.yml
b/.github/workflows/utitcase-flink-1.x-others.yml
deleted file mode 100644
index d042f1a8ed..0000000000
--- a/.github/workflows/utitcase-flink-1.x-others.yml
+++ /dev/null
@@ -1,75 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-name: UTCase and ITCase Flink 1.x Others
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 8
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
- timeout-minutes: 60
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Flink
- run: |
- mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3
-
- - name: Test Flink
- run: |
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- test_modules=""
- for suffix in cdc 1.16 1.17 1.18 1.19 1.20; do
- test_modules+="org.apache.paimon:paimon-flink-${suffix},"
- done
- test_modules="${test_modules%,}"
- mvn -T 2C -B -ntp test verify -Pflink1,spark3 -pl "${test_modules}"
-Duser.timezone=$jvm_timezone
- env:
- MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2
diff --git a/.github/workflows/utitcase-flink-2.x-jdk11.yml
b/.github/workflows/utitcase-flink-2.x-jdk11.yml
deleted file mode 100644
index b1b6005867..0000000000
--- a/.github/workflows/utitcase-flink-2.x-jdk11.yml
+++ /dev/null
@@ -1,76 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase and ITCase Flink 2.x on JDK 11
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 11
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event.pull_request.number ||
github.ref }}
- cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
-jobs:
- build:
- runs-on: ubuntu-latest
- timeout-minutes: 120
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Flink
- run: |
- mvn -T 2C -B -ntp clean install -DskipTests -Pflink2,spark3
-
- - name: Test Flink
- run: |
- # run tests with random timezone to find out timezone related bugs
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- test_modules=""
- for suffix in 2.0 2.1 2.2 common; do
- test_modules+="org.apache.paimon:paimon-flink-${suffix},"
- done
- test_modules="${test_modules%,}"
- mvn -T 2C -B -ntp test verify -Pflink2,spark3 -pl "${test_modules}"
-Duser.timezone=$jvm_timezone
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utitcase-iceberg-ga.yml
b/.github/workflows/utitcase-iceberg-ga.yml
deleted file mode 100644
index deca2434fc..0000000000
--- a/.github/workflows/utitcase-iceberg-ga.yml
+++ /dev/null
@@ -1,66 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase Iceberg GA row lineage on JDK 17
-
-on:
- push:
- paths:
- - 'paimon-iceberg/**'
- - 'paimon-core/**'
- - 'paimon-common/**'
- - 'paimon-api/**'
- - 'paimon-format/**'
- - 'pom.xml'
- - '.github/workflows/utitcase-iceberg-ga.yml'
- pull_request:
- paths:
- - 'paimon-iceberg/**'
- - 'paimon-core/**'
- - 'paimon-common/**'
- - 'paimon-api/**'
- - 'paimon-format/**'
- - 'pom.xml'
- - '.github/workflows/utitcase-iceberg-ga.yml'
-
-env:
- JDK_VERSION: 17
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build:
- runs-on: ubuntu-latest
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
- - name: Build
- run: mvn -T 1C -B -ntp clean install -DskipTests -pl paimon-iceberg
-am -Ppaimon-iceberg,iceberg-ga
- - name: Test against GA Iceberg
- run: mvn -B -ntp test -pl paimon-iceberg -Ppaimon-iceberg,iceberg-ga
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utitcase-jdk11.yml
b/.github/workflows/utitcase-jdk11.yml
deleted file mode 100644
index a6ae09a52b..0000000000
--- a/.github/workflows/utitcase-jdk11.yml
+++ /dev/null
@@ -1,72 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase and ITCase Others on JDK 11
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 11
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build:
- runs-on: ubuntu-latest
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
- - name: Build
- run: mvn -T 1C -B -ntp clean install -DskipTests -Pflink1,spark3
- - name: Test
- run: |
- # run tests with random timezone to find out timezone related bugs
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
-
test_modules="!paimon-e2e-tests,!org.apache.paimon:paimon-hive-connector-3.1,"
- for suffix in ut 3.5 3.4 3.3 3.2; do
- test_modules+="!org.apache.paimon:paimon-spark-${suffix}_2.12,"
- done
- test_modules="${test_modules%,}"
- mvn -T 1C -B -ntp clean install -pl "${test_modules}"
-Pflink1,spark3 -Pskip-paimon-flink-tests -Duser.timezone=$jvm_timezone
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utitcase-spark-3.x.yml
b/.github/workflows/utitcase-spark-3.x.yml
deleted file mode 100644
index 0fe7923060..0000000000
--- a/.github/workflows/utitcase-spark-3.x.yml
+++ /dev/null
@@ -1,79 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase and ITCase Spark 3.x
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 8
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
- strategy:
- fail-fast: true
- matrix:
- scala_version: [ '2.12', '2.13' ]
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Spark 3 with ${{ matrix.scala_version }}
- run: mvn -T 2C -B -ntp clean install -DskipTests
-Pspark3,flink1,scala-${{ matrix.scala_version }}
-
- - name: Test Spark 3 with ${{ matrix.scala_version }}
- timeout-minutes: 60
- run: |
- # run tests with random timezone to find out timezone related bugs
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- test_modules=""
- for suffix in ut 3.5 3.4 3.3 3.2; do
- test_modules+="org.apache.paimon:paimon-spark-${suffix}_${{
matrix.scala_version }},"
- done
- test_modules="${test_modules%,}"
- mvn -T 2C -B -ntp verify -pl "${test_modules}"
-Duser.timezone=$jvm_timezone -Pspark3,flink1,scala-${{ matrix.scala_version }}
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utitcase-spark-4.x.yml
b/.github/workflows/utitcase-spark-4.x.yml
deleted file mode 100644
index 72c43c018e..0000000000
--- a/.github/workflows/utitcase-spark-4.x.yml
+++ /dev/null
@@ -1,76 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase and ITCase Spark 4.x
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 17
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Spark
- run: mvn -T 2C -B -ntp clean install -DskipTests -Pspark4,flink1
-
- - name: Test Spark
- timeout-minutes: 60
- run: |
- # run tests with random timezone to find out timezone related bugs
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
- test_modules=""
- for suffix in ut 4.0 4.1; do
- test_modules+="org.apache.paimon:paimon-spark-${suffix}_2.13,"
- done
- test_modules="${test_modules%,}"
- mvn -T 2C -B -ntp verify -pl "${test_modules}"
-Duser.timezone=$jvm_timezone -Pspark4,flink1
- env:
- MAVEN_OPTS: -Xmx4096m
diff --git a/.github/workflows/utitcase.yml b/.github/workflows/utitcase.yml
deleted file mode 100644
index e83be074f4..0000000000
--- a/.github/workflows/utitcase.yml
+++ /dev/null
@@ -1,78 +0,0 @@
-################################################################################
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-################################################################################
-
-name: UTCase and ITCase Others
-
-on:
- push:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
- pull_request:
- paths-ignore:
- - 'docs/**'
- - '**/*.md'
- - 'paimon-python/**'
- - '.github/workflows/paimon-python-checks.yml'
- - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java'
-
-env:
- JDK_VERSION: 8
- MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30
-Dmaven.wagon.http.retryHandler.requestSentEnabled=true
-
-concurrency:
- group: ${{ github.workflow }}-${{ github.event_name }}-${{
github.event.number || github.run_id }}
- cancel-in-progress: true
-
-jobs:
- build_test:
- runs-on: ubuntu-latest
-
- steps:
- - name: Checkout code
- uses: actions/checkout@v6
-
- - name: Set up JDK ${{ env.JDK_VERSION }}
- uses: actions/setup-java@v5
- with:
- java-version: ${{ env.JDK_VERSION }}
- distribution: 'temurin'
-
- - name: Build Others
- run: |
- echo "Start compiling modules"
- mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3
-
- - name: Test Others
- timeout-minutes: 60
- run: |
- . .github/workflows/utils.sh
- jvm_timezone=$(random_timezone)
- echo "JVM timezone is set to $jvm_timezone"
-
- TEST_MODULES="!paimon-e2e-tests,"
- for suffix in ut 3.5 3.4 3.3 3.2; do
- TEST_MODULES+="!org.apache.paimon:paimon-spark-${suffix}_2.12,"
- done
- TEST_MODULES="${TEST_MODULES%,}"
- mvn -T 2C -B -ntp clean install -pl "${TEST_MODULES}"
-Pskip-paimon-flink-tests -Duser.timezone=$jvm_timezone
- env:
- MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2
diff --git a/docs/docs/project/verifying-a-release-candidate.md
b/docs/docs/project/verifying-a-release-candidate.md
index c0a5b55933..f3cd609281 100644
--- a/docs/docs/project/verifying-a-release-candidate.md
+++ b/docs/docs/project/verifying-a-release-candidate.md
@@ -343,7 +343,7 @@ archive; the PyPaimon source distribution must not be used
by itself for an
independent release. Run the tests from
`paimon-PAIMON_VERSION/paimon-python` on as many supported Python versions as
your environment allows. Use the Python matrix in the candidate's
-`.github/workflows/paimon-python-checks.yml` to select relevant versions rather
+`.github/workflows/ci-python.yml` to select relevant versions rather
than assuming that a matrix from another release applies:
```shell
diff --git a/tools/ci/plan.py b/tools/ci/plan.py
new file mode 100644
index 0000000000..b539f45a60
--- /dev/null
+++ b/tools/ci/plan.py
@@ -0,0 +1,180 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Select CI suites from the complete git diff, without API pagination
limits."""
+
+import json
+import os
+from pathlib import Path
+import re
+import subprocess
+
+
+# Keep the existing JDK, engine and Scala coverage. A suite is a test boundary,
+# not just a source directory: core also owns Hive and generated-doc tests.
+SUITES = [
+ ("core", "Core and integrations / JDK 8", 8, "2.12", 90),
+ ("core", "Core and integrations / JDK 11", 11, "2.12", 90),
+ ("flink1-common", "Flink 1 / Common", 8, "2.12", 100),
+ ("flink1-connectors", "Flink 1 / Connectors and CDC", 8, "2.12", 60),
+ ("flink2", "Flink 2 / Common and connectors", 11, "2.12", 120),
+ ("spark3", "Spark 3 / Scala 2.12", 8, "2.12", 90),
+ ("spark3", "Spark 3 / Scala 2.13", 8, "2.13", 90),
+ ("spark4", "Spark 4 / Scala 2.13", 17, "2.13", 90),
+ ("e2e-flink1", "E2E / Flink 1 and Spark interoperability", 8, "2.12", 60),
+ ("e2e-flink2", "E2E / Flink 2", 11, "2.12", 60),
+ ("iceberg", "Iceberg GA / JDK 17", 17, "2.12", 60),
+ ("eslib", "ESLib / JDK 11", 11, "2.12", 30),
+ ("full-text", "Full text / JDK 8", 8, "2.12", 30),
+]
+ALL_JAVA = {suite[0] for suite in SUITES}
+FLINK = {"flink1-common", "flink1-connectors", "flink2"}
+SPARK = {"spark3", "spark4"}
+E2E = {"e2e-flink1", "e2e-flink2"}
+
+
+def select(paths, full=False):
+ selected = set()
+ docs = python = licensing = False
+ if full:
+ selected.update(ALL_JAVA)
+ docs = python = licensing = True
+
+ for path in paths:
+ if path.startswith("docs/"):
+ docs = True
+ continue
+ if path.endswith(".md"):
+ continue
+ if path.startswith("paimon-python/") or path ==
".github/workflows/ci-python.yml":
+ python = True
+ continue
+ if re.search(r"/src/test/.*?/JavaPy[^/]*\.java$", path):
+ python = True
+ continue
+ if path == ".github/workflows/ci-docs.yml":
+ docs = True
+ continue
+ if path == ".github/workflows/ci-licensing.yml":
+ licensing = True
+ continue
+ # Changes to CI machinery must validate every consumer of that
machinery.
+ if path.startswith((".github/workflows/ci", "tools/ci/")):
+ selected.update(ALL_JAVA)
+ docs = python = licensing = True
+ continue
+ if path.startswith((".github/", ".asf.yaml")):
+ continue
+
+ licensing = True
+ if path.startswith("paimon-spark/"):
+ # Core includes spark-common tests and paimon-docs completeness
tests.
+ # SparkE2eTest is disabled on JDK 11, so only the Flink 1 E2E lane
applies.
+ selected.update(SPARK | {"core", "e2e-flink1"})
+ elif path.startswith("paimon-flink/"):
+ # Hive and Iceberg integration tests depend on flink-common.
+ selected.update(FLINK | E2E | {"core", "iceberg"})
+ elif path.startswith("paimon-e2e-tests/"):
+ selected.update(E2E)
+ elif path.startswith("paimon-eslib/"):
+ selected.add("eslib")
+ elif path.startswith("paimon-iceberg/"):
+ selected.update({"core", "iceberg"})
+ elif path.startswith("paimon-full-text/"):
+ selected.update({"core", "full-text"})
+ else:
+ # Shared modules, POMs outside a known leaf, build configuration
and
+ # new/unknown directories fall back to all Java suites.
+ selected.update(ALL_JAVA)
+ if path == "pom.xml":
+ python = True
+
+ matrix = [
+ dict(suite=suite, name=name, java=java, scala=scala, timeout=timeout)
+ for suite, name, java, scala, timeout in SUITES
+ if suite in selected
+ ]
+ return dict(java=bool(matrix), matrix={"include": matrix}, docs=docs,
+ python=python, licensing=licensing)
+
+
+def changed_paths(event_name, event):
+ if event_name == "pull_request":
+ base = event["pull_request"]["base"]["sha"]
+ head = event["pull_request"]["head"]["sha"]
+ separator = "..."
+ elif event_name == "push":
+ base, head = event["before"], event["after"]
+ separator = ".."
+ else:
+ return None
+ for sha in (base, head):
+ if not re.fullmatch(r"[0-9a-f]{40}", sha) or sha == "0" * 40:
+ return None
+ try:
+ # Disabling rename detection includes both the old and new path. -z
also
+ # handles whitespace, newlines and non-ASCII names without shell
parsing.
+ output = subprocess.check_output([
+ "git", "diff", "--no-renames", "--name-only", "-z",
+ base + separator + head, "--",
+ ])
+ except subprocess.CalledProcessError:
+ print("Cannot resolve the complete diff; selecting full CI.")
+ return None
+ return [os.fsdecode(path) for path in output.split(b"\0") if path]
+
+
+def check_file_sizes(paths, root=Path(".")):
+ for path in paths:
+ candidate = root / path
+ # Git symlinks contain a link target; don't follow them outside
checkout.
+ if candidate.is_symlink() or not candidate.is_file():
+ continue
+ if candidate.stat().st_size > 1048576:
+ raise ValueError("Changed file exceeds 1 MiB: " + repr(path))
+
+
+def main():
+ event_name = os.environ["GITHUB_EVENT_NAME"]
+ event = json.loads(Path(os.environ["GITHUB_EVENT_PATH"]).read_text())
+ paths = changed_paths(event_name, event)
+ plan = select(paths or [], full=paths is None)
+ size_warning = None
+ if event_name == "pull_request":
+ if paths is None:
+ # Existing oversized files must not block the full-CI fallback.
+ size_warning = "Skipping changed-file size check because the PR
diff is unavailable."
+ print("::warning::" + size_warning)
+ else:
+ check_file_sizes(paths)
+ with open(os.environ["GITHUB_OUTPUT"], "a") as output:
+ for key, value in plan.items():
+ output.write(key + "=" + json.dumps(value, separators=(",", ":"))
+ "\n")
+ with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as summary:
+ summary.write("## CI selection\n\n")
+ summary.write("Full run (manual or unavailable diff).\n\n" if paths is
None
+ else "Compared {} changed paths.\n\n".format(len(paths)))
+ if size_warning:
+ summary.write("**Warning:** " + size_warning + "\n\n")
+ for lane in plan["matrix"]["include"]:
+ summary.write("- " + lane["name"] + "\n")
+ for key in ("python", "docs", "licensing"):
+ summary.write("- {}: {}\n".format(key, "run" if plan[key] else
"skip"))
+ print(json.dumps(plan, indent=2))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tools/ci/run-java-tests.sh b/tools/ci/run-java-tests.sh
new file mode 100644
index 0000000000..b5a7de266c
--- /dev/null
+++ b/tools/ci/run-java-tests.sh
@@ -0,0 +1,129 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euo pipefail
+
+suite=${1:?Specify a CI suite}
+scala=${2:-2.12}
+java=${3:-8}
+profiles=flink1,spark3
+threads=2C
+goal=verify
+test_options=(-Dstyle.color=never)
+modules=()
+
+case "$suite" in
+ core)
+ # Keep spark-common and paimon-docs here: they have their own tests,
+ # including JDK 11 coverage, outside the Spark connector test modules.
+ modules=('!paimon-e2e-tests')
+ for version in ut 3.5 3.4 3.3 3.2; do
+ modules+=("!org.apache.paimon:paimon-spark-${version}_2.12")
+ done
+ if [[ "$java" == 11 ]]; then
+ modules+=('!org.apache.paimon:paimon-hive-connector-3.1')
+ threads=1C
+ fi
+ test_options+=(-Pskip-paimon-flink-tests)
+ ;;
+ flink1-common)
+ modules=(paimon-flink/paimon-flink-common)
+ ;;
+ flink1-connectors)
+ for version in cdc 1.16 1.17 1.18 1.19 1.20; do
+ modules+=("org.apache.paimon:paimon-flink-${version}")
+ done
+ ;;
+ flink2)
+ profiles=flink2,spark3
+ for version in 2.0 2.1 2.2 common; do
+ modules+=("org.apache.paimon:paimon-flink-${version}")
+ done
+ ;;
+ spark3)
+ profiles="flink1,spark3,scala-${scala}"
+ for version in ut 3.5 3.4 3.3 3.2; do
+ modules+=("org.apache.paimon:paimon-spark-${version}_${scala}")
+ done
+ ;;
+ spark4)
+ profiles=flink1,spark4
+ for version in ut 4.0 4.1; do
+ modules+=("org.apache.paimon:paimon-spark-${version}_2.13")
+ done
+ ;;
+ e2e-flink1)
+ profiles=flink1,spark3,flink-1.20
+ modules=(paimon-e2e-tests)
+ threads=1C
+ goal=test
+ ;;
+ e2e-flink2)
+ profiles=flink2,spark3,flink-2.2,java11
+ modules=(paimon-e2e-tests)
+ threads=1C
+ goal=test
+ ;;
+ iceberg)
+ profiles=flink1,spark3,paimon-iceberg,iceberg-ga
+ modules=(paimon-iceberg)
+ threads=1C
+ goal=test
+ ;;
+ eslib)
+ profiles=paimon-eslib
+ modules=(paimon-eslib)
+ threads=1
+ ;;
+ full-text)
+ modules=(paimon-full-text)
+ threads=1
+ ;;
+ *)
+ echo "Unknown CI suite: $suite" >&2
+ exit 1
+ ;;
+esac
+
+module_list=$(IFS=,; echo "${modules[*]}")
+
+run_maven() {
+ if [[ "${CI_DRY_RUN:-false}" == true ]]; then
+ printf '%q ' mvn "$@"
+ printf '\n'
+ else
+ mvn "$@"
+ fi
+}
+
+# -am compiles and installs the selected tests' dependency closure, including
+# test jars. Only the selected modules run tests in the second invocation.
+run_maven -T 2C -B -ntp install -DskipTests -P"$profiles" -pl "$module_list"
-am
+
+jvm_timezone=GMT+00:00
+if [[ "${CI_DRY_RUN:-false}" != true ]]; then
+ # Preserve the native-library setup used by the original engine/core lanes.
+ if [[ "$suite" != eslib && "$suite" != full-text ]]; then
+ . tools/ci/utils.sh
+ jvm_timezone=$(random_timezone)
+ fi
+ echo "JVM timezone is set to $jvm_timezone"
+fi
+
+# verify already includes the test phase; don't execute `test verify` or clean
+# away the dependencies and test classes just built in this fresh checkout.
+run_maven -T "$threads" -B -ntp "$goal" -P"$profiles" -pl "$module_list" \
+ "${test_options[@]}" -Duser.timezone="$jvm_timezone"
diff --git a/tools/ci/test_java_commands.py b/tools/ci/test_java_commands.py
new file mode 100644
index 0000000000..5cfc7aa487
--- /dev/null
+++ b/tools/ci/test_java_commands.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Validate suite coverage and Maven command boundaries without running
Maven."""
+
+import os
+from pathlib import Path
+import re
+import shlex
+import subprocess
+import unittest
+import xml.etree.ElementTree as ET
+
+from plan import SUITES
+
+
+SCRIPT = Path(__file__).with_name('run-java-tests.sh')
+
+
+class MavenCommandsTest(unittest.TestCase):
+ def commands(self, suite, scala='2.12', java=8):
+ output = subprocess.check_output(
+ ['bash', str(SCRIPT), suite, scala, str(java)],
+ env=dict(os.environ, CI_DRY_RUN='true'), text=True)
+ return [shlex.split(line) for line in output.splitlines()]
+
+ def modules(self, command):
+ return set(command[command.index('-pl') + 1].split(','))
+
+ def
test_every_lane_builds_dependencies_but_only_tests_selected_modules(self):
+ for suite, name, java, scala, timeout in SUITES:
+ with self.subTest(name=name):
+ build, test = self.commands(suite, scala, java)
+ self.assertIn('-am', build)
+ self.assertIn('-DskipTests', build)
+ self.assertNotIn('-am', test)
+ self.assertNotIn('-DskipTests', test)
+ self.assertEqual(self.modules(build), self.modules(test))
+ self.assertNotIn('clean', test)
+ self.assertFalse('test' in test and 'verify' in test)
+ self.assertFalse(any('fast-build' in arg for arg in build +
test))
+
+ def test_spark_retains_all_connector_versions_and_scala_variants(self):
+ for scala in ('2.12', '2.13'):
+ test = self.commands('spark3', scala)[1]
+ for version in ('ut', '3.2', '3.3', '3.4', '3.5'):
+ self.assertIn('org.apache.paimon:paimon-spark-' + version +
'_' + scala,
+ self.modules(test))
+ self.assertIn('-Pflink1,spark3,scala-' + scala, test)
+ self.assertEqual(self.modules(self.commands('spark4', '2.13', 17)[1]),
+ {'org.apache.paimon:paimon-spark-' + version + '_2.13'
+ for version in ('ut', '4.0', '4.1')})
+
+ def
test_core_keeps_common_and_docs_tests_and_existing_jdk11_exclusion(self):
+ for java in (8, 11):
+ test = self.commands('core', java=java)[1]
+ self.assertIn('-Pskip-paimon-flink-tests', test)
+ self.assertNotIn('!org.apache.paimon:paimon-spark-common_2.12',
self.modules(test))
+ self.assertNotIn('!paimon-docs', self.modules(test))
+ self.assertEqual('!org.apache.paimon:paimon-hive-connector-3.1' in
self.modules(test),
+ java == 11)
+
+ def test_flink_coverage_includes_cdc_and_both_major_versions(self):
+ self.assertEqual(self.modules(self.commands('flink1-connectors')[1]),
+ {'org.apache.paimon:paimon-flink-' + version
+ for version in ('cdc', '1.16', '1.17', '1.18',
'1.19', '1.20')})
+ self.assertEqual(self.modules(self.commands('flink2', java=11)[1]),
+ {'org.apache.paimon:paimon-flink-' + version
+ for version in ('common', '2.0', '2.1', '2.2')})
+
+ def test_e2e_uses_the_same_version_when_building_and_testing(self):
+ for suite, java, version in [('e2e-flink1', 8, '1.20'), ('e2e-flink2',
11, '2.2')]:
+ build, test = self.commands(suite, java=java)
+ self.assertTrue(any('flink-' + version in arg for arg in build))
+ self.assertEqual([arg for arg in build if arg.startswith('-P')],
+ [arg for arg in test if arg.startswith('-P')])
+ self.assertIn('test', test)
+
+ def test_unknown_suite_fails_instead_of_silently_running_no_tests(self):
+ result = subprocess.run(['bash', str(SCRIPT), 'typo'],
+ env=dict(os.environ, CI_DRY_RUN='true'),
capture_output=True)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertIn(b'Unknown CI suite', result.stderr)
+
+ def
test_python_build_selects_runnable_modules_not_just_aggregator_poms(self):
+ root = SCRIPT.resolve().parents[2]
+ workflow = (root / '.github/workflows/ci-python.yml').read_text()
+ modules = set(re.search(r'-pl ([^\s]+) -am',
workflow).group(1).split(','))
+ self.assertTrue({'paimon-core', 'paimon-lance', 'paimon-lumina',
+ 'paimon-vortex/paimon-vortex-format',
'paimon-full-text',
+ 'paimon-vector'}.issubset(modules))
+ for module in modules:
+ pom = ET.parse(root / module / 'pom.xml').getroot()
+ packaging =
pom.findtext('{http://maven.apache.org/POM/4.0.0}packaging', 'jar')
+ self.assertNotEqual(packaging, 'pom', module)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tools/ci/test_plan.py b/tools/ci/test_plan.py
new file mode 100644
index 0000000000..4d0c26a174
--- /dev/null
+++ b/tools/ci/test_plan.py
@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Behavioral checks for CI routing and the complete PR/push diff."""
+
+import contextlib
+import io
+import json
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+import unittest
+from unittest.mock import patch
+
+import plan
+
+
+class SelectionTest(unittest.TestCase):
+ def suites(self, paths):
+ return {lane['suite'] for lane in
plan.select(paths)['matrix']['include']}
+
+ def test_spark_keeps_common_docs_and_interoperability_tests(self):
+ suites =
self.suites(['paimon-spark/paimon-spark-common/src/main/scala/Change.scala'])
+ self.assertEqual(suites, {'core', 'spark3', 'spark4', 'e2e-flink1'})
+
+ def test_flink_keeps_hive_iceberg_and_both_e2e_versions(self):
+ suites =
self.suites(['paimon-flink/paimon-flink-common/src/main/java/Change.java'])
+ self.assertEqual(suites, {'core', 'flink1-common', 'flink1-connectors',
+ 'flink2', 'e2e-flink1', 'e2e-flink2',
'iceberg'})
+
+ def test_shared_and_unknown_changes_run_all_java(self):
+ for path in ['paimon-core/Change.java', 'paimon-api/Change.java',
+ 'paimon-format/pom.xml', 'paimon-new-module/pom.xml',
+ '.mvn/jvm.config', '.scalafmt.conf', 'copyright.txt']:
+ with self.subTest(path=path):
+ self.assertEqual(self.suites([path]), plan.ALL_JAVA)
+
+ def test_parent_pom_also_validates_python_build(self):
+ self.assertTrue(plan.select(['pom.xml'])['python'])
+ self.assertEqual(self.suites(['pom.xml']), plan.ALL_JAVA)
+
+ def test_docs_and_markdown_do_not_start_maven(self):
+ result = plan.select(['docs/start.md', 'README.md'])
+ self.assertTrue(result['docs'])
+ self.assertFalse(result['java'])
+ self.assertFalse(result['python'])
+ self.assertFalse(result['licensing'])
+ self.assertFalse(plan.select(['paimon-python/README.md'])['python'])
+
+ def test_mixed_changes_take_union(self):
+ self.assertEqual(self.suites(['docs/intro.md', 'paimon-spark/pom.xml',
+ 'paimon-flink/pom.xml']),
+ {'core', 'spark3', 'spark4', 'flink1-common',
+ 'flink1-connectors', 'flink2', 'e2e-flink1',
+ 'e2e-flink2', 'iceberg'})
+ result = plan.select(['paimon-python/setup.py',
'paimon-core/Change.java'])
+ self.assertTrue(result['python'])
+ self.assertTrue(result['java'])
+
+ def test_java_python_fixtures_run_python(self):
+ for module, name in [('paimon-core', 'JavaPyE2ETest'),
+ ('paimon-lance', 'JavaPyLanceE2ETest'),
+ ('paimon-lumina', 'JavaPyLuminaE2ETest')]:
+ result = plan.select([module + '/src/test/java/org/apache/paimon/'
+ name + '.java'])
+ self.assertTrue(result['python'])
+ self.assertFalse(result['java'])
+
+ def test_leaf_modules_and_workflows(self):
+ self.assertEqual(self.suites(['paimon-eslib/pom.xml']), {'eslib'})
+ self.assertEqual(self.suites(['paimon-e2e-tests/pom.xml']),
+ {'e2e-flink1', 'e2e-flink2'})
+
self.assertTrue(plan.select(['.github/workflows/ci-python.yml'])['python'])
+
self.assertFalse(plan.select(['.github/workflows/ci-python.yml'])['java'])
+ self.assertTrue(plan.select(['.github/workflows/ci-docs.yml'])['docs'])
+
self.assertFalse(plan.select(['.github/workflows/release-java.yml'])['java'])
+
+ def test_ci_logic_and_manual_run_validate_every_consumer(self):
+ for result in [plan.select(['tools/ci/plan.py']),
+ plan.select(['.github/workflows/ci-java.yml']),
+ plan.select([], full=True)]:
+ self.assertTrue(all(result[key] for key in ('java', 'python',
'docs', 'licensing')))
+ self.assertEqual(len(result['matrix']['include']), 13)
+
+ def test_empty_diff_does_not_run_expensive_jobs(self):
+ result = plan.select([])
+ self.assertFalse(any(result[key] for key in ('java', 'python', 'docs',
'licensing')))
+ self.assertEqual(result['matrix'], {'include': []})
+
+ def test_size_guard_handles_spaces_deletions_and_symlinks(self):
+ with tempfile.TemporaryDirectory() as temp:
+ root = Path(temp)
+ large = root / 'large file.txt'
+ large.write_bytes(b'x' * (1048576 + 1))
+ (root / 'link').symlink_to(large)
+ plan.check_file_sizes(['deleted.txt', 'link'], root)
+ with self.assertRaisesRegex(ValueError, 'large file'):
+ plan.check_file_sizes(['large file.txt'], root)
+
+
+class GitDiffTest(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory()
+ self.addCleanup(self.temp.cleanup)
+ self.previous = Path.cwd()
+ os.chdir(self.temp.name)
+ self.addCleanup(os.chdir, self.previous)
+ self.git('init', '-q')
+ self.git('config', 'user.email', '[email protected]')
+ self.git('config', 'user.name', 'CI test')
+ self.commit('base.txt', 'base')
+ self.base = self.git('rev-parse', 'HEAD')
+
+ def git(self, *args):
+ return subprocess.check_output(['git', *args],
stderr=subprocess.DEVNULL).decode().strip()
+
+ def commit(self, name, content):
+ path = Path(name)
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_text(content)
+ self.git('add', '--', name)
+ self.git('commit', '-qm', 'Test commit')
+
+ def test_pr_uses_merge_base_instead_of_including_base_branch_changes(self):
+ self.commit('base-only.txt', 'base change')
+ base_tip = self.git('rev-parse', 'HEAD')
+ self.git('checkout', '-q', '--detach', self.base)
+ self.commit('paimon-spark/name with spaces\nand newline.scala',
'spark')
+ head = self.git('rev-parse', 'HEAD')
+ paths = plan.changed_paths('pull_request', {
+ 'pull_request': {'base': {'sha': base_tip}, 'head': {'sha':
head}}})
+ self.assertEqual(paths, ['paimon-spark/name with spaces\nand
newline.scala'])
+
+ def test_push_includes_all_commits_and_both_sides_of_renames(self):
+ self.commit('paimon-spark/A.scala', 'spark')
+ before = self.git('rev-parse', 'HEAD')
+ Path('paimon-flink').mkdir()
+ self.git('mv', 'paimon-spark/A.scala', 'paimon-flink/A.java')
+ self.git('commit', '-qm', 'Move across modules')
+ self.commit('docs/intro.md', 'docs')
+ paths = plan.changed_paths('push', {'before': before,
+ 'after': self.git('rev-parse',
'HEAD')})
+ self.assertEqual(set(paths), {'paimon-spark/A.scala',
'paimon-flink/A.java',
+ 'docs/intro.md'})
+
+ def test_large_diffs_do_not_truncate_late_core_changes(self):
+ for index in range(350):
+ Path('doc{:03d}.md'.format(index)).write_text('docs')
+ self.commit('z-new-core/Change.java', 'core')
+ self.git('add', '.')
+ self.git('commit', '-qm', 'Many docs')
+ paths = plan.changed_paths('push', {'before': self.base,
+ 'after': self.git('rev-parse',
'HEAD')})
+ self.assertEqual(len(paths), 351)
+ self.assertTrue(plan.select(paths)['java'])
+
+ def test_new_branch_missing_commits_and_manual_run_fall_back(self):
+ self.assertIsNone(plan.changed_paths('push', {'before': '0' * 40,
'after': self.base}))
+ self.assertIsNone(plan.changed_paths('workflow_dispatch', {}))
+ with patch('subprocess.check_output',
side_effect=subprocess.CalledProcessError(128, 'git')):
+ self.assertIsNone(plan.changed_paths('push', {'before': 'a' * 40,
'after': self.base}))
+
+ def run_pr_plan(self, base, head):
+ Path('event.json').write_text(json.dumps({
+ 'pull_request': {'base': {'sha': base}, 'head': {'sha': head}}}))
+ env = {'GITHUB_EVENT_NAME': 'pull_request', 'GITHUB_EVENT_PATH':
'event.json',
+ 'GITHUB_OUTPUT': 'output.txt', 'GITHUB_STEP_SUMMARY':
'summary.md'}
+ with patch.dict(os.environ, env),
contextlib.redirect_stdout(io.StringIO()) as log:
+ plan.main()
+ outputs = dict(line.split('=', 1) for line in
Path('output.txt').read_text().splitlines())
+ return {key: json.loads(value) for key, value in outputs.items()},
log.getvalue()
+
+ def
test_pr_unavailable_diff_runs_full_ci_with_unchanged_oversized_file(self):
+ self.commit('docs/static/old image.png', 'x' * (1048576 + 1))
+ base = self.git('rev-parse', 'HEAD')
+ self.commit('paimon-spark/Change.scala', 'spark')
+ head = self.git('rev-parse', 'HEAD')
+ git_output = subprocess.check_output
+
+ def unavailable_diff(command, *args, **kwargs):
+ if command[:2] == ['git', 'diff']:
+ raise subprocess.CalledProcessError(128, command)
+ return git_output(command, *args, **kwargs)
+
+ with patch('subprocess.check_output', side_effect=unavailable_diff):
+ outputs, log = self.run_pr_plan(base, head)
+ self.assertTrue(all(outputs[key] for key in ('java', 'python', 'docs',
'licensing')))
+ self.assertEqual(len(outputs['matrix']['include']), 13)
+ summary = Path('summary.md').read_text()
+ self.assertIn('Full run (manual or unavailable diff).', summary)
+ self.assertIn('Skipping changed-file size check', log)
+ self.assertIn('Skipping changed-file size check', summary)
+
+ def
test_pr_size_guard_ignores_unchanged_but_rejects_changed_oversized_files(self):
+ self.commit('docs/static/old image.png', 'x' * (1048576 + 1))
+ base = self.git('rev-parse', 'HEAD')
+ self.commit('paimon-spark/Change.scala', 'spark')
+ outputs, _ = self.run_pr_plan(base, self.git('rev-parse', 'HEAD'))
+ self.assertEqual(len(outputs['matrix']['include']), 6)
+
+ for path in ('docs/static/old image.png', 'docs/static/new image.png'):
+ with self.subTest(path=path):
+ base = self.git('rev-parse', 'HEAD')
+ self.commit(path, 'y' * (1048576 + 1))
+ with self.assertRaisesRegex(ValueError, 'Changed file exceeds
1 MiB') as error:
+ self.run_pr_plan(base, self.git('rev-parse', 'HEAD'))
+ self.assertIn(path, str(error.exception))
+
+ def
test_github_outputs_are_valid_json_and_summary_explains_selection(self):
+ self.commit('paimon-spark/Change.scala', 'spark')
+ Path('event.json').write_text(json.dumps({'before': self.base,
+ 'after': self.git('rev-parse',
'HEAD')}))
+ env = {'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': 'event.json',
+ 'GITHUB_OUTPUT': 'output.txt', 'GITHUB_STEP_SUMMARY':
'summary.md'}
+ with patch.dict(os.environ, env), contextlib.redirect_stdout(None):
+ plan.main()
+ outputs = dict(line.split('=', 1) for line in
Path('output.txt').read_text().splitlines())
+ self.assertTrue(json.loads(outputs['java']))
+ self.assertFalse(json.loads(outputs['python']))
+ self.assertEqual(len(json.loads(outputs['matrix'])['include']), 6)
+ self.assertIn('Spark interoperability', Path('summary.md').read_text())
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/tools/ci/test_result.py b/tools/ci/test_result.py
new file mode 100644
index 0000000000..ff373e3216
--- /dev/null
+++ b/tools/ci/test_result.py
@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""The required CI check must not pass when a selected group did not pass."""
+
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+import textwrap
+import unittest
+
+
+class ResultTest(unittest.TestCase):
+ def run_gate(self, plan_result='success', **outcomes):
+ workflow = Path(__file__).resolve().parents[2] /
'.github/workflows/ci.yml'
+ source = workflow.read_text().split("python3 - <<'PYTHON'\n", 1)[1]
+ source = textwrap.dedent(source.split(' PYTHON', 1)[0])
+ flags = dict(java='true', python='false', docs='false',
licensing='true')
+ results = dict(plan=dict(result=plan_result, outputs=flags),
+ java=dict(result='success'),
python=dict(result='skipped'),
+ docs=dict(result='skipped'),
licensing=dict(result='success'))
+ for name, result in outcomes.items():
+ results[name]['result'] = result
+ return subprocess.run([sys.executable, '-c', source],
capture_output=True,
+ env=dict(os.environ,
RESULTS=json.dumps(results)))
+
+ def test_selected_groups_pass_and_unselected_groups_skip(self):
+ self.assertEqual(self.run_gate().returncode, 0)
+
+ def test_selected_failure_cancellation_and_unexpected_skip_fail(self):
+ for outcome in ('failure', 'cancelled', 'skipped'):
+ with self.subTest(outcome=outcome):
+ self.assertNotEqual(self.run_gate(java=outcome).returncode, 0)
+
+ def test_planner_failure_fails_even_if_other_groups_report_success(self):
+ for outcome in ('failure', 'cancelled', 'skipped'):
+ self.assertNotEqual(self.run_gate(plan_result=outcome).returncode,
0)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/.github/workflows/utils.sh b/tools/ci/utils.sh
similarity index 100%
rename from .github/workflows/utils.sh
rename to tools/ci/utils.sh
