This is an automated email from the ASF dual-hosted git repository. XiaoHongbo-Hope pushed a commit to branch release-2.1 in repository https://gitbox.apache.org/repos/asf/paimon.git
commit 8993507fa7627f0fc8fa8c6b159039a1ca23ef8e Author: Jingsong Lee <[email protected]> AuthorDate: Sun Sep 13 17:55:19 2026 +0800 [ci] Run affected test suites through reusable workflows (#9787) --- .github/CI.md | 83 +++++++ .github/workflows/{docs-tests.yml => ci-docs.yml} | 15 +- .github/workflows/ci-java.yml | 73 +++++++ .../{check-licensing.yml => ci-licensing.yml} | 31 ++- .../{paimon-python-checks.yml => ci-python.yml} | 43 ++-- .github/workflows/ci.yml | 112 ++++++++++ .github/workflows/e2e-tests-flink-1.x.yml | 80 ------- .github/workflows/e2e-tests-flink-2.x-jdk11.yml | 81 ------- .github/workflows/file-size-check.yml | 48 ----- .../{stale-pr.yml => maintenance-stale-pr.yml} | 2 +- ...pshot-jdk17.yml => publish-snapshot-spark4.yml} | 14 +- .../{publish_snapshot.yml => publish-snapshot.yml} | 14 +- .github/workflows/release-java.yml | 2 +- .github/workflows/release-python-publish.yml | 2 +- .github/workflows/release-python.yml | 2 +- .github/workflows/utcase-eslib.yml | 73 ------- .github/workflows/utcase-full-text.yml | 61 ------ .github/workflows/utitcase-flink-1.x-common.yml | 75 ------- .github/workflows/utitcase-flink-1.x-others.yml | 75 ------- .github/workflows/utitcase-flink-2.x-jdk11.yml | 76 ------- .github/workflows/utitcase-iceberg-ga.yml | 66 ------ .github/workflows/utitcase-jdk11.yml | 72 ------- .github/workflows/utitcase-spark-3.x.yml | 79 ------- .github/workflows/utitcase-spark-4.x.yml | 76 ------- .github/workflows/utitcase.yml | 78 ------- docs/docs/project/verifying-a-release-candidate.md | 2 +- tools/ci/plan.py | 180 ++++++++++++++++ tools/ci/run-java-tests.sh | 129 +++++++++++ tools/ci/test_java_commands.py | 112 ++++++++++ tools/ci/test_plan.py | 238 +++++++++++++++++++++ tools/ci/test_result.py | 56 +++++ {.github/workflows => tools/ci}/utils.sh | 0 32 files changed, 1051 insertions(+), 999 deletions(-) diff --git a/.github/CI.md b/.github/CI.md new file mode 100644 index 0000000000..d0383aa5ff --- /dev/null +++ b/.github/CI.md @@ -0,0 +1,83 @@ +# Continuous integration + +`workflows/ci.yml` is the entry point for pull requests, pushes to any branch or +tag, and manual full runs. Contributors can validate changes in their own forks +before opening an upstream PR, once GitHub Actions is enabled in the fork. +Both push and PR runs select tests by changed paths. Updating a PR cancels its +previous CI run; push and publishing runs are not cancelled in progress. + +The entry point calls `ci-java.yml`, `ci-python.yml`, `ci-docs.yml` and +`ci-licensing.yml`. `publish-snapshot*.yml`, `release*.yml` and +`maintenance-stale-pr.yml` have separate publishing or maintenance triggers. + +## Selecting tests + +`tools/ci/plan.py` is the single source of path routing and the Java matrix. +Its selection is shown in the workflow run summary. It uses the PR merge-base +diff, or the entire before/after push diff. Both sides of a rename are included. +Diffs are read from Git without a changed-file API limit. If a complete diff is +unavailable (including a new branch), CI runs all suites. Manual dispatch also +runs everything. + +PRs check changed files against a 1 MiB size limit. When the PR diff is unavailable, +this check is skipped with a warning in the log and run summary so that existing +oversized files do not block the full CI run. + +| Changed paths | Java tests | +| --- | --- | +| `paimon-spark/**` | Spark 3/4, core and integrations, Flink 1/Spark E2E | +| `paimon-flink/**` | Flink 1/2, core and integrations, both E2E groups, Iceberg GA | +| `paimon-e2e-tests/**` | Both E2E groups | +| `paimon-eslib/**` | ESLib | +| `paimon-iceberg/**` | Core and integrations, Iceberg GA | +| `paimon-full-text/**` | Core and integrations, full text | +| Shared modules, root/build configuration, unclassified source paths | All Java groups | + +Java changes retain the artifact licensing check. Changes to `docs/**` build the +website; other Markdown-only changes skip expensive jobs. Changes to +`paimon-python/**` and Java/Python `JavaPy*` test fixtures run Python checks. +Root POM changes also run Python to validate its Java build. Mixed changes take +the union of all required groups. Changes to CI routing or execution scripts +validate every consumer, including Python and documentation. + +The core and integrations jobs deliberately still run for Spark/Flink changes: +`paimon-spark-common` owns tests outside the Spark connector test jobs, +`paimon-docs` tests engine configuration documentation, and Hive/Iceberg tests +depend on Flink. Do not remove these jobs by looking only at the workflow name. +`SparkE2eTest` runs in the JDK 8 E2E job and is disabled on JDK 11. + +## Builds and caches + +`tools/ci/run-java-tests.sh` owns Java build/test commands. Each group installs +its selected modules and their dependencies with `-pl ... -am -DskipTests`, then +tests only its modules. The test invocation does not clean away the first build +or run both `test` and `verify`. Normal Maven checks remain enabled. + +Java dependency caches are isolated by suite, JDK and Scala. This prevents a +small group's cache from permanently starving other groups of their dependencies. +Failed Java jobs upload Surefire reports with a seven-day retention period. +Python builds only the Java modules used by the mixed-language tests and their +dependencies; its Python version and compatibility coverage is preserved. + +## Required checks and maintenance + +Configure branch protection to require **CI result** when adopting this layout. +Remove any requirements referring to the retired individual workflow/job names. +The result job always runs and requires the planner and every selected group to +succeed. Unselected groups must be skipped. A planner failure, failed test, +unexpected skip or cancellation cannot produce a green result. + +Every run validates the workflow syntax with pinned actionlint 1.7.11 and runs +the routing/command tests before starting expensive jobs. Locally: + +```sh +python3 -m unittest discover -s tools/ci -p 'test_*.py' +actionlint -shellcheck= -pyflakes= +CI_DRY_RUN=true bash tools/ci/run-java-tests.sh spark3 2.12 8 +``` + +When adding an engine version, update both the matrix in `plan.py` and the +module lists in `run-java-tests.sh`, including the core exclusions where needed. +Add behavioral tests for any new routing boundary and retain shared-dependency +fallbacks. Use a manual CI run to validate the complete matrix before adopting +changes to build profiles or test ownership. diff --git a/.github/workflows/docs-tests.yml b/.github/workflows/ci-docs.yml similarity index 84% rename from .github/workflows/docs-tests.yml rename to .github/workflows/ci-docs.yml index d6d1df1987..e9d77e644b 100644 --- a/.github/workflows/docs-tests.yml +++ b/.github/workflows/ci-docs.yml @@ -13,16 +13,13 @@ # See the License for the specific language governing permissions and # limitations under the License. -name: docs-tests +name: CI / Documentation + on: - push: - pull_request: - paths: - - 'docs/**' + workflow_call: -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true +permissions: + contents: read jobs: build: @@ -31,7 +28,7 @@ jobs: steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Setup Node.js uses: actions/setup-node@v4 diff --git a/.github/workflows/ci-java.yml b/.github/workflows/ci-java.yml new file mode 100644 index 0000000000..1283bbe19f --- /dev/null +++ b/.github/workflows/ci-java.yml @@ -0,0 +1,73 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +name: CI / Java + +on: + workflow_call: + inputs: + matrix: + description: Java suites selected by tools/ci/plan.py + required: true + type: string + +permissions: + contents: read + +env: + MAVEN_OPTS: -Xmx4096m -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true + +jobs: + test: + name: ${{ matrix.name }} + runs-on: ubuntu-latest + timeout-minutes: ${{ matrix.timeout }} + strategy: + # Keep independent results when one engine fails. + fail-fast: false + matrix: ${{ fromJSON(inputs.matrix) }} + steps: + - uses: actions/checkout@v6 + with: + persist-credentials: false + - name: Set up JDK ${{ matrix.java }} + uses: actions/setup-java@v5 + with: + java-version: ${{ matrix.java }} + distribution: temurin + - name: Cache Maven dependencies + uses: actions/cache@v5 + with: + path: ~/.m2/repository + key: ci-maven-${{ runner.os }}-${{ runner.arch }}-${{ matrix.suite }}-jdk${{ matrix.java }}-scala${{ matrix.scala }}-${{ hashFiles('**/pom.xml') }} + restore-keys: | + ci-maven-${{ runner.os }}-${{ runner.arch }}-${{ matrix.suite }}-jdk${{ matrix.java }}-scala${{ matrix.scala }}- + - name: Build dependencies and run tests + env: + CI_SUITE: ${{ matrix.suite }} + CI_SCALA: ${{ matrix.scala }} + CI_JAVA: ${{ matrix.java }} + run: bash tools/ci/run-java-tests.sh "$CI_SUITE" "$CI_SCALA" "$CI_JAVA" + - name: Upload failed test reports + if: failure() + uses: actions/upload-artifact@v6 + with: + name: test-reports-${{ matrix.suite }}-jdk${{ matrix.java }}-scala${{ matrix.scala }} + path: '**/target/surefire-reports/**' + if-no-files-found: ignore + retention-days: 7 diff --git a/.github/workflows/check-licensing.yml b/.github/workflows/ci-licensing.yml similarity index 82% rename from .github/workflows/check-licensing.yml rename to .github/workflows/ci-licensing.yml index 207cdd13fa..3857c2022e 100644 --- a/.github/workflows/check-licensing.yml +++ b/.github/workflows/ci-licensing.yml @@ -13,29 +13,21 @@ # See the License for the specific language governing permissions and # limitations under the License. -name: Check licensing +name: CI / Licensing on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' + workflow_call: + +permissions: + contents: read env: MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - jobs: - build: + check: + name: Artifact licensing + timeout-minutes: 60 runs-on: ubuntu-latest env: MVN_COMMON_OPTIONS: -U -B --no-transfer-progress @@ -50,6 +42,13 @@ jobs: with: java-version: 8 distribution: 'temurin' + - name: Cache Maven dependencies + uses: actions/cache@v5 + with: + path: ~/.m2/repository + key: ci-maven-licensing-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/pom.xml') }} + restore-keys: | + ci-maven-licensing-${{ runner.os }}-${{ runner.arch }}- - name: Build run: | set -o pipefail diff --git a/.github/workflows/paimon-python-checks.yml b/.github/workflows/ci-python.yml old mode 100755 new mode 100644 similarity index 95% rename from .github/workflows/paimon-python-checks.yml rename to .github/workflows/ci-python.yml index ac48f730fc..daffd050ad --- a/.github/workflows/paimon-python-checks.yml +++ b/.github/workflows/ci-python.yml @@ -16,19 +16,13 @@ # limitations under the License. ################################################################################ -name: Python Check Code Style and Test +name: CI / Python on: - push: - paths: - - 'paimon-python/**' - - '!**/*.md' - - '.github/workflows/paimon-python-checks.yml' - pull_request: - paths: - - 'paimon-python/**' - - '!**/*.md' - - '.github/workflows/paimon-python-checks.yml' + workflow_call: + +permissions: + contents: read env: JDK_VERSION: 8 @@ -37,13 +31,11 @@ env: PYPAIMON_RUST_REV: be5456e96b326939570b6ef5d63acddda3311c4f -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - jobs: # Lint + test on selected versions only (3.6 / 3.7 low end, 3.10 / 3.11 / 3.12 / 3.13 current), not every Python version. - lint-python: + test: + name: Tests / Python ${{ matrix.python-version }} + timeout-minutes: 90 runs-on: ubuntu-latest container: "python:${{ matrix.python-version }}-slim" strategy: @@ -61,6 +53,14 @@ jobs: java-version: ${{ env.JDK_VERSION }} distribution: 'temurin' + - name: Cache Maven dependencies + uses: actions/cache@v5 + with: + path: ~/.m2/repository + key: ci-maven-python-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/pom.xml') }} + restore-keys: | + ci-maven-python-${{ runner.os }}-${{ runner.arch }}- + - name: Set up Maven uses: stCarolas/setup-maven@v5 with: @@ -116,7 +116,8 @@ jobs: - name: Build Java run: | echo "Start compiling modules" - mvn -T 2C -B -ntp clean install -DskipTests + mvn -T 2C -B -ntp install -DskipTests \ + -pl paimon-core,paimon-lance,paimon-vortex/paimon-vortex-format,paimon-full-text,paimon-lumina,paimon-vector -am - name: Install Python dependencies shell: bash @@ -184,7 +185,9 @@ jobs: fi ./paimon-python/dev/lint-python.sh -e pytest_torch - torch_test: + torch: + name: PyTorch / Python 3.10 + timeout-minutes: 60 runs-on: ubuntu-latest container: "python:3.10-slim" @@ -219,7 +222,9 @@ jobs: ./paimon-python/dev/lint-python.sh -i pytest_torch # One job: check dev/requirements.txt on each Python version in sequence, then Ray version test on 3.10. - requirement_version_compatible_test: + compatibility: + name: Requirements and Ray compatibility + timeout-minutes: 60 runs-on: ubuntu-latest steps: - name: Checkout code diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000000..f250062d4b --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,112 @@ +################################################################################ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +################################################################################ + +name: CI + +on: + pull_request: + push: + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ci-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + plan: + name: Select tests and validate CI + runs-on: ubuntu-latest + timeout-minutes: 10 + outputs: + java: ${{ steps.plan.outputs.java }} + matrix: ${{ steps.plan.outputs.matrix }} + python: ${{ steps.plan.outputs.python }} + docs: ${{ steps.plan.outputs.docs }} + licensing: ${{ steps.plan.outputs.licensing }} + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + persist-credentials: false + - name: Test CI selection and Maven commands + run: python3 -m unittest discover -s tools/ci -p 'test_*.py' + - name: Validate workflow syntax + env: + GH_TOKEN: ${{ github.token }} + run: | + gh release download v1.7.11 --repo rhysd/actionlint \ + --pattern 'actionlint_1.7.11_linux_amd64.tar.gz' --dir "$RUNNER_TEMP" + tar -xzf "$RUNNER_TEMP/actionlint_1.7.11_linux_amd64.tar.gz" -C "$RUNNER_TEMP" actionlint + "$RUNNER_TEMP/actionlint" -shellcheck= -pyflakes= + - name: Select affected tests and check changed file sizes + id: plan + run: python3 tools/ci/plan.py + + java: + name: Java + needs: plan + if: needs.plan.outputs.java == 'true' + uses: ./.github/workflows/ci-java.yml + with: + matrix: ${{ needs.plan.outputs.matrix }} + + python: + name: Python + needs: plan + if: needs.plan.outputs.python == 'true' + uses: ./.github/workflows/ci-python.yml + + docs: + name: Documentation + needs: plan + if: needs.plan.outputs.docs == 'true' + uses: ./.github/workflows/ci-docs.yml + + licensing: + name: Licensing + needs: plan + if: needs.plan.outputs.licensing == 'true' + uses: ./.github/workflows/ci-licensing.yml + + result: + # Use this stable check for branch protection, not individual matrix jobs. + name: CI result + if: always() + needs: [plan, java, python, docs, licensing] + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Require all selected jobs to pass + env: + RESULTS: ${{ toJSON(needs) }} + run: | + python3 - <<'PYTHON' + import json + import os + results = json.loads(os.environ['RESULTS']) + if results['plan']['result'] != 'success': + raise SystemExit('CI selection or validation failed') + selected = results['plan']['outputs'] + for name in ('java', 'python', 'docs', 'licensing'): + expected = 'success' if selected[name] == 'true' else 'skipped' + if results[name]['result'] != expected: + raise SystemExit(name + ': ' + results[name]['result'] + ', expected ' + expected) + PYTHON diff --git a/.github/workflows/e2e-tests-flink-1.x.yml b/.github/workflows/e2e-tests-flink-1.x.yml deleted file mode 100644 index 3d7c25b600..0000000000 --- a/.github/workflows/e2e-tests-flink-1.x.yml +++ /dev/null @@ -1,80 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: End to End Tests Flink 1.x - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 8 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -jobs: - build_test: - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: true - matrix: - # Only Test Latest Version - flink_version: [ '1.20' ] - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Flink - run: mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }} - - - name: Test Flink - run: | - # run tests with random timezone to find out timezone related bugs - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - profile="flink-${{ matrix.flink_version }}" - if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1] }}" ]; then - mvn -T 1C -B -ntp test -Pflink1,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone - else - mvn -T 1C -B -ntp test -Pflink1,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile} - fi - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/e2e-tests-flink-2.x-jdk11.yml b/.github/workflows/e2e-tests-flink-2.x-jdk11.yml deleted file mode 100644 index 548916d106..0000000000 --- a/.github/workflows/e2e-tests-flink-2.x-jdk11.yml +++ /dev/null @@ -1,81 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: End to End Tests Flink 2.x on JDK 11 - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 11 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -jobs: - build_test: - runs-on: ubuntu-latest - timeout-minutes: 60 - strategy: - fail-fast: true - matrix: - # Only Test Latest Version - flink_version: [ '2.2' ] - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Flink - run: | - mvn -T 2C -B -ntp clean install -DskipTests -Pflink2,spark3 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11 - - - name: Test Flink - run: | - # run tests with random timezone to find out timezone related bugs - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - profile="flink-${{ matrix.flink_version }}" - if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1] }}" ]; then - mvn -T 1C -B -ntp test -Pflink2,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -Pjava11 - else - mvn -T 1C -B -ntp test -Pflink2,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile},java11 - fi - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/file-size-check.yml b/.github/workflows/file-size-check.yml deleted file mode 100644 index 9b4f073b0a..0000000000 --- a/.github/workflows/file-size-check.yml +++ /dev/null @@ -1,48 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: Check File Size - -on: - pull_request: - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -jobs: - check-file-size: - runs-on: ubuntu-latest - steps: - - name: Checkout code - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - - name: Check file size - run: | - files=$(git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }}) - for file in $files; do - if [ -f "$file" ]; then - size=$(ls -l "$file" | awk '{print $5}') - if [ "$size" -gt 1048576 ]; then - echo "Error: File $file is larger than 1MB ($size bytes)." - exit 1 - fi - fi - done diff --git a/.github/workflows/stale-pr.yml b/.github/workflows/maintenance-stale-pr.yml similarity index 98% rename from .github/workflows/stale-pr.yml rename to .github/workflows/maintenance-stale-pr.yml index ca5c3dae46..252f8c7237 100644 --- a/.github/workflows/stale-pr.yml +++ b/.github/workflows/maintenance-stale-pr.yml @@ -19,7 +19,7 @@ # # See [email protected] "Stale PR cleanup for Paimon" thread. -name: Stale PR reminder +name: Maintenance / Stale PR reminder on: schedule: diff --git a/.github/workflows/publish_snapshot-jdk17.yml b/.github/workflows/publish-snapshot-spark4.yml similarity index 89% rename from .github/workflows/publish_snapshot-jdk17.yml rename to .github/workflows/publish-snapshot-spark4.yml index 5835a4d14b..c1b8eabeb0 100644 --- a/.github/workflows/publish_snapshot-jdk17.yml +++ b/.github/workflows/publish-snapshot-spark4.yml @@ -16,7 +16,7 @@ # limitations under the License. ################################################################################ -name: Publish Snapshot on JDK 17 +name: Publish / Snapshot / Spark 4 on: schedule: @@ -29,12 +29,16 @@ env: MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true + group: publish-snapshot-spark4-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read jobs: publish-snapshot: if: github.repository == 'apache/paimon' + timeout-minutes: 120 runs-on: ubuntu-latest steps: - name: Checkout code @@ -48,9 +52,9 @@ jobs: uses: actions/cache@v5 with: path: ~/.m2/repository - key: snapshot-maven-${{ hashFiles('**/pom.xml') }} + key: snapshot-maven-spark4-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/pom.xml') }} restore-keys: | - snapshot-maven- + snapshot-maven-spark4-${{ runner.os }}-${{ runner.arch }}- - name: Publish snapshot env: ASF_USERNAME: ${{ secrets.NEXUS_USER }} diff --git a/.github/workflows/publish_snapshot.yml b/.github/workflows/publish-snapshot.yml similarity index 89% rename from .github/workflows/publish_snapshot.yml rename to .github/workflows/publish-snapshot.yml index b1a9335dc4..b1f201593c 100644 --- a/.github/workflows/publish_snapshot.yml +++ b/.github/workflows/publish-snapshot.yml @@ -16,7 +16,7 @@ # limitations under the License. ################################################################################ -name: Publish Snapshot +name: Publish / Snapshot / JDK 8 on: schedule: @@ -29,12 +29,16 @@ env: MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true + group: publish-snapshot-${{ github.ref }} + cancel-in-progress: false + +permissions: + contents: read jobs: publish-snapshot: if: github.repository == 'apache/paimon' + timeout-minutes: 120 runs-on: ubuntu-latest steps: - name: Checkout code @@ -50,9 +54,9 @@ jobs: uses: actions/cache@v5 with: path: ~/.m2/repository - key: snapshot-maven-${{ hashFiles('**/pom.xml') }} + key: snapshot-maven-jdk8-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/pom.xml') }} restore-keys: | - snapshot-maven- + snapshot-maven-jdk8-${{ runner.os }}-${{ runner.arch }}- - name: Publish snapshot env: ASF_USERNAME: ${{ secrets.NEXUS_USER }} diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml index 12160e9808..fe6ed56e0f 100644 --- a/.github/workflows/release-java.yml +++ b/.github/workflows/release-java.yml @@ -16,7 +16,7 @@ # limitations under the License. ################################################################################ -name: Release Java +name: Release / Java packages on: workflow_call: diff --git a/.github/workflows/release-python-publish.yml b/.github/workflows/release-python-publish.yml index 9ae869a590..a153ff2d4c 100644 --- a/.github/workflows/release-python-publish.yml +++ b/.github/workflows/release-python-publish.yml @@ -16,7 +16,7 @@ # limitations under the License. ################################################################################ -name: Release PyPaimon Publish +name: Release / Python publish on: workflow_call: diff --git a/.github/workflows/release-python.yml b/.github/workflows/release-python.yml index 0dc5112821..c8d3ef32e2 100644 --- a/.github/workflows/release-python.yml +++ b/.github/workflows/release-python.yml @@ -19,7 +19,7 @@ # Build the PyPaimon source distribution and universal wheel. Publishing is # handled by release-python-publish.yml after this package job passes. -name: Release PyPaimon Package +name: Release / Python packages on: workflow_call: diff --git a/.github/workflows/utcase-eslib.yml b/.github/workflows/utcase-eslib.yml deleted file mode 100644 index a5dc7d9d20..0000000000 --- a/.github/workflows/utcase-eslib.yml +++ /dev/null @@ -1,73 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase ESLib on JDK 11 - -on: - push: - paths: - - 'paimon-api/**' - - 'paimon-common/**' - - 'paimon-eslib/**' - - 'paimon-test-utils/**' - - 'pom.xml' - - '.github/workflows/utcase-eslib.yml' - pull_request: - paths: - - 'paimon-api/**' - - 'paimon-common/**' - - 'paimon-eslib/**' - - 'paimon-test-utils/**' - - 'pom.xml' - - '.github/workflows/utcase-eslib.yml' - -permissions: - contents: read - -env: - JDK_VERSION: 11 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - eslib_test: - runs-on: ubuntu-latest - timeout-minutes: 30 - - steps: - - name: Checkout code - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 - with: - persist-credentials: false - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build ESLib dependencies - run: mvn -T 2C -B -ntp clean install -DskipTests -Ppaimon-eslib -pl paimon-eslib -am - - - name: Test ESLib - run: mvn -B -ntp verify -Ppaimon-eslib -pl paimon-eslib - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utcase-full-text.yml b/.github/workflows/utcase-full-text.yml deleted file mode 100644 index c9904529cf..0000000000 --- a/.github/workflows/utcase-full-text.yml +++ /dev/null @@ -1,61 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase Full Text - -on: - push: - paths: - - 'paimon-full-text/**' - - 'pom.xml' - - '.github/workflows/utcase-full-text.yml' - pull_request: - paths: - - 'paimon-full-text/**' - - 'pom.xml' - - '.github/workflows/utcase-full-text.yml' - -env: - JDK_VERSION: 8 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - full_text_test: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build and test full-text module - timeout-minutes: 30 - run: | - mvn -T 2C -B -ntp clean install -DskipTests - mvn -B -ntp verify -pl paimon-full-text -Dcheckstyle.skip=true -Dspotless.check.skip=true - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utitcase-flink-1.x-common.yml b/.github/workflows/utitcase-flink-1.x-common.yml deleted file mode 100644 index 59d124804d..0000000000 --- a/.github/workflows/utitcase-flink-1.x-common.yml +++ /dev/null @@ -1,75 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ -name: UTCase and ITCase Flink 1.x Common - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 8 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build_test: - runs-on: ubuntu-latest - timeout-minutes: 100 - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Flink - run: | - COMPILE_MODULE="org.apache.paimon:paimon-flink-common" - echo "Start compiling modules: $COMPILE_MODULE" - mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3 -pl "${COMPILE_MODULE}" -am - - - name: Test Flink - run: | - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - TEST_MODULE="org.apache.paimon:paimon-flink-common" - echo "Start testing module: $TEST_MODULE" - mvn -T 2C -B -ntp test verify -Pflink1,spark3 -pl "${TEST_MODULE}" -Duser.timezone=$jvm_timezone - echo "All modules tested" - env: - MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2 diff --git a/.github/workflows/utitcase-flink-1.x-others.yml b/.github/workflows/utitcase-flink-1.x-others.yml deleted file mode 100644 index d042f1a8ed..0000000000 --- a/.github/workflows/utitcase-flink-1.x-others.yml +++ /dev/null @@ -1,75 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ -name: UTCase and ITCase Flink 1.x Others - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 8 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build_test: - runs-on: ubuntu-latest - timeout-minutes: 60 - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Flink - run: | - mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3 - - - name: Test Flink - run: | - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - test_modules="" - for suffix in cdc 1.16 1.17 1.18 1.19 1.20; do - test_modules+="org.apache.paimon:paimon-flink-${suffix}," - done - test_modules="${test_modules%,}" - mvn -T 2C -B -ntp test verify -Pflink1,spark3 -pl "${test_modules}" -Duser.timezone=$jvm_timezone - env: - MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2 diff --git a/.github/workflows/utitcase-flink-2.x-jdk11.yml b/.github/workflows/utitcase-flink-2.x-jdk11.yml deleted file mode 100644 index b1b6005867..0000000000 --- a/.github/workflows/utitcase-flink-2.x-jdk11.yml +++ /dev/null @@ -1,76 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase and ITCase Flink 2.x on JDK 11 - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 11 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -jobs: - build: - runs-on: ubuntu-latest - timeout-minutes: 120 - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Flink - run: | - mvn -T 2C -B -ntp clean install -DskipTests -Pflink2,spark3 - - - name: Test Flink - run: | - # run tests with random timezone to find out timezone related bugs - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - test_modules="" - for suffix in 2.0 2.1 2.2 common; do - test_modules+="org.apache.paimon:paimon-flink-${suffix}," - done - test_modules="${test_modules%,}" - mvn -T 2C -B -ntp test verify -Pflink2,spark3 -pl "${test_modules}" -Duser.timezone=$jvm_timezone - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utitcase-iceberg-ga.yml b/.github/workflows/utitcase-iceberg-ga.yml deleted file mode 100644 index deca2434fc..0000000000 --- a/.github/workflows/utitcase-iceberg-ga.yml +++ /dev/null @@ -1,66 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase Iceberg GA row lineage on JDK 17 - -on: - push: - paths: - - 'paimon-iceberg/**' - - 'paimon-core/**' - - 'paimon-common/**' - - 'paimon-api/**' - - 'paimon-format/**' - - 'pom.xml' - - '.github/workflows/utitcase-iceberg-ga.yml' - pull_request: - paths: - - 'paimon-iceberg/**' - - 'paimon-core/**' - - 'paimon-common/**' - - 'paimon-api/**' - - 'paimon-format/**' - - 'pom.xml' - - '.github/workflows/utitcase-iceberg-ga.yml' - -env: - JDK_VERSION: 17 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - name: Build - run: mvn -T 1C -B -ntp clean install -DskipTests -pl paimon-iceberg -am -Ppaimon-iceberg,iceberg-ga - - name: Test against GA Iceberg - run: mvn -B -ntp test -pl paimon-iceberg -Ppaimon-iceberg,iceberg-ga - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utitcase-jdk11.yml b/.github/workflows/utitcase-jdk11.yml deleted file mode 100644 index a6ae09a52b..0000000000 --- a/.github/workflows/utitcase-jdk11.yml +++ /dev/null @@ -1,72 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase and ITCase Others on JDK 11 - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 11 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - name: Build - run: mvn -T 1C -B -ntp clean install -DskipTests -Pflink1,spark3 - - name: Test - run: | - # run tests with random timezone to find out timezone related bugs - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - test_modules="!paimon-e2e-tests,!org.apache.paimon:paimon-hive-connector-3.1," - for suffix in ut 3.5 3.4 3.3 3.2; do - test_modules+="!org.apache.paimon:paimon-spark-${suffix}_2.12," - done - test_modules="${test_modules%,}" - mvn -T 1C -B -ntp clean install -pl "${test_modules}" -Pflink1,spark3 -Pskip-paimon-flink-tests -Duser.timezone=$jvm_timezone - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utitcase-spark-3.x.yml b/.github/workflows/utitcase-spark-3.x.yml deleted file mode 100644 index 0fe7923060..0000000000 --- a/.github/workflows/utitcase-spark-3.x.yml +++ /dev/null @@ -1,79 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase and ITCase Spark 3.x - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 8 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build_test: - runs-on: ubuntu-latest - strategy: - fail-fast: true - matrix: - scala_version: [ '2.12', '2.13' ] - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Spark 3 with ${{ matrix.scala_version }} - run: mvn -T 2C -B -ntp clean install -DskipTests -Pspark3,flink1,scala-${{ matrix.scala_version }} - - - name: Test Spark 3 with ${{ matrix.scala_version }} - timeout-minutes: 60 - run: | - # run tests with random timezone to find out timezone related bugs - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - test_modules="" - for suffix in ut 3.5 3.4 3.3 3.2; do - test_modules+="org.apache.paimon:paimon-spark-${suffix}_${{ matrix.scala_version }}," - done - test_modules="${test_modules%,}" - mvn -T 2C -B -ntp verify -pl "${test_modules}" -Duser.timezone=$jvm_timezone -Pspark3,flink1,scala-${{ matrix.scala_version }} - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utitcase-spark-4.x.yml b/.github/workflows/utitcase-spark-4.x.yml deleted file mode 100644 index 72c43c018e..0000000000 --- a/.github/workflows/utitcase-spark-4.x.yml +++ /dev/null @@ -1,76 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase and ITCase Spark 4.x - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 17 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build_test: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Spark - run: mvn -T 2C -B -ntp clean install -DskipTests -Pspark4,flink1 - - - name: Test Spark - timeout-minutes: 60 - run: | - # run tests with random timezone to find out timezone related bugs - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - test_modules="" - for suffix in ut 4.0 4.1; do - test_modules+="org.apache.paimon:paimon-spark-${suffix}_2.13," - done - test_modules="${test_modules%,}" - mvn -T 2C -B -ntp verify -pl "${test_modules}" -Duser.timezone=$jvm_timezone -Pspark4,flink1 - env: - MAVEN_OPTS: -Xmx4096m diff --git a/.github/workflows/utitcase.yml b/.github/workflows/utitcase.yml deleted file mode 100644 index e83be074f4..0000000000 --- a/.github/workflows/utitcase.yml +++ /dev/null @@ -1,78 +0,0 @@ -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ - -name: UTCase and ITCase Others - -on: - push: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - pull_request: - paths-ignore: - - 'docs/**' - - '**/*.md' - - 'paimon-python/**' - - '.github/workflows/paimon-python-checks.yml' - - 'paimon-core/src/test/java/org/apache/paimon/JavaPyE2ETest.java' - -env: - JDK_VERSION: 8 - MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=30 -Dmaven.wagon.http.retryHandler.requestSentEnabled=true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.number || github.run_id }} - cancel-in-progress: true - -jobs: - build_test: - runs-on: ubuntu-latest - - steps: - - name: Checkout code - uses: actions/checkout@v6 - - - name: Set up JDK ${{ env.JDK_VERSION }} - uses: actions/setup-java@v5 - with: - java-version: ${{ env.JDK_VERSION }} - distribution: 'temurin' - - - name: Build Others - run: | - echo "Start compiling modules" - mvn -T 2C -B -ntp clean install -DskipTests -Pflink1,spark3 - - - name: Test Others - timeout-minutes: 60 - run: | - . .github/workflows/utils.sh - jvm_timezone=$(random_timezone) - echo "JVM timezone is set to $jvm_timezone" - - TEST_MODULES="!paimon-e2e-tests," - for suffix in ut 3.5 3.4 3.3 3.2; do - TEST_MODULES+="!org.apache.paimon:paimon-spark-${suffix}_2.12," - done - TEST_MODULES="${TEST_MODULES%,}" - mvn -T 2C -B -ntp clean install -pl "${TEST_MODULES}" -Pskip-paimon-flink-tests -Duser.timezone=$jvm_timezone - env: - MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2 diff --git a/docs/docs/project/verifying-a-release-candidate.md b/docs/docs/project/verifying-a-release-candidate.md index c0a5b55933..f3cd609281 100644 --- a/docs/docs/project/verifying-a-release-candidate.md +++ b/docs/docs/project/verifying-a-release-candidate.md @@ -343,7 +343,7 @@ archive; the PyPaimon source distribution must not be used by itself for an independent release. Run the tests from `paimon-PAIMON_VERSION/paimon-python` on as many supported Python versions as your environment allows. Use the Python matrix in the candidate's -`.github/workflows/paimon-python-checks.yml` to select relevant versions rather +`.github/workflows/ci-python.yml` to select relevant versions rather than assuming that a matrix from another release applies: ```shell diff --git a/tools/ci/plan.py b/tools/ci/plan.py new file mode 100644 index 0000000000..b539f45a60 --- /dev/null +++ b/tools/ci/plan.py @@ -0,0 +1,180 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Select CI suites from the complete git diff, without API pagination limits.""" + +import json +import os +from pathlib import Path +import re +import subprocess + + +# Keep the existing JDK, engine and Scala coverage. A suite is a test boundary, +# not just a source directory: core also owns Hive and generated-doc tests. +SUITES = [ + ("core", "Core and integrations / JDK 8", 8, "2.12", 90), + ("core", "Core and integrations / JDK 11", 11, "2.12", 90), + ("flink1-common", "Flink 1 / Common", 8, "2.12", 100), + ("flink1-connectors", "Flink 1 / Connectors and CDC", 8, "2.12", 60), + ("flink2", "Flink 2 / Common and connectors", 11, "2.12", 120), + ("spark3", "Spark 3 / Scala 2.12", 8, "2.12", 90), + ("spark3", "Spark 3 / Scala 2.13", 8, "2.13", 90), + ("spark4", "Spark 4 / Scala 2.13", 17, "2.13", 90), + ("e2e-flink1", "E2E / Flink 1 and Spark interoperability", 8, "2.12", 60), + ("e2e-flink2", "E2E / Flink 2", 11, "2.12", 60), + ("iceberg", "Iceberg GA / JDK 17", 17, "2.12", 60), + ("eslib", "ESLib / JDK 11", 11, "2.12", 30), + ("full-text", "Full text / JDK 8", 8, "2.12", 30), +] +ALL_JAVA = {suite[0] for suite in SUITES} +FLINK = {"flink1-common", "flink1-connectors", "flink2"} +SPARK = {"spark3", "spark4"} +E2E = {"e2e-flink1", "e2e-flink2"} + + +def select(paths, full=False): + selected = set() + docs = python = licensing = False + if full: + selected.update(ALL_JAVA) + docs = python = licensing = True + + for path in paths: + if path.startswith("docs/"): + docs = True + continue + if path.endswith(".md"): + continue + if path.startswith("paimon-python/") or path == ".github/workflows/ci-python.yml": + python = True + continue + if re.search(r"/src/test/.*?/JavaPy[^/]*\.java$", path): + python = True + continue + if path == ".github/workflows/ci-docs.yml": + docs = True + continue + if path == ".github/workflows/ci-licensing.yml": + licensing = True + continue + # Changes to CI machinery must validate every consumer of that machinery. + if path.startswith((".github/workflows/ci", "tools/ci/")): + selected.update(ALL_JAVA) + docs = python = licensing = True + continue + if path.startswith((".github/", ".asf.yaml")): + continue + + licensing = True + if path.startswith("paimon-spark/"): + # Core includes spark-common tests and paimon-docs completeness tests. + # SparkE2eTest is disabled on JDK 11, so only the Flink 1 E2E lane applies. + selected.update(SPARK | {"core", "e2e-flink1"}) + elif path.startswith("paimon-flink/"): + # Hive and Iceberg integration tests depend on flink-common. + selected.update(FLINK | E2E | {"core", "iceberg"}) + elif path.startswith("paimon-e2e-tests/"): + selected.update(E2E) + elif path.startswith("paimon-eslib/"): + selected.add("eslib") + elif path.startswith("paimon-iceberg/"): + selected.update({"core", "iceberg"}) + elif path.startswith("paimon-full-text/"): + selected.update({"core", "full-text"}) + else: + # Shared modules, POMs outside a known leaf, build configuration and + # new/unknown directories fall back to all Java suites. + selected.update(ALL_JAVA) + if path == "pom.xml": + python = True + + matrix = [ + dict(suite=suite, name=name, java=java, scala=scala, timeout=timeout) + for suite, name, java, scala, timeout in SUITES + if suite in selected + ] + return dict(java=bool(matrix), matrix={"include": matrix}, docs=docs, + python=python, licensing=licensing) + + +def changed_paths(event_name, event): + if event_name == "pull_request": + base = event["pull_request"]["base"]["sha"] + head = event["pull_request"]["head"]["sha"] + separator = "..." + elif event_name == "push": + base, head = event["before"], event["after"] + separator = ".." + else: + return None + for sha in (base, head): + if not re.fullmatch(r"[0-9a-f]{40}", sha) or sha == "0" * 40: + return None + try: + # Disabling rename detection includes both the old and new path. -z also + # handles whitespace, newlines and non-ASCII names without shell parsing. + output = subprocess.check_output([ + "git", "diff", "--no-renames", "--name-only", "-z", + base + separator + head, "--", + ]) + except subprocess.CalledProcessError: + print("Cannot resolve the complete diff; selecting full CI.") + return None + return [os.fsdecode(path) for path in output.split(b"\0") if path] + + +def check_file_sizes(paths, root=Path(".")): + for path in paths: + candidate = root / path + # Git symlinks contain a link target; don't follow them outside checkout. + if candidate.is_symlink() or not candidate.is_file(): + continue + if candidate.stat().st_size > 1048576: + raise ValueError("Changed file exceeds 1 MiB: " + repr(path)) + + +def main(): + event_name = os.environ["GITHUB_EVENT_NAME"] + event = json.loads(Path(os.environ["GITHUB_EVENT_PATH"]).read_text()) + paths = changed_paths(event_name, event) + plan = select(paths or [], full=paths is None) + size_warning = None + if event_name == "pull_request": + if paths is None: + # Existing oversized files must not block the full-CI fallback. + size_warning = "Skipping changed-file size check because the PR diff is unavailable." + print("::warning::" + size_warning) + else: + check_file_sizes(paths) + with open(os.environ["GITHUB_OUTPUT"], "a") as output: + for key, value in plan.items(): + output.write(key + "=" + json.dumps(value, separators=(",", ":")) + "\n") + with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as summary: + summary.write("## CI selection\n\n") + summary.write("Full run (manual or unavailable diff).\n\n" if paths is None + else "Compared {} changed paths.\n\n".format(len(paths))) + if size_warning: + summary.write("**Warning:** " + size_warning + "\n\n") + for lane in plan["matrix"]["include"]: + summary.write("- " + lane["name"] + "\n") + for key in ("python", "docs", "licensing"): + summary.write("- {}: {}\n".format(key, "run" if plan[key] else "skip")) + print(json.dumps(plan, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/tools/ci/run-java-tests.sh b/tools/ci/run-java-tests.sh new file mode 100644 index 0000000000..b5a7de266c --- /dev/null +++ b/tools/ci/run-java-tests.sh @@ -0,0 +1,129 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +suite=${1:?Specify a CI suite} +scala=${2:-2.12} +java=${3:-8} +profiles=flink1,spark3 +threads=2C +goal=verify +test_options=(-Dstyle.color=never) +modules=() + +case "$suite" in + core) + # Keep spark-common and paimon-docs here: they have their own tests, + # including JDK 11 coverage, outside the Spark connector test modules. + modules=('!paimon-e2e-tests') + for version in ut 3.5 3.4 3.3 3.2; do + modules+=("!org.apache.paimon:paimon-spark-${version}_2.12") + done + if [[ "$java" == 11 ]]; then + modules+=('!org.apache.paimon:paimon-hive-connector-3.1') + threads=1C + fi + test_options+=(-Pskip-paimon-flink-tests) + ;; + flink1-common) + modules=(paimon-flink/paimon-flink-common) + ;; + flink1-connectors) + for version in cdc 1.16 1.17 1.18 1.19 1.20; do + modules+=("org.apache.paimon:paimon-flink-${version}") + done + ;; + flink2) + profiles=flink2,spark3 + for version in 2.0 2.1 2.2 common; do + modules+=("org.apache.paimon:paimon-flink-${version}") + done + ;; + spark3) + profiles="flink1,spark3,scala-${scala}" + for version in ut 3.5 3.4 3.3 3.2; do + modules+=("org.apache.paimon:paimon-spark-${version}_${scala}") + done + ;; + spark4) + profiles=flink1,spark4 + for version in ut 4.0 4.1; do + modules+=("org.apache.paimon:paimon-spark-${version}_2.13") + done + ;; + e2e-flink1) + profiles=flink1,spark3,flink-1.20 + modules=(paimon-e2e-tests) + threads=1C + goal=test + ;; + e2e-flink2) + profiles=flink2,spark3,flink-2.2,java11 + modules=(paimon-e2e-tests) + threads=1C + goal=test + ;; + iceberg) + profiles=flink1,spark3,paimon-iceberg,iceberg-ga + modules=(paimon-iceberg) + threads=1C + goal=test + ;; + eslib) + profiles=paimon-eslib + modules=(paimon-eslib) + threads=1 + ;; + full-text) + modules=(paimon-full-text) + threads=1 + ;; + *) + echo "Unknown CI suite: $suite" >&2 + exit 1 + ;; +esac + +module_list=$(IFS=,; echo "${modules[*]}") + +run_maven() { + if [[ "${CI_DRY_RUN:-false}" == true ]]; then + printf '%q ' mvn "$@" + printf '\n' + else + mvn "$@" + fi +} + +# -am compiles and installs the selected tests' dependency closure, including +# test jars. Only the selected modules run tests in the second invocation. +run_maven -T 2C -B -ntp install -DskipTests -P"$profiles" -pl "$module_list" -am + +jvm_timezone=GMT+00:00 +if [[ "${CI_DRY_RUN:-false}" != true ]]; then + # Preserve the native-library setup used by the original engine/core lanes. + if [[ "$suite" != eslib && "$suite" != full-text ]]; then + . tools/ci/utils.sh + jvm_timezone=$(random_timezone) + fi + echo "JVM timezone is set to $jvm_timezone" +fi + +# verify already includes the test phase; don't execute `test verify` or clean +# away the dependencies and test classes just built in this fresh checkout. +run_maven -T "$threads" -B -ntp "$goal" -P"$profiles" -pl "$module_list" \ + "${test_options[@]}" -Duser.timezone="$jvm_timezone" diff --git a/tools/ci/test_java_commands.py b/tools/ci/test_java_commands.py new file mode 100644 index 0000000000..5cfc7aa487 --- /dev/null +++ b/tools/ci/test_java_commands.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Validate suite coverage and Maven command boundaries without running Maven.""" + +import os +from pathlib import Path +import re +import shlex +import subprocess +import unittest +import xml.etree.ElementTree as ET + +from plan import SUITES + + +SCRIPT = Path(__file__).with_name('run-java-tests.sh') + + +class MavenCommandsTest(unittest.TestCase): + def commands(self, suite, scala='2.12', java=8): + output = subprocess.check_output( + ['bash', str(SCRIPT), suite, scala, str(java)], + env=dict(os.environ, CI_DRY_RUN='true'), text=True) + return [shlex.split(line) for line in output.splitlines()] + + def modules(self, command): + return set(command[command.index('-pl') + 1].split(',')) + + def test_every_lane_builds_dependencies_but_only_tests_selected_modules(self): + for suite, name, java, scala, timeout in SUITES: + with self.subTest(name=name): + build, test = self.commands(suite, scala, java) + self.assertIn('-am', build) + self.assertIn('-DskipTests', build) + self.assertNotIn('-am', test) + self.assertNotIn('-DskipTests', test) + self.assertEqual(self.modules(build), self.modules(test)) + self.assertNotIn('clean', test) + self.assertFalse('test' in test and 'verify' in test) + self.assertFalse(any('fast-build' in arg for arg in build + test)) + + def test_spark_retains_all_connector_versions_and_scala_variants(self): + for scala in ('2.12', '2.13'): + test = self.commands('spark3', scala)[1] + for version in ('ut', '3.2', '3.3', '3.4', '3.5'): + self.assertIn('org.apache.paimon:paimon-spark-' + version + '_' + scala, + self.modules(test)) + self.assertIn('-Pflink1,spark3,scala-' + scala, test) + self.assertEqual(self.modules(self.commands('spark4', '2.13', 17)[1]), + {'org.apache.paimon:paimon-spark-' + version + '_2.13' + for version in ('ut', '4.0', '4.1')}) + + def test_core_keeps_common_and_docs_tests_and_existing_jdk11_exclusion(self): + for java in (8, 11): + test = self.commands('core', java=java)[1] + self.assertIn('-Pskip-paimon-flink-tests', test) + self.assertNotIn('!org.apache.paimon:paimon-spark-common_2.12', self.modules(test)) + self.assertNotIn('!paimon-docs', self.modules(test)) + self.assertEqual('!org.apache.paimon:paimon-hive-connector-3.1' in self.modules(test), + java == 11) + + def test_flink_coverage_includes_cdc_and_both_major_versions(self): + self.assertEqual(self.modules(self.commands('flink1-connectors')[1]), + {'org.apache.paimon:paimon-flink-' + version + for version in ('cdc', '1.16', '1.17', '1.18', '1.19', '1.20')}) + self.assertEqual(self.modules(self.commands('flink2', java=11)[1]), + {'org.apache.paimon:paimon-flink-' + version + for version in ('common', '2.0', '2.1', '2.2')}) + + def test_e2e_uses_the_same_version_when_building_and_testing(self): + for suite, java, version in [('e2e-flink1', 8, '1.20'), ('e2e-flink2', 11, '2.2')]: + build, test = self.commands(suite, java=java) + self.assertTrue(any('flink-' + version in arg for arg in build)) + self.assertEqual([arg for arg in build if arg.startswith('-P')], + [arg for arg in test if arg.startswith('-P')]) + self.assertIn('test', test) + + def test_unknown_suite_fails_instead_of_silently_running_no_tests(self): + result = subprocess.run(['bash', str(SCRIPT), 'typo'], + env=dict(os.environ, CI_DRY_RUN='true'), capture_output=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn(b'Unknown CI suite', result.stderr) + + def test_python_build_selects_runnable_modules_not_just_aggregator_poms(self): + root = SCRIPT.resolve().parents[2] + workflow = (root / '.github/workflows/ci-python.yml').read_text() + modules = set(re.search(r'-pl ([^\s]+) -am', workflow).group(1).split(',')) + self.assertTrue({'paimon-core', 'paimon-lance', 'paimon-lumina', + 'paimon-vortex/paimon-vortex-format', 'paimon-full-text', + 'paimon-vector'}.issubset(modules)) + for module in modules: + pom = ET.parse(root / module / 'pom.xml').getroot() + packaging = pom.findtext('{http://maven.apache.org/POM/4.0.0}packaging', 'jar') + self.assertNotEqual(packaging, 'pom', module) + + +if __name__ == '__main__': + unittest.main() diff --git a/tools/ci/test_plan.py b/tools/ci/test_plan.py new file mode 100644 index 0000000000..4d0c26a174 --- /dev/null +++ b/tools/ci/test_plan.py @@ -0,0 +1,238 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Behavioral checks for CI routing and the complete PR/push diff.""" + +import contextlib +import io +import json +import os +from pathlib import Path +import subprocess +import tempfile +import unittest +from unittest.mock import patch + +import plan + + +class SelectionTest(unittest.TestCase): + def suites(self, paths): + return {lane['suite'] for lane in plan.select(paths)['matrix']['include']} + + def test_spark_keeps_common_docs_and_interoperability_tests(self): + suites = self.suites(['paimon-spark/paimon-spark-common/src/main/scala/Change.scala']) + self.assertEqual(suites, {'core', 'spark3', 'spark4', 'e2e-flink1'}) + + def test_flink_keeps_hive_iceberg_and_both_e2e_versions(self): + suites = self.suites(['paimon-flink/paimon-flink-common/src/main/java/Change.java']) + self.assertEqual(suites, {'core', 'flink1-common', 'flink1-connectors', + 'flink2', 'e2e-flink1', 'e2e-flink2', 'iceberg'}) + + def test_shared_and_unknown_changes_run_all_java(self): + for path in ['paimon-core/Change.java', 'paimon-api/Change.java', + 'paimon-format/pom.xml', 'paimon-new-module/pom.xml', + '.mvn/jvm.config', '.scalafmt.conf', 'copyright.txt']: + with self.subTest(path=path): + self.assertEqual(self.suites([path]), plan.ALL_JAVA) + + def test_parent_pom_also_validates_python_build(self): + self.assertTrue(plan.select(['pom.xml'])['python']) + self.assertEqual(self.suites(['pom.xml']), plan.ALL_JAVA) + + def test_docs_and_markdown_do_not_start_maven(self): + result = plan.select(['docs/start.md', 'README.md']) + self.assertTrue(result['docs']) + self.assertFalse(result['java']) + self.assertFalse(result['python']) + self.assertFalse(result['licensing']) + self.assertFalse(plan.select(['paimon-python/README.md'])['python']) + + def test_mixed_changes_take_union(self): + self.assertEqual(self.suites(['docs/intro.md', 'paimon-spark/pom.xml', + 'paimon-flink/pom.xml']), + {'core', 'spark3', 'spark4', 'flink1-common', + 'flink1-connectors', 'flink2', 'e2e-flink1', + 'e2e-flink2', 'iceberg'}) + result = plan.select(['paimon-python/setup.py', 'paimon-core/Change.java']) + self.assertTrue(result['python']) + self.assertTrue(result['java']) + + def test_java_python_fixtures_run_python(self): + for module, name in [('paimon-core', 'JavaPyE2ETest'), + ('paimon-lance', 'JavaPyLanceE2ETest'), + ('paimon-lumina', 'JavaPyLuminaE2ETest')]: + result = plan.select([module + '/src/test/java/org/apache/paimon/' + name + '.java']) + self.assertTrue(result['python']) + self.assertFalse(result['java']) + + def test_leaf_modules_and_workflows(self): + self.assertEqual(self.suites(['paimon-eslib/pom.xml']), {'eslib'}) + self.assertEqual(self.suites(['paimon-e2e-tests/pom.xml']), + {'e2e-flink1', 'e2e-flink2'}) + self.assertTrue(plan.select(['.github/workflows/ci-python.yml'])['python']) + self.assertFalse(plan.select(['.github/workflows/ci-python.yml'])['java']) + self.assertTrue(plan.select(['.github/workflows/ci-docs.yml'])['docs']) + self.assertFalse(plan.select(['.github/workflows/release-java.yml'])['java']) + + def test_ci_logic_and_manual_run_validate_every_consumer(self): + for result in [plan.select(['tools/ci/plan.py']), + plan.select(['.github/workflows/ci-java.yml']), + plan.select([], full=True)]: + self.assertTrue(all(result[key] for key in ('java', 'python', 'docs', 'licensing'))) + self.assertEqual(len(result['matrix']['include']), 13) + + def test_empty_diff_does_not_run_expensive_jobs(self): + result = plan.select([]) + self.assertFalse(any(result[key] for key in ('java', 'python', 'docs', 'licensing'))) + self.assertEqual(result['matrix'], {'include': []}) + + def test_size_guard_handles_spaces_deletions_and_symlinks(self): + with tempfile.TemporaryDirectory() as temp: + root = Path(temp) + large = root / 'large file.txt' + large.write_bytes(b'x' * (1048576 + 1)) + (root / 'link').symlink_to(large) + plan.check_file_sizes(['deleted.txt', 'link'], root) + with self.assertRaisesRegex(ValueError, 'large file'): + plan.check_file_sizes(['large file.txt'], root) + + +class GitDiffTest(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.previous = Path.cwd() + os.chdir(self.temp.name) + self.addCleanup(os.chdir, self.previous) + self.git('init', '-q') + self.git('config', 'user.email', '[email protected]') + self.git('config', 'user.name', 'CI test') + self.commit('base.txt', 'base') + self.base = self.git('rev-parse', 'HEAD') + + def git(self, *args): + return subprocess.check_output(['git', *args], stderr=subprocess.DEVNULL).decode().strip() + + def commit(self, name, content): + path = Path(name) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(content) + self.git('add', '--', name) + self.git('commit', '-qm', 'Test commit') + + def test_pr_uses_merge_base_instead_of_including_base_branch_changes(self): + self.commit('base-only.txt', 'base change') + base_tip = self.git('rev-parse', 'HEAD') + self.git('checkout', '-q', '--detach', self.base) + self.commit('paimon-spark/name with spaces\nand newline.scala', 'spark') + head = self.git('rev-parse', 'HEAD') + paths = plan.changed_paths('pull_request', { + 'pull_request': {'base': {'sha': base_tip}, 'head': {'sha': head}}}) + self.assertEqual(paths, ['paimon-spark/name with spaces\nand newline.scala']) + + def test_push_includes_all_commits_and_both_sides_of_renames(self): + self.commit('paimon-spark/A.scala', 'spark') + before = self.git('rev-parse', 'HEAD') + Path('paimon-flink').mkdir() + self.git('mv', 'paimon-spark/A.scala', 'paimon-flink/A.java') + self.git('commit', '-qm', 'Move across modules') + self.commit('docs/intro.md', 'docs') + paths = plan.changed_paths('push', {'before': before, + 'after': self.git('rev-parse', 'HEAD')}) + self.assertEqual(set(paths), {'paimon-spark/A.scala', 'paimon-flink/A.java', + 'docs/intro.md'}) + + def test_large_diffs_do_not_truncate_late_core_changes(self): + for index in range(350): + Path('doc{:03d}.md'.format(index)).write_text('docs') + self.commit('z-new-core/Change.java', 'core') + self.git('add', '.') + self.git('commit', '-qm', 'Many docs') + paths = plan.changed_paths('push', {'before': self.base, + 'after': self.git('rev-parse', 'HEAD')}) + self.assertEqual(len(paths), 351) + self.assertTrue(plan.select(paths)['java']) + + def test_new_branch_missing_commits_and_manual_run_fall_back(self): + self.assertIsNone(plan.changed_paths('push', {'before': '0' * 40, 'after': self.base})) + self.assertIsNone(plan.changed_paths('workflow_dispatch', {})) + with patch('subprocess.check_output', side_effect=subprocess.CalledProcessError(128, 'git')): + self.assertIsNone(plan.changed_paths('push', {'before': 'a' * 40, 'after': self.base})) + + def run_pr_plan(self, base, head): + Path('event.json').write_text(json.dumps({ + 'pull_request': {'base': {'sha': base}, 'head': {'sha': head}}})) + env = {'GITHUB_EVENT_NAME': 'pull_request', 'GITHUB_EVENT_PATH': 'event.json', + 'GITHUB_OUTPUT': 'output.txt', 'GITHUB_STEP_SUMMARY': 'summary.md'} + with patch.dict(os.environ, env), contextlib.redirect_stdout(io.StringIO()) as log: + plan.main() + outputs = dict(line.split('=', 1) for line in Path('output.txt').read_text().splitlines()) + return {key: json.loads(value) for key, value in outputs.items()}, log.getvalue() + + def test_pr_unavailable_diff_runs_full_ci_with_unchanged_oversized_file(self): + self.commit('docs/static/old image.png', 'x' * (1048576 + 1)) + base = self.git('rev-parse', 'HEAD') + self.commit('paimon-spark/Change.scala', 'spark') + head = self.git('rev-parse', 'HEAD') + git_output = subprocess.check_output + + def unavailable_diff(command, *args, **kwargs): + if command[:2] == ['git', 'diff']: + raise subprocess.CalledProcessError(128, command) + return git_output(command, *args, **kwargs) + + with patch('subprocess.check_output', side_effect=unavailable_diff): + outputs, log = self.run_pr_plan(base, head) + self.assertTrue(all(outputs[key] for key in ('java', 'python', 'docs', 'licensing'))) + self.assertEqual(len(outputs['matrix']['include']), 13) + summary = Path('summary.md').read_text() + self.assertIn('Full run (manual or unavailable diff).', summary) + self.assertIn('Skipping changed-file size check', log) + self.assertIn('Skipping changed-file size check', summary) + + def test_pr_size_guard_ignores_unchanged_but_rejects_changed_oversized_files(self): + self.commit('docs/static/old image.png', 'x' * (1048576 + 1)) + base = self.git('rev-parse', 'HEAD') + self.commit('paimon-spark/Change.scala', 'spark') + outputs, _ = self.run_pr_plan(base, self.git('rev-parse', 'HEAD')) + self.assertEqual(len(outputs['matrix']['include']), 6) + + for path in ('docs/static/old image.png', 'docs/static/new image.png'): + with self.subTest(path=path): + base = self.git('rev-parse', 'HEAD') + self.commit(path, 'y' * (1048576 + 1)) + with self.assertRaisesRegex(ValueError, 'Changed file exceeds 1 MiB') as error: + self.run_pr_plan(base, self.git('rev-parse', 'HEAD')) + self.assertIn(path, str(error.exception)) + + def test_github_outputs_are_valid_json_and_summary_explains_selection(self): + self.commit('paimon-spark/Change.scala', 'spark') + Path('event.json').write_text(json.dumps({'before': self.base, + 'after': self.git('rev-parse', 'HEAD')})) + env = {'GITHUB_EVENT_NAME': 'push', 'GITHUB_EVENT_PATH': 'event.json', + 'GITHUB_OUTPUT': 'output.txt', 'GITHUB_STEP_SUMMARY': 'summary.md'} + with patch.dict(os.environ, env), contextlib.redirect_stdout(None): + plan.main() + outputs = dict(line.split('=', 1) for line in Path('output.txt').read_text().splitlines()) + self.assertTrue(json.loads(outputs['java'])) + self.assertFalse(json.loads(outputs['python'])) + self.assertEqual(len(json.loads(outputs['matrix'])['include']), 6) + self.assertIn('Spark interoperability', Path('summary.md').read_text()) + + +if __name__ == '__main__': + unittest.main() diff --git a/tools/ci/test_result.py b/tools/ci/test_result.py new file mode 100644 index 0000000000..ff373e3216 --- /dev/null +++ b/tools/ci/test_result.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""The required CI check must not pass when a selected group did not pass.""" + +import json +import os +from pathlib import Path +import subprocess +import sys +import textwrap +import unittest + + +class ResultTest(unittest.TestCase): + def run_gate(self, plan_result='success', **outcomes): + workflow = Path(__file__).resolve().parents[2] / '.github/workflows/ci.yml' + source = workflow.read_text().split("python3 - <<'PYTHON'\n", 1)[1] + source = textwrap.dedent(source.split(' PYTHON', 1)[0]) + flags = dict(java='true', python='false', docs='false', licensing='true') + results = dict(plan=dict(result=plan_result, outputs=flags), + java=dict(result='success'), python=dict(result='skipped'), + docs=dict(result='skipped'), licensing=dict(result='success')) + for name, result in outcomes.items(): + results[name]['result'] = result + return subprocess.run([sys.executable, '-c', source], capture_output=True, + env=dict(os.environ, RESULTS=json.dumps(results))) + + def test_selected_groups_pass_and_unselected_groups_skip(self): + self.assertEqual(self.run_gate().returncode, 0) + + def test_selected_failure_cancellation_and_unexpected_skip_fail(self): + for outcome in ('failure', 'cancelled', 'skipped'): + with self.subTest(outcome=outcome): + self.assertNotEqual(self.run_gate(java=outcome).returncode, 0) + + def test_planner_failure_fails_even_if_other_groups_report_success(self): + for outcome in ('failure', 'cancelled', 'skipped'): + self.assertNotEqual(self.run_gate(plan_result=outcome).returncode, 0) + + +if __name__ == '__main__': + unittest.main() diff --git a/.github/workflows/utils.sh b/tools/ci/utils.sh similarity index 100% rename from .github/workflows/utils.sh rename to tools/ci/utils.sh
