voonhous commented on code in PR #19217:
URL: https://github.com/apache/hudi/pull/19217#discussion_r3701773133
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -0,0 +1,157 @@
+name: Hudi Trino E2E
+
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ # TEMPORARY: validate the workflow on the dev branch; drop this line
+ # (and the commit that added it) before merging upstream.
+ - trino-e2e-testcontainers
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ workflow_dispatch:
+
+concurrency:
+ group: hudi-trino-e2e-${{ github.ref }}
+ cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'release-') }}
+
+env:
+ MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log
Review Comment:
Copied `bot.yml`'s retry block into `MVN_ARGS` verbatim. One nit: those
flags are `maven.wagon.httpconnectionManager.ttlSeconds` /
`maven.wagon.http.retryHandler.count`, not `aether.connector.*` -- carried them
exactly as `bot.yml` has them.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -0,0 +1,157 @@
+name: Hudi Trino E2E
+
+on:
+ push:
+ branches:
+ - master
+ - 'release-*'
+ # TEMPORARY: validate the workflow on the dev branch; drop this line
+ # (and the commit that added it) before merging upstream.
+ - trino-e2e-testcontainers
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ pull_request:
+ branches:
+ - master
+ - 'release-*'
+ paths:
+ - 'hudi-trino/**'
+ - 'docker/trino/**'
+ - 'docker/compose/docker-compose_hadoop340_hive2310_spark402*'
+ - 'docker/demo/sparksql-stock-ticks-trino.commands'
+ - 'hudi-integ-test/src/test/java/org/apache/hudi/integ2/**'
+ - '.github/workflows/hudi_trino_e2e.yml'
+ workflow_dispatch:
+
+concurrency:
+ group: hudi-trino-e2e-${{ github.ref }}
+ cancel-in-progress: ${{ !contains(github.ref, 'master') &&
!contains(github.ref, 'release-') }}
+
+env:
+ MVN_ARGS: -e -ntp -B -V -Dgpg.skip -Djacoco.skip -Pwarn-log
+ SCALA_PROFILE: -Dscala-2.13 -Dscala.binary.version=2.13
+ COMPOSE_PREFIX: docker-compose_hadoop340_hive2310_spark402
+
+jobs:
+ trino-e2e:
+ # Testcontainers E2E for the RFC-105 native trino-hudi connector: builds
+ # hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
+ # (docker/trino/shim, standing in for the not-yet-released upstream
+ # trinodb/trino plugin/trino-hudi shim), bakes it into a local
+ # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
+ # spark402 compose stack (the only pair with the trinocoordinator service).
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v5
+ - name: Free disk space
+ run: |
+ sudo rm -rf /usr/share/dotnet
+ sudo rm -rf /usr/local/lib/android
+ sudo rm -rf /opt/ghc
+ sudo rm -rf /usr/local/share/boost
+ docker system prune --all --force --volumes
+ - name: Pre-pull compose images (fails fast if not published)
+ run: |
+ # Surface a missing sparkadhoc image before the long Maven install.
The
+ # remaining stack images are pulled by docker-compose at test time;
the
+ # trino image is built locally below, never pulled.
+ docker pull
apachehudi/hudi-hadoop_3.4.0-hive_2.3.10-sparkadhoc_4.0.2:latest
+ - name: Set up JDK 17
+ uses: actions/setup-java@v5
+ with:
+ java-version: '17'
+ distribution: 'temurin'
+ architecture: x64
+ cache: maven
+ - name: Build and install Hudi artifacts (JDK 17)
+ # Full reactor: the compose containers mount the workspace and the
tests
+ # use bundles staged by the -Pintegration-tests build (e.g.
+ # docker/hoodie/hadoop/hive_base/target/hoodie-spark-bundle.jar).
+ run:
+ mvn clean install -T 2 $SCALA_PROFILE -Dspark4.0 -Dflink1.20
-Pintegration-tests -DskipTests=true -Ddocker.compose.skip=true $MVN_ARGS
+ - name: Set up JDK 25
+ uses: actions/setup-java@v5
+ with:
+ java-version: '25'
+ distribution: 'temurin'
+ cache: maven
+ - name: Build hudi-trino connector (JDK 25)
+ # No trinodb/trino checkout needed: the unpublished Trino test-jars sit
+ # behind the off-by-default hudi-trino-tests profile and packaging
+ # resolves entirely from Maven Central.
+ run:
+ mvn $MVN_ARGS -Phudi-trino -pl hudi-trino install
-Dmaven.test.skip=true
+ - name: Assemble trino-hudi plugin dir via in-repo shim (JDK 25)
+ # package, NOT install: installing would shadow the real
+ # io.trino:trino-hudi release coordinates in the local m2.
+ run:
+ mvn $MVN_ARGS -f docker/trino/shim/pom.xml clean package -DskipTests
+ - name: Build apachehudi/hudi-trino_481 image
+ run: |
+ docker/trino/build_image.sh --plugin-dir
docker/trino/shim/target/trino-hudi-481
+ # Sanity: the shim must have produced a populated plugin dir with a
+ # service descriptor jar, or Trino cannot load the plugin at boot.
+ echo "plugin dir jar count: $(ls
docker/trino/shim/target/trino-hudi-481 | wc -l)"
+ ls docker/trino/shim/target/trino-hudi-481/*services*.jar
+ - name: Smoke-boot the Trino image standalone
+ # Catches image-level boot failures (plugin load errors, bad etc/
config)
+ # ~30 min before the IT step would, with the full boot log on screen.
+ # --hostname trinocoordinator makes the baked discovery.uri
self-resolve.
+ run: |
+ docker run -d --name trino-smoke --hostname trinocoordinator \
+ apachehudi/hudi-trino_481:latest
+ ok=""
+ for i in $(seq 1 18); do
+ if [ "$(docker inspect -f '{{.State.Running}}' trino-smoke)" !=
"true" ]; then
+ echo "trino-smoke container died during startup" >&2
+ break
+ fi
+ if docker exec trino-smoke trino --server localhost:8080 \
+ --execute "SELECT 1" >/dev/null 2>&1; then
+ ok=1; echo "Trino answered SELECT 1 (attempt $i)"; break
+ fi
+ sleep 10
+ done
+ if [ -z "$ok" ]; then
+ echo "==== trino-smoke boot log ===="
+ docker logs trino-smoke 2>&1 | tail -200
+ docker rm -f trino-smoke >/dev/null 2>&1 || true
+ exit 1
+ fi
+ docker rm -f trino-smoke
+ - name: Set up JDK 17 (restore for the IT run)
+ # setup-java resets JAVA_HOME on each call; hudi-integ-test needs 17.
+ uses: actions/setup-java@v5
+ with:
+ java-version: '17'
+ distribution: 'temurin'
+ architecture: x64
+ - name: Run Trino E2E ITs (JDK 17)
+ run: |
+ # -DskipITs=false overrides the spark4.0 profile's skipITs=true
default
+ # (see root pom.xml). -Dcompose.profiles=trino starts the
profile-gated
+ # trinocoordinator service; without it every ITTestTrino* class
skips.
+ mvn verify $SCALA_PROFILE -Dspark4.0 -Pintegration-tests \
+ -pl hudi-integ-test \
+ -DskipITs=false \
+ -Ddocker.compose.skip=true \
+ -Dit.test='ITTestTrino*' \
+ -Dcompose.profiles=trino \
+ -Dspark.docker.compose.prefix=$COMPOSE_PREFIX \
+ $MVN_ARGS
+ - name: Dump failsafe test outputs on failure
+ # Surefire redirects test stdout (incl. the streamed trinocoordinator
+ # boot/query logs) into per-class output files; print their tails so
+ # server-side failures are readable straight from the workflow log.
+ if: failure()
+ run: |
+ for f in hudi-integ-test/target/failsafe-reports/*-output.txt; do
Review Comment:
Added the redirect to the IT step so the dump step has `*-output.txt` files
to tail. Note: the user property is `maven.test.redirectTestOutputToFile`
(shared by surefire and failsafe); `surefire.redirectTestOutputToFile` does not
exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
