wombatu-kun commented on code in PR #19642:
URL: https://github.com/apache/hudi/pull/19642#discussion_r3793518602
##########
.github/workflows/hudi_trino_compat.yml:
##########
@@ -59,18 +68,61 @@ jobs:
echo "trino_version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Detected Trino version: $VERSION"
- name: Install Trino modules from master (JDK 25)
- working-directory: trino
- # hudi-trino compiles against these plus their transitive modules
(spi, cache, metastore,
- # hive-formats, memory-context). They must come from the master
checkout -- resolving from
- # Maven Central would defeat the point of the drift check.
- run: mvn $MVN_ARGS install -pl
:trino-hive,:trino-filesystem-manager,:trino-parquet,:trino-plugin-toolkit -am
-DskipTests -Dair.check.skip-all=true
+ # Same module set the pinned build uses, so a green compile here is a
promotable pin. The
+ # script purges ~/.m2/repository/io/trino itself and only warns when
master's version has
+ # rolled past the pinned trino.version.
+ env:
+ HEAD_SHA: ${{ steps.trino-head.outputs.head_sha }}
+ run: hudi/scripts/trino/bootstrap_trino.sh trino --skip-checkout --ref
"$HEAD_SHA"
- name: Compile hudi-trino against current Trino SPI (JDK 25)
id: compile
working-directory: hudi
run: |
mvn $MVN_ARGS -Phudi-trino \
-Dtrino.version=${{ steps.trino-version.outputs.trino_version }} \
-pl hudi-trino compile
+ - name: Save Trino artifacts under the candidate pin
+ # Pre-seeds the gating CI cache, which keys on the sha, so the pin
advance below does not
+ # make every PR rebuild Trino from source.
+ if: steps.compile.outcome == 'success'
+ uses: actions/cache/save@v4
+ with:
+ path: ~/.m2/repository/io/trino
+ key: trino-m2-v1-${{ steps.trino-head.outputs.head_sha }}
+ - name: Propose pin advance
+ # Pin PRs are human-merged by policy. Org settings may forbid
GITHUB_TOKEN from creating
+ # PRs; the branch still lands and the PR is then opened by hand.
+ working-directory: hudi
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ HEAD_SHA: ${{ steps.trino-head.outputs.head_sha }}
+ TRINO_VERSION: ${{ steps.trino-version.outputs.trino_version }}
+ BASE_BRANCH: ${{ github.ref_name }}
+ run: |
+ set -euo pipefail
+ PINNED_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ git config user.name 'github-actions[bot]'
+ git config user.email
'41898282+github-actions[bot]@users.noreply.github.com'
+ git checkout -B bot/trino-pin
+ sed -i
"s|<trino.sha>.*</trino.sha>|<trino.sha>${HEAD_SHA}</trino.sha>|" pom.xml
Review Comment:
The pin advance rewrites `trino.sha` (and `trino.version` on rollover) but
never `trino.e2e.version`, so the root pom's claim that it is "the latest
released Trino" holds only until the next Trino release. Have this job also set
it to the newest released trinodb/trino tag, or drop the claim and name the
owner in the module README.
##########
.github/workflows/hudi_trino_compat.yml:
##########
@@ -59,18 +68,61 @@ jobs:
echo "trino_version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Detected Trino version: $VERSION"
- name: Install Trino modules from master (JDK 25)
- working-directory: trino
- # hudi-trino compiles against these plus their transitive modules
(spi, cache, metastore,
- # hive-formats, memory-context). They must come from the master
checkout -- resolving from
- # Maven Central would defeat the point of the drift check.
- run: mvn $MVN_ARGS install -pl
:trino-hive,:trino-filesystem-manager,:trino-parquet,:trino-plugin-toolkit -am
-DskipTests -Dair.check.skip-all=true
+ # Same module set the pinned build uses, so a green compile here is a
promotable pin. The
+ # script purges ~/.m2/repository/io/trino itself and only warns when
master's version has
+ # rolled past the pinned trino.version.
+ env:
+ HEAD_SHA: ${{ steps.trino-head.outputs.head_sha }}
+ run: hudi/scripts/trino/bootstrap_trino.sh trino --skip-checkout --ref
"$HEAD_SHA"
- name: Compile hudi-trino against current Trino SPI (JDK 25)
id: compile
working-directory: hudi
run: |
mvn $MVN_ARGS -Phudi-trino \
-Dtrino.version=${{ steps.trino-version.outputs.trino_version }} \
-pl hudi-trino compile
+ - name: Save Trino artifacts under the candidate pin
+ # Pre-seeds the gating CI cache, which keys on the sha, so the pin
advance below does not
+ # make every PR rebuild Trino from source.
+ if: steps.compile.outcome == 'success'
+ uses: actions/cache/save@v4
+ with:
+ path: ~/.m2/repository/io/trino
+ key: trino-m2-v1-${{ steps.trino-head.outputs.head_sha }}
+ - name: Propose pin advance
+ # Pin PRs are human-merged by policy. Org settings may forbid
GITHUB_TOKEN from creating
+ # PRs; the branch still lands and the PR is then opened by hand.
+ working-directory: hudi
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ HEAD_SHA: ${{ steps.trino-head.outputs.head_sha }}
+ TRINO_VERSION: ${{ steps.trino-version.outputs.trino_version }}
+ BASE_BRANCH: ${{ github.ref_name }}
+ run: |
+ set -euo pipefail
+ PINNED_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ git config user.name 'github-actions[bot]'
+ git config user.email
'41898282+github-actions[bot]@users.noreply.github.com'
+ git checkout -B bot/trino-pin
+ sed -i
"s|<trino.sha>.*</trino.sha>|<trino.sha>${HEAD_SHA}</trino.sha>|" pom.xml
+ if [ "$TRINO_VERSION" != "$PINNED_VERSION" ]; then
+ # Version rollover: the shim's parent tracks trino.version.
+ sed -i
"s|<trino.version>.*</trino.version>|<trino.version>${TRINO_VERSION}</trino.version>|"
pom.xml
+ sed -i "/<parent>/,/<\/parent>/
s|<version>.*</version>|<version>${TRINO_VERSION}</version>|"
docker/trino/shim/pom.xml
+ fi
+ if git diff --quiet; then
+ echo "Pin is already at ${HEAD_SHA}; nothing to propose."
+ exit 0
+ fi
+ TITLE="build(trino): advance trino master pin to ${HEAD_SHA:0:12}"
+ git commit -am "$TITLE"
+ git push --force origin bot/trino-pin
+ if [ -n "$(gh pr list --head bot/trino-pin --state open --json
number --jq '.[].number')" ]; then
+ echo "Open pin PR picked up the force-push."
+ exit 0
+ fi
+ gh pr create --base "$BASE_BRANCH" --head bot/trino-pin --title
"$TITLE" \
Review Comment:
A PR opened with `secrets.GITHUB_TOKEN` does not start workflow runs, so
none of the five `validate-*` contexts `.asf.yaml` requires on master would
ever report and the pin PR could not be merged; the title type `build` is also
outside `validate-pr-title`'s allowed list. Push the branch and let a human
open the PR (or use the `hudi-bot` collaborator's PAT), and switch the title to
`chore(trino):`.
##########
hudi-trino/src/test/java/io/trino/plugin/hudi/HudiQueryRunner.java:
##########
@@ -56,14 +56,14 @@ public static Builder builder()
return new Builder("local:///");
}
- public static Builder builder(Hive3MinioDataLake hiveMinioDataLake)
+ public static Builder builder(Hive3FlociDataLake hiveFlociDataLake)
{
- return new Builder("s3://" + hiveMinioDataLake.getBucketName() + "/")
- .addConnectorProperty("fs.native-s3.enabled", "true")
- .addConnectorProperty("s3.aws-access-key", MINIO_ROOT_USER)
- .addConnectorProperty("s3.aws-secret-key", MINIO_ROOT_PASSWORD)
- .addConnectorProperty("s3.region", MINIO_REGION)
- .addConnectorProperty("s3.endpoint",
hiveMinioDataLake.getMinio().getMinioAddress())
+ return new Builder("s3://" + hiveFlociDataLake.getBucketName() + "/")
+ .addConnectorProperty("fs.s3.enabled", "true")
Review Comment:
The 481 -> pin fixups stop at src/main and `test-hudi-trino-plugin` is red:
the Alluxio cache moved out of trino-filesystem-manager into the loadable
`trino-blob-cache-alluxio` plugin (hive, iceberg and delta-lake all declare it
test-scope), `TestTupleDomainUtilsTest` still hands a `String` to
`Domain.multipleValues`, and the memory-cache expectations predate the
`FileSystemCache.cacheInput` / `Input.readTail` split. Add the missing
test-scope dependency and fix the other two so the connector lane is green
before the pin model lands.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -62,50 +62,131 @@ jobs:
# hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
# (docker/trino/shim, standing in for the not-yet-released upstream
# trinodb/trino plugin/trino-hudi shim), bakes it into a local
- # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
- # spark402 compose stack (the only pair with the trinocoordinator service).
+ # apachehudi/hudi-trino-e2e image on top of the released trino.e2e.version
+ # server, and runs ITTestTrino* against the spark402 compose stack (the
only
+ # pair with the trinocoordinator service).
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+ - name: Read Trino pin
+ id: trino-pin
+ run: |
+ set -euo pipefail
+ TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
pom.xml)
+ TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ E2E_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p' pom.xml)
+ echo "Connector builds at $TRINO_VERSION ($TRINO_SHA); server image
is $E2E_VERSION"
+ echo "trino_sha=$TRINO_SHA" >> "$GITHUB_OUTPUT"
+ echo "trino_version=$TRINO_VERSION" >> "$GITHUB_OUTPUT"
+ echo "e2e_version=$E2E_VERSION" >> "$GITHUB_OUTPUT"
+ - name: SPI drift gate
+ id: spi-drift
+ # The plugin is built at the pin but loaded by the released
trino.e2e.version server, so
+ # any SPI / filesystem change between the two can make the image
unbootable. Skip the run
+ # instead of reporting a failure that no connector change caused.
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
+ TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
+ E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
+ run: |
+ set -euo pipefail
+ COMPARE="$RUNNER_TEMP/trino-compare.json"
+ gh api "repos/trinodb/trino/compare/${E2E_VERSION}...${TRINO_SHA}" >
"$COMPARE"
+ COUNT=$(jq '.files | length' "$COMPARE")
+ DRIFTED=false
+ # The compare API caps the file list at 300; treat a capped list as
drifted rather
+ # than trusting a truncated sample.
+ if [ "$COUNT" -ge 300 ]; then
Review Comment:
A single Trino release cycle already exceeds the compare API's 300-file cap
(482...483 returns 300), so this branch fires for any pin more than a release
old and the trino-e2e check on this PR went green in 13s without running
anything. Decide drift from the four SPI paths directly -
`repos/trinodb/trino/commits?sha=<pin>&path=<p>&since=<e2e tag date>` is
uncapped and paginates - instead of one compare call's truncated file list.
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -16,9 +16,9 @@ on:
# which compiles the whole module, so a break
# anywhere in it fails this pipeline -- not just
# under integ2/.
- # pom.xml owns trino.version, which the shim pom's
parent,
- # the Dockerfile TRINO_VERSION arg and the
- # hardcoded 481 paths below all track by hand.
+ # pom.xml owns trino.version / trino.sha /
trino.e2e.version:
+ # the shim pom's parent, the plugin dir path and
the
+ # server image version below are all derived
from them.
Review Comment:
The e2e job now runs `scripts/trino/bootstrap_trino.sh`, but neither
`paths:` list here covers `scripts/trino/**` the way hudi_trino_ci.yml's does.
Add it to both push and pull_request so a bootstrap change retriggers this
workflow.
##########
hudi-trino/README.md:
##########
@@ -83,7 +99,7 @@ mvn verify -pl hudi-integ-test -Dscala-2.13
-Dscala.binary.version=2.13 -Dspark4
```
Fast iteration loop: after changing connector code, redo steps 2-3, then add
-`-Dtrino.plugin.dir=$PWD/docker/trino/shim/target/trino-hudi-481` to step 5.
The
+`-Dtrino.plugin.dir=$PWD/docker/trino/shim/target/trino-hudi-$TRINO_VERSION`
to step 5. The
Review Comment:
This loop says to skip step 4, but both `$TRINO_VERSION` and the exploded
`trino-hudi-$TRINO_VERSION` directory are produced only inside step 4 (the sed
and the unzip), and step 3's `clean` removes any directory left from a previous
run. Move the version read and the unzip out of step 4, or point the loop at
the zip.
##########
.github/workflows/hudi_trino_compat.yml:
##########
@@ -5,10 +5,12 @@ on:
- cron: '17 4 * * *'
workflow_dispatch:
-# The failure handler files/updates a drift report issue.
+# The failure handler files/updates a drift report issue; on success the job
pushes the
+# pin-advance branch and opens its PR.
permissions:
- contents: read
+ contents: write
Review Comment:
This is the first workflow in the repo with `contents: write`, and the same
job runs `./mvnw` from a fresh `trinodb/trino` master checkout while holding a
push credential for apache/hudi. Split the build into a read-only job and keep
the write token in a separate propose step, and confirm with a PMC member that
bot branches on a canonical ASF repo are acceptable.
##########
release/release_guide.md:
##########
@@ -429,6 +429,23 @@ Set up a few environment variables to simplify Maven
commands that follow. This
and `./scripts/release/deploy_staging_jars_java25.sh 2>&1 | tee -a
"/tmp/${RELEASE_VERSION}-${RC_NUM}.deploy3.log"`.
This step must run after the Java 11 step in 9.4.1, which installs
the upstream Hudi modules that hudi-trino
resolves from the local m2 (the script does not pass `-am` because
Lombok cannot run on JDK 25).
+ 4. hudi-trino Trino pin-back, to be done on the release branch before
9.4.3. On master hudi-trino tracks
Review Comment:
This block sits in step 9 (deploy), but the source tarball that gets staged
and voted on is generated back in step 6, so the RC source release would ship
`trino.version=484-SNAPSHOT` and could not be built with `-Phudi-trino` from
Central. Move the pin-back to "Cut a release branch" or make it step 1 of
"Build a release candidate" - as written it also says "before 9.4.3" while
sitting after it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
