voonhous commented on code in PR #19642:
URL: https://github.com/apache/hudi/pull/19642#discussion_r3793861814
##########
.github/workflows/hudi_trino_e2e.yml:
##########
@@ -62,50 +62,131 @@ jobs:
# hudi-trino at HEAD, assembles the plugin dir via the in-repo shim
# (docker/trino/shim, standing in for the not-yet-released upstream
# trinodb/trino plugin/trino-hudi shim), bakes it into a local
- # apachehudi/hudi-trino_481 image, and runs ITTestTrino* against the
- # spark402 compose stack (the only pair with the trinocoordinator service).
+ # apachehudi/hudi-trino-e2e image on top of the released trino.e2e.version
+ # server, and runs ITTestTrino* against the spark402 compose stack (the
only
+ # pair with the trinocoordinator service).
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
+ - name: Read Trino pin
+ id: trino-pin
+ run: |
+ set -euo pipefail
+ TRINO_SHA=$(sed -n 's|.*<trino.sha>\(.*\)</trino.sha>.*|\1|p'
pom.xml)
+ TRINO_VERSION=$(sed -n
's|.*<trino.version>\(.*\)</trino.version>.*|\1|p' pom.xml)
+ E2E_VERSION=$(sed -n
's|.*<trino.e2e.version>\(.*\)</trino.e2e.version>.*|\1|p' pom.xml)
+ echo "Connector builds at $TRINO_VERSION ($TRINO_SHA); server image
is $E2E_VERSION"
+ echo "trino_sha=$TRINO_SHA" >> "$GITHUB_OUTPUT"
+ echo "trino_version=$TRINO_VERSION" >> "$GITHUB_OUTPUT"
+ echo "e2e_version=$E2E_VERSION" >> "$GITHUB_OUTPUT"
+ - name: SPI drift gate
+ id: spi-drift
+ # The plugin is built at the pin but loaded by the released
trino.e2e.version server, so
+ # any SPI / filesystem change between the two can make the image
unbootable. Skip the run
+ # instead of reporting a failure that no connector change caused.
+ env:
+ GH_TOKEN: ${{ github.token }}
+ TRINO_SHA: ${{ steps.trino-pin.outputs.trino_sha }}
+ TRINO_VERSION: ${{ steps.trino-pin.outputs.trino_version }}
+ E2E_VERSION: ${{ steps.trino-pin.outputs.e2e_version }}
+ run: |
+ set -euo pipefail
+ COMPARE="$RUNNER_TEMP/trino-compare.json"
+ gh api "repos/trinodb/trino/compare/${E2E_VERSION}...${TRINO_SHA}" >
"$COMPARE"
+ COUNT=$(jq '.files | length' "$COMPARE")
+ DRIFTED=false
+ # The compare API caps the file list at 300; treat a capped list as
drifted rather
+ # than trusting a truncated sample.
+ if [ "$COUNT" -ge 300 ]; then
Review Comment:
You are right -- `482...483` alone saturates the cap, so the gate was
effectively always-drifted. Rewrote it in 00c42eebfa51 to per-path
`commits?sha=<pin>&path=<p>&since=<tag date>` queries (excluding the tag commit
itself); existence decides drift, so truncation cannot produce a false pass.
Note today's honest answer is still "drifted" (the blob-cache migration touched
`lib/trino-filesystem`), so e2e legitimately skips on this PR either way.
##########
hudi-trino/README.md:
##########
@@ -83,7 +99,7 @@ mvn verify -pl hudi-integ-test -Dscala-2.13
-Dscala.binary.version=2.13 -Dspark4
```
Fast iteration loop: after changing connector code, redo steps 2-3, then add
-`-Dtrino.plugin.dir=$PWD/docker/trino/shim/target/trino-hudi-481` to step 5.
The
+`-Dtrino.plugin.dir=$PWD/docker/trino/shim/target/trino-hudi-$TRINO_VERSION`
to step 5. The
Review Comment:
Fixed in 00c42eebfa51: the `TRINO_VERSION` read and the unzip moved into
step 3 (with a note that `clean` wipes the exploded dir), so the loop's "redo
steps 2-3" now leaves a valid plugin dir and step 4 is only the image build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
