felipepessoto commented on code in PR #12388:
URL: https://github.com/apache/gluten/pull/12388#discussion_r3661688269
##########
.github/workflows/velox_backend_x86.yml:
##########
@@ -101,6 +106,70 @@ jobs:
path: ./cpp/build/
if-no-files-found: error
+ # Gate the (expensive) Delta Spark UT suite so per-PR it runs only when the
PR
+ # touches high-signal Delta paths -- the Delta integration code
+ # (backends-velox/src-delta*), the gluten-delta module, or this pipeline's
own
+ # files -- or carries the `run-delta-ci` opt-in label. Changes to general
+ # Velox/core/native code can also affect Delta offload but are touched
+ # constantly, so per-PR they skip it; the nightly full run
(delta_spark_ut.yml
+ # `schedule`) and the opt-in label are the safety nets. This keeps GHA usage
+ # down. NOTE: the label is read from the event that triggered this run, so
add
+ # it before/with a push; labeling an already-finished run needs a new push.
+ delta-changes:
+ runs-on: ubuntu-22.04
+ outputs:
+ run_delta: ${{ steps.filter.outputs.run_delta }}
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ fetch-depth: 0
+ - name: Detect Delta-relevant changes / opt-in label
+ id: filter
+ env:
+ HAS_LABEL: ${{ contains(github.event.pull_request.labels.*.name,
'run-delta-ci') }}
+ BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ run: |
+ set -euo pipefail
+ # Opt-in label forces the suite even with no Delta-relevant path
change.
+ if [ "$HAS_LABEL" = "true" ]; then
+ echo "run-delta-ci label present -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ # Fail open if we can't determine the PR range (e.g. a non-PR
trigger):
+ # never silently skip coverage.
+ if [ -z "${BASE_SHA:-}" ] || [ -z "${HEAD_SHA:-}" ]; then
+ echo "no PR base/head sha -> running Delta suite (fail-open)"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
+ fi
+ BASE=$(git merge-base "$BASE_SHA" "$HEAD_SHA" 2>/dev/null || echo
"$BASE_SHA")
+ echo "diff base=$BASE head=$HEAD_SHA"
+ # High-signal Delta paths only: the Delta integration code
+ # (backends-velox/src-delta*), the Delta module, and this pipeline's
own
+ # files. A change to general Velox/core/native code can also affect
Delta
+ # offload, but those are touched constantly; per-PR we skip them (the
+ # nightly full run + the `run-delta-ci` label are the safety nets) to
+ # keep GHA usage down.
+ if git diff --name-only "$BASE" "$HEAD_SHA" | grep -Eq \
+
'^(\.github/workflows/velox_backend_x86\.yml|\.github/workflows/delta_spark_ut\.yml|\.github/workflows/util/delta-spark-ut/|gluten-delta/|backends-velox/src-delta)';
then
+ echo "Delta-relevant paths changed -> running Delta suite"
+ echo "run_delta=true" >> "$GITHUB_OUTPUT"
+ else
+ echo "No Delta-relevant paths changed and no opt-in label ->
skipping Delta suite"
+ echo "run_delta=false" >> "$GITHUB_OUTPUT"
+ fi
Review Comment:
Valid, and a good catch — the gate really did fail *closed*, which is the
opposite of what the comment right above it promises. Fixed in 7618d6f0d.
Reproduced it first:
```console
$ git diff --name-only <badsha> <badsha> | grep -Eq '^gluten-delta/'
fatal: bad object deadbeefdeadbeefdeadbeefdeadbeefdeadbeef
run_delta=FALSE <-- silently skips Delta coverage
```
The reason it slips through is that git's failure leaves `grep` with empty
input, so `grep -Eq` exits 1 — indistinguishable from "no Delta paths changed".
`set -euo pipefail` doesn't help either, since a command used as an `if`
condition is explicitly allowed to fail. Worth noting this is reachable: the
`git merge-base ... || echo "$BASE_SHA"` fallback just above already
anticipates the shas being unusable, and it feeds straight into this diff.
The fix is what you suggested — capture the diff, treat a git failure as
fail-open, then grep the captured output:
```bash
if ! CHANGED=$(git diff --name-only "$BASE" "$HEAD_SHA"); then
echo "git diff failed -> running Delta suite (fail-open)"
echo "run_delta=true" >> "$GITHUB_OUTPUT"; exit 0
fi
if printf '%s\n' "$CHANGED" | grep -Eq '^(...)'; then
```
Verified against a real repo across 8 scenarios: the three git-error cases
(bad base+head, valid base + bad head, empty shas) now all fail open, and the
five matching cases are unchanged (`gluten-delta/` → true,
`backends-velox/src-delta40` → true, docs-only → false, `cpp/velox` only →
false, no changes → false). I ran the old logic through the same harness to
confirm it fails exactly where the new one passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
