andygrove commented on code in PR #5963:
URL: https://github.com/apache/datafusion-comet/pull/5963#discussion_r4018335380
##########
.github/workflows/ci.yml:
##########
@@ -204,6 +225,21 @@ jobs:
# every output true for this event so a manual run can exercise
# any gated job.
:
+ elif [[ "$EVENT_NAME" == "schedule" ]]; then
+ # The nightly's base is the commit the last successful scheduled
+ # run tested, so everything that landed since is covered exactly
+ # once and a red nightly keeps its commits in scope until a green
+ # one supersedes it. Without such a run (the first nightly, an API
+ # error, or a base no longer on main) fall back to the tip as of
+ # the previous tick, with 30 minutes of slack for scheduling
+ # jitter. A quiet day diffs to nothing and runs nothing; so does
+ # a docs-only day.
+ prev=$(python3 dev/ci/nightly-base.py)
+ if [[ -z "$prev" ]] || ! git merge-base --is-ancestor "$prev" HEAD
2>/dev/null; then
+ prev=$(git rev-list -1 --before="24 hours 30 minutes ago" HEAD)
Review Comment:
Good catch, and thanks for the reproduction — that is exactly the failure.
Fixed in 01d000fc6 by taking your first option: when there is no base the run
can trust, it lists the whole tree and runs the entire nightly tier. The time
window is gone rather than widened, because any guessed base has the same shape
of bug at some history, and a run that covered everything is a base tomorrow
can rely on.
I ran your scenario against the previous revision to confirm it, then
against the fix, using the real helper, a local API server and the bash from
`ci.yml`:
```
=== BEFORE (time-window fallback), API down ===
::warning::could not list previous scheduled runs (HTTP Error 503: Service
Unavailable)
Nightly base: uncovered source change
selected:
=== AFTER (whole-tree fallback), API down ===
Nightly base unknown; running the whole nightly tier
selected: build_linux_all_profiles spark_3_5 spark_4_0 iceberg_1_8
iceberg_1_9 iceberg_1_10
```
The other three paths still behave: a successful lookup diffs against it and
selects the same six, a lookup that returns `HEAD` (a quiet day) selects
nothing, and a base that has left `main` takes the fallback. Note that the
whole-tree list goes through `FILTERS` and `POLICY` unchanged, so it can only
ever reach the nightly tier — nothing in the queue tier comes back.
Since the thing that went wrong here is silent, I also added
`check_nightly_base_fallback` to `check-ci-config.py`. It pins the `schedule`
case to the `nightly-base.py` lookup and the whole-tree fallback, and rejects a
base derived from the clock (`--before`/`--since`/`--after`/`--until`) with
your scenario as the error message. Mutation-tested three ways: restoring the
time window, replacing the lookup with a fixed revision, and renaming the case
all fail the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
