parshimers commented on code in PR #7172:
URL: https://github.com/apache/texera/pull/7172#discussion_r3696635561
##########
.github/workflows/precheck.yml:
##########
@@ -314,6 +341,169 @@ jobs:
runPyrightLanguageService = false;
}
+ // Runner tiers: merge queue entries default to the
+ // self-hosted scale sets (the queue is the latency-critical
+ // serialized path most hurt by hosted-pool congestion);
+ // PR events opt in via the ci:self-hosted label. Everything
+ // else (push / dispatch) stays GitHub-hosted. Tiers:
+ // arc-heavy-linux for the amber/sbt stacks, arc-medium-linux
+ // for platform and pyamber, arc-light-linux for the small
+ // service jobs.
+ //
+ // Fallback probing, two levels, evaluated per tier:
+ //
+ // Level 1 (availability): a set with no online runner (cluster
+ // down, scale set removed) falls back to GitHub-hosted so a
+ // labeled PR never queues forever. Relies on each set keeping
+ // warm runners (minRunners > 0) — an idle scale-set with
+ // minRunners: 0 registers no runners and would look
+ // unavailable.
+ //
+ // Level 2 (congestion): predict queueing before it happens by
+ // comparing this round's demand (the stack decisions above
+ // tell us how many jobs each tier is about to receive)
+ // against idle supply (runner busy flags) plus the backlog
+ // of already-queued jobs waiting for the same label. Blind
+ // spot: k8s-side capacity (Pending pods, scale-up headroom)
+ // is invisible to the GitHub API.
+ //
+ // Runner listing needs ARC_STATUS_TOKEN (administration:read);
+ // without it the label is trusted as-is. Any probe error
+ // degrades to trusting the label.
+ const selfHosted = eventName === "merge_group" ||
labels.includes("ci:self-hosted");
+ const tiers = {
+ heavy_runner: selfHosted ? "arc-heavy-linux" : "ubuntu-latest",
+ medium_runner: selfHosted ? "arc-medium-linux" : "ubuntu-latest",
+ light_runner: selfHosted ? "arc-light-linux" : "ubuntu-latest",
+ };
+ // Per-tier job counts this round. PLATFORM_SERVICES mirrors the
+ // service matrix size in build.yml — keep in sync.
+ const PLATFORM_SERVICES = 6;
+ const demand = {
+ "arc-heavy-linux":
+ (runAmber ? 1 : 0) + (runAmberIntegration ? 1 : 0) +
(runFrontend ? 1 : 0),
+ "arc-medium-linux":
+ (runPlatform ? PLATFORM_SERVICES : 0) +
+ (runPlatformIntegration ? PLATFORM_SERVICES : 0) +
+ (runPyamber ? 3 : 0),
+ "arc-light-linux":
+ (runInfra ? 1 : 0) +
+ (runAgentService ? 1 : 0) +
+ (runPyrightLanguageService ? 1 : 0),
+ };
+ // Kill switch: ARC_AVAILABLE=false (manual, or flipped by the
+ // runner-heartbeat watchdog) forces every tier to GitHub-hosted.
+ if (selfHosted && process.env.ARC_AVAILABLE === "false") {
+ for (const k of Object.keys(tiers)) tiers[k] = "ubuntu-latest";
+ core.warning("ARC_AVAILABLE=false; routing all Linux tiers to
GitHub-hosted runners.");
+ }
+ // Heartbeat fallback: runner-heartbeat.yml proves fleet liveness
+ // every 15 minutes. Stale last-success (with history present)
+ // means the fleet is down — fall back without needing any extra
+ // credential (plain GITHUB_TOKEN, actions: read).
+ if (selfHosted && tiers.light_runner !== "ubuntu-latest") {
+ try {
+ const STALE_MS = 35 * 60 * 1000;
+ const ok = await github.rest.actions.listWorkflowRuns({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ workflow_id: "runner-heartbeat.yml",
+ status: "success",
+ per_page: 1,
+ });
+ const any = await github.rest.actions.listWorkflowRuns({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ workflow_id: "runner-heartbeat.yml",
+ per_page: 1,
+ });
+ const hasHistory = (any.data.workflow_runs ?? []).length > 0;
+ const last = ok.data.workflow_runs?.[0];
+ const lastOk = last ? new Date(last.updated_at).getTime() : 0;
+ if (hasHistory && Date.now() - lastOk > STALE_MS) {
+ for (const k of Object.keys(tiers)) tiers[k] =
"ubuntu-latest";
+ core.warning("Runner heartbeat is stale; routing all Linux
tiers to GitHub-hosted runners.");
+ }
+ } catch (e) {
+ core.info(`Heartbeat check skipped (${e.message}).`);
+ }
+ }
+ if (selfHosted && process.env.ARC_STATUS_TOKEN) {
+ try {
+ const res = await fetch(
+
`https://api.github.com/repos/${context.repo.owner}/${context.repo.repo}/actions/runners?per_page=100`,
+ {
+ headers: {
+ authorization: `Bearer ${process.env.ARC_STATUS_TOKEN}`,
+ accept: "application/vnd.github+json",
+ },
+ }
Review Comment:
i don't know if there's an alternative but this seems like it might end up
being brittle. is `github.rest.actions` just a sugar/wrapper for a call like
this, or does it have more handling of the requests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
