DanielLeens opened a new issue, #12985: URL: https://github.com/apache/seatunnel/issues/12985
### Code of Conduct - [x] I agree to follow this projects [Code of Conduct](https://www.apache.org/foundation/policies/conduct). ### Search before asking - [x] I searched the existing Zeta admission, fixed-slot, and STIP issues before filing this proposal. - [x] This proposal is intentionally separate from #12119. That issue defines how a queued job stops retaining a fully initialized JobMaster; this one defines the admission order after a queued job has a resource demand. ## Abstract Zeta has a fixed-slot admission path for `dynamic-slot: false` with `job-schedule-strategy: WAIT`, but it has only one queue policy: retain the queue head and retry it. This is safe in the sense that it does not silently run a job with only part of its required resources, but it creates two operationally different failure modes: 1. a job whose request can never fit in the eligible fixed-slot capacity waits forever; and 2. a temporarily blocked large job prevents later smaller jobs from using currently available slots, even when those jobs can run safely. This STIP introduces an explicit, starvation-safe admission policy for fixed slots. It preserves current FIFO behavior by default and offers an opt-in bounded-backfill mode. Bounded backfill can admit later runnable jobs while the head job is temporarily blocked, but an aging limit turns the head into an admission gate so it cannot starve. ## Current behavior and problem statement This section was verified against `dev` commit [`7604a83ec802a0b148e98e557fb9184c48127988`](https://github.com/apache/seatunnel/tree/7604a83ec802a0b148e98e557fb9184c48127988). - `job-schedule-strategy` has only `REJECT` and `WAIT` semantics; it does not select among queued jobs. The documented default is `REJECT` ([`ServerConfigOptions.java`](https://github.com/apache/seatunnel/blob/7604a83ec802a0b148e98e557fb9184c48127988/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java#L322-L328)). - Fixed-slot capacity is selected by `dynamic-slot: false` and `slot-num`; `slot-num` is explicitly meaningful only in that mode ([`ServerConfigOptions.java`](https://github.com/apache/seatunnel/blob/7604a83ec802a0b148e98e557fb9184c48127988/seatunnel-engine/seatunnel-engine-common/src/main/java/org/apache/seatunnel/engine/common/config/server/ServerConfigOptions.java#L565-L576)). - `CoordinatorService` owns a single `PeekBlockingQueue<PendingJobInfo>` ([`CoordinatorService.java`](https://github.com/apache/seatunnel/blob/7604a83ec802a0b148e98e557fb9184c48127988/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L187-L207)). Its scheduler reserves the first queued job, calls `JobMaster.preApplyResources()`, and, under `WAIT`, leaves that same job in place and sleeps for three seconds when allocation fails ([`CoordinatorService.java`](https://github.com/apache/seatunnel/blob/7604a83ec802a0b148e98e557fb9184c48127988/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/CoordinatorService.java#L319-L519)). - The queue is a `LinkedBlockingQueue`, and `peekBlocking` iterates it in FIFO order ([`PeekBlockingQueue.java`](https://github.com/apache/seatunnel/blob/7604a83ec802a0b148e98e557fb9184c48127988/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/utils/PeekBlockingQueue.java#L41-L125)). - Resource pre-application is all-or-nothing for a job: it requests all task-group resources and releases any successfully allocated subset if any request fails ([`JobMaster.java`](https://github.com/apache/seatunnel/blob/7604a83ec802a0b148e98e557fb9184c48127988/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/master/JobMaster.java#L509-L659)). This property must be retained. For example, assume a fixed-slot cluster has at least ten eligible slots in total, but only three are currently free. Job A is first in the queue and needs ten slots. Job B is behind it and needs one slot. In `WAIT`, A is retried and B is not evaluated, although B could execute now. If the entire eligible static capacity is only three slots, A cannot ever run, but current `WAIT` semantics still retry it indefinitely. This is a design gap, not a claim that every observed `PENDING` job is a scheduler defect. #10993 is related operational context, while this STIP specifies the missing contract for the current engine path. ## Scope and non-goals ### Scope - Zeta fixed slots only: `dynamic-slot: false` and `job-schedule-strategy: WAIT`. - Queue admission after the job resource demand is known. - Correct classification of permanently unschedulable versus temporarily unavailable fixed-slot requests. - FIFO compatibility, bounded backfill, fairness, cancellation, failover, diagnostics, and regression coverage. ### Non-goals - Dynamic-slot behavior, autoscaling, or worker creation. Those remain outside this proposal and are related to #12087. - Online rescaling, task migration, or changing the parallelism of a running job. - Replacing worker selection strategies such as `RANDOM`, `SLOT_RATIO`, or `SYSTEM_LOAD`. - Tenant quotas, user-defined priorities, or a general-purpose batch scheduler in the first delivery. ## Proposed admission contract ### 1. Classify the resource request before deciding its queue behavior The admission layer must distinguish these outcomes using the full resource shape and tag constraints, not only a raw count of free slots: | Outcome | Meaning | Required result | | --- | --- | --- | | `ADMITTED` | All required task-group resources can be atomically allocated now. | Bind the allocation and start the job. | | `TEMPORARILY_UNAVAILABLE` | The eligible fixed-slot topology can satisfy the request, but currently assigned work prevents an atomic allocation. | Keep the job pending and apply the selected admission policy. | | `UNSCHEDULABLE` | No eligible fixed-slot topology can ever satisfy the request, including resource-profile and tag constraints. | Complete the job as `FAILED` with a stable structured unschedulable-resource cause; do not retry forever. | The final admission attempt remains authoritative and atomic. A capacity snapshot can shortlist candidates but must not make the final decision because a snapshot can become stale. A failed attempt must preserve the current rollback guarantee: no partial slot assignment may remain attached to the candidate. This directly resolves the ambiguous `three free slots, ten requested slots` case: it is temporary only when the cluster can eventually provide ten eligible slots. It is unschedulable when the entire matching static capacity cannot provide them. ### 2. Add an explicit policy, preserving todays behavior by default The policy is active only for fixed slots with `WAIT`: | Policy | Semantics | | --- | --- | | `FIFO` | Default. Evaluate only the oldest pending job. If it is temporarily unavailable, do not admit a later job. This is the current behavior. | | `BOUNDED_BACKFILL` | When the oldest job is temporarily unavailable, evaluate later jobs in original enqueue order and admit the first job whose full resource request succeeds. Backfill is bounded by the aging gate below. | The configuration names are illustrative; the implementation must expose one documented policy option plus a maximum bypass count and maximum head-wait duration. Existing configurations must resolve to `FIFO`, and `REJECT` and dynamic slots must remain unchanged. The scheduler must not pick the globally smallest job or continually prefer newly submitted jobs. It scans in enqueue order and starts at most the first later candidate that can complete a full atomic admission. That keeps the scheduling decision deterministic and avoids converting backfill into a shortest-job policy. ### 3. Bound backfill so the head job cannot starve Every admission record needs immutable enqueue sequence and enqueue time, plus mutable bypass count, first-blocked time, and gate state. 1. Evaluate the oldest pending job H. 2. If H is `ADMITTED`, run H. 3. If H is `UNSCHEDULABLE`, fail H with the structured cause and continue with the next job. 4. If H is temporarily unavailable under `FIFO`, keep H at the head and wait for a resource-change trigger or bounded retry. 5. If H is temporarily unavailable under `BOUNDED_BACKFILL` and its gate is not active, scan later jobs in enqueue order. Admit only the first candidate whose final atomic admission succeeds. Record that H was bypassed. 6. Activate the H gate as soon as either its actual bypass count reaches the configured limit or its waiting age reaches the configured limit. While the gate is active, no later job may be admitted until H is admitted, cancelled, or becomes unschedulable. The gate is deliberately passive: it stops newer admissions so slots released by running jobs accumulate for H. It does not retain a partial allocation for H. This preserves the existing all-or-nothing resource model and gives H a bounded route to the next admission opportunity. No policy can make a job run if running jobs never release slots or if its eligible capacity is impossible. Subject to those conditions, the aging gate prevents perpetual bypass by smaller jobs. ## Durable state, recovery, and lifecycle `PendingJobInfo` currently keeps enqueue timestamp and diagnostic counters in process memory. FIFO and starvation guarantees cannot depend on a local queue because active-master recovery rebuilds pending jobs. This proposal therefore depends on the durable acceptance/admission record designed in #12119. The record must preserve at least: - job id, immutable configuration or artifact digest, original enqueue sequence and time; - derived resource-demand snapshot and applicable tags; - selected admission policy, bypass count, first-blocked time, and gate state; - admission phase (`PENDING`, `ADMITTING`, `ADMITTED`, terminal reason) and master epoch. Only the active master may advance this record. On master change, the new epoch must reconcile any `ADMITTING` record with actual slot ownership by job id before either continuing it or releasing it and retrying once. It must not double-assign slots, lose the original queue order, or reset a starvation gate merely because the master changed. Cancellation removes the durable admission record, releases a pending gate immediately, and triggers a reschedule. The public legacy `JobStatus` contract remains unchanged: queued jobs are still `PENDING`, admitted jobs follow the existing initialization and running path, and a permanently unschedulable job uses terminal `FAILED` with an additional machine-readable reason rather than a new ordinal. ## Observability The pending-job REST response and metrics should expose optional admission diagnostics without breaking existing fields: - policy, queue sequence, pending age, request summary, and eligible-capacity verdict; - `TEMPORARILY_UNAVAILABLE` versus `UNSCHEDULABLE` reason; - bypass count, gate state, and number of backfilled admissions; - oldest-pending age, gated-head count, unschedulable rejection count, and admission-attempt outcome counters. Logs must record state transitions and reason codes, not emit one unbounded high-volume message for every retry. ## Delivery plan and acceptance tests 1. Define the durable admission record and lifecycle boundary with #12119. 2. Add fixed-slot feasibility classification and structured diagnostics while retaining the current atomic rollback behavior. 3. Add `FIFO` and opt-in `BOUNDED_BACKFILL` selection with the aging gate. 4. Add REST and metric observability, then master-failover reconciliation. GitHub CI must cover at least: - a request larger than all matching static capacity fails promptly and never retries indefinitely; - strict FIFO keeps the later one-slot job pending behind a temporarily blocked ten-slot head; - bounded backfill admits that one-slot job while the head gate is not active; - after either aging bound is reached, newer jobs stop being admitted and released slots accumulate until the head is atomically admitted; - cancellation, tag and resource-shape mismatch, and partial-allocation rollback; - active-master failover during pending, backfilled, gated, and admitting states, with no duplicate slot ownership and preserved order; - legacy configuration with no new option remains behaviorally equivalent to current FIFO `WAIT`. ## Related work - #12119: durable two-phase admission and queued JobMaster lifetime. This STIP consumes that record; it does not duplicate the memory-lifetime design. - #12126: reliability-hardening umbrella that already tracks the admission path. - #10993: historical fixed-slot `PENDING` operational report; it is not treated here as proof of this design gap. - #12087: worker-capacity autoscaling, intentionally out of scope. ### Are you willing to submit PR? - [x] Yes, after design review and agreement on the durable admission-record dependency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
