This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/main by this push:
new 696ca29b3 fix(executor): keep heartbeating while all task slots are
busy (#2159)
696ca29b3 is described below
commit 696ca29b3c1fe3013238822b7f5d8cdb918fdaa3
Author: Andy Grove <[email protected]>
AuthorDate: Thu Jul 23 08:38:18 2026 -0600
fix(executor): keep heartbeating while all task slots are busy (#2159)
Under pull-based scheduling the executor's only heartbeat is `poll_work`,
but
the poll loop blocked on `free_vcores.acquire().await` before polling. When
every vcore is held by a task running longer than the scheduler's
`executor_timeout` (e.g. a long stage-1 scan on a wide TPC-H query), the
loop
blocked on acquire and never polled, so the scheduler stopped receiving
heartbeats and wrongly marked the healthy-but-busy executor dead, resetting
its
in-flight (and completed) tasks. When a stage's tasks consistently exceed
the
timeout this livelocks: reset -> re-dispatch -> reset, never converging.
Cap the vcore wait at `HEARTBEAT_POLL_INTERVAL` (5s); on timeout, poll
anyway
reporting `num_free_vcores: 0` (heartbeat only), so executor liveness no
longer
depends on vcore availability. A freed vcore still wakes the wait
immediately,
so scheduling responsiveness is unchanged.
---
ballista/executor/src/execution_loop.rs | 36 ++++++++++++++++++++++++++-------
1 file changed, 29 insertions(+), 7 deletions(-)
diff --git a/ballista/executor/src/execution_loop.rs
b/ballista/executor/src/execution_loop.rs
index c4fac99d2..e0eea30e4 100644
--- a/ballista/executor/src/execution_loop.rs
+++ b/ballista/executor/src/execution_loop.rs
@@ -49,6 +49,13 @@ use std::{sync::Arc, time::Duration};
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
use tonic::codegen::{Body, Bytes, StdError};
+/// Maximum time the poll loop waits for a free vcore before polling the
+/// scheduler anyway. `poll_work` doubles as the executor's heartbeat under
+/// pull-based scheduling, so a fully-busy executor must keep polling
(reporting
+/// zero free vcores) or the scheduler times it out and resets its tasks. Kept
+/// well below the scheduler's executor timeout.
+const HEARTBEAT_POLL_INTERVAL: Duration = Duration::from_secs(5);
+
/// Main execution loop that polls the scheduler for available tasks.
///
/// Runs indefinitely, periodically asking the scheduler for work. When tasks
@@ -114,13 +121,28 @@ where
DedicatedExecutor::new("task_runner", executor_specification.vcores as
usize);
loop {
- // Wait for a vcore permit before asking for new work.
- let permit = free_vcores
- .acquire()
- .await
- .map_err(|_| BallistaError::Internal("vcore semaphore
closed".to_string()))?;
- // Make the vcore available again for the actual bind below.
- drop(permit);
+ // Wait for a vcore permit before asking for new work, but cap the wait
+ // so a fully-busy executor still polls the scheduler periodically.
+ // `poll_work` is the executor's ONLY heartbeat under pull-based
+ // scheduling (the scheduler records a heartbeat on every poll). If
every
+ // vcore is held by a task running longer than the scheduler's executor
+ // timeout, blocking here indefinitely stops heartbeats, so the
scheduler
+ // wrongly marks this healthy-but-busy executor dead and resets its
+ // in-flight tasks. On timeout we poll anyway below, reporting
+ // `num_free_vcores: 0`, so liveness no longer depends on vcore
+ // availability.
+ match tokio::time::timeout(HEARTBEAT_POLL_INTERVAL,
free_vcores.acquire()).await {
+ // A vcore is free; release it so the bind below can claim it.
+ Ok(Ok(permit)) => drop(permit),
+ // Semaphore closed (executor shutting down).
+ Ok(Err(_)) => {
+ return Err(BallistaError::Internal(
+ "vcore semaphore closed".to_string(),
+ ));
+ }
+ // No free vcore within the interval; poll anyway to stay alive.
+ Err(_) => {}
+ }
// Keeps track of whether we received task in last iteration
// to avoid going in sleep mode between polling
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
