andygrove commented on code in PR #2698:
URL:
https://github.com/apache/datafusion-ballista/pull/2698#discussion_r4643078521
##########
ballista/scheduler/src/state/mod.rs:
##########
@@ -547,6 +547,18 @@ impl<T: 'static + AsLogicalPlan, U: 'static +
AsExecutionPlan> SchedulerState<T,
error!(
"Failed to launch new task, could not get executor
metadata: {e}"
);
+ // Bound tasks were already marked Running at bind
+ // time. Return them to pending before refunding
+ // vcores, otherwise they hang forever (#2691).
+ let reset = state
+ .task_manager
+ .reset_unlaunched_tasks(&executor_id,
&reservations)
Review Comment:
One small question for a possible follow-up. If the metadata lookup kept
failing while the executor's vcores stayed in the budget, I think we'd now loop
bind, miss, reset, refund and `ReviveOffers` back to back. Before, that loop
stopped once pending drained. With the in-memory cluster this only looks
reachable as a brief race with `remove_executor`, so I don't think it should
block. Would a short backoff, or removing the executor after a few misses in a
row, be worth an issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]