andygrove commented on code in PR #2698:
URL: 
https://github.com/apache/datafusion-ballista/pull/2698#discussion_r4579320780


##########
ballista/scheduler/src/state/execution_graph.rs:
##########
@@ -316,6 +330,27 @@ pub trait ExecutionGraph: Debug {
     }
 }
 
+/// Reset a still-Running task_info slot so its partitions return to pending.
+pub(crate) fn reset_unlaunched_task_in_stages(
+    stages: &mut HashMap<usize, ExecutionStage>,
+    key: &TaskKey,
+) -> bool {
+    let Some(ExecutionStage::Running(stage)) = stages.get_mut(&key.stage_id) 
else {
+        return false;
+    };

Review Comment:
   Small hardening idea. `reset_unlaunched_task_in_stages` matches any 
`Running` slot at `(stage_id, task_id)`. Task ids restart after a rollback 
though. The reservation ledger keys on `stage_attempt_num` for exactly that 
reason (see the comment on `record_task_reservations`). Could we pass the 
`TaskDescription` plus `executor_id` and only reset when the stage attempt 
matches and the slot is `Running { executor_id }` for this executor? Then a 
late reset can never touch a task that really launched elsewhere.



##########
ballista/scheduler/src/state/mod.rs:
##########
@@ -926,6 +938,14 @@ mod tests {
         let (tx_event, _rx_event) = tokio::sync::mpsc::channel(100);
         let sender = EventSender::new(tx_event);
         state.task_manager.record_task_reservations(&bound);
+
+        let available_before =

Review Comment:
   When I merge this with current `main`, 
`launch_tasks_refunds_vcores_when_executor_metadata_missing` fails at the 
`available_before == 0` assert. The plan now has 16 pending partitions and the 
bind only takes 8 of them. That also makes `available_after > 0` pass even 
without the fix. Could you rebase and assert the exact count instead? Something 
like summing `global_input_partition_ids.len()` over `bound` and checking 
`available_after == available_before + bound_partitions`. I checked locally and 
the fix gives 8 before and 16 after, so only the test needs adjusting.



##########
ballista/scheduler/src/state/mod.rs:
##########
@@ -540,6 +540,18 @@ impl<T: 'static + AsLogicalPlan, U: 'static + 
AsExecutionPlan> SchedulerState<T,
                         error!(
                             "Failed to launch new task, could not get executor 
metadata: {e}"
                         );
+                        // Bound tasks were already marked Running at bind
+                        // time. Return them to pending before refunding
+                        // vcores, otherwise they hang forever (#2691).
+                        let reset = state
+                            .task_manager
+                            .reset_unlaunched_tasks(&reservations)
+                            .await;
+                        if reset > 0 {

Review Comment:
   Question, maybe for a follow-up. Pipelined jobs fall through to the default 
`reset_unlaunched_task` and return `false`. A temporary metadata miss would 
leave that job holding its reservation, with peers waiting on tasks that never 
start. Would it make sense to put pipelined tasks' job ids into the returned 
failed set in this branch so `JobRunningFailed` fires? Happy for that to be a 
separate issue.



##########
ballista/scheduler/src/state/execution_graph.rs:
##########
@@ -1394,6 +1429,10 @@ impl ExecutionGraph for StaticExecutionGraph {
         }
     }
 
+    fn reset_unlaunched_task(&mut self, key: &TaskKey) -> bool {

Review Comment:
   Since AQE is on by default, the test only covers `AdaptiveExecutionGraph`. 
Would you mind adding a variant with `ballista.planner.adaptive.enabled = 
false` so this `StaticExecutionGraph` override gets exercised too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to