andygrove commented on code in PR #2887:
URL:
https://github.com/apache/datafusion-ballista/pull/2887#discussion_r4607322712
##########
ballista/scheduler/src/state/aqe/mod.rs:
##########
@@ -297,12 +298,20 @@ impl AdaptiveExecutionGraph {
/// Processing stage status update after task status changing
fn processing_stages_update(
&mut self,
- updated_stages: UpdatedStages,
+ mut updated_stages: UpdatedStages,
) -> ballista_core::error::Result<Vec<QueryStageSchedulerEvent>> {
let job_id = self.job_id().to_owned();
let mut has_resolved = false;
let mut job_err_msg = "".to_owned();
+ // A fetch failure asks to rerun the finished stage whose output was
+ // lost, which AQE cannot do yet; fail the job instead of hanging it.
+ for stage_id in mem::take(&mut
updated_stages.resubmit_successful_stages) {
+ updated_stages
+ .failed_stages
+ .insert(stage_id, cannot_rerun_finished_stage(stage_id));
Review Comment:
Small wording thing. On the fetch-failure path the job error ends up as `Job
failed due to stage 0 failed: the output of finished stage 0 was lost ...`.
Stage 0 actually succeeded. It was the reader stage whose task failed. Would it
read better to key this entry on the reader stage, or to adjust the message so
it doesn't say stage 0 failed?
##########
ballista/scheduler/src/state/aqe/test/job_failure.rs:
##########
@@ -52,6 +56,76 @@ async fn test_join_plan(partition: usize) ->
AdaptiveExecutionGraph {
.unwrap()
}
+/// Builds an adaptive graph for an aggregation: a map stage, then a reduce
+/// stage reading its shuffle output.
+async fn test_aggregation_plan() -> AdaptiveExecutionGraph {
+ let ctx = SessionContext::new_with_config(mock_context().copied_config());
+ ctx.register_batch("t", mock_batch().unwrap()).unwrap();
+ let sql = "select c, sum(a) from t group by c";
+ let plan = ctx.sql(sql).await.unwrap().logical_plan().clone();
+ AdaptiveExecutionGraph::try_new(
+ "localhost:50050",
+ &"job".into(),
+ "",
+ &ctx,
+ &plan,
+ 0,
+ None,
+ )
+ .await
+ .unwrap()
+}
+
+/// Runs the map stage of an aggregation on executor "a", leaving the reduce
+/// stage, which reads its output, running.
+async fn aggregation_with_map_output_on_a() -> AdaptiveExecutionGraph {
+ let mut graph = test_aggregation_plan().await;
+ revive_graph_and_complete_next_stage_with_executor(
+ &mut graph,
+ &mock_executor("a".to_string()),
+ )
+ .unwrap();
+ graph.revive();
+ assert!(matches!(graph.stages[&0], ExecutionStage::Successful(_)));
+ assert!(matches!(graph.stages[&1], ExecutionStage::Running(_)));
+ graph
+}
+
+const CANNOT_RERUN: &str = "cannot rerun a finished stage";
+
+// AQE cannot rerun a finished stage whose output was lost (#1986): its
+// planner no longer tracks the stage, so the rerun's tasks could not report
+// and the job hung (#2539). Losing the executor with that output fails the
+// rollback instead, and the task manager fails the job on that error.
+#[tokio::test]
+async fn losing_the_output_of_a_finished_stage_fails_the_rollback() {
+ let mut graph = aggregation_with_map_output_on_a().await;
+
+ let err = graph.reset_stages_on_lost_executor("a").unwrap_err();
+
+ assert!(err.to_string().contains(CANNOT_RERUN), "{err}");
+}
+
+// The same loss, found by a reduce task failing to fetch the map output: the
+// job fails instead of hanging.
+#[tokio::test]
+async fn a_fetch_failure_on_the_output_of_a_finished_stage_fails_the_job() ->
Result<()> {
+ let mut graph = aggregation_with_map_output_on_a().await;
+ let b = mock_executor("b".to_string());
+ let task = graph.pop_next_task(&b.id)?.unwrap();
+ let status = mock_failed_task(task, fetch_partition_failure("a", 0, 0));
+
+ graph.update_task_status(&b, vec![status], 4, 4)?;
Review Comment:
Could this also assert that `update_task_status` returns a
`QueryStageSchedulerEvent::JobRunningFailed`? That event is what actually
aborts the job and cancels its tasks. Checking it would guard the "no longer
hangs" part directly, since the status alone could be set without the event
going out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]