andygrove commented on issue #1986: URL: https://github.com/apache/datafusion-ballista/issues/1986#issuecomment-6171010086
Some specifics from reading `AdaptiveExecutionGraph::update_task_status` side by side with the static graph, at commit ea1126c0649b, that are relevant here: - **The reset-running-stages path is dead code on the AQE graph.** `reset_running_stages` is declared as an immutable empty map ([aqe/mod.rs#L817](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/scheduler/src/state/aqe/mod.rs#L817)), so the reset loop at L1106-L1130 never runs and the `!reset_running_stages.contains_key(&stage_id)` check at L1013 is always true. In the static graph the same map is `mut` and is populated by the late fetch-failure handling. - **The late fetch-failure path for UnResolved stages is missing.** `StaticExecutionGraph::update_task_status` handles fetch failures reported for stages that are still UnResolved ([execution_graph.rs#L1016-L1088](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/scheduler/src/state/execution_graph.rs#L1016-L1088)); the AQE copy has no equivalent, which is the main source of the dead code above. - **The max-task-failures message is less useful on AQE.** The AQE copy reports only the internal task slot id ([aqe/mod.rs#L942-L948](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/scheduler/src/state/aqe/mod.rs#L942-L948)), while the static copy reports the partitions that exceeded the limit ([execution_graph.rs#L910-L937](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/scheduler/src/state/execution_graph.rs#L910-L937)). These are all consequences of the two `update_task_status` implementations being separate copies. #2595 tracks sharing the execution-graph code between the static and adaptive graphs, which would let the recovery logic here be implemented once. A related AQE-only bug where a stage is dropped from the graph on a failed conversion is #2594. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
