Richard-Durkee opened a new pull request, #2875:
URL: https://github.com/apache/datafusion-ballista/pull/2875

   # Which issue does this PR close?
   
   Closes #2833.
   
   # Rationale for this change
   
   `RunningStage::update_task_metrics` took the metrics shape of whichever task 
reported first. Every later task with a different number of operator metrics 
sets was rejected. Since #2828 a rejected task is logged and skipped instead of 
hanging the job, but which tasks win is still arbitrary.
   
   Tasks of one stage can report different counts. Restricting a task's plan to 
its partitions can swap an `EmptyExec`, which has no metrics, for a source 
stand-in that has them. If one of those tasks reports first, its shape wins and 
all the normal tasks after it are dropped. The stage then ends up with metrics 
that don't line up with its plan. `merge_stage_metrics` in `display.rs` 
requires the two counts to match, so the plan display and the REST API show no 
metrics for that stage at all.
   
   # What changes are included in this PR?
   
   - `RunningStage` now counts the operator metrics sets of its plan once, in 
`new` and in `SuccessfulStage::to_running`, with `collect_plan_metrics`. 
`update_task_metrics` checks each task against that count instead of against 
the first task to report. Tasks that match the stage plan always merge, 
whatever order they report in. The ones that don't are skipped with an error, 
as before, and counted in `metrics_skipped_tasks`.
   - The stage plan is the right reference because the display code already 
merges against it, and because the executor builds every writer around the 
stage plan's child. All four writers report metrics, so a task that ran the 
stage plan as is reports exactly that many sets.
   - The warning from #2828 fired once per task. A wide stage where many tasks 
don't match could log thousands of identical lines. The three callers, in 
`execution_graph.rs`, `aqe/mod.rs` and `pipelined_graph.rs`, now go through 
`RunningStage::log_skipped_metrics`, which warns for the first skipped task of 
a stage and logs the rest at debug level.
   
   The error message changes from `task metrics array size N does not equal 
with the stage metrics array size M for task T` to `task T reported N operator 
metrics sets but the stage plan has M`.
   
   Tests: the stage tests built their stage over a bare `EmptyExec`, which has 
no metrics, so they now build it over a `ShuffleWriterExec`, like a real stage 
plan. A new unit test has a mismatched task report first and checks that the 
matching task still merges. A new adaptive graph test does the same through 
`update_task_status` and checks that every multi-task stage finishes with one 
metrics set per operator of its plan. That test fails without the fix.
   
   # Are there any user-facing changes?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to