Imran Rashid commented on SPARK-17911:

bq. In other words, handling a ResubmitFailedStages event should be quick, and 
causes failedStages to be cleared, allowing the next ResubmitFailedStages event 
to be posted from the handling of another FetchFailed. If there are the 
expected lot of fetch failures for a single stage, and there is no 
RESUBMIT_TIMEOUT, then it is quite likely that there will be a burst of 
resubmit events (and corresponding log messages) and submitStage calls made in 
rapid succession.

Lemme rephrase your comment to make sure I understand it.

The messageScheduler and delay do *not* effective correctness, or even what 
actually gets resubmitted.  When we resubmit {{mapStage}}, we always resubmit 
all tasks corresponding to shuffle map output on the failed executor.  And when 
we resubmit the {{failedStage}}, there is probably a long enough delay from 
{{mapStage}} that waiting 200ms is relatively inconsequential.

However, it *does* effect the logging.  If the scheduler event queue is 
relatively empty, then as the fetch failures trickle in, for each one we'd post 
a Resubmit event which gets handled relatively quickly.  So each fetch failure 
would trigger another resubmit event and more logging.  Which is also 
undesirable, both because of the noise in the logs, and b/c it would creates an 
unnecessary flood of events on the scheduler event queue.

Is that a fair summary?

I agree with everything said there, but then I'd request we take one of two 

1) change the resubmit logic -- in addition to checking failed stages, you can 
also check {{waitingStages}} and {{runningStages}}.  that is what happens 
eventually anyway inside {{resubmitFailedStages}}.  This would actually be even 
better for decreasing noise in the logs etc.
I know this may seem like a small thing to make such a big deal about, but I 
honestly think this is confusing enough that its worth cleaning up -- 
eliminating an unneeded event queue I think is a significant win.

2) If we don't do that, lets at least add in a better comment explaining the 
purpose.  (Maybe just a pointer to this jira at this point)

> Scheduler does not need messageScheduler for ResubmitFailedStages
> -----------------------------------------------------------------
>                 Key: SPARK-17911
>                 URL: https://issues.apache.org/jira/browse/SPARK-17911
>             Project: Spark
>          Issue Type: Improvement
>          Components: Scheduler
>    Affects Versions: 2.0.0
>            Reporter: Imran Rashid
> Its not totally clear what the purpose of the {{messageScheduler}} is in 
> {{DAGScheduler}}.  It can perhaps be eliminated completely; or perhaps we 
> should just clearly document its purpose.
> This comes from a long discussion w/ [~markhamstra] on an unrelated PR here: 
> https://github.com/apache/spark/pull/15335/files/c80ad22a242255cac91cce2c7c537f9b21100f70#diff-6a9ff7fb74fd490a50462d45db2d5e11
> But its tricky so breaking it out here for archiving the discussion.
> Note: this issue requires a decision on what to do before a code change, so 
> lets just discuss it on jira first.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to