Github user suyanNone commented on the pull request:
https://github.com/apache/spark/pull/4055#issuecomment-118758477
@squito about stage.pendingTask and shuffleMapStage.isAvaiable
for use `isAvailable` to instead of `stage.pendingTask's`, may need more
careful to do with that.
there is a comments in the code `// Some tasks had failed; let's resubmit
this shuffleStage`, I can't recognize the situation or context... if we
change to `isAvailable`, then the following code
`if(shuffleStage.outputLocs.contains(Nil))` may be removed? and also may remove
all variable `stage.pendingTask` related code, because is no reader any more.
the logical of dagScheduler is complicated and always has some confusing
code...
```
if (runningStages.contains(shuffleStage) &&
shuffleStage.pendingTasks.isEmpty) { //
clearCacheLocs()
if (shuffleStage.outputLocs.contains(Nil)) {
// Some tasks had failed; let's resubmit this shuffleStage
// TODO: Lower-level scheduler should also deal with this
logInfo("Resubmitting " + shuffleStage + " (" + shuffleStage.name + ")
because some of its tasks had failed: " +
shuffleStage.outputLocs.zipWithIndex.filter(_._1.isEmpty).map(_._2).mkString(",
"))
submitStage(shuffleStage)
} else {
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
