Re: [DAGScheduler] resubmitFailedStages, failedStages.clear() and submitStage

2015-12-24 Thread Ted Yu
getMissingParentStages(stage) would be called for the stage (being re-submitted) If there is no missing parents, submitMissingTasks() would be called. If there is missing parent(s), the parent would go through the same flow. I don't see issue in this part of the code. Cheers On Thu, Dec 24, 201

[DAGScheduler] resubmitFailedStages, failedStages.clear() and submitStage

2015-12-24 Thread Jacek Laskowski
Hi, While reviewing DAGScheduler, and where failedStages internal collection of failed staged ready for resubmission is used, I came across a question for which I'm looking an answer to. Any hints would be greatly appreciated. When resubmitFailedStages [1] is executed, and there are any failed st