cloud-fan commented on issue #24359: Revert [SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions URL: https://github.com/apache/spark/pull/24359#issuecomment-482935316 @squito the event seq is pretty simple: 1. a task of a stage failed with `FetchFailure` 2. `TaskSetManager` marks the corresponding task set as zombie 3. `TaskSchedulerImp` does not clean up the `stageIdToFinishedPartitions` of that stage 4. `DAGScheduler` re-submit the stage, and mark some finished partitions to unfinished. 5. `TaskSchedulerImp` creates another task set for this stage, but mistakenly mark some partitions as finished. I think `3` is the root cause here, but I'm not sure this is the only problem that can make `TaskSchedulerImp` and `DAGScheduler` inconsistent. Anyway, I'm reverting first. Let's think of a better fix later.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
