cloud-fan commented on issue #23871: [SPARK-23433][SPARK-25250] [CORE] Later created TaskSet should learn about the finished partitions URL: https://github.com/apache/spark/pull/23871#issuecomment-468590197 Let's take a step back and re-understand the initial problem. My understanding is: a stage can have multiple task set managers with one of them being active. However, the zombie task set managers can still have running tasks. This can lead to two problems: 1. A task from the zombie TSM finishes, later the active TSM launches a new task for the same partition, which keeps failing 2. A task from the zombie TSM finishes. But right before it finishes, the active TSM launches a new task for the same partition. After the task from zombie TSM finishes, the new task keeps failing. The recent work only focuses on case 1, and doesn't work for case 2 IIUC. Is it the right understanding or I miss something?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
