Github user squito commented on the issue:
https://github.com/apache/spark/pull/17208
This looks like the right change. In fact, I could have sworn we had
recently merged in something like this -- maybe there is another pr still in
flight which includes this? @jinxing64 perhaps this is in one of your open prs?
The description needs to be updated, and we really should have a unit test
(though with a very quick look I don't see a good way to test, I'll need to
think about that part). Here is my suggestion for the description:
> We must set the taskset to zombie before the dagscheduler handles the
`taskEnded` event, because that event may cause the dagscheduler to launch
another task attempt. If that happens before the taskSet has been set to
zombie, it will appear that we have conflicting task sets.
> The code worked before this change because dagScheduler.taskEnded() is
async, so the task ended was almost always processed after the zombie status
had been updated. However, that left a race, which would occasionally go the
wrong way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
