cloud-fan commented on issue #26975: [SPARK-30325][CORE] markPartitionCompleted cause task status inconsistent URL: https://github.com/apache/spark/pull/26975#issuecomment-569590023 Note that, a task roughly has 2 states: running and finished. a partition has 2 states as well: successful or not. `TaskSetManager.successful` tracks the successfulness of each partition, and is kind of synced among all TSMs, via `TaskSetManager.markPartitionCompleted`. It's obviously a bug that `TaskSetManager.executorLost` checks the status of task and partition separately. A task may satisfy both conditions and be handled twice. For case 1: I think it's OK as the new task will override the map status (the new task has a bigger epoch). It's a waste of resources, but it's better than hang. For case 2: I don't think it can happen. If T1 finished first, the partition in TSM2 will be marked as successful too.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
