[GitHub] [spark] Ngone51 commented on a diff in pull request #39011: [SPARK-41469][CORE] Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated

2022-12-13 Thread GitBox
Ngone51 commented on code in PR #39011: URL: https://github.com/apache/spark/pull/39011#discussion_r1047069384 ## core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala: ## @@ -51,6 +52,7 @@ import org.apache.spark.rdd.RDD * at the same time for

[GitHub] [spark] Ngone51 commented on a diff in pull request #39011: [SPARK-41469][CORE] Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated

2022-12-12 Thread GitBox
Ngone51 commented on code in PR #39011: URL: https://github.com/apache/spark/pull/39011#discussion_r1045505675 ## core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala: ## @@ -51,6 +52,7 @@ import org.apache.spark.rdd.RDD * at the same time for

[GitHub] [spark] Ngone51 commented on a diff in pull request #39011: [SPARK-41469][CORE] Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated

2022-12-12 Thread GitBox
Ngone51 commented on code in PR #39011: URL: https://github.com/apache/spark/pull/39011#discussion_r1045505675 ## core/src/main/scala/org/apache/spark/scheduler/ShuffleMapTask.scala: ## @@ -51,6 +52,7 @@ import org.apache.spark.rdd.RDD * at the same time for

[GitHub] [spark] Ngone51 commented on a diff in pull request #39011: [SPARK-41469][CORE] Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated

2022-12-12 Thread GitBox
Ngone51 commented on code in PR #39011: URL: https://github.com/apache/spark/pull/39011#discussion_r1045502828 ## core/src/main/scala/org/apache/spark/MapOutputTracker.scala: ## @@ -1128,6 +1128,17 @@ private[spark] class MapOutputTrackerMaster( } } + /** + * Get

[GitHub] [spark] Ngone51 commented on a diff in pull request #39011: [SPARK-41469][CORE] Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated

2022-12-09 Thread GitBox
Ngone51 commented on code in PR #39011: URL: https://github.com/apache/spark/pull/39011#discussion_r1044572392 ## core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala: ## @@ -1046,17 +1046,46 @@ private[spark] class TaskSetManager( /** Called by