[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647869895 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1739,18 +1756,24 @@ private[spark] class DAGScheduler(

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647869229 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2000,6 +2023,133 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647868756 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1678,38 +1717,16 @@ private[spark] class DAGScheduler(

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647868756 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1678,38 +1717,16 @@ private[spark] class DAGScheduler(

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647868622 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1739,18 +1756,24 @@ private[spark] class DAGScheduler(

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647868344 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1298,7 +1329,7 @@ private[spark] class DAGScheduler( //

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647763338 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -254,6 +259,29 @@ private[spark] class DAGScheduler( private

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r647758831 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2000,6 +2023,133 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r646974281 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2000,6 +2025,149 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r643661092 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -737,25 +737,26 @@ private[spark] class MapOutputTrackerMaster( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r643574768 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1743,6 +1762,10 @@ private[spark] class DAGScheduler(

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r643344276 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1743,6 +1762,10 @@ private[spark] class DAGScheduler(

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r641707976 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2000,6 +2023,147 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r641095507 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2004,6 +2020,131 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r641094842 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2136,9 +2137,24 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r640027450 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2136,9 +2137,24 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r639104565 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -2004,6 +2020,131 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r629848900 ## File path: .idea/vcs.xml ## @@ -1,36 +0,0 @@ - Review comment: Sorry my bad, I think it got added as part of this `[SPARK-35223] Add

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r629845232 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1271,21 +1302,28 @@ private[spark] class DAGScheduler( *

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r629844923 ## File path: .idea/vcs.xml ## @@ -1,36 +0,0 @@ - Review comment: Somehow my idea file got added and pushed. I think I removed it. Isn't it?

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r628401105 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -254,6 +259,28 @@ private[spark] class DAGScheduler( private

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r626774584 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -2075,6 +2075,25 @@ package object config { .booleanConf

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r626773776 ## File path: core/src/main/scala/org/apache/spark/Dependency.scala ## @@ -102,6 +102,12 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r626224298 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -3393,6 +3406,271 @@ class DAGSchedulerSuite extends

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r626223337 ## File path: core/src/test/resources/META-INF/services/org.apache.spark.scheduler.ExternalClusterManager ## @@ -1,3 +1,4 @@

[GitHub] [spark] venkata91 commented on a change in pull request #30691: [SPARK-32920][SHUFFLE] Finalization of Shuffle push/merge with Push based shuffle and preparation step for the reduce stage

venkata91 commented on a change in pull request #30691: URL: https://github.com/apache/spark/pull/30691#discussion_r626222342 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -689,7 +716,7 @@ private[spark] class DAGScheduler(