[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973642323 ## core/src/main/scala/org/apache/spark/internal/config/package.scala: ## @@ -2311,6 +2311,15 @@ package object config { .intConf .createWithDefault(3)

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973642320 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,60 +2251,57 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973642279 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -282,13 +286,19 @@ private[spark] class DAGScheduler( None } - // Use

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973642094 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973642086 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973641582 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973641546 ## core/src/main/scala/org/apache/spark/internal/config/package.scala: ## @@ -2309,7 +2309,17 @@ package object config { " shuffle is enabled.")

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r973488467 ## core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala: ## @@ -4443,36 +4443,115 @@ class DAGSchedulerSuite extends SparkFunSuite with

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r968109053 ## core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala: ## @@ -4440,6 +4443,37 @@ class DAGSchedulerSuite extends SparkFunSuite with

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r968109053 ## core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala: ## @@ -4440,6 +4443,37 @@ class DAGSchedulerSuite extends SparkFunSuite with

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r968109053 ## core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala: ## @@ -4440,6 +4443,37 @@ class DAGSchedulerSuite extends SparkFunSuite with

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r968089690 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2249,94 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r966495158 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r965431205 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r964091915 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r964091915 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r963281413 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,70 +2252,110 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r963274409 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -282,13 +285,18 @@ private[spark] class DAGScheduler( None } - // Use

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r960287381 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,60 +2251,57 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r960285819 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2242,60 +2251,57 @@ private[spark] class DAGScheduler( val numMergers =

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r958010820 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -282,13 +285,19 @@ private[spark] class DAGScheduler( None } - // Use

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956887936 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r956329357 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r955615397 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r955615397 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r955615397 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r955615397 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r954454349 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r954454349 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r954454349 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }

[GitHub] [spark] mridulm commented on a diff in pull request #37533: [SPARK-40096]Fix finalize shuffle stage slow due to connection creation slow

mridulm commented on code in PR #37533: URL: https://github.com/apache/spark/pull/37533#discussion_r954454349 ## core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala: ## @@ -2259,37 +2259,51 @@ private[spark] class DAGScheduler( }