venkata91 commented on a change in pull request #30164:
URL: https://github.com/apache/spark/pull/30164#discussion_r518280848
##########
File path: core/src/main/scala/org/apache/spark/Dependency.scala
##########
@@ -95,6 +97,30 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
val shuffleHandle: ShuffleHandle =
_rdd.context.env.shuffleManager.registerShuffle(
shuffleId, this)
+ // By default, shuffle merge is enabled for ShuffleDependency if push based
shuffle is enabled
+ private[spark] var _shuffleMergeEnabled =
Review comment:
`_` is used so that we can distinguish between class variables and the
method arguments as all these are just some getters and setters. This is not
the case with other variables. If not we have to change the name to something
different that would make it difficult quickly inspect the code.
##########
File path: core/src/main/scala/org/apache/spark/Dependency.scala
##########
@@ -95,6 +97,30 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
val shuffleHandle: ShuffleHandle =
_rdd.context.env.shuffleManager.registerShuffle(
shuffleId, this)
+ // By default, shuffle merge is enabled for ShuffleDependency if push based
shuffle is enabled
+ private[spark] var _shuffleMergeEnabled =
+ Utils.isPushBasedShuffleEnabled(rdd.sparkContext.getConf)
+
+ def setShuffleMergeEnabled(shuffleMergeEnabled: Boolean): Unit = {
+ _shuffleMergeEnabled = shuffleMergeEnabled
+ }
+
+ def shuffleMergeEnabled : Boolean = _shuffleMergeEnabled
+
+ /**
+ * Stores the location of the list of chosen external shuffle services for
handling the
+ * shuffle merge requests from mappers in this shuffle map stage.
+ */
+ private[spark] var _mergerLocs: Seq[BlockManagerId] = Nil
Review comment:
Same as above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
