mridulm commented on a change in pull request #33034:
URL: https://github.com/apache/spark/pull/33034#discussion_r676108210
##########
File path: core/src/main/scala/org/apache/spark/Dependency.scala
##########
@@ -122,6 +120,14 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
*/
private[this] var _shuffleMergedFinalized: Boolean = false
+ /**
+ * shuffleMergeId is used to give temporal ordering to the executions of a
ShuffleDependency.
+ * This is required in order to handle indeterministic stage retries for
push-based shuffle.
+ */
+ private[this] var _shuffleMergeId: Int = -1
Review comment:
start with `0`
Also, if there is intention to differentiate between deterministic and other
stages, this needs to be initialized appropriately - which is not yet done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
