jerrypeng commented on code in PR #57286: URL: https://github.com/apache/spark/pull/57286#discussion_r3617533547
########## core/src/main/scala/org/apache/spark/Dependency.scala: ########## @@ -258,6 +261,71 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag]( } +/** + * :: DeveloperApi :: + * A [[ShuffleDependency]] whose output can be read incrementally: a consumer stage may begin + * reading the shuffle output while the producer stage is still running, rather than waiting for the + * producer's full, materialized output. + * + * This is a subtype of [[ShuffleDependency]] -- and thus, like it, a first-class dependency kind + * alongside [[NarrowDependency]] under [[Dependency]]. It is intended to be the marker the + * `DAGScheduler` will use to decide that the producer and consumer + * stages connected by this edge may run concurrently (a "pipelined group"), and that the shuffle + * layer should serve this shuffle with an incremental shuffle implementation. A plain + * [[ShuffleDependency]] keeps the existing semantics: its output is fully materialized before any + * consumer reads it. + * + * This class only declares the capability. On its own it behaves exactly like its parent Review Comment: will fix the comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
