cloud-fan commented on code in PR #57179:
URL: https://github.com/apache/spark/pull/57179#discussion_r3573995927
##########
core/src/main/scala/org/apache/spark/Dependency.scala:
##########
@@ -258,6 +258,59 @@ class ShuffleDependency[K: ClassTag, V: ClassTag, C:
ClassTag](
}
+/**
+ * :: DeveloperApi ::
+ * A [[ShuffleDependency]] whose output can be read incrementally: a consumer
stage may begin
+ * reading the shuffle output while the producer stage is still running,
rather than waiting for the
+ * producer's full, materialized output.
+ *
+ * This is a first-class dependency kind, peer to [[NarrowDependency]] and
[[ShuffleDependency]]. It
Review Comment:
This says the class is "peer to `[[ShuffleDependency]]`", but line 274 calls
`ShuffleDependency` "its parent" and the class `extends ShuffleDependency[K, V,
C]` just below. It's a *subtype* of `ShuffleDependency` — a peer only of
`NarrowDependency` (one level up, both under `Dependency[T]`). Worth getting
exact here since the subtype relationship is precisely what lets it inherit
stage-boundary splitting and the transport path.
```suggestion
* This is a subtype of [[ShuffleDependency]] -- and thus, like it, a
first-class dependency kind
* alongside [[NarrowDependency]] under [[Dependency]]. It
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
