Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4145#discussion_r27923915
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala ---
@@ -47,9 +48,14 @@ private[spark] case class NarrowCoGroupSplitDep(
}
}
-private[spark] case class ShuffleCoGroupSplitDep(handle: ShuffleHandle)
extends CoGroupSplitDep
-
-private[spark] class CoGroupPartition(idx: Int, val deps:
Array[CoGroupSplitDep])
+/**
+ * Stores information about the narrow dependencies used by a
CoGroupedRdd. narrowDeps maps to
+ * the dependencies variable in the parent RDD: for each one to one
dependency in dependencies,
+ * narrowDeps has a NarrowCoGroupSplitDep (describing the partition for
that dependency) at the
+ * corresponding index.
+ */
+private[spark] class CoGroupPartition(
+ idx: Int, val narrowDeps: Array[Option[NarrowCoGroupSplitDep]])
--- End diff --
as discussed offline, let's make it explicit that the size of the array ==
number of parents.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
