Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3054#discussion_r20125180
--- Diff:
graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartition.scala ---
@@ -285,50 +337,116 @@ class EdgePartition[
}
/**
- * Upgrade the given edge iterator into a triplet iterator.
+ * Send messages along edges and aggregate them at the receiving
vertices. Implemented by scanning
+ * all edges sequentially and filtering them with `idPred`.
+ *
+ * @param mapFunc the edge map function which generates messages to
neighboring vertices
+ * @param reduceFunc the combiner applied to messages destined to the
same vertex
+ * @param mapUsesSrcAttr whether or not `mapFunc` uses the edge's source
vertex attribute
+ * @param mapUsesDstAttr whether or not `mapFunc` uses the edge's
destination vertex attribute
+ * @param idPred a predicate to filter edges based on their source and
destination vertex ids
*
- * Be careful not to keep references to the objects from this iterator.
To improve GC performance
- * the same object is re-used in `next()`.
+ * @return iterator aggregated messages keyed by the receiving vertex id
*/
- def upgradeIterator(
- edgeIter: Iterator[Edge[ED]], includeSrc: Boolean = true,
includeDst: Boolean = true)
- : Iterator[EdgeTriplet[VD, ED]] = {
- new ReusingEdgeTripletIterator(edgeIter, this, includeSrc, includeDst)
+ def mapReduceTriplets[A: ClassTag](
+ mapFunc: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)],
+ reduceFunc: (A, A) => A,
+ mapUsesSrcAttr: Boolean,
+ mapUsesDstAttr: Boolean,
+ idPred: (VertexId, VertexId) => Boolean): Iterator[(VertexId, A)] = {
--- End diff --
it'd be great to do away with the closure here since this requires caller
to understand the internals of EdgePartition. we can just have two boolean
fields
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
