[
https://issues.apache.org/jira/browse/SPARK-39356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-39356:
---------------------------------
Target Version/s: (was: 3.3.0)
> Add option to skip initial message in Pregel API
> ------------------------------------------------
>
> Key: SPARK-39356
> URL: https://issues.apache.org/jira/browse/SPARK-39356
> Project: Spark
> Issue Type: Improvement
> Components: GraphX
> Affects Versions: 3.2.1
> Reporter: Aaron Zolnai-Lucas
> Priority: Minor
> Labels: graphx, pregel
>
> The current (3.2.1) [Pregel
> API|https://github.com/apache/spark/blob/5a3ba9b0b301a3b0c43f8d0d88e2b6bdce57d0e6/graphx/src/main/scala/org/apache/spark/graphx/Pregel.scala#L117]
> takes a parameter {{initialMsg: A}} where {{A : scala.reflect.ClassTag}} is
> the message type for the Pregel iterations. At the start of the iterative
> process, the user-supplied vertex update method {{vprog}} is called with the
> initial message.
> However, in some cases, the start point for a message passing scheme is best
> described by starting with the {{message}} phase rather than the {{vprog}}
> phase, and in many cases the first message depends on individual vertex data
> (instead of a static initial message). In these cases, users are forced to
> add boilerplate to their {{vprog}} function to check if the message received
> is the {{initialMessage}} and ignore the message (leave the node state
> unchanged) if it is. This leads to less efficient (due to extra iteration and
> check) and less readable code.
>
> My proposed solution is to change {{initialMsg}} to a parameter of type
> {{Option[A]}} with default {{{}None{}}}, and then inside {{Pregel.apply}}
> function, set:
> {code:scala}
> var g = initialMsg match {
> case Some(msg) => graph.mapVertices((vid, vdata) => vprog(vid, vdata, msg))
> case _ => graph
> }
> {code}
> This way, the user chooses whether to start the iteration from the
> {{message}} or {{vprog}} phase. I believe this small change could improve
> user code readability and efficiency.
> Note: The signature of {{GraphOps.pregel}} would have to be changed to match
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
