xs-li opened a new pull request #27113: [SPARK-30432][Graphx]reduce degree recomputation in StronglyConnectedComponents URL: https://github.com/apache/spark/pull/27113 So the computation happens every time in the do-while loop, the first time the outer while loop executes. although just once per do-while loop after, it seems, but It does reduce a lot of recomputation；because every time it jump out of the do-while loop，there are no vertices have only out-degree or in-degree，so it's no need to recompute degree to tag the vertices true. I have done a small code proposal, because there is a problem when the pregel executions have done, the degree no need to be recomputed. for example，the Email-EuAll data set：http://snap.stanford.edu/data/email-EuAll.html do-while loop execute 10 times，and the reduce logic happend 8 times；so it would be helpful when computing StronglyConnectedComponents to reduce degree computation. I created a branch in my fork: https://github.com/xs-li/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala I hope you can consider this small code proposal.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
