[
https://issues.apache.org/jira/browse/SPARK-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225788#comment-14225788
]
Reynold Xin commented on SPARK-3936:
------------------------------------
BTW message from Taobao: 根据团队小伙伴的测试，[SPARK-3936] Add aggregateMessages, which
supersedes
mapReduceTriplets这个issue确实Work，可以提升30%的性能。以KCore为例，在standord的100w的数据集上运行作业，运行时间由
37mins减到22mins。使用GraphX的同学可以早日升级。#给周末此时还在加班做Spark和图计算的同学#
i.e. switching from mapReduceTriplets to aggregateMessages improved performance
by 30% in their kcore implementation.
> Remove auto join elimination and introduce aggregateMessages
> ------------------------------------------------------------
>
> Key: SPARK-3936
> URL: https://issues.apache.org/jira/browse/SPARK-3936
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Reporter: Pedro Rodriguez
> Assignee: Ankur Dave
> Priority: Blocker
> Fix For: 1.2.0
>
>
> This is actually a ticket with two separate problems:
> 1. Remove auto join elimination
> 2. Introduce a new fundamental primitive aggregateMessages
> For the first one, description provided by Pedro:
> There seems to be a bug with the GraphX byte code inspection, specifically in
> BytecodeUtils.
> These are the unit tests I wrote to expose the problem:
> https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121
> The first two tests pass, the second two tests fail. This exposes a problem
> with inspection of methods in closures, in this case within maps.
> Specifically, it seems like there is a problem with inspection of non-inline
> methods in a closure.
> For the 2nd one, see pull request https://github.com/apache/spark/pull/3100
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
