[
https://issues.apache.org/jira/browse/SPARK-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-3936:
-------------------------------
Description:
This is actually a ticket with two separate problems:
1. Remove auto join elimination
2. Introduce a new fundamental primitive aggregateMessages
For the first one, description provided by Pedro:
There seems to be a bug with the GraphX byte code inspection, specifically in
BytecodeUtils.
These are the unit tests I wrote to expose the problem:
https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121
The first two tests pass, the second two tests fail. This exposes a problem
with inspection of methods in closures, in this case within maps. Specifically,
it seems like there is a problem with inspection of non-inline methods in a
closure.
For the 2nd one, see pull request https://github.com/apache/spark/pull/3100
was:
There seems to be a bug with the GraphX byte code inspection, specifically in
BytecodeUtils.
These are the unit tests I wrote to expose the problem:
https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121
The first two tests pass, the second two tests fail. This exposes a problem
with inspection of methods in closures, in this case within maps. Specifically,
it seems like there is a problem with inspection of non-inline methods in a
closure.
> Remove auto join elimination and introduce aggregateMessages
> ------------------------------------------------------------
>
> Key: SPARK-3936
> URL: https://issues.apache.org/jira/browse/SPARK-3936
> Project: Spark
> Issue Type: Bug
> Components: GraphX
> Reporter: Pedro Rodriguez
> Assignee: Ankur Dave
> Priority: Blocker
> Fix For: 1.2.0
>
>
> This is actually a ticket with two separate problems:
> 1. Remove auto join elimination
> 2. Introduce a new fundamental primitive aggregateMessages
> For the first one, description provided by Pedro:
> There seems to be a bug with the GraphX byte code inspection, specifically in
> BytecodeUtils.
> These are the unit tests I wrote to expose the problem:
> https://github.com/EntilZha/spark/blob/a3c38a8329545c034fae2458df134fa3829d08fb/graphx/src/test/scala/org/apache/spark/graphx/util/BytecodeUtilsSuite.scala#L93-L121
> The first two tests pass, the second two tests fail. This exposes a problem
> with inspection of methods in closures, in this case within maps.
> Specifically, it seems like there is a problem with inspection of non-inline
> methods in a closure.
> For the 2nd one, see pull request https://github.com/apache/spark/pull/3100
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
