GitHub user nitin2goyal opened a pull request:
https://github.com/apache/spark/pull/9253
[SPARK-7970] Skip closure cleaning for SQL operations
Also introduces new spark private API in RDD.scala with name
'mapPartitionsInternal' which doesn't closure cleans the RDD elements.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nitin2goyal/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9253.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9253
----
commit 4ee8058447b5e7eff242960aae6fb3333d56631b
Author: nitin.goyal <[email protected]>
Date: 2015-10-19T06:51:42Z
SPARK-11179: Push filters through aggregate if filters are subset of 'group
by' attribute set
commit 3b016b73c239ce9cdc85a5edb1a2127c1f67433a
Author: nitin goyal <[email protected]>
Date: 2015-10-20T07:19:53Z
SPARK-11179: Push filters through aggregate if filters are subset of 'group
by' attribute set
commit 671fbb31d7c908668526bdc146e0168ffb3014a8
Author: nitin goyal <[email protected]>
Date: 2015-10-20T10:17:41Z
SPARK-11179: Push filters through aggregate if filters are subset of 'group
by' attribute set
commit f422aa81e10ad01762847c71e678c3b2ef85a926
Author: nitin goyal <[email protected]>
Date: 2015-10-20T18:32:47Z
[SPARK-11179] [SQL] Push filters through aggregate
Push conjunctive predicates though Aggregate operators when their
references are a subset of the groupingExpressions.
Query plan before optimisation :-
Filter ((c#138L = 2) && (a#0 = 3))
Aggregate [a#0], [a#0,count(b#1) AS c#138L]
Project [a#0,b#1]
LocalRelation [a#0,b#1,c#2]
Query plan after optimisation :-
Filter (c#138L = 2)
Aggregate [a#0], [a#0,count(b#1) AS c#138L]
Filter (a#0 = 3)
Project [a#0,b#1]
LocalRelation [a#0,b#1,c#2]
commit 82fc386675ea2bcd5123d3abd83f6565669fcd69
Author: nitin goyal <[email protected]>
Date: 2015-10-21T04:39:56Z
[SPARK-11179] [SQL] Push filters through aggregate
Push conjunctive predicates though Aggregate operators when their
references are a subset of the groupingExpressions.
Query plan before optimisation :-
Filter ((c#138L = 2) && (a#0 = 3))
Aggregate [a#0], [a#0,count(b#1) AS c#138L]
Project [a#0,b#1]
LocalRelation [a#0,b#1,c#2]
Query plan after optimisation :-
Filter (c#138L = 2)
Aggregate [a#0], [a#0,count(b#1) AS c#138L]
Filter (a#0 = 3)
Project [a#0,b#1]
LocalRelation [a#0,b#1,c#2]
commit 20cf7226f80707bfb6c4164effab50edbea4dce2
Author: nitin goyal <[email protected]>
Date: 2015-10-23T15:19:35Z
Merge remote-tracking branch 'upstream/master'
commit ca487cbae6ba4eb2d14d7b007eb54ccc4dd3ee3a
Author: nitin goyal <[email protected]>
Date: 2015-10-23T15:26:33Z
[SPARK-7970] Skip closure cleaning for SQL operations
Also introduces new spark private API in RDD.scala with name
'mapPartitionsInternal' which doesn't closure cleans the RDD elements.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
