Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14876
closing, @maropu will take over
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14876
yea, pushing down partial aggregate below exchange is a good idea, but I
think it's out of the scope of SPARK-12978, which is aim to remove unnecessary
partial aggregate right?
---
If your
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/14876
yea, I thinks so. I like the approach in this pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/14876
On the other hand, when caching the already-partitioned input table, we
cannot push-down them;
```
(0 to 1000).map(x => (x % 2, x.toString)).toDF("a",
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/14876
I found that we need to push-down partial aggregation below exchange
operators instead of merging them? For example, in the spark v2.0 branch,
```
(0 to 1000).map(x => (x % 2,
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/14876
Thank you for your concrete example! I'll check in hours.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14876
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14876
**[Test build #64660 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64660/consoleFull)**
for PR 14876 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14876
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64660/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14876
**[Test build #64660 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64660/consoleFull)**
for PR 14876 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14876
cc @maropu @hvanhovell @yhuai @liancheng @clockfly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
11 matches
Mail list logo
