[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85530/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85476/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85476/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85476/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85446/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85446/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85442/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85442/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 @ueshin I pushed some more change to address your comments. There is one regression in existing test `SQLTests.test_udf_with_aggregate_function`. I will try to fix it tomorrow. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85446/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85442/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85152/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85152/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85152/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19872 @ramacode2014 Hi, I'm not sure why you received notifications from this PR, but I guess you can unsubscribe by the "Unsubscribe" button in the right column of this page. Sorry for the inconvenience.

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user ramacode2014 commented on the issue: https://github.com/apache/spark/pull/19872 Please unsubscribe me from this spam On Wed, Dec 20, 2017 at 10:46 AM, Takuya UESHIN wrote: > *@ueshin* commented on this pull request. >

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85138/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85138/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85137/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85137 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85137/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85138/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85137 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85137/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85136/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85136/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85136/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19872 @icexelloss I'm sorry for the late response. Actually I tried to implement prototypes of Pandas UDAF with partial aggregation and combining existing aggregate functions, but they are still much

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84632 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84632/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84632/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84632 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84632/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84631/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84631/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84631/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84630/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84630/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84630/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 And to @holdenk 's question. Pandas group_agg udf fundamentally uses different physical plan than the existing java/scala udf and therefore it's hard to combine them together. I don't know a

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 I end up removing `UDFColumn` class and using the existing `Aggregate` logical plan for pandas group_agg UDF. This reuses a lot of code being to existing `Aggregate` and minimize the code

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84628/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84628 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84628/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84628/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84446/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84446/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84446/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 > I thought @ueshin is working on this BTW. Oh, I certainly don't want to duplicate @ueshin 's work. I am under the impression that @ueshin is working on two-stage PySpark UDAF with

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84415/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84415/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84415/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19872 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84414/ Test FAILed. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84414/testReport)** for PR 19872 at commit

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 cc @HyukjinKwon @holdenk @ueshin Passing some basic tests. I will work on this more next week to clean up and add more testing. ---

[GitHub] spark issue #19872: WIP: [SPARK-22274][PySpark] User-defined aggregation fun...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #84414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84414/testReport)** for PR 19872 at commit