Re: About introduce function sum0 to Spark

2018-10-22 Thread Mark Hamstra
That's a horrible name. This is just a fold. On Mon, Oct 22, 2018 at 7:39 PM 陶 加涛 wrote: > Hi, in calcite, has the concept of sum0, here I quote the definition of > sum0: > > > > Sum0 is an aggregator which returns the sum of the values which > > go into it like Sum. It differs in that when no

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Darcy Shen
+1 On Tue, 23 Oct 2018 01:42:06 +0800 Wenchen Fan wrote Please vote on releasing the following candidate as Apache Spark version 2.4.0.The vote is open until October 26 PST and passes if a majority +1 PMC votes are cast, witha minimum of 3 +1 votes.[ ]

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Aron.tao
+1 -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

About introduce function sum0 to Spark

2018-10-22 Thread 陶 加涛
Hi, in calcite, has the concept of sum0, here I quote the definition of sum0: Sum0 is an aggregator which returns the sum of the values which go into it like Sum. It differs in that when no non null values are applied zero is returned instead of null.. One scenario is that we can use sum0 to

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Imran Rashid
+1 No blockers and our internal tests are all passing. (I did file https://issues.apache.org/jira/browse/SPARK-25805, but this is just a minor issue with a flaky test) On Mon, Oct 22, 2018 at 12:42 PM Wenchen Fan wrote: > Please vote on releasing the following candidate as Apache Spark version

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Wenchen Fan
Regarding the doc tickets, I vaguely remember that we can merge doc PRs after release and publish doc to spark website later. Can anyone confirm? On Tue, Oct 23, 2018 at 8:30 AM Sean Owen wrote: > This is what I got from a straightforward build of the source distro > here ... really, ideally,

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Sean Owen
This is what I got from a straightforward build of the source distro here ... really, ideally, it builds as-is from source. You're saying someone would have to first build a k8s distro from source too? It's not a 'must' that this be automatic but nothing else fails out of the box. I feel like I

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Stavros Kontopoulos
> tar (child): Error is not recoverable: exiting now > tar: Child returned status 2 > tar: Error is not recoverable: exiting now > scripts/setup-integration-test-env.sh: line 85: > /home/srowen/spark-2.4.0/resource-managers/kubernetes/ >

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Sean Owen
Provisionally looking good to me, but I had a few questions. We have these open for 2.4, but I presume they aren't actually going to be in 2.4 and should be untargeted: SPARK-25507 Update documents for the new features in 2.4 release SPARK-25179 Document the features that require Pyarrow 0.10

Re: GitHub is out of order

2018-10-22 Thread shane knapp
ok, it looks like webhooks are caught up... turning jenkins back on now. if anyone sees any strange behavior, please reply back immediately to this thread and i'll see what i can do on my end to mitigate any further issues (aka 'service jenkins stop'). shane On Mon, Oct 22, 2018 at 2:14 PM

Re: GitHub is out of order

2018-10-22 Thread shane knapp
quick update: the github status messages have gone from "red" to "orange", which to me still means "we're broken". :) https://status.github.com/messages i'm still holding off on restarting jenkins, but if things don't change within a couple of hours, i'll start it back up and keep an eye on

Re: [VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Wenchen Fan
Since GitHub and Jenkins are in a chaotic state, I didn't wait for a green Jenkins QA job for the RC4 commit. We should fail this RC if the Jenkins is broken (very unlikely). I'm adding my own +1, all known blockers are resolved. On Tue, Oct 23, 2018 at 1:42 AM Wenchen Fan wrote: > Please vote

[VOTE] SPARK 2.4.0 (RC4)

2018-10-22 Thread Wenchen Fan
Please vote on releasing the following candidate as Apache Spark version 2.4.0. The vote is open until October 26 PST and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 2.4.0 [ ] -1 Do not release this package because ... To

Re: GitHub is out of order

2018-10-22 Thread shane knapp
i've actually taken jenkins down completely. after i see an 'all clear' on their status page, i'll bring jenkins back up. On Mon, Oct 22, 2018 at 9:34 AM shane knapp wrote: > jenkins is no longer accepting new builds. > > On Mon, Oct 22, 2018 at 8:55 AM Hyukjin Kwon wrote: > >> It's chaotic

Re: GitHub is out of order

2018-10-22 Thread shane knapp
jenkins is no longer accepting new builds. On Mon, Oct 22, 2018 at 8:55 AM Hyukjin Kwon wrote: > It's chaotic now.. can we turn off the Jenkins for a while if the Github > is being out of order for a while? My notifications are full of > AmblapJenkins bot messages ... > > On Mon, 22 Oct 2018,

Re: GitHub is out of order

2018-10-22 Thread Hyukjin Kwon
It's chaotic now.. can we turn off the Jenkins for a while if the Github is being out of order for a while? My notifications are full of AmblapJenkins bot messages ... On Mon, 22 Oct 2018, 1:13 pm Hyukjin Kwon, wrote: > Yea.. please ignore my duplicated comments if they exist. I didn't know >

Re: SPIP: SPARK-25728 Structured Intermediate Representation (Tungsten IR) for generating Java code

2018-10-22 Thread Xiao Li
Hi, Kazuaki, Thanks for your great SPIP! I am willing to be the shepherd of this SPIP. Cheers, Xiao On Mon, Oct 22, 2018 at 12:05 AM Kazuaki Ishizaki wrote: > Hi Yamamuro-san, > Thank you for your comments. This SPIP gets several valuable comments and > feedback on Google Doc: >

Re: SPIP: SPARK-25728 Structured Intermediate Representation (Tungsten IR) for generating Java code

2018-10-22 Thread Kazuaki Ishizaki
Hi Yamamuro-san, Thank you for your comments. This SPIP gets several valuable comments and feedback on Google Doc: https://docs.google.com/document/d/1Jzf56bxpMpSwsGV_hSzl9wQG22hyI731McQcjognqxY/edit?usp=sharing . I hope that this SPIP could go forward based on these feedback. Based on this