Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

2017-11-07 Thread Saisai Shao
+1, looking forward to more design details of this feature. Thanks Jerry On Wed, Nov 8, 2017 at 6:40 AM, Shixiong(Ryan) Zhu wrote: > +1 > > On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley > wrote: > >> +1 >> >> On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust >> wrote: >> >>> +1 >>> >>> On Sat

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

2017-11-07 Thread Reynold Xin
The vote has passed with the following +1s: Reynold Xin* Debasish Das Noman Khan Wenchen Fan* Matei Zaharia* Weichen Xu Vaquar Khan Burak Yavuz Xiao Li Tom Graves* Michael Armbrust* Joseph Bradley* Shixiong Zhu* And the following +0s: Sean Owen* Thanks for the feedback! On Wed, Nov 1, 2017

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Felix Cheung
We actually have some immediate needs for custom config for some upcoming integration tests. I don't know if such changes are possible in ASF Jenkins but the work is in progress in RISELab Jenkins :) From: holden.ka...@gmail.com on behalf of Holden Karau Sen

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Holden Karau
That makes sense, in that case do we know how hard it would be to make the necessary hands to the AMP Lab Jenkins to support this? On Tue, Nov 7, 2017 at 4:04 PM Reynold Xin wrote: > My understanding is that AMP actually can provide more resources or adapt > changes, while ASF needs to manage 20

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Reynold Xin
My understanding is that AMP actually can provide more resources or adapt changes, while ASF needs to manage 200+ projects and it's hard to accommodate much. I could be wrong though. On Tue, Nov 7, 2017 at 2:14 PM, Holden Karau wrote: > True, I think we've seen that the Amp Lab Jenkins needs to

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Holden Karau
True, I think we've seen that the Amp Lab Jenkins needs to be more focused on running AMP Lab projects, and while I don't know how difficult the ASF Jenkins is I assume it might be an easier place to make changes going forward? (Of course this could be the grass is greener on the other side and I d

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

2017-11-07 Thread Shixiong(Ryan) Zhu
+1 On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley wrote: > +1 > > On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust > wrote: > >> +1 >> >> On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li wrote: >> >>> +1 >>> >>> 2017-11-04 11:00 GMT-07:00 Burak Yavuz : >>> +1 On Fri, Nov 3, 2017 at 10:0

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Sean Owen
Faster tests would be great. I recall that the straightforward ways to parallelize via Maven haven't worked because many tests collide with one another. Is this about running each module's tests in a container? that should work. I can see how this is becoming essential for repeatable and reliable

Re: [ML] Migrating transformers from mllib to ml

2017-11-07 Thread Joseph Bradley
Hi, we do still want to do this migration; it's just been a bit stalled due to low bandwidth. There are still a few feature parity items which need to be completed, so the deprecation will likely not happen until after 2.3. Joseph On Tue, Nov 7, 2017 at 12:38 AM, 颜发才(Yan Facai) wrote: > Hi, I h

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

2017-11-07 Thread Joseph Bradley
+1 On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust wrote: > +1 > > On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li wrote: > >> +1 >> >> 2017-11-04 11:00 GMT-07:00 Burak Yavuz : >> >>> +1 >>> >>> On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan >>> wrote: >>> +1 On Fri, Nov 3, 2017 at 8:14

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Holden Karau
Just wanting to +1 this idea. One potential option would be to look at migrating away from the AMP Lab Jenkins infra into the ASF infra. I've added Josh, Shane, and Sean to the CC line explicitly since I think they might have opinions about this. On Tue, Oct 31, 2017 at 11:05 PM, Xin Lu wrote:

Re: Jenkins upgrade/Test Parallelization & Containerization

2017-11-07 Thread Holden Karau
Oh and to be clear part of my +1 is dockerized Spark builds would simplify a lot of headaches we face trying to coordinate changes on the PySpark side (it's not just oooh shiny faster build times, although that's pretty compelling in its self). On Tue, Nov 7, 2017 at 11:20 AM, Holden Karau wrote:

Re: [ML] Migrating transformers from mllib to ml

2017-11-07 Thread Yan Facai
Hi, I have migrated HashingTF from mllib to ml, and wait for review. see: [SPARK-21748][ML] Migrate the implementation of HashingTF from MLlib to ML #18998 https://github.com/apache/spark/pull/18998 On Mon, Nov 6, 2017 at 10:58 PM, Marco Gaido wrote: > Hello, > > I saw that there are several