Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Saisai Shao]

+1, looking forward to more design details of this feature.

Thanks
Jerry

On Wed, Nov 8, 2017 at 6:40 AM, Shixiong(Ryan) Zhu 
wrote:

> +1
>
> On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley 
> wrote:
>
>> +1
>>
>> On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust 
>> wrote:
>>
>>> +1
>>>
>>> On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li  wrote:
>>>
 +1

 2017-11-04 11:00 GMT-07:00 Burak Yavuz :

> +1
>
> On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan 
> wrote:
>
>> +1
>>
>> On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu > > wrote:
>>
>>> +1.
>>>
>>> On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <
>>> matei.zaha...@gmail.com> wrote:
>>>
 +1 from me too.

 Matei

 > On Nov 3, 2017, at 4:59 PM, Wenchen Fan 
 wrote:
 >
 > +1.
 >
 > I think this architecture makes a lot of sense to let executors
 talk to source/sink directly, and bring very low latency.
 >
 > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen 
 wrote:
 > +0 simply because I don't feel I know enough to have an opinion.
 I have no reason to doubt the change though, from a skim through the 
 doc.
 >
 >
 > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin 
 wrote:
 > Earlier I sent out a discussion thread for CP in Structured
 Streaming:
 >
 > https://issues.apache.org/jira/browse/SPARK-20928
 >
 > It is meant to be a very small, surgical change to Structured
 Streaming to enable ultra-low latency. This is great timing because we 
 are
 also designing and implementing data source API v2. If designed 
 properly,
 we can have the same data source API working for both streaming and 
 batch.
 >
 >
 > Following the SPIP process, I'm putting this SPIP up for a vote.
 >
 > +1: Let's go ahead and design / implement the SPIP.
 > +0: Don't really care.
 > -1: I do not think this is a good idea for the following reasons.
 >
 >
 >


 
 -
 To unsubscribe e-mail: dev-unsubscr...@spark.apache.org


>>>
>>
>>
>> --
>> Regards,
>> Vaquar Khan
>> +1 -224-436-0783 <(224)%20436-0783>
>> Greater Chicago
>>
>
>

>>>
>>
>>
>> --
>>
>> Joseph Bradley
>>
>> Software Engineer - Machine Learning
>>
>> Databricks, Inc.
>>
>> [image: http://databricks.com] 
>>
>
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Reynold Xin]

The vote has passed with the following +1s:

Reynold Xin*
Debasish Das
Noman Khan
Wenchen Fan*
Matei Zaharia*
Weichen Xu
Vaquar Khan
Burak Yavuz
Xiao Li
Tom Graves*
Michael Armbrust*
Joseph Bradley*
Shixiong Zhu*


And the following +0s:

Sean Owen*


Thanks for the feedback!


On Wed, Nov 1, 2017 at 8:37 AM, Reynold Xin  wrote:

> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to
> enable ultra-low latency. This is great timing because we are also
> designing and implementing data source API v2. If designed properly, we can
> have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Shixiong(Ryan) Zhu]

+1

On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley 
wrote:

> +1
>
> On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust 
> wrote:
>
>> +1
>>
>> On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li  wrote:
>>
>>> +1
>>>
>>> 2017-11-04 11:00 GMT-07:00 Burak Yavuz :
>>>
 +1

 On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan 
 wrote:

> +1
>
> On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu 
> wrote:
>
>> +1.
>>
>> On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <
>> matei.zaha...@gmail.com> wrote:
>>
>>> +1 from me too.
>>>
>>> Matei
>>>
>>> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan 
>>> wrote:
>>> >
>>> > +1.
>>> >
>>> > I think this architecture makes a lot of sense to let executors
>>> talk to source/sink directly, and bring very low latency.
>>> >
>>> > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen 
>>> wrote:
>>> > +0 simply because I don't feel I know enough to have an opinion. I
>>> have no reason to doubt the change though, from a skim through the doc.
>>> >
>>> >
>>> > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin 
>>> wrote:
>>> > Earlier I sent out a discussion thread for CP in Structured
>>> Streaming:
>>> >
>>> > https://issues.apache.org/jira/browse/SPARK-20928
>>> >
>>> > It is meant to be a very small, surgical change to Structured
>>> Streaming to enable ultra-low latency. This is great timing because we 
>>> are
>>> also designing and implementing data source API v2. If designed 
>>> properly,
>>> we can have the same data source API working for both streaming and 
>>> batch.
>>> >
>>> >
>>> > Following the SPIP process, I'm putting this SPIP up for a vote.
>>> >
>>> > +1: Let's go ahead and design / implement the SPIP.
>>> > +0: Don't really care.
>>> > -1: I do not think this is a good idea for the following reasons.
>>> >
>>> >
>>> >
>>>
>>>
>>> 
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>
>
> --
> Regards,
> Vaquar Khan
> +1 -224-436-0783 <(224)%20436-0783>
> Greater Chicago
>


>>>
>>
>
>
> --
>
> Joseph Bradley
>
> Software Engineer - Machine Learning
>
> Databricks, Inc.
>
> [image: http://databricks.com] 
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Joseph Bradley]

+1

On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust 
wrote:

> +1
>
> On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li  wrote:
>
>> +1
>>
>> 2017-11-04 11:00 GMT-07:00 Burak Yavuz :
>>
>>> +1
>>>
>>> On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan 
>>> wrote:
>>>
 +1

 On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu 
 wrote:

> +1.
>
> On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia  > wrote:
>
>> +1 from me too.
>>
>> Matei
>>
>> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan 
>> wrote:
>> >
>> > +1.
>> >
>> > I think this architecture makes a lot of sense to let executors
>> talk to source/sink directly, and bring very low latency.
>> >
>> > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen 
>> wrote:
>> > +0 simply because I don't feel I know enough to have an opinion. I
>> have no reason to doubt the change though, from a skim through the doc.
>> >
>> >
>> > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin 
>> wrote:
>> > Earlier I sent out a discussion thread for CP in Structured
>> Streaming:
>> >
>> > https://issues.apache.org/jira/browse/SPARK-20928
>> >
>> > It is meant to be a very small, surgical change to Structured
>> Streaming to enable ultra-low latency. This is great timing because we 
>> are
>> also designing and implementing data source API v2. If designed properly,
>> we can have the same data source API working for both streaming and 
>> batch.
>> >
>> >
>> > Following the SPIP process, I'm putting this SPIP up for a vote.
>> >
>> > +1: Let's go ahead and design / implement the SPIP.
>> > +0: Don't really care.
>> > -1: I do not think this is a good idea for the following reasons.
>> >
>> >
>> >
>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>


 --
 Regards,
 Vaquar Khan
 +1 -224-436-0783 <(224)%20436-0783>
 Greater Chicago

>>>
>>>
>>
>


-- 

Joseph Bradley

Software Engineer - Machine Learning

Databricks, Inc.

[image: http://databricks.com] 

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Michael Armbrust]

+1

On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li  wrote:

> +1
>
> 2017-11-04 11:00 GMT-07:00 Burak Yavuz :
>
>> +1
>>
>> On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu 
>>> wrote:
>>>
 +1.

 On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia 
 wrote:

> +1 from me too.
>
> Matei
>
> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan  wrote:
> >
> > +1.
> >
> > I think this architecture makes a lot of sense to let executors talk
> to source/sink directly, and bring very low latency.
> >
> > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen 
> wrote:
> > +0 simply because I don't feel I know enough to have an opinion. I
> have no reason to doubt the change though, from a skim through the doc.
> >
> >
> > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin 
> wrote:
> > Earlier I sent out a discussion thread for CP in Structured
> Streaming:
> >
> > https://issues.apache.org/jira/browse/SPARK-20928
> >
> > It is meant to be a very small, surgical change to Structured
> Streaming to enable ultra-low latency. This is great timing because we are
> also designing and implementing data source API v2. If designed properly,
> we can have the same data source API working for both streaming and batch.
> >
> >
> > Following the SPIP process, I'm putting this SPIP up for a vote.
> >
> > +1: Let's go ahead and design / implement the SPIP.
> > +0: Don't really care.
> > -1: I do not think this is a good idea for the following reasons.
> >
> >
> >
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

>>>
>>>
>>> --
>>> Regards,
>>> Vaquar Khan
>>> +1 -224-436-0783 <(224)%20436-0783>
>>> Greater Chicago
>>>
>>
>>
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Reynold Xin]

Thanks Tom. I'd imagine more details belong either in a full design doc, or
a PR description. Might make sense to do an additional design doc, if there
is enough delta from the current sketch doc.


On Mon, Nov 6, 2017 at 7:29 AM, Tom Graves  wrote:

> +1 for the idea and feature, but I think the design is definitely lacking
> detail on the internal changes needed and how the execution pieces work and
> the communication.  Are you planning on posting more of those details or
> were you just planning on discussing in PR?
>
> Tom
>
> On Wednesday, November 1, 2017, 11:29:21 AM CDT, Debasish Das <
> debasish.da...@gmail.com> wrote:
>
>
> +1
>
> Is there any design doc related to API/internal changes ? Will CP be the
> default in structured streaming or it's a mode in conjunction with
> exisiting behavior.
>
> Thanks.
> Deb
>
> On Nov 1, 2017 8:37 AM, "Reynold Xin"  wrote:
>
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/ jira/browse/SPARK-20928
> 
>
> It is meant to be a very small, surgical change to Structured Streaming to
> enable ultra-low latency. This is great timing because we are also
> designing and implementing data source API v2. If designed properly, we can
> have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Tom Graves]

 +1 for the idea and feature, but I think the design is definitely lacking 
detail on the internal changes needed and how the execution pieces work and the 
communication.  Are you planning on posting more of those details or were you 
just planning on discussing in PR?
Tom
On Wednesday, November 1, 2017, 11:29:21 AM CDT, Debasish Das 
 wrote:  
 
 +1
Is there any design doc related to API/internal changes ? Will CP be the 
default in structured streaming or it's a mode in conjunction with exisiting 
behavior.
Thanks.Deb
On Nov 1, 2017 8:37 AM, "Reynold Xin"  wrote:

Earlier I sent out a discussion thread for CP in Structured Streaming:
https://issues.apache.org/ jira/browse/SPARK-20928
It is meant to be a very small, surgical change to Structured Streaming to 
enable ultra-low latency. This is great timing because we are also designing 
and implementing data source API v2. If designed properly, we can have the same 
data source API working for both streaming and batch.

Following the SPIP process, I'm putting this SPIP up for a vote.
+1: Let's go ahead and design / implement the SPIP.+0: Don't really care.-1: I 
do not think this is a good idea for the following reasons.



  

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Burak Yavuz]

+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan  wrote:

> +1
>
> On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu 
> wrote:
>
>> +1.
>>
>> On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia 
>> wrote:
>>
>>> +1 from me too.
>>>
>>> Matei
>>>
>>> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan  wrote:
>>> >
>>> > +1.
>>> >
>>> > I think this architecture makes a lot of sense to let executors talk
>>> to source/sink directly, and bring very low latency.
>>> >
>>> > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen  wrote:
>>> > +0 simply because I don't feel I know enough to have an opinion. I
>>> have no reason to doubt the change though, from a skim through the doc.
>>> >
>>> >
>>> > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin 
>>> wrote:
>>> > Earlier I sent out a discussion thread for CP in Structured Streaming:
>>> >
>>> > https://issues.apache.org/jira/browse/SPARK-20928
>>> >
>>> > It is meant to be a very small, surgical change to Structured
>>> Streaming to enable ultra-low latency. This is great timing because we are
>>> also designing and implementing data source API v2. If designed properly,
>>> we can have the same data source API working for both streaming and batch.
>>> >
>>> >
>>> > Following the SPIP process, I'm putting this SPIP up for a vote.
>>> >
>>> > +1: Let's go ahead and design / implement the SPIP.
>>> > +0: Don't really care.
>>> > -1: I do not think this is a good idea for the following reasons.
>>> >
>>> >
>>> >
>>>
>>>
>>> -
>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>>
>>>
>>
>
>
> --
> Regards,
> Vaquar Khan
> +1 -224-436-0783 <(224)%20436-0783>
> Greater Chicago
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[vaquar khan]

+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu 
wrote:

> +1.
>
> On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia 
> wrote:
>
>> +1 from me too.
>>
>> Matei
>>
>> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan  wrote:
>> >
>> > +1.
>> >
>> > I think this architecture makes a lot of sense to let executors talk to
>> source/sink directly, and bring very low latency.
>> >
>> > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen  wrote:
>> > +0 simply because I don't feel I know enough to have an opinion. I have
>> no reason to doubt the change though, from a skim through the doc.
>> >
>> >
>> > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin  wrote:
>> > Earlier I sent out a discussion thread for CP in Structured Streaming:
>> >
>> > https://issues.apache.org/jira/browse/SPARK-20928
>> >
>> > It is meant to be a very small, surgical change to Structured Streaming
>> to enable ultra-low latency. This is great timing because we are also
>> designing and implementing data source API v2. If designed properly, we can
>> have the same data source API working for both streaming and batch.
>> >
>> >
>> > Following the SPIP process, I'm putting this SPIP up for a vote.
>> >
>> > +1: Let's go ahead and design / implement the SPIP.
>> > +0: Don't really care.
>> > -1: I do not think this is a good idea for the following reasons.
>> >
>> >
>> >
>>
>>
>> -
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>>
>


-- 
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Weichen Xu]

+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia 
wrote:

> +1 from me too.
>
> Matei
>
> > On Nov 3, 2017, at 4:59 PM, Wenchen Fan  wrote:
> >
> > +1.
> >
> > I think this architecture makes a lot of sense to let executors talk to
> source/sink directly, and bring very low latency.
> >
> > On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen  wrote:
> > +0 simply because I don't feel I know enough to have an opinion. I have
> no reason to doubt the change though, from a skim through the doc.
> >
> >
> > On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin  wrote:
> > Earlier I sent out a discussion thread for CP in Structured Streaming:
> >
> > https://issues.apache.org/jira/browse/SPARK-20928
> >
> > It is meant to be a very small, surgical change to Structured Streaming
> to enable ultra-low latency. This is great timing because we are also
> designing and implementing data source API v2. If designed properly, we can
> have the same data source API working for both streaming and batch.
> >
> >
> > Following the SPIP process, I'm putting this SPIP up for a vote.
> >
> > +1: Let's go ahead and design / implement the SPIP.
> > +0: Don't really care.
> > -1: I do not think this is a good idea for the following reasons.
> >
> >
> >
>
>
> -
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Matei Zaharia]

+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan  wrote:
> 
> +1.
> 
> I think this architecture makes a lot of sense to let executors talk to 
> source/sink directly, and bring very low latency.
> 
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen  wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no 
> reason to doubt the change though, from a skim through the doc.
> 
> 
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin  wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
> 
> https://issues.apache.org/jira/browse/SPARK-20928
> 
> It is meant to be a very small, surgical change to Structured Streaming to 
> enable ultra-low latency. This is great timing because we are also designing 
> and implementing data source API v2. If designed properly, we can have the 
> same data source API working for both streaming and batch.
> 
> 
> Following the SPIP process, I'm putting this SPIP up for a vote.
> 
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
> 
> 
> 


-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Wenchen Fan]

+1.

I think this architecture makes a lot of sense to let executors talk to
source/sink directly, and bring very low latency.

On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen  wrote:

> +0 simply because I don't feel I know enough to have an opinion. I have no
> reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin  wrote:
>
>> Earlier I sent out a discussion thread for CP in Structured Streaming:
>>
>> https://issues.apache.org/jira/browse/SPARK-20928
>>
>> It is meant to be a very small, surgical change to Structured Streaming
>> to enable ultra-low latency. This is great timing because we are also
>> designing and implementing data source API v2. If designed properly, we can
>> have the same data source API working for both streaming and batch.
>>
>>
>> Following the SPIP process, I'm putting this SPIP up for a vote.
>>
>> +1: Let's go ahead and design / implement the SPIP.
>> +0: Don't really care.
>> -1: I do not think this is a good idea for the following reasons.
>>
>>
>>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Sean Owen]

+0 simply because I don't feel I know enough to have an opinion. I have no
reason to doubt the change though, from a skim through the doc.

On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin  wrote:

> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to
> enable ultra-low latency. This is great timing because we are also
> designing and implementing data source API v2. If designed properly, we can
> have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Noman Khan]

+1
for ultra-low latency

Thanks & Regards
Noman

Get Outlook for Android<https://aka.ms/ghei36>



From: Reynold Xin
Sent: Wednesday, 1 November, 21:07
Subject: [Vote] SPIP: Continuous Processing Mode for Structured Streaming
To: dev@spark.apache.org


Earlier I sent out a discussion thread for CP in Structured Streaming:

https://issues.apache.org/jira/browse/SPARK-20928

It is meant to be a very small, surgical change to Structured Streaming to 
enable ultra-low latency. This is great timing because we are also designing 
and implementing data source API v2. If designed properly, we can have the same 
data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Reynold Xin]

I just replied.


On Wed, Nov 1, 2017 at 5:50 PM, Cody Koeninger  wrote:

> Was there any answer to my question around the effect of changes to
> the sink api regarding access to underlying offsets?
>
> On Wed, Nov 1, 2017 at 11:32 AM, Reynold Xin  wrote:
> > Most of those should be answered by the attached design sketch in the
> JIRA
> > ticket.
> >
> > On Wed, Nov 1, 2017 at 5:29 PM Debasish Das 
> > wrote:
> >>
> >> +1
> >>
> >> Is there any design doc related to API/internal changes ? Will CP be the
> >> default in structured streaming or it's a mode in conjunction with
> exisiting
> >> behavior.
> >>
> >> Thanks.
> >> Deb
> >>
> >> On Nov 1, 2017 8:37 AM, "Reynold Xin"  wrote:
> >>
> >> Earlier I sent out a discussion thread for CP in Structured Streaming:
> >>
> >> https://issues.apache.org/jira/browse/SPARK-20928
> >>
> >> It is meant to be a very small, surgical change to Structured Streaming
> to
> >> enable ultra-low latency. This is great timing because we are also
> designing
> >> and implementing data source API v2. If designed properly, we can have
> the
> >> same data source API working for both streaming and batch.
> >>
> >>
> >> Following the SPIP process, I'm putting this SPIP up for a vote.
> >>
> >> +1: Let's go ahead and design / implement the SPIP.
> >> +0: Don't really care.
> >> -1: I do not think this is a good idea for the following reasons.
> >>
> >>
> >>
> >
>

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Cody Koeninger]

Was there any answer to my question around the effect of changes to
the sink api regarding access to underlying offsets?

On Wed, Nov 1, 2017 at 11:32 AM, Reynold Xin  wrote:
> Most of those should be answered by the attached design sketch in the JIRA
> ticket.
>
> On Wed, Nov 1, 2017 at 5:29 PM Debasish Das 
> wrote:
>>
>> +1
>>
>> Is there any design doc related to API/internal changes ? Will CP be the
>> default in structured streaming or it's a mode in conjunction with exisiting
>> behavior.
>>
>> Thanks.
>> Deb
>>
>> On Nov 1, 2017 8:37 AM, "Reynold Xin"  wrote:
>>
>> Earlier I sent out a discussion thread for CP in Structured Streaming:
>>
>> https://issues.apache.org/jira/browse/SPARK-20928
>>
>> It is meant to be a very small, surgical change to Structured Streaming to
>> enable ultra-low latency. This is great timing because we are also designing
>> and implementing data source API v2. If designed properly, we can have the
>> same data source API working for both streaming and batch.
>>
>>
>> Following the SPIP process, I'm putting this SPIP up for a vote.
>>
>> +1: Let's go ahead and design / implement the SPIP.
>> +0: Don't really care.
>> -1: I do not think this is a good idea for the following reasons.
>>
>>
>>
>

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Debasish Das]

+1

Is there any design doc related to API/internal changes ? Will CP be the
default in structured streaming or it's a mode in conjunction with
exisiting behavior.

Thanks.
Deb

On Nov 1, 2017 8:37 AM, "Reynold Xin"  wrote:

Earlier I sent out a discussion thread for CP in Structured Streaming:

https://issues.apache.org/jira/browse/SPARK-20928

It is meant to be a very small, surgical change to Structured Streaming to
enable ultra-low latency. This is great timing because we are also
designing and implementing data source API v2. If designed properly, we can
have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.

[Vote] SPIP: Continuous Processing Mode for Structured Streaming

[Reynold Xin]

Earlier I sent out a discussion thread for CP in Structured Streaming:

https://issues.apache.org/jira/browse/SPARK-20928

It is meant to be a very small, surgical change to Structured Streaming to
enable ultra-low latency. This is great timing because we are also
designing and implementing data source API v2. If designed properly, we can
have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.