Thanks to everyone that attended the sync! We had some good discussions.
Here are my notes for anyone that missed it or couldn’t join the live
stream. If anyone wants to add to this, please send additional thoughts or
corrections.
*Attendees:*
- Ryan Blue - Netflix - Using v2 to integrate
Thanks for bringing up the custom metrics API in the list, its something
that needs to be addressed.
A couple more items worth considering,
1. Possibility to unify the batch, micro-batch and continuous sources.
(similar to SPARK-25000)
Right now now there is significant code duplication even
Hi all,
I spent some time thinking about the roadmap, and came up with an initial
list:
SPARK-25390: data source V2 API refactoring
SPARK-24252: add catalog support
SPARK-25531: new write APIs for data source v2
SPARK-25190: better operator pushdown API
Streaming rate control API
Custom metrics
Everyone,
There are now 25 guests invited, which is a lot of people to actively
participate in a sync like this.
For those of you who probably won't actively participate, I've added a live
stream. If you don't plan to talk, please use the live stream instead of
the meet/hangout so that we don't
Responding for invite
On Fri, Oct 26, 2018, 12:34 PM Ryan Blue wrote:
> Looks like the majority opinion is for Wednesday. I've sent out an invite
> to everyone that replied and will add more people as I hear more responses.
>
> Thanks, everyone!
>
> On Fri, Oct 26, 2018 at 3:23 AM Gengliang
Looks like the majority opinion is for Wednesday. I've sent out an invite
to everyone that replied and will add more people as I hear more responses.
Thanks, everyone!
On Fri, Oct 26, 2018 at 3:23 AM Gengliang Wang wrote:
> +1
>
> On Oct 26, 2018, at 8:45 AM, Hyukjin Kwon wrote:
>
> I didn't
+1
> On Oct 26, 2018, at 8:45 AM, Hyukjin Kwon wrote:
>
> I didn't know I live in the same timezone with you Wenchen :D.
> Monday or Wednesday at 5PM PDT sounds good to me too FWIW.
>
> 2018년 10월 26일 (금) 오전 8:29, Ryan Blue 님이 작성:
> Good point. How about Monday or Wednesday at 5PM PDT then?
>
Ditto, I’d also like to join and am in Seattle, generally afternoons work
better for me.
Sent from my iPhone
On Oct 25, 2018, at 5:02 PM, Wenchen Fan
mailto:cloud0...@gmail.com>> wrote:
Big +1 on this!
I live in UTC+8 and I'm available from 8 am, which is 5 pm in the bay area.
Hopefully we
I didn't know I live in the same timezone with you Wenchen :D.
Monday or Wednesday at 5PM PDT sounds good to me too FWIW.
2018년 10월 26일 (금) 오전 8:29, Ryan Blue 님이 작성:
> Good point. How about Monday or Wednesday at 5PM PDT then?
>
> Everyone, please reply to me (no need to spam the list) with
Good point. How about Monday or Wednesday at 5PM PDT then?
Everyone, please reply to me (no need to spam the list) with which option
works for you and I'll send an invite for the one with the most votes.
On Thu, Oct 25, 2018 at 5:14 PM Wenchen Fan wrote:
> Friday at the bay area is Saturday at
Friday at the bay area is Saturday at my side, it will be great if we can
pick a day from Monday to Thursday.
On Fri, Oct 26, 2018 at 8:08 AM Ryan Blue wrote:
> Since not many people have replied with a time window, how about we aim
> for 5PM PDT? That should work for Wenchen and most people
Since not many people have replied with a time window, how about we aim for
5PM PDT? That should work for Wenchen and most people here in the bay area.
If that makes it so some people can't attend, we can do the next one
earlier for people in Europe.
If we go with 5PM PDT, then what day works
Big +1 on this!
I live in UTC+8 and I'm available from 8 am, which is 5 pm in the bay area.
Hopefully we can coordinate a time that fits everyone.
Thanks
Wenchen
On Fri, Oct 26, 2018 at 7:21 AM Dongjoon Hyun
wrote:
> +1. Thank you for volunteering, Ryan!
>
> Bests,
> Dongjoon.
>
>
> On Thu,
+1 !
2018년 10월 26일 (금) 오전 7:21, Dongjoon Hyun 님이 작성:
> +1. Thank you for volunteering, Ryan!
>
> Bests,
> Dongjoon.
>
>
> On Thu, Oct 25, 2018 at 4:19 PM Xiao Li wrote:
>
>> +1
>>
>> Reynold Xin 于2018年10月25日周四 下午4:16写道：
>>
>>> +1
>>>
>>>
>>>
>>> On Thu, Oct 25, 2018 at 4:12 PM Li Jin wrote:
+1. Thank you for volunteering, Ryan!
Bests,
Dongjoon.
On Thu, Oct 25, 2018 at 4:19 PM Xiao Li wrote:
> +1
>
> Reynold Xin 于2018年10月25日周四 下午4:16写道：
>
>> +1
>>
>>
>>
>> On Thu, Oct 25, 2018 at 4:12 PM Li Jin wrote:
>>
>>> Although I am not specifically involved in DSv2, I think having this
+1
Reynold Xin 于2018年10月25日周四 下午4:16写道：
> +1
>
>
>
> On Thu, Oct 25, 2018 at 4:12 PM Li Jin wrote:
>
>> Although I am not specifically involved in DSv2, I think having this kind
>> of meeting is definitely helpful to discuss, move certain effort forward
>> and keep people on the same page.
+1
On Thu, Oct 25, 2018 at 4:12 PM Li Jin wrote:
> Although I am not specifically involved in DSv2, I think having this kind
> of meeting is definitely helpful to discuss, move certain effort forward
> and keep people on the same page. Glad to see this kind of working group
> happening.
>
>
Although I am not specifically involved in DSv2, I think having this kind
of meeting is definitely helpful to discuss, move certain effort forward
and keep people on the same page. Glad to see this kind of working group
happening.
On Thu, Oct 25, 2018 at 5:58 PM John Zhuge wrote:
> Great idea!
Great idea!
On Thu, Oct 25, 2018 at 1:10 PM Ryan Blue wrote:
> Hi everyone,
>
> There's been some great discussion for DataSourceV2 in the last few
> months, but it has been difficult to resolve some of the discussions and I
> don't think that we have a very clear roadmap for getting the work
Yes please!
From: Ryan Blue
Sent: Thursday, October 25, 2018 1:10 PM
To: Spark Dev List
Subject: DataSourceV2 hangouts sync
Hi everyone,
There's been some great discussion for DataSourceV2 in the last few months, but
it has been difficult to resolve some
Hi everyone,
There's been some great discussion for DataSourceV2 in the last few months,
but it has been difficult to resolve some of the discussions and I don't
think that we have a very clear roadmap for getting the work done.
To coordinate better as a community, I'd like to start a regular
21 matches
Mail list logo
