Re: Community hackathon

2017-04-26 Thread Davor Bonaci 
Let's start!

A few quick notes on how to get started: https://docs.google.
com/document/d/1UKC2R_9FkSdMVTz2nt2sIW18KoLbIu6w0aj9bwSSPiw/edit

There's no particular progress on the Jenkins infrastructure issue. Current
workaround: ask on Slack, and a committer can manually kick-off a build and
give you a link to the Jenkins job. Inspect the job to see the results.

It's getting late on the US West Coast, but I plan to be available for the
next few hours. Then, I'll disappear for a few hours and then show up in
the morning.

Hope to see many of you on Slack, and let's make this a success!

Davor

On Wed, Apr 26, 2017 at 2:02 PM, Davor Bonaci  wrote:

> The outage is still ongoing, unfortunately... I think we should make one
> more (and final) delay.
>
> New (final) start time: 11 PM Pacific, and we start regardless of the
> outage.
>
> On Wed, Apr 26, 2017 at 10:04 AM, Davor Bonaci  wrote:
>
>> *** DELAYED START ***
>>
>> Unfortunately, due to Jenkins infrastructure issue currently ongoing,
>> we'll have to delay the start of the hackthon.
>>
>> New (tentative) start time: 2 PM Pacific (4 hours from now).
>>
>> I apologize for the delay.
>>
>> Davor
>>
>> On Wed, Apr 26, 2017 at 2:03 AM, Tibor Kiss  wrote:
>>
>>> I've created a meetup in Budapest for this event:
>>> https://www.meetup.com/futureofdata-budapest/events/239504356/
>>>
>>> We (folks from the Hortonworks office @ Budapest) will try to prep a demo
>>> for the event and if time allows we'll jump into open issues.
>>>
>>> On Tue, Apr 25, 2017 at 7:54 AM, Davor Bonaci  wrote:
>>>
>>> > Thanks everyone for the enthusiasm!
>>> >
>>> > Let's go with this Wednesday, 4/26, starting at 10 AM Pacific time, and
>>> > running for the following 24 hours. I'll try to seed the
>>> > instructions/starting point, and then let's take it from there.
>>> >
>>> > (Michael, invite sent.)
>>> >
>>> > Davor
>>> >
>>> > On Mon, Apr 24, 2017 at 7:47 PM, Michael Huston 
>>> > wrote:
>>> >
>>> > > Could you please add me to the Slack channel also? My apologizes for
>>> the
>>> > > noise on this mailing list and if there is a better way to request
>>> > access.
>>> > >
>>> > > Cheers,
>>> > > Michael
>>> > >
>>> > > On Mon, Apr 24, 2017 at 6:15 PM, Lukasz Cwik
>>> 
>>> > > wrote:
>>> > >
>>> > > > Dylan, sent you invite to slack channel.
>>> > > >
>>> > > > On Mon, Apr 24, 2017 at 5:18 PM, Dylan Raithel <
>>> dylanrait...@gmail.com
>>> > >
>>> > > > wrote:
>>> > > >
>>> > > > > Can you please add me to the Slack channel?
>>> > > > >
>>> > > > > On Apr 24, 2017 12:51 AM, "Jean-Baptiste Onofré" <
>>> j...@nanthrax.net>
>>> > > > wrote:
>>> > > > >
>>> > > > > > That's a wonderful idea !
>>> > > > > >
>>> > > > > > I think the easiest way to organize this event is using the
>>> Slack
>>> > > > > channels
>>> > > > > > to discuss, help each other, and sync together.
>>> > > > > >
>>> > > > > > Regards
>>> > > > > > JB
>>> > > > > >
>>> > > > > > On 04/24/2017 09:48 AM, Davor Bonaci wrote:
>>> > > > > >
>>> > > > > >> We've been working as a community towards the first stable
>>> release
>>> > > > for a
>>> > > > > >> while now, and I think we made a ton of progress across the
>>> board
>>> > > over
>>> > > > > the
>>> > > > > >> last few weeks.
>>> > > > > >>
>>> > > > > >> We could try to organize a community-wide hackathon to
>>> identify
>>> > and
>>> > > > fix
>>> > > > > >> those last few issues, as well as to get a better sense of the
>>> > > overall
>>> > > > > >> project quality as it stands right now.
>>> > > > > >>
>>> > > > > >> This could be a self-organized event, and coordinated via the
>>> > Slack
>>> > > > > >> channel. For example, we (as a community and participants)
>>> can try
>>> > > out
>>> > > > > the
>>> > > > > >> project in various ways -- quickstart, examples, different
>>> > runners,
>>> > > > > >> different platforms -- immediately fixing issues as we run
>>> into
>>> > > them.
>>> > > > It
>>> > > > > >> could last, say, 24 hours, with people from different time
>>> zones
>>> > > > > >> participating at the time of their choosing.
>>> > > > > >>
>>> > > > > >> Thoughts?
>>> > > > > >>
>>> > > > > >> Davor
>>> > > > > >>
>>> > > > > >>
>>> > > > > > --
>>> > > > > > Jean-Baptiste Onofré
>>> > > > > > jbono...@apache.org
>>> > > > > > http://blog.nanthrax.net
>>> > > > > > Talend - http://www.talend.com
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> Kiss Tibor
>>>
>>> +36 70 275 9863
>>> tibor.k...@gmail.com
>>>
>>
>>
>

Re: Community hackathon

2017-04-26 Thread Davor Bonaci 
The outage is still ongoing, unfortunately... I think we should make one
more (and final) delay.

New (final) start time: 11 PM Pacific, and we start regardless of the
outage.

On Wed, Apr 26, 2017 at 10:04 AM, Davor Bonaci  wrote:

> *** DELAYED START ***
>
> Unfortunately, due to Jenkins infrastructure issue currently ongoing,
> we'll have to delay the start of the hackthon.
>
> New (tentative) start time: 2 PM Pacific (4 hours from now).
>
> I apologize for the delay.
>
> Davor
>
> On Wed, Apr 26, 2017 at 2:03 AM, Tibor Kiss  wrote:
>
>> I've created a meetup in Budapest for this event:
>> https://www.meetup.com/futureofdata-budapest/events/239504356/
>>
>> We (folks from the Hortonworks office @ Budapest) will try to prep a demo
>> for the event and if time allows we'll jump into open issues.
>>
>> On Tue, Apr 25, 2017 at 7:54 AM, Davor Bonaci  wrote:
>>
>> > Thanks everyone for the enthusiasm!
>> >
>> > Let's go with this Wednesday, 4/26, starting at 10 AM Pacific time, and
>> > running for the following 24 hours. I'll try to seed the
>> > instructions/starting point, and then let's take it from there.
>> >
>> > (Michael, invite sent.)
>> >
>> > Davor
>> >
>> > On Mon, Apr 24, 2017 at 7:47 PM, Michael Huston 
>> > wrote:
>> >
>> > > Could you please add me to the Slack channel also? My apologizes for
>> the
>> > > noise on this mailing list and if there is a better way to request
>> > access.
>> > >
>> > > Cheers,
>> > > Michael
>> > >
>> > > On Mon, Apr 24, 2017 at 6:15 PM, Lukasz Cwik > >
>> > > wrote:
>> > >
>> > > > Dylan, sent you invite to slack channel.
>> > > >
>> > > > On Mon, Apr 24, 2017 at 5:18 PM, Dylan Raithel <
>> dylanrait...@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Can you please add me to the Slack channel?
>> > > > >
>> > > > > On Apr 24, 2017 12:51 AM, "Jean-Baptiste Onofré" > >
>> > > > wrote:
>> > > > >
>> > > > > > That's a wonderful idea !
>> > > > > >
>> > > > > > I think the easiest way to organize this event is using the
>> Slack
>> > > > > channels
>> > > > > > to discuss, help each other, and sync together.
>> > > > > >
>> > > > > > Regards
>> > > > > > JB
>> > > > > >
>> > > > > > On 04/24/2017 09:48 AM, Davor Bonaci wrote:
>> > > > > >
>> > > > > >> We've been working as a community towards the first stable
>> release
>> > > > for a
>> > > > > >> while now, and I think we made a ton of progress across the
>> board
>> > > over
>> > > > > the
>> > > > > >> last few weeks.
>> > > > > >>
>> > > > > >> We could try to organize a community-wide hackathon to identify
>> > and
>> > > > fix
>> > > > > >> those last few issues, as well as to get a better sense of the
>> > > overall
>> > > > > >> project quality as it stands right now.
>> > > > > >>
>> > > > > >> This could be a self-organized event, and coordinated via the
>> > Slack
>> > > > > >> channel. For example, we (as a community and participants) can
>> try
>> > > out
>> > > > > the
>> > > > > >> project in various ways -- quickstart, examples, different
>> > runners,
>> > > > > >> different platforms -- immediately fixing issues as we run into
>> > > them.
>> > > > It
>> > > > > >> could last, say, 24 hours, with people from different time
>> zones
>> > > > > >> participating at the time of their choosing.
>> > > > > >>
>> > > > > >> Thoughts?
>> > > > > >>
>> > > > > >> Davor
>> > > > > >>
>> > > > > >>
>> > > > > > --
>> > > > > > Jean-Baptiste Onofré
>> > > > > > jbono...@apache.org
>> > > > > > http://blog.nanthrax.net
>> > > > > > Talend - http://www.talend.com
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Kiss Tibor
>>
>> +36 70 275 9863
>> tibor.k...@gmail.com
>>
>
>

Re: An Update on Jenkins

2017-04-26 Thread Jason Kuster 
Yup, not. Update as of this morning: Infra has contacted Github support
about the throttling issue; they're hoping that support will be able to
point to the issue. They are continuing to see rolling problems as the
result of some certificate upgrades they performed over the weekend.

In terms of the hackathon today, several of the committers with Jenkins
access have graciously agreed to manually trigger builds via the Jenkins UI
if anyone needs a PR validated. Reach out on the dev list or on slack and
someone will help you out. :)

Best,

Jason

On Wed, Apr 26, 2017 at 9:18 AM, Dan Halperin 
wrote:

> > If not, feel free to reply to this thread
>
> ... not. :) :(
>
> On Tue, Apr 25, 2017 at 9:58 PM, Jean-Baptiste Onofré 
> wrote:
>
> > Thanks for the update !
> >
> > Regards
> > JB
> >
> > On Apr 26, 2017, 05:51, at 05:51, Jason Kuster  INVALID>
> > wrote:
> > >Hey folks,
> > >
> > >There have been a couple of different issues over the last couple of
> > >days
> > >related to some necessary updates Infra has been working on. We've
> > >tracked
> > >down the last couple of issues, and the latest one seems to be that
> > >we're
> > >being hit by the rate limiter as a result of everything starting back
> > >up
> > >again. They expect that waiting a couple of hours should solve the
> > >problem,
> > >so hopefully by tomorrow things will be back to normal. If not, feel
> > >free
> > >to reply to this thread, and I'll try to keep things up to date with
> > >status.
> > >
> > >Best,
> > >
> > >Jason
> > >
> > >--
> > >---
> > >Jason Kuster
> > >Apache Beam / Google Cloud Dataflow
> >
>



-- 
---
Jason Kuster
Apache Beam / Google Cloud Dataflow

Re: An Update on Jenkins

2017-04-26 Thread Dan Halperin 
> If not, feel free to reply to this thread

... not. :) :(

On Tue, Apr 25, 2017 at 9:58 PM, Jean-Baptiste Onofré 
wrote:

> Thanks for the update !
>
> Regards
> JB
>
> On Apr 26, 2017, 05:51, at 05:51, Jason Kuster 
> 
> wrote:
> >Hey folks,
> >
> >There have been a couple of different issues over the last couple of
> >days
> >related to some necessary updates Infra has been working on. We've
> >tracked
> >down the last couple of issues, and the latest one seems to be that
> >we're
> >being hit by the rate limiter as a result of everything starting back
> >up
> >again. They expect that waiting a couple of hours should solve the
> >problem,
> >so hopefully by tomorrow things will be back to normal. If not, feel
> >free
> >to reply to this thread, and I'll try to keep things up to date with
> >status.
> >
> >Best,
> >
> >Jason
> >
> >--
> >---
> >Jason Kuster
> >Apache Beam / Google Cloud Dataflow
>

Re: Towards a spec for robust streaming SQL, Part 1

2017-04-26 Thread Tyler Akidau 
No worries, thanks for the heads up. Good luck wrapping all that stuff up.

-Tyler

On Tue, Apr 25, 2017 at 12:07 AM Fabian Hueske  wrote:

> Hi Tyler,
>
> thanks for pushing this effort and including the Flink list.
> I haven't managed to read the doc yet, but just wanted to thank you for the
> write-up and let you know that I'm very interested in this discussion.
>
> We are very close to the feature freeze of Flink 1.3 and I'm quite busy
> getting as many contributions merged before the release is forked off.
> When that happened, I'll have more time to read and comment.
>
> Thanks,
> Fabian
>
>
> 2017-04-22 0:16 GMT+02:00 Tyler Akidau :
>
> > Good point, when you start talking about anything less than a full join,
> > triggers get involved to describe how one actually achieves the desired
> > semantics, and they may end up being tied to just one of the inputs
> (e.g.,
> > you may only care about the watermark for one side of the join). Am
> > expecting us to address these sorts of details more precisely in doc #2.
> >
> > -Tyler
> >
> > On Fri, Apr 21, 2017 at 2:26 PM Kenneth Knowles 
> > wrote:
> >
> > > There's something to be said about having different triggering
> depending
> > on
> > > which side of a join data comes from, perhaps?
> > >
> > > (delightful doc, as usual)
> > >
> > > Kenn
> > >
> > > On Fri, Apr 21, 2017 at 1:33 PM, Tyler Akidau
>  > >
> > > wrote:
> > >
> > > > Thanks for reading, Luke. The simple answer is that CoGBK is
> basically
> > > > flatten + GBK. Flatten is a non-grouping operation that merges the
> > input
> > > > streams into a single output stream. GBK then groups the data within
> > that
> > > > single union stream as you might otherwise expect, yielding a single
> > > table.
> > > > So I think it doesn't really impact things much. Grouping,
> aggregation,
> > > > window merging etc all just act upon the merged stream and generate
> > what
> > > is
> > > > effectively a merged table.
> > > >
> > > > -Tyler
> > > >
> > > > On Fri, Apr 21, 2017 at 12:36 PM Lukasz Cwik
>  > >
> > > > wrote:
> > > >
> > > > > The doc is a good read.
> > > > >
> > > > > I think you do a great job of explaining table -> stream, stream ->
> > > > stream,
> > > > > and stream -> table when there is only one stream.
> > > > > But when there are multiple streams reading/writing to a table, how
> > > does
> > > > > that impact what occurs?
> > > > > For example, with CoGBK you have multiple streams writing to a
> table,
> > > how
> > > > > does that impact window merging?
> > > > >
> > > > > On Thu, Apr 20, 2017 at 5:57 PM, Tyler Akidau
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello Beam, Calcite, and Flink dev lists!
> > > > > >
> > > > > > Apologies for the big cross post, but I thought this might be
> > > something
> > > > > all
> > > > > > three communities would find relevant.
> > > > > >
> > > > > > Beam is finally making progress on a SQL DSL utilizing Calcite,
> > > thanks
> > > > to
> > > > > > Mingmin Xu. As you can imagine, we need to come to some
> conclusion
> > > > about
> > > > > > how to elegantly support the full suite of streaming
> functionality
> > in
> > > > the
> > > > > > Beam model in via Calcite SQL. You folks in the Flink community
> > have
> > > > been
> > > > > > pushing on this (e.g., adding windowing constructs, amongst
> others,
> > > > thank
> > > > > > you! :-), but from my understanding we still don't have a full
> spec
> > > for
> > > > > how
> > > > > > to support robust streaming in SQL (including but not limited to,
> > > > e.g., a
> > > > > > triggers analogue such as EMIT).
> > > > > >
> > > > > > I've been spending a lot of time thinking about this and have
> some
> > > > > opinions
> > > > > > about how I think it should look that I've already written down,
> > so I
> > > > > > volunteered to try to drive forward agreement on a general
> > streaming
> > > > SQL
> > > > > > spec between our three communities (well, technically I
> volunteered
> > > to
> > > > do
> > > > > > that w/ Beam and Calcite, but I figured you Flink folks might
> want
> > to
> > > > > join
> > > > > > in since you're going that direction already anyway and will have
> > > > useful
> > > > > > insights :-).
> > > > > >
> > > > > > My plan was to do this by sharing two docs:
> > > > > >
> > > > > >1. The Beam Model : Streams & Tables - This one is for
> context,
> > > and
> > > > > >really only mentions SQL in passing. But it describes the
> > > > relationship
> > > > > >between the Beam Model and the "streams & tables" way of
> > thinking,
> > > > > which
> > > > > >turns out to be useful in understanding what robust streaming
> in
> > > SQL
> > > > > > might
> > > > > >look like. Many of you probably already know some or all of
> > what's
> > > > in
> > > > > > here,
> > > > > >but I felt it

Re: Community hackathon

2017-04-26 Thread Tibor Kiss 
I've created a meetup in Budapest for this event:
https://www.meetup.com/futureofdata-budapest/events/239504356/

We (folks from the Hortonworks office @ Budapest) will try to prep a demo
for the event and if time allows we'll jump into open issues.

On Tue, Apr 25, 2017 at 7:54 AM, Davor Bonaci  wrote:

> Thanks everyone for the enthusiasm!
>
> Let's go with this Wednesday, 4/26, starting at 10 AM Pacific time, and
> running for the following 24 hours. I'll try to seed the
> instructions/starting point, and then let's take it from there.
>
> (Michael, invite sent.)
>
> Davor
>
> On Mon, Apr 24, 2017 at 7:47 PM, Michael Huston 
> wrote:
>
> > Could you please add me to the Slack channel also? My apologizes for the
> > noise on this mailing list and if there is a better way to request
> access.
> >
> > Cheers,
> > Michael
> >
> > On Mon, Apr 24, 2017 at 6:15 PM, Lukasz Cwik 
> > wrote:
> >
> > > Dylan, sent you invite to slack channel.
> > >
> > > On Mon, Apr 24, 2017 at 5:18 PM, Dylan Raithel  >
> > > wrote:
> > >
> > > > Can you please add me to the Slack channel?
> > > >
> > > > On Apr 24, 2017 12:51 AM, "Jean-Baptiste Onofré" 
> > > wrote:
> > > >
> > > > > That's a wonderful idea !
> > > > >
> > > > > I think the easiest way to organize this event is using the Slack
> > > > channels
> > > > > to discuss, help each other, and sync together.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On 04/24/2017 09:48 AM, Davor Bonaci wrote:
> > > > >
> > > > >> We've been working as a community towards the first stable release
> > > for a
> > > > >> while now, and I think we made a ton of progress across the board
> > over
> > > > the
> > > > >> last few weeks.
> > > > >>
> > > > >> We could try to organize a community-wide hackathon to identify
> and
> > > fix
> > > > >> those last few issues, as well as to get a better sense of the
> > overall
> > > > >> project quality as it stands right now.
> > > > >>
> > > > >> This could be a self-organized event, and coordinated via the
> Slack
> > > > >> channel. For example, we (as a community and participants) can try
> > out
> > > > the
> > > > >> project in various ways -- quickstart, examples, different
> runners,
> > > > >> different platforms -- immediately fixing issues as we run into
> > them.
> > > It
> > > > >> could last, say, 24 hours, with people from different time zones
> > > > >> participating at the time of their choosing.
> > > > >>
> > > > >> Thoughts?
> > > > >>
> > > > >> Davor
> > > > >>
> > > > >>
> > > > > --
> > > > > Jean-Baptiste Onofré
> > > > > jbono...@apache.org
> > > > > http://blog.nanthrax.net
> > > > > Talend - http://www.talend.com
> > > > >
> > > >
> > >
> >
>



-- 
Kiss Tibor

+36 70 275 9863
tibor.k...@gmail.com