Cloud Next 2018 : Catch Up

[Gaurav Thakur]

Hi Everyone,

Are any of us going to be at CloudNext 2018? Would you want to point out
any interesting sessions?

I think it would be a great idea to catch up in person if a few of us are
going to be there?

Excuse me, if this is not the right forum for something like this.

Thanks, gaurav

Re: CODEOWNERS for apache/beam repo

[Eugene Kirpichov]

Hi Udi,

I see that the PR was merged - thanks! However it seems to have some
unintended effects.

On my PR https://github.com/apache/beam/pull/5940 , I assigned a reviewer
manually, but the moment I pushed a new commit, it auto-assigned a lot of
other people to it, and I had to remove them. This seems like a big
inconvenience to me, is there a way to disable this?

Thanks.

On Thu, Jul 12, 2018 at 2:53 PM Udi Meiri  wrote:

> :/ That makes it a little less useful.
>
> On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson 
> wrote:
>
>> Hi Udi
>>
>> I asked the GH helpdesk and they confirmed that only people with write
>> access will actually be automatically chosen.
>>
>> It don't expect it should stop us using it, but we should be aware that
>> there are non-committers also willing to review.
>>
>> Thanks,
>> Tim
>>
>> On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin 
>> wrote:
>>
>>> Idea looks good in general.
>>>
>>> Did you look into ways to keep this file up-to-date? For example we can
>>> run monthly job to see if owner was active during this period.
>>>
>>> --Mikhail
>>>
>>> Have feedback ?
>>>
>>>
>>> On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  wrote:
>>>
 Thanks all!
 I'll try to get the file merged today and see how it works out.
 Please surface any issues, such as with auto-assignment, here or in
 JIRA.

 On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot 
 wrote:

> Hi,
>
> I added myself as a reviewer for some modules.
>
> Etienne
>
> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>
> Hi everyone,
>
> I'm proposing to add auto-reviewer-assignment using Github's
> CODEOWNERS mechanism.
> Initial version is here: *https://github.com/apache/beam/pull/5909/files
> *
>
> I need help from the community in determining owners for each
> component.
> Feel free to directly edit the PR (if you have permission) or add a
> comment.
>
>
> Background
> The idea is to:
> 1. Document good review candidates for each component.
> 2. Help choose reviewers using the auto-assignment mechanism. The
> suggestion is in no way binding.
>
>
>
>>

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

:/ That makes it a little less useful.

On Thu, Jul 12, 2018 at 11:14 AM Tim Robertson 
wrote:

> Hi Udi
>
> I asked the GH helpdesk and they confirmed that only people with write
> access will actually be automatically chosen.
>
> It don't expect it should stop us using it, but we should be aware that
> there are non-committers also willing to review.
>
> Thanks,
> Tim
>
> On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin 
> wrote:
>
>> Idea looks good in general.
>>
>> Did you look into ways to keep this file up-to-date? For example we can
>> run monthly job to see if owner was active during this period.
>>
>> --Mikhail
>>
>> Have feedback ?
>>
>>
>> On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  wrote:
>>
>>> Thanks all!
>>> I'll try to get the file merged today and see how it works out.
>>> Please surface any issues, such as with auto-assignment, here or in JIRA.
>>>
>>> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot 
>>> wrote:
>>>
 Hi,

 I added myself as a reviewer for some modules.

 Etienne

 Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :

 Hi everyone,

 I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
 mechanism.
 Initial version is here: *https://github.com/apache/beam/pull/5909/files
 *

 I need help from the community in determining owners for each component.
 Feel free to directly edit the PR (if you have permission) or add a
 comment.


 Background
 The idea is to:
 1. Document good review candidates for each component.
 2. Help choose reviewers using the auto-assignment mechanism. The
 suggestion is in no way binding.



>


smime.p7s
Description: S/MIME Cryptographic Signature

[Call for Volunteers] Apache Beam Summit Europe 2018

[Griselda Cuevas]

*Hi Beam Community: If you’d like to sponsor/help organize the Europe
Apache Beam Summit Europe 2018, please reach out to gris [at] apache [dot]
org with your contact info and what areas would you like to help with.
We’ll be hosting a kick-off meeting with the organizing committee the last
week of July. Cheers, G*

Re: Vendoring / Shading Protobuf and gRPC

[Ismaël Mejía]

Seems reasonable, but why exactly may we need the model (or protobuf
related things) in the future in the SDK ? wasn’t it supposed to be
translated into the Pipeline proto representation via the runners (and
in this case the dep reside in the runner side) ?
On Thu, Jul 12, 2018 at 2:50 AM Lukasz Cwik  wrote:
>
> Got a fix[1] for Andrews issue which turned out to be a release blocker since 
> it broke performing the release. Also fixed several minor things like javadoc 
> that were wrong with the release. Solving it allowed me to do the publishing 
> in parallel and cut the release time from 20+ mins to 8 mins on my machine.
>
> 1: https://github.com/apache/beam/pull/5936
>
> On Wed, Jul 11, 2018 at 3:51 PM Andrew Pilloud  wrote:
>>
>> We discussed this in person, sounds like my issue is known and will be fixed 
>> shortly. I'm running builds with '-Ppublishing' because I need to generate 
>> release artifacts for bundling the Beam SQL shell with the Google Cloud SDK. 
>> Hope to eventually just use the Beam release, but we are currently cutting a 
>> release off master every week to quickly iterate on bug fixes.
>>
>> Andrew
>>
>> On Wed, Jul 11, 2018 at 1:39 PM Lukasz Cwik  wrote:
>>>
>>> Andrew, to my knowledge it seems as though your running into BEAM-4744, is 
>>> there a reason you need to specify -Ppublishing?
>>>
>>> No particular reason to using ByteString within ByteKey and TextSource. 
>>> Note that we currently do shade away protobuf in sdks/java/core so we could 
>>> either migrate to using a vendored version or re-implement the 
>>> functionality to not use ByteString. Note that sdks/java/core can now 
>>> dependend on the model/* classes and perform the Pipeline -> Proto 
>>> translation as this will be needed to support portability efforts so I 
>>> would prefer just migrating to use the vendored versions of the code. Filed 
>>> BEAM-4766.
>>>
>>> As for the IO module, I was referring to the upstream bigtable/bigquery/... 
>>> libraries vended by Google. If they trimmed their API surface to not expose 
>>> gRPC or protobuf, then we wouldn't have to worry about having the shading 
>>> logic within sdks/java/io/google-cloud-platform. I know that this will be 
>>> impossible for some connectors without backwards incompatible changes since 
>>> they exposed protobuf on their API surface. I know that Chamikara was 
>>> looking to shade this away in the sdks/java/io/google-cloud-platform but 
>>> only had limited success in the past.
>>>
>>> On Wed, Jul 11, 2018 at 1:14 PM Ismaël Mejía  wrote:

 This is great news in particular for runners (Spark) where the leaking of 
 some grpc subdependencies caused stability issues and required extra 
 shading. Great !

 About the other modules

 > Note, these are the following modules that still depend on protobuf that 
 > are shaded away and could move to use a vendored variant of protobuf:
 > * sdks/java/core
 > * sdks/java/extensions/sql

 For sdks/java/core the dependency in protobuf seems to be minor, from a 
 quick look it seems that it is only used to import ByteString in two 
 classes: ByteKey and TextSource so hopefully we can rewrite both and get 
 rid of the dependency altogether (making core smaller which is always a 
 win).
 Can we fill a JIRA for this or do I miss other reasons to depend on 
 protobuf in core?

 For sdks/java/extensions/sql I don’t know if I am missing something, but I 
 don’t see any code use of protobuf and I doubt that calcite uses protobuf 
 so maybe it is there just because it was leaking from somewhere else in 
 Beam, we should better check this first.

 > These modules expose protobuf because it is part of the API surface:
 > * sdks/java/extensions/protobuf
 > * sdks/java/io/google-cloud-platform (I believe that gRPC could be 
 > shaded here but preferrably the IO module would do it so we wouldn't 
 > have this maintenance burden.)

 Can you please elaborate on ‘but preferrably the IO module would do it so 
 we wouldn't have this maintenance burden’. I remember there was an issue 
 when running the examples in the spark runner examples because of 
 sdks/java/io/google-cloud-platform leaking netty via gRPC (BEAM-3519) 
 [Note that this is hidden at this moment because of pure luck Spark 2.3.x 
 and Beam are aligned on netty version but this can change in the future so 
 hopefully this can be shaded/controlled].

 On Wed, Jul 11, 2018 at 8:55 PM Andrew Pilloud  wrote:
>
> This is really cool and should cut down our artifact size significantly! 
> Thanks Luke!
>
> I am running into one issue after this: builds with the publishing flag 
> no longer work. (We run './gradlew -Ppublishing shadowJar' to generate 
> release artifacts for the Beam SQL shell.) I get a bunch of errors like 
> this:
>
> 

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Ahmet Altay]

Thank you Etienne! This looks great.

I hope we can get other languages to have benchmarks at this level soon
enough.

Ahmet

On Thu, Jul 12, 2018 at 1:45 PM, Ismaël Mejía  wrote:

> That’s great to see in action, great work Etienne!
>
> Is there any document on how to integrate ‘stuff’ into the dashboards?
> I think this is worth having for people willing to do so like Kai or
> Andrew. Are there any docs on this? or maybe Lukasz Gajowy know ?
> On Thu, Jul 12, 2018 at 9:04 PM Andrew Pilloud 
> wrote:
> >
> > This is great, should make performance work much easier! I'm going to
> get the Beam SQL Nexmark jobs publishing as well. (Opened
> https://issues.apache.org/jira/browse/BEAM-4774 to track.) I might take
> on the Dataflow runner as well if no one else volunteers.
> >
> > I am curious as to why you have two separate graphs for runtime and
> count rather then graphing runtime/count to get the throughput rate for
> each run? Or should that be a third graph? Looks like it would just be a
> small tweak to the query in perfkit.
> >
> > Andrew
> >
> > On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada 
> wrote:
> >>
> >> This is really cool Etienne : ) thanks for working on this.
> >> Our of curiosity, do you know how often the tests run on each runner?
> >>
> >> Best
> >> -P.
> >>
> >> On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
> >>>
> >>> Awesome Etienne, this is really important for the (user) community to
> have that visibility since it is one of the most important aspect of the
> Beam's quality, kudo!
> >>>
> >>>
> >>> Romain Manni-Bucau
> >>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >>>
> >>>
> >>> Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré 
> a écrit :
> 
>  It's really great to have these dashboards and integration in Jenkins
> !
> 
>  Thanks Etienne for driving this !
> 
>  Regards
>  JB
> 
>  On 11/07/2018 15:13, Etienne Chauchot wrote:
>  >
>  > Hi guys,
>  >
>  > I'm glad to announce that the CI of Beam has much improved ! Indeed
>  > Nexmark is now included in the perfkit dashboards.
>  >
>  > At each commit on master, nexmark suites are run and plots are
> created
>  > on the graphs.
>  >
>  > I've created 2 kind of dashboards:
>  > - one for performances (run times of the queries)
>  > - one for the size of the output PCollection (which should be
> constant)
>  >
>  > There are dashboards for these runners:
>  > - spark
>  > - flink
>  > - direct runner
>  >
>  > Each dashboard contains:
>  > - graphs in batch mode
>  > - graphs in streaming mode
>  > - graphs for the 13 queries.
>  >
>  > That gives more than a hundred of graphs (my right finger hurts
> after so
>  > many clics on the mouse :) ). It is detailed that much so that
> anyone
>  > can focus on the area they have interest in.
>  > Feel free to also create new dashboards with more aggregated data.
>  >
>  > Thanks to Lukasz and Cham for reviewing my PRs and showing how to
> use
>  > perfkit dashboards.
>  >
>  > Dashboards are there:
>  >
>  > https://apache-beam-testing.appspot.com/explore?dashboard=
> 5084698770407424
>  > https://apache-beam-testing.appspot.com/explore?dashboard=
> 5699257587728384
>  >  5138380291571712>https://apache-beam-testing.appspot.
> com/explore?dashboard=5138380291571712
>  >
>  > https://apache-beam-testing.appspot.com/explore?dashboard=
> 5099379773931520
>  > https://apache-beam-testing.appspot.com/explore?dashboard=
> 5731568492478464
>  > https://apache-beam-testing.appspot.com/explore?dashboard=
> 5163657986048000
>  >
>  >
>  > Enjoy,
>  >
>  > Etienne
>  >
>  >
> 
>  --
>  Jean-Baptiste Onofré
>  jbono...@apache.org
>  http://blog.nanthrax.net
>  Talend - http://www.talend.com
> >>
> >> --
> >> Got feedback? go/pabloem-feedback
>

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Ismaël Mejía]

That’s great to see in action, great work Etienne!

Is there any document on how to integrate ‘stuff’ into the dashboards?
I think this is worth having for people willing to do so like Kai or
Andrew. Are there any docs on this? or maybe Lukasz Gajowy know ?
On Thu, Jul 12, 2018 at 9:04 PM Andrew Pilloud  wrote:
>
> This is great, should make performance work much easier! I'm going to get the 
> Beam SQL Nexmark jobs publishing as well. (Opened 
> https://issues.apache.org/jira/browse/BEAM-4774 to track.) I might take on 
> the Dataflow runner as well if no one else volunteers.
>
> I am curious as to why you have two separate graphs for runtime and count 
> rather then graphing runtime/count to get the throughput rate for each run? 
> Or should that be a third graph? Looks like it would just be a small tweak to 
> the query in perfkit.
>
> Andrew
>
> On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada  wrote:
>>
>> This is really cool Etienne : ) thanks for working on this.
>> Our of curiosity, do you know how often the tests run on each runner?
>>
>> Best
>> -P.
>>
>> On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau  
>> wrote:
>>>
>>> Awesome Etienne, this is really important for the (user) community to have 
>>> that visibility since it is one of the most important aspect of the Beam's 
>>> quality, kudo!
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>>
>>>
>>> Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a 
>>> écrit :

 It's really great to have these dashboards and integration in Jenkins !

 Thanks Etienne for driving this !

 Regards
 JB

 On 11/07/2018 15:13, Etienne Chauchot wrote:
 >
 > Hi guys,
 >
 > I'm glad to announce that the CI of Beam has much improved ! Indeed
 > Nexmark is now included in the perfkit dashboards.
 >
 > At each commit on master, nexmark suites are run and plots are created
 > on the graphs.
 >
 > I've created 2 kind of dashboards:
 > - one for performances (run times of the queries)
 > - one for the size of the output PCollection (which should be constant)
 >
 > There are dashboards for these runners:
 > - spark
 > - flink
 > - direct runner
 >
 > Each dashboard contains:
 > - graphs in batch mode
 > - graphs in streaming mode
 > - graphs for the 13 queries.
 >
 > That gives more than a hundred of graphs (my right finger hurts after so
 > many clics on the mouse :) ). It is detailed that much so that anyone
 > can focus on the area they have interest in.
 > Feel free to also create new dashboards with more aggregated data.
 >
 > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
 > perfkit dashboards.
 >
 > Dashboards are there:
 >
 > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
 > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
 > https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
 >
 > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
 > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
 > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
 >
 >
 > Enjoy,
 >
 > Etienne
 >
 >

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com
>>
>> --
>> Got feedback? go/pabloem-feedback

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Andrew Pilloud]

This is great, should make performance work much easier! I'm going to get
the Beam SQL Nexmark jobs publishing as well. (Opened
https://issues.apache.org/jira/browse/BEAM-4774 to track.) I might take on
the Dataflow runner as well if no one else volunteers.

I am curious as to why you have two separate graphs for runtime and count
rather then graphing runtime/count to get the throughput rate for each run?
Or should that be a third graph? Looks like it would just be a small tweak
to the query in perfkit.

Andrew

On Thu, Jul 12, 2018 at 11:40 AM Pablo Estrada  wrote:

> This is really cool Etienne : ) thanks for working on this.
> Our of curiosity, do you know how often the tests run on each runner?
>
> Best
> -P.
>
> On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau 
> wrote:
>
>> Awesome Etienne, this is really important for the (user) community to
>> have that visibility since it is one of the most important aspect of the
>> Beam's quality, kudo!
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau  |  Blog
>>  | Old Blog
>>  | Github
>>  | LinkedIn
>>  | Book
>> 
>>
>>
>> Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a
>> écrit :
>>
>>> It's really great to have these dashboards and integration in Jenkins !
>>>
>>> Thanks Etienne for driving this !
>>>
>>> Regards
>>> JB
>>>
>>> On 11/07/2018 15:13, Etienne Chauchot wrote:
>>> >
>>> > Hi guys,
>>> >
>>> > I'm glad to announce that the CI of Beam has much improved ! Indeed
>>> > Nexmark is now included in the perfkit dashboards.
>>> >
>>> > At each commit on master, nexmark suites are run and plots are created
>>> > on the graphs.
>>> >
>>> > I've created 2 kind of dashboards:
>>> > - one for performances (run times of the queries)
>>> > - one for the size of the output PCollection (which should be constant)
>>> >
>>> > There are dashboards for these runners:
>>> > - spark
>>> > - flink
>>> > - direct runner
>>> >
>>> > Each dashboard contains:
>>> > - graphs in batch mode
>>> > - graphs in streaming mode
>>> > - graphs for the 13 queries.
>>> >
>>> > That gives more than a hundred of graphs (my right finger hurts after
>>> so
>>> > many clics on the mouse :) ). It is detailed that much so that anyone
>>> > can focus on the area they have interest in.
>>> > Feel free to also create new dashboards with more aggregated data.
>>> >
>>> > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
>>> > perfkit dashboards.
>>> >
>>> > Dashboards are there:
>>> >
>>> >
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
>>> >
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
>>> > <
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
>>> >
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
>>> >
>>> >
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
>>> >
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
>>> >
>>> https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
>>> >
>>> >
>>> > Enjoy,
>>> >
>>> > Etienne
>>> >
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>> --
> Got feedback? go/pabloem-feedback
> 
>

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Pablo Estrada]

This is really cool Etienne : ) thanks for working on this.
Our of curiosity, do you know how often the tests run on each runner?

Best
-P.

On Thu, Jul 12, 2018 at 2:15 AM Romain Manni-Bucau 
wrote:

> Awesome Etienne, this is really important for the (user) community to have
> that visibility since it is one of the most important aspect of the Beam's
> quality, kudo!
>
>
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> 
>
>
> Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a
> écrit :
>
>> It's really great to have these dashboards and integration in Jenkins !
>>
>> Thanks Etienne for driving this !
>>
>> Regards
>> JB
>>
>> On 11/07/2018 15:13, Etienne Chauchot wrote:
>> >
>> > Hi guys,
>> >
>> > I'm glad to announce that the CI of Beam has much improved ! Indeed
>> > Nexmark is now included in the perfkit dashboards.
>> >
>> > At each commit on master, nexmark suites are run and plots are created
>> > on the graphs.
>> >
>> > I've created 2 kind of dashboards:
>> > - one for performances (run times of the queries)
>> > - one for the size of the output PCollection (which should be constant)
>> >
>> > There are dashboards for these runners:
>> > - spark
>> > - flink
>> > - direct runner
>> >
>> > Each dashboard contains:
>> > - graphs in batch mode
>> > - graphs in streaming mode
>> > - graphs for the 13 queries.
>> >
>> > That gives more than a hundred of graphs (my right finger hurts after so
>> > many clics on the mouse :) ). It is detailed that much so that anyone
>> > can focus on the area they have interest in.
>> > Feel free to also create new dashboards with more aggregated data.
>> >
>> > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
>> > perfkit dashboards.
>> >
>> > Dashboards are there:
>> >
>> >
>> https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
>> >
>> https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
>> > <
>> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
>> >
>> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
>> >
>> >
>> https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
>> >
>> https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
>> >
>> https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
>> >
>> >
>> > Enjoy,
>> >
>> > Etienne
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
> --
Got feedback? go/pabloem-feedback

Re: CODEOWNERS for apache/beam repo

[Tim Robertson]

Hi Udi

I asked the GH helpdesk and they confirmed that only people with write
access will actually be automatically chosen.

It don't expect it should stop us using it, but we should be aware that
there are non-committers also willing to review.

Thanks,
Tim

On Thu, Jul 12, 2018 at 7:24 PM, Mikhail Gryzykhin 
wrote:

> Idea looks good in general.
>
> Did you look into ways to keep this file up-to-date? For example we can
> run monthly job to see if owner was active during this period.
>
> --Mikhail
>
> Have feedback ?
>
>
> On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  wrote:
>
>> Thanks all!
>> I'll try to get the file merged today and see how it works out.
>> Please surface any issues, such as with auto-assignment, here or in JIRA.
>>
>> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot 
>> wrote:
>>
>>> Hi,
>>>
>>> I added myself as a reviewer for some modules.
>>>
>>> Etienne
>>>
>>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>>>
>>> Hi everyone,
>>>
>>> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
>>> mechanism.
>>> Initial version is here: *https://github.com/apache/beam/pull/5909/files
>>> *
>>>
>>> I need help from the community in determining owners for each component.
>>> Feel free to directly edit the PR (if you have permission) or add a
>>> comment.
>>>
>>>
>>> Background
>>> The idea is to:
>>> 1. Document good review candidates for each component.
>>> 2. Help choose reviewers using the auto-assignment mechanism. The
>>> suggestion is in no way binding.
>>>
>>>
>>>

[Events] Save the Date, Apache Beam Summit Europe 2018

[Griselda Cuevas]

*Hi Beam Community, We are happy to announce that the Apache Beam Summit
Europe will be hosted in London on October 1st & 2nd. Please mark your
calendars and keep an eye open for the call for speakers and the
registration announcement.If you have questions or suggestions, please let
us know. Gris on behalf of the Summit Organizers*

Re: CODEOWNERS for apache/beam repo

[Mikhail Gryzykhin]

Idea looks good in general.

Did you look into ways to keep this file up-to-date? For example we can run
monthly job to see if owner was active during this period.

--Mikhail

Have feedback ?


On Thu, Jul 12, 2018 at 9:56 AM Udi Meiri  wrote:

> Thanks all!
> I'll try to get the file merged today and see how it works out.
> Please surface any issues, such as with auto-assignment, here or in JIRA.
>
> On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot 
> wrote:
>
>> Hi,
>>
>> I added myself as a reviewer for some modules.
>>
>> Etienne
>>
>> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>>
>> Hi everyone,
>>
>> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
>> mechanism.
>> Initial version is here: *https://github.com/apache/beam/pull/5909/files
>> *
>>
>> I need help from the community in determining owners for each component.
>> Feel free to directly edit the PR (if you have permission) or add a
>> comment.
>>
>>
>> Background
>> The idea is to:
>> 1. Document good review candidates for each component.
>> 2. Help choose reviewers using the auto-assignment mechanism. The
>> suggestion is in no way binding.
>>
>>
>>

Re: CODEOWNERS for apache/beam repo

[Udi Meiri]

Thanks all!
I'll try to get the file merged today and see how it works out.
Please surface any issues, such as with auto-assignment, here or in JIRA.

On Thu, Jul 12, 2018 at 2:12 AM Etienne Chauchot 
wrote:

> Hi,
>
> I added myself as a reviewer for some modules.
>
> Etienne
>
> Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
>
> Hi everyone,
>
> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS
> mechanism.
> Initial version is here: *https://github.com/apache/beam/pull/5909/files
> *
>
> I need help from the community in determining owners for each component.
> Feel free to directly edit the PR (if you have permission) or add a
> comment.
>
>
> Background
> The idea is to:
> 1. Document good review candidates for each component.
> 2. Help choose reviewers using the auto-assignment mechanism. The
> suggestion is in no way binding.
>
>
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: 2.5.0

[Alexey Romanenko]

Great that it helped. Probably, it was a dependency conflict in your local 
maven cache.

> On 12 Jul 2018, at 14:47, Chaim Turkel  wrote:
> 
> not sure what the problem was, but after deleting beam from my .m2 and
> recompiling, it worked
> thanks
> chaim
> On Thu, Jul 12, 2018 at 3:18 PM Alexey Romanenko
>  wrote:
>> 
>> Hi Chaim,
>> 
>> Let me ask you some questions:
>> - From which version you are trying to upgrade?
>> - Did you properly set your environment (JDK, maven)?
>> - Could you send an example of compile issues that you have?
>> - Which command do you use to build and to run Quickstart example? I checked 
>> it with Direct runner on my side - it works fine.
>> 
>> Alexey
>> 
>>> On 12 Jul 2018, at 10:00, Chaim Turkel  wrote:
>>> 
>>> Hi,
>>> I have been trying to upgrade to 2.5.0 but I am having a lot of
>>> compile issues.
>>> So i used the quick start, and have the same issues:
>>> 
>>> https://beam.apache.org/get-started/quickstart-java/
>>> 
>>> Any ideas?
>>> 
>>> chaim
>>> 
>>> --
>>> 
>>> 
>>> Loans are funded by
>>> FinWise Bank, a Utah-chartered bank located in Sandy,
>>> Utah, member FDIC, Equal
>>> Opportunity Lender. Merchant Cash Advances are
>>> made by Behalf. For more
>>> information on ECOA, click here
>>> . For important information about
>>> opening a new
>>> account, review Patriot Act procedures here
>>> .
>>> Visit Legal
>>>  to
>>> review our comprehensive program terms,
>>> conditions, and disclosures.
>> 
> 
> -- 
> 
> 
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy, 
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are 
> made by Behalf. For more
> information on ECOA, click here 
> . For important information about 
> opening a new
> account, review Patriot Act procedures here 
> .
> Visit Legal 
>  to
> review our comprehensive program terms, 
> conditions, and disclosures. 

Re: 2.5.0

[Chaim Turkel]

not sure what the problem was, but after deleting beam from my .m2 and
recompiling, it worked
thanks
chaim
On Thu, Jul 12, 2018 at 3:18 PM Alexey Romanenko
 wrote:
>
> Hi Chaim,
>
> Let me ask you some questions:
> - From which version you are trying to upgrade?
> - Did you properly set your environment (JDK, maven)?
> - Could you send an example of compile issues that you have?
> - Which command do you use to build and to run Quickstart example? I checked 
> it with Direct runner on my side - it works fine.
>
> Alexey
>
> > On 12 Jul 2018, at 10:00, Chaim Turkel  wrote:
> >
> > Hi,
> >  I have been trying to upgrade to 2.5.0 but I am having a lot of
> > compile issues.
> > So i used the quick start, and have the same issues:
> >
> > https://beam.apache.org/get-started/quickstart-java/
> >
> > Any ideas?
> >
> > chaim
> >
> > --
> >
> >
> > Loans are funded by
> > FinWise Bank, a Utah-chartered bank located in Sandy,
> > Utah, member FDIC, Equal
> > Opportunity Lender. Merchant Cash Advances are
> > made by Behalf. For more
> > information on ECOA, click here
> > . For important information about
> > opening a new
> > account, review Patriot Act procedures here
> > .
> > Visit Legal
> >  to
> > review our comprehensive program terms,
> > conditions, and disclosures.
>

-- 


Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy, 
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are 
made by Behalf. For more
information on ECOA, click here 
. For important information about 
opening a new
account, review Patriot Act procedures here 
.
Visit Legal 
 to
review our comprehensive program terms, 
conditions, and disclosures. 

Re: 2.5.0

[Chaim Turkel]

Hi,
  I am trying to upgrade from version 2.4.0

Though also i ran the command:

mvn archetype:generate \
  -DarchetypeGroupId=org.apache.beam \
  -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
  -DarchetypeVersion=2.5.0 \
  -DgroupId=org.example \
  -DartifactId=word-count-beam \
  -Dversion="0.1" \
  -Dpackage=org.apache.beam.examples \
  -DinteractiveMode=false

and then mvn clean package
it failes (using mvn 3.5.0 java 8)

example of failure:
word-count-beam/src/main/java/org/apache/beam/examples/complete/game/UserScore.java:[25,34]
package org.apache.beam.sdk.coders does not exist
word-count-beam/src/main/java/org/apache/beam/examples/complete/game/utils/WriteWindowedToBigQuery.java:[25,38]
package org.apache.beam.sdk.transforms does not exist
word-count-beam/src/main/java/org/apache/beam/examples/complete/game/utils/WriteToText.java:[29,44]
package org.apache.beam.sdk.io.FileBasedSink does not exist
On Thu, Jul 12, 2018 at 3:18 PM Alexey Romanenko
 wrote:
>
> Hi Chaim,
>
> Let me ask you some questions:
> - From which version you are trying to upgrade?
> - Did you properly set your environment (JDK, maven)?
> - Could you send an example of compile issues that you have?
> - Which command do you use to build and to run Quickstart example? I checked 
> it with Direct runner on my side - it works fine.
>
> Alexey
>
> > On 12 Jul 2018, at 10:00, Chaim Turkel  wrote:
> >
> > Hi,
> >  I have been trying to upgrade to 2.5.0 but I am having a lot of
> > compile issues.
> > So i used the quick start, and have the same issues:
> >
> > https://beam.apache.org/get-started/quickstart-java/
> >
> > Any ideas?
> >
> > chaim
> >
> > --
> >
> >
> > Loans are funded by
> > FinWise Bank, a Utah-chartered bank located in Sandy,
> > Utah, member FDIC, Equal
> > Opportunity Lender. Merchant Cash Advances are
> > made by Behalf. For more
> > information on ECOA, click here
> > . For important information about
> > opening a new
> > account, review Patriot Act procedures here
> > .
> > Visit Legal
> >  to
> > review our comprehensive program terms,
> > conditions, and disclosures.
>

-- 


Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy, 
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are 
made by Behalf. For more
information on ECOA, click here 
. For important information about 
opening a new
account, review Patriot Act procedures here 
.
Visit Legal 
 to
review our comprehensive program terms, 
conditions, and disclosures. 

Re: 2.5.0

[Alexey Romanenko]

Hi Chaim,

Let me ask you some questions:
- From which version you are trying to upgrade?
- Did you properly set your environment (JDK, maven)? 
- Could you send an example of compile issues that you have?
- Which command do you use to build and to run Quickstart example? I checked it 
with Direct runner on my side - it works fine.

Alexey

> On 12 Jul 2018, at 10:00, Chaim Turkel  wrote:
> 
> Hi,
>  I have been trying to upgrade to 2.5.0 but I am having a lot of
> compile issues.
> So i used the quick start, and have the same issues:
> 
> https://beam.apache.org/get-started/quickstart-java/
> 
> Any ideas?
> 
> chaim
> 
> -- 
> 
> 
> Loans are funded by
> FinWise Bank, a Utah-chartered bank located in Sandy, 
> Utah, member FDIC, Equal
> Opportunity Lender. Merchant Cash Advances are 
> made by Behalf. For more
> information on ECOA, click here 
> . For important information about 
> opening a new
> account, review Patriot Act procedures here 
> .
> Visit Legal 
>  to
> review our comprehensive program terms, 
> conditions, and disclosures. 

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Romain Manni-Bucau]

Awesome Etienne, this is really important for the (user) community to have
that visibility since it is one of the most important aspect of the Beam's
quality, kudo!

Romain Manni-Bucau
@rmannibucau  |  Blog
 | Old Blog
 | Github  |
LinkedIn  | Book



Le jeu. 12 juil. 2018 à 10:59, Jean-Baptiste Onofré  a
écrit :

> It's really great to have these dashboards and integration in Jenkins !
>
> Thanks Etienne for driving this !
>
> Regards
> JB
>
> On 11/07/2018 15:13, Etienne Chauchot wrote:
> >
> > Hi guys,
> >
> > I'm glad to announce that the CI of Beam has much improved ! Indeed
> > Nexmark is now included in the perfkit dashboards.
> >
> > At each commit on master, nexmark suites are run and plots are created
> > on the graphs.
> >
> > I've created 2 kind of dashboards:
> > - one for performances (run times of the queries)
> > - one for the size of the output PCollection (which should be constant)
> >
> > There are dashboards for these runners:
> > - spark
> > - flink
> > - direct runner
> >
> > Each dashboard contains:
> > - graphs in batch mode
> > - graphs in streaming mode
> > - graphs for the 13 queries.
> >
> > That gives more than a hundred of graphs (my right finger hurts after so
> > many clics on the mouse :) ). It is detailed that much so that anyone
> > can focus on the area they have interest in.
> > Feel free to also create new dashboards with more aggregated data.
> >
> > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
> > perfkit dashboards.
> >
> > Dashboards are there:
> >
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
> > <
> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
> >
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
> >
> https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
> >
> >
> > Enjoy,
> >
> > Etienne
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Etienne Chauchot]

+1,
Thanks for volunteering Pablo, thanks also to have caught tickets that I forgot 
to close :)
Etienne
Le mercredi 11 juillet 2018 à 12:55 -0700, Alan Myrvold a écrit :
> +1 Thanks for volunteering, Pablo
> 
> On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster  wrote:
> > +1 sounds great
> > 
> > On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:
> > > +1
> > > 
> > > Thanks for volunteering, Pablo!
> > > On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré  
> > > wrote:
> > > > +1
> > > > 
> > > > 
> > > > 
> > > > I planned to send the proposal as well ;)
> > > > 
> > > > 
> > > > 
> > > > Regards
> > > > 
> > > > JB
> > > > 
> > > > 
> > > > 
> > > > On 09/07/2018 23:16, Pablo Estrada wrote:
> > > > 
> > > > > Hello everyone!
> > > > 
> > > > > 
> > > > 
> > > > > As per the previously agreed-upon schedule for Beam releases, the
> > > > 
> > > > > process for the 2.6.0 Beam release should start on July 17th.
> > > > 
> > > > > 
> > > > 
> > > > > I volunteer to perform this release. 
> > > > 
> > > > > 
> > > > 
> > > > > Here is the schedule that I have in mind:
> > > > 
> > > > > 
> > > > 
> > > > > - We start triaging JIRA issues this week.
> > > > 
> > > > > - I will cut a release branch on July 17.
> > > > 
> > > > > - After July 17, any blockers will need to be cherry-picked into the
> > > > 
> > > > > release branch.
> > > > 
> > > > > - As soon as tests look good, and blockers have been addressed, I will
> > > > 
> > > > > perform the other release tasks.
> > > > 
> > > > > 
> > > > 
> > > > > Does that seem reasonable to the community?
> > > > 
> > > > > 
> > > > 
> > > > > Best
> > > > 
> > > > > -P.
> > > > 
> > > > > -- 
> > > > 
> > > > > Got feedback? go/pabloem-feedback
> > > > 
> > > > 
> > > > 
> > > > -- 
> > > > 
> > > > Jean-Baptiste Onofré
> > > > 
> > > > jbono...@apache.org
> > > > 
> > > > http://blog.nanthrax.net
> > > > 
> > > > Talend - http://www.talend.com
> > > > 
> > 
> > 

Re: CODEOWNERS for apache/beam repo

[Etienne Chauchot]

Hi,
I added myself as a reviewer for some modules.
Etienne
Le lundi 09 juillet 2018 à 17:06 -0700, Udi Meiri a écrit :
> Hi everyone,
> I'm proposing to add auto-reviewer-assignment using Github's CODEOWNERS 
> mechanism.
> Initial version is here: https://github.com/apache/beam/pull/5909/files
> I need help from the community in determining owners for each component.
> Feel free to directly edit the PR (if you have permission) or add a comment.
> 
> 
> Background
> The idea is to:
> 1. Document good review candidates for each component.
> 2. Help choose reviewers using the auto-assignment mechanism. The suggestion 
> is in no way binding.
> 
> 
> 

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Łukasz Gajowy]

+1, Thanks Pablo!

śr., 11 lip 2018 o 21:55 Alan Myrvold  napisał(a):

> +1 Thanks for volunteering, Pablo
>
> On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster 
> wrote:
>
>> +1 sounds great
>>
>> On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:
>>
>>> +1
>>>
>>> Thanks for volunteering, Pablo!
>>>
>>> On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré 
>>> wrote:
>>>
 +1

 I planned to send the proposal as well ;)

 Regards
 JB

 On 09/07/2018 23:16, Pablo Estrada wrote:
 > Hello everyone!
 >
 > As per the previously agreed-upon schedule for Beam releases, the
 > process for the 2.6.0 Beam release should start on July 17th.
 >
 > I volunteer to perform this release.
 >
 > Here is the schedule that I have in mind:
 >
 > - We start triaging JIRA issues this week.
 > - I will cut a release branch on July 17.
 > - After July 17, any blockers will need to be cherry-picked into the
 > release branch.
 > - As soon as tests look good, and blockers have been addressed, I will
 > perform the other release tasks.
 >
 > Does that seem reasonable to the community?
 >
 > Best
 > -P.
 > --
 > Got feedback? go/pabloem-feedback
 

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com

>>>
>>
>> --
>> ---
>> Jason Kuster
>> Apache Beam / Google Cloud Dataflow
>>
>> See something? Say something. go/jasonkuster-feedback
>> 
>>
>

Re: Broken seed job

[Łukasz Gajowy]

It didn't. After running the standalone job (even multiple times before I
started this thread) I still couldn't run the seed job from master branch
because it was expecting changes from my branch.

I created an issue for this to document this better:
https://issues.apache.org/jira/browse/BEAM-4770. I left it unassigned -
feel free to take this.

Thanks,
Łukasz




śr., 11 lip 2018 o 20:25 Alan Myrvold  napisał(a):

> The outage likely fixed it.
> I would have expected running the standalone job with the trigger phrase
> should have fixed it. Did it not?
>
> On Wed, Jul 11, 2018 at 8:45 AM Lukasz Cwik  wrote:
>
>> I believe there was an outage a few hours ago.
>>
>> On Wed, Jul 11, 2018 at 8:36 AM Łukasz Gajowy 
>> wrote:
>>
>>> It's totally fine. The problem is now gone even though I have taken no
>>> action to fix it. I suppose Jenkins was restarted?
>>>
>>> BTW: Is it restarted only on demand by those who have access, or it's
>>> done once in a while (periodically)?
>>>
>>> śr., 11 lip 2018 o 17:07 Lukasz Cwik  napisał(a):
>>>
 Ah, sorry for my confusion.

 On Tue, Jul 10, 2018 at 4:59 PM Łukasz Gajowy 
 wrote:

> I didn't edit the "Standalone Seed job", only the "SeedJob". Now every
> time someone tries to run the seed job ("Run seed job") it results in an
> error even despite prior running the standalone job from master branch the
> way you described.
>
>
>
> śr., 11 lip 2018 o 01:25 Lukasz Cwik  napisał(a):
>
>> job_seed_standalone should only be edited when we know that the
>> regular seed job is in a healthy state.
>>
>> Note, that you can always recover back to what is checked in master
>> by:
>> 1) Creating an empty PR
>> 2) Using the standalone seed job trigger phrase: "Run Standalone Seed
>> Job"
>>
>> I kicked one off right now:
>>
>> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob_Standalone/1289/
>>
>> On Tue, Jul 10, 2018 at 4:17 PM Łukasz Gajowy <
>> lukasz.gaj...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> while working on Jenkins' seed job, some changes I introduced didn't
>>> get reverted even after running the seed job again from the
>>> master branch. This is why seed job is now failing. More details here 
>>> [1]
>>> and here [2].
>>>
>>> Since I can operate only with jobs phrase-triggered from GitHub's PR,
>>> I think there's nothing more I can do than I already tried. In my
>>> opinion, fixing this issue requires an aid of a person with some greater
>>> access to Jenkins. Can someone help with that?
>>>
>>> Sorry for the inconvenience - I didn't expect that such situation
>>> can occur. "job_seed_standalone" works fine and it can be used
>>> instead (until the issue is fixed).
>>>
>>> [1] https://github.com/apache/beam/pull/5915
>>> [2]
>>> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob/2190/console
>>>
>>>
>>> Best regards,
>>> Łukasz
>>>
>>

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Jean-Baptiste Onofré]

It's really great to have these dashboards and integration in Jenkins !

Thanks Etienne for driving this !

Regards
JB

On 11/07/2018 15:13, Etienne Chauchot wrote:
> 
> Hi guys,
> 
> I'm glad to announce that the CI of Beam has much improved ! Indeed
> Nexmark is now included in the perfkit dashboards.
> 
> At each commit on master, nexmark suites are run and plots are created
> on the graphs.
> 
> I've created 2 kind of dashboards:
> - one for performances (run times of the queries)
> - one for the size of the output PCollection (which should be constant)
> 
> There are dashboards for these runners:
> - spark
> - flink
> - direct runner
> 
> Each dashboard contains:
> - graphs in batch mode 
> - graphs in streaming mode
> - graphs for the 13 queries.
> 
> That gives more than a hundred of graphs (my right finger hurts after so
> many clics on the mouse :) ). It is detailed that much so that anyone
> can focus on the area they have interest in.
> Feel free to also create new dashboards with more aggregated data.
> 
> Thanks to Lukasz and Cham for reviewing my PRs and showing how to use
> perfkit dashboards.
> 
> Dashboards are there:
> 
> https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
> https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
> https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
> 
> https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
> https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
> https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
> 
> 
> Enjoy, 
> 
> Etienne
> 
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [ANNOUNCEMENT] Nexmark included to the CI

[Etienne Chauchot]

Hi Kai,
Cool for TPC-H, it will be complementary to Nexmark.Regarding Dataflow we can 
run Nexmark on dataflow, note that it was
the original target of Nexmark port by Mark. We have not done it because we 
have no DF environment available. 
Is someone from google willing to run nexmark on dataflow and add the 
postCommit script and the perfkit dashboards?
Thanks
Etienne
Le mercredi 11 juillet 2018 à 20:11 -0700, Kai Jiang a écrit :
> Hi Etienne,
> It's awesome for working on these useful dashboards. I am getting TPC-H 
> benchmark running on Flink and Dataflow
> Runner. I could work on similar dashboards for TPC benchmark after code 
> merged.
> Also, it's great to have a dashboards for Dataflow.
> 
> Best,
> Kaiᐧ
> On Wed, Jul 11, 2018 at 6:35 AM Etienne Chauchot  wrote:
> > First catch of the nexmark-CI:It seems that there was a change in the 
> > direct runner.
> > Query3 (exercise state and timers) - output size should be constant but has 
> > increased today => Was there a change in
> > state and timer related code?- the output size of this query is different 
> > between batch and streaming modes on
> > direct runner.
> > Etienne
> > Le mercredi 11 juillet 2018 à 15:25 +0200, Etienne Chauchot a écrit :
> > > Is someone interested in creating the scripts and dashboards for the 
> > > other runners? They can be created by copying
> > > the existing scripts and dashboards and changing one gradle parameter in 
> > > the scripts and the table name in the
> > > dashboards. 
> > > I have created the 
> > > tickets:https://issues.apache.org/jira/browse/BEAM-4763https://issues.apache.org/jira/browse/BE
> > > AM-4762https://issues.apache.org/jira/browse/BEAM-4761https://issues.apache.org/jira/browse/BEAM-4760
> > > Etienne Le mercredi 11 juillet 2018 à 15:13 +0200, Etienne Chauchot a 
> > > écrit :
> > > > Hi guys, 
> > > > 
> > > > I'm glad to announce that the CI of Beam has much improved !  Indeed 
> > > > Nexmark is now included in the perfkit
> > > > dashboards.
> > > > 
> > > > At each commit on master, nexmark suites are run and plots are created 
> > > > on the graphs.
> > > > 
> > > > I've created 2 kind of dashboards:
> > > > - one for performances (run times of the queries)
> > > > - one for the size of the output PCollection (which  should be constant)
> > > > 
> > > > There are dashboards for these runners:
> > > > - spark
> > > > - flink
> > > > - direct runner
> > > > 
> > > > Each dashboard contains:
> > > > - graphs in batch mode 
> > > > - graphs in streaming mode
> > > > - graphs for the 13 queries.
> > > > 
> > > > That gives more than a hundred of graphs (my right finger hurts after 
> > > > so many clics on the mouse :) ). It is
> > > > detailed that much so that anyone can focus on the area they have 
> > > > interest in.
> > > > Feel free to also create new dashboards with more aggregated data.  
> > > > 
> > > > Thanks to Lukasz and Cham for reviewing my PRs and showing how to use 
> > > > perfkit dashboards.
> > > > 
> > > > Dashboards are there: 
> > > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5084698770407424
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5699257587728384
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5138380291571712
> > > > 
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5099379773931520
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5731568492478464
> > > > https://apache-beam-testing.appspot.com/explore?dashboard=5163657986048000
> > > > 
> > > > 
> > > > Enjoy, 
> > > > 
> > > > Etienne
> > > > 
> > > > 

Re: Beam's recent community development work

[Bertrand Delacretaz]

(note the mix of public and private lists)

On Tue, Jul 10, 2018 at 9:56 PM Karl Fogel  wrote:
> ...Kenneth, are you planning to turn your email into a blog post or
> other easily-pointable-at-on-the-web thing?  I'm referring to your original 
> email.

+1 for a blog post and you're welcome at
https://blogs.apache.org/comdev/ if desired, the dev@community.a.o
team can give you access.

-Bertrand

2.5.0

[Chaim Turkel]

Hi,
  I have been trying to upgrade to 2.5.0 but I am having a lot of
compile issues.
So i used the quick start, and have the same issues:

https://beam.apache.org/get-started/quickstart-java/

Any ideas?

chaim

-- 


Loans are funded by
FinWise Bank, a Utah-chartered bank located in Sandy, 
Utah, member FDIC, Equal
Opportunity Lender. Merchant Cash Advances are 
made by Behalf. For more
information on ECOA, click here 
. For important information about 
opening a new
account, review Patriot Act procedures here 
.
Visit Legal 
 to
review our comprehensive program terms, 
conditions, and disclosures. 

Jenkins build is back to normal : beam_Release_Gradle_NightlySnapshot #98

[Apache Jenkins Server]

See