Re: Greetings from Tyson

[Alan Myrvold]

Welcome, Tyson!

On Wed, Apr 29, 2020 at 3:15 PM Rui Wang  wrote:

> Welcome!
>
> -Rui
>
> On Wed, Apr 29, 2020, 3:13 PM Brian Hulette  wrote:
>
>> Welcome Tyson!
>>
>> On Wed, Apr 29, 2020 at 2:54 PM Ahmet Altay  wrote:
>>
>>> Welcome!
>>>
>>> On Tue, Apr 28, 2020 at 3:06 PM Hannah Jiang 
>>> wrote:
>>>
 Welcome to the community!


 On Tue, Apr 28, 2020 at 2:45 PM Tyson Hamilton 
 wrote:

> Hello Beam Community,
>
> This is just a simple 'Hello' to introduce myself. I'm a Software
> Engineer at Google and have worked with data processing languages and
> runtime systems on and off during my career. I now have the pleasure of
> dedicating more time towards working with you lovely folks on Beam and I'm
> really excited!
>
> I hope you're all doing well and staying safe in these difficult times.
>
> -Tyson
>
>
>
>

Re: No space left on device - beam-jenkins 1 and 7

[Alan Myrvold]

Did a one time cleanup of tmp files owned by jenkins older than 3 days.
Agree that we need a longer term solution.

Passing recent tests on all executors except jenkins-12, which has not
scheduled recent builds for the past 13 days. Not scheduling:
https://builds.apache.org/computer/apache-beam-jenkins-12/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-12/builds=D>
Recent passing builds:
https://builds.apache.org/computer/apache-beam-jenkins-1/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-1/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-2/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-2/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-3/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-3/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-4/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-4/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-5/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-5/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-6/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-6/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-7/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-7/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-8/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-8/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-9/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-9/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-10/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-10/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-11/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-11/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-13/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-13/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-14/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-14/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-15/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-15/builds=D>
https://builds.apache.org/computer/apache-beam-jenkins-16/builds
<https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-16/builds=D>

On Fri, Mar 6, 2020 at 11:54 AM Ahmet Altay  wrote:

> +Alan Myrvold  is doing a one time cleanup. I agree
> that we need to have a solution to automate this task or address the root
> cause of the buildup.
>
> On Thu, Mar 5, 2020 at 2:47 AM Michał Walenia 
> wrote:
>
>> Hi there,
>> it seems we have a problem with Jenkins workers again. Nodes 1 and 7 both
>> fail jobs with "No space left on device".
>> Who is the best person to contact in these cases (someone with access
>> permissions to the workers).
>>
>> I also noticed that such errors are becoming more and more frequent
>> recently and I'd like to discuss how can this be remedied. Can a cleanup
>> task be automated on Jenkins somehow?
>>
>> Regards
>> Michal
>>
>> --
>>
>> Michał Walenia
>> Polidea <https://www.polidea.com/> | Software Engineer
>>
>> M: +48 791 432 002 <+48791432002>
>> E: michal.wale...@polidea.com
>>
>> Unique Tech
>> Check out our projects! <https://www.polidea.com/our-work>
>>
>

Re: Jenkins problems: javaPreCommitPortabilityApiJava11 and No Space left

[Alan Myrvold]

The disk on jenkins5 was 100% full. I deleted some /tmp files owned by
jenkins > 3d old, bringing it down to 90%. There a are still quite a few
large files in the jenkins workspaces. I can look into ways to keep this
cleaned up.

On Wed, Feb 26, 2020 at 5:38 PM Kyle Weaver  wrote:

> It looks like apache-beam-jenkins-5 is back to normal now:
> https://builds.apache.org/computer/apache-beam-jenkins-5/builds
>
> > Java PreCommit also seems to be failing due to a couple of errors in
> 'BigQureyIO" and "SpannerIO".
>
> Judging by the log output, the job failed because of a timeout, possibly
> (probably?) related to Rabbit mq test:
>
> *10:52:08* >* Task :sdks:java:testing:test-utils:buildDependents**12:11:39* 
> Build timed out (after 120 minutes). Marking the build as aborted.*12:11:39* 
> Build was aborted*12:11:39* Recording test results*12:11:41* >* Task 
> :sdks:java:io:rabbitmq:test* FAILED*12:11:41* *12:11:41* FAILURE: Build 
> failed with an exception.
>
>
>
> On Wed, Feb 26, 2020 at 5:22 PM Pulasthi Supun Wickramasinghe <
> pulasthi...@gmail.com> wrote:
>
>> I got the same build issues for my pull request as well. Java PreCommit
>> also seems to be failing due to a couple of errors in 'BigQureyIO" and
>> "SpannerIO".
>>
>> [1]
>> https://builds.apache.org/job/beam_PreCommit_Java_Commit/10160/java/fixed/
>>
>> Best Regards,
>> Pulasthi
>>
>> On Wed, Feb 26, 2020 at 1:27 PM Alex Van Boxel  wrote:
>>
>>> I see 2 problems with Jenkins popping up. The root beam project is
>>> missing:
>>>
>>>
>>> JavaPortabilityApiJava11 ("Run JavaPortabilityApiJava11 PreCommit")  is
>>> running, but:
>>>
>>> Task 'javaPreCommitPortabilityApiJava11' not found in root project 'beam'
>>>
>>> And we have no space left on device:
>>>
>>> Example:
>>>
>>> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Commit/9287/console
>>>
>>>
>>>
>>>  _/
>>> _/ Alex Van Boxel
>>>
>>
>>
>> --
>> Pulasthi S. Wickramasinghe
>> PhD Candidate  | Research Assistant
>> School of Informatics and Computing | Digital Science Center
>> Indiana University, Bloomington
>> cell: 224-386-9035 <(224)%20386-9035>
>>
>

Re: Github flaky?

[Alan Myrvold]

There is an active incident, from https://www.githubstatus.com/

On Tue, Feb 25, 2020 at 10:10 AM Luke Cwik  wrote:

> I have been getting errors from the Github UI when attempting to post a
> comment / merge a commit / 500 errors when reloading a page. Anyone else
> seeing this?
>

Re: [ANNOUNCE] New committer: Hannah Jiang

[Alan Myrvold]

Congrats, Hannah

On Tue, Jan 28, 2020 at 4:46 PM Connell O'Callaghan 
wrote:

> Thank you for sharing Luke!!!
>
> Well done and congratulations Hannah!!
>
> On Tue, Jan 28, 2020 at 4:45 PM Heejong Lee  wrote:
>
>> Congratulations! :)
>>
>> On Tue, Jan 28, 2020 at 4:43 PM Yichi Zhang  wrote:
>>
>>> Congrats Hannah!
>>>
>>> On Tue, Jan 28, 2020 at 3:57 PM Yifan Zou  wrote:
>>>
 Congratulations Hannah!!

 On Tue, Jan 28, 2020 at 3:55 PM Boyuan Zhang 
 wrote:

> Thanks for all your contributions! Congratulations~
>
> On Tue, Jan 28, 2020 at 3:44 PM Pablo Estrada 
> wrote:
>
>> yoooho : D
>>
>> On Tue, Jan 28, 2020 at 3:21 PM Luke Cwik  wrote:
>>
>>> Hi everyone,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Hannah Jiang
>>>
>>> Hannah has contributed to Beam in many ways, including work on
>>> building and releasing the Apache Beam SDK containers.
>>>
>>> In consideration of their contributions, the Beam PMC trusts them
>>> with the responsibilities of a Beam committer[1].
>>>
>>> Thanks for your contributions Hannah!
>>>
>>> Luke, on behalf of the Apache Beam PMC.
>>>
>>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: Jenkins jobs not running for my PR 10438

[Alan Myrvold]

Oh, the PR 9903 run is quite old; I don't see a recent one yet.

On Thu, Jan 2, 2020 at 2:48 PM Alan Myrvold  wrote:

> For PR 10427, I see
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1593/
> For PR 9903, I see
> https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Flink_PR/22/
>
> Maybe the PR status is not being updated when the jobs run?
>
>
> On Thu, Jan 2, 2020 at 2:37 PM Kai Jiang  wrote:
>
>> same for https://github.com/apache/beam/pull/9903 as well
>>
>> On Thu, Jan 2, 2020 at 1:40 PM Chamikara Jayalath 
>> wrote:
>>
>>> Seems like Jenkins tests are not being triggered for this PR as well:
>>> https://github.com/apache/beam/pull/10427
>>>
>>> On Fri, Dec 20, 2019 at 2:16 PM Tomo Suzuki  wrote:
>>>
>>>> Jenkins started working. Thank you for whoever fixed it.
>>>>
>>>> On Fri, Dec 20, 2019 at 1:42 PM Boyuan Zhang 
>>>> wrote:
>>>> >
>>>> > Same here. Even the phrase trigger doesn't work.
>>>> >
>>>> > On Fri, Dec 20, 2019 at 10:16 AM Luke Cwik  wrote:
>>>> >>
>>>> >> I'm also affected by this.
>>>> >>
>>>> >> On Fri, Dec 20, 2019 at 10:13 AM Tomo Suzuki 
>>>> wrote:
>>>> >>>
>>>> >>> Hi Beam developers,
>>>> >>>
>>>> >>> Does anybody know why my PR does not trigger Jenkins jobs today?
>>>> >>> https://github.com/apache/beam/pull/10438
>>>> >>>
>>>> >>> --
>>>> >>> Regards,
>>>> >>> Tomo
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>

Re: Jenkins jobs not running for my PR 10438

[Alan Myrvold]

For PR 10427, I see
https://builds.apache.org/job/beam_PreCommit_Java_Phrase/1593/
For PR 9903, I see
https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Flink_PR/22/

Maybe the PR status is not being updated when the jobs run?


On Thu, Jan 2, 2020 at 2:37 PM Kai Jiang  wrote:

> same for https://github.com/apache/beam/pull/9903 as well
>
> On Thu, Jan 2, 2020 at 1:40 PM Chamikara Jayalath 
> wrote:
>
>> Seems like Jenkins tests are not being triggered for this PR as well:
>> https://github.com/apache/beam/pull/10427
>>
>> On Fri, Dec 20, 2019 at 2:16 PM Tomo Suzuki  wrote:
>>
>>> Jenkins started working. Thank you for whoever fixed it.
>>>
>>> On Fri, Dec 20, 2019 at 1:42 PM Boyuan Zhang  wrote:
>>> >
>>> > Same here. Even the phrase trigger doesn't work.
>>> >
>>> > On Fri, Dec 20, 2019 at 10:16 AM Luke Cwik  wrote:
>>> >>
>>> >> I'm also affected by this.
>>> >>
>>> >> On Fri, Dec 20, 2019 at 10:13 AM Tomo Suzuki 
>>> wrote:
>>> >>>
>>> >>> Hi Beam developers,
>>> >>>
>>> >>> Does anybody know why my PR does not trigger Jenkins jobs today?
>>> >>> https://github.com/apache/beam/pull/10438
>>> >>>
>>> >>> --
>>> >>> Regards,
>>> >>> Tomo
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>

Re: Beam Testing Tools FAQ

[Alan Myrvold]

Nice, thanks!

On Tue, Nov 26, 2019 at 8:04 AM Robert Bradshaw  wrote:

> Thanks!
>
> On Tue, Nov 26, 2019 at 7:43 AM Łukasz Gajowy  wrote:
> >
> > Hi all,
> >
> > our documentation (either confluence or the website docs) describes how
> to create various integration and performance tests - there already are
> core operations tests, nexmark and IO test documentation pages. However, we
> are lacking some general docs to describe what tools do we have and what is
> the purpose of them.
> >
> > Therefore, I took the liberty of creating the Beam Testing Tools FAQ on
> our confluence:
> > https://cwiki.apache.org/confluence/display/BEAM/Beam+Testing+Tools+FAQ
> >
> > Hopefully, this is helpful and sheds some more light on that important
> part of our infrastructure. If you feel that something is missing there,
> feel free to let me know or add it yourself. :)
> >
> > Thanks,
> > Łukasz
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Alan Myrvold]

Congratulations, Brian!

On Fri, Nov 15, 2019 at 11:11 AM Yifan Zou  wrote:

> Well deserved. Congratulations!
>
> On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov 
> wrote:
>
>> Congratulations, Brian!
>>
>> On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:
>>
>>> Congrats Brian!
>>>
>>>
>>> On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang  wrote:
>>>
 Congrats Brian!

 On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:

> Congrats, Brian!
>
> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
> danolive...@google.com> wrote:
>
>> Congratulations Brian! It's well deserved.
>>
>> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
>> aromanenko@gmail.com> wrote:
>>
>>> Congratulations, Brian!
>>>
>>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>>
>>> Congrats!
>>>
>>>
>>> -Rui
>>>
>>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>>
 Congratulations!


 On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done Brian!!!
>
> Kenn thank you for sharing
>
> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
> wrote:
>
>> Congrats Brian!
>>
>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
>> wrote:
>>
>>> Congratulations Brian!
>>> Happy to see this happening and eager to see more of your work!
>>>
>>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>>> wrote:
>>> >
>>> > Congrats Brian!
>>> >
>>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
>>> wrote:
>>> >>
>>> >> Congrats Brian!
>>> >>
>>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>>> >>
>>> >> Great news!
>>> >>
>>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>>> wrote:
>>> >>>
>>> >>> Congratulations!
>>> >>>
>>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>> 
>>>  Congratulations, Brian!
>>> 
>>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>>> sunjincheng...@gmail.com> wrote:
>>> >
>>> > Congratulation Brian!
>>> >
>>> > Best,
>>> > Jincheng
>>> >
>>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>>> >>
>>> >> Thanks for your contributions and congrats Brian!
>>> >>
>>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>>> k...@apache.org> wrote:
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> Please join me and the rest of the Beam PMC in welcoming
>>> a new committer: Brian Hulette
>>> >>>
>>> >>> Brian introduced himself to dev@ earlier this year and
>>> has been contributing since then. His contributions to Beam include
>>> explorations of integration with Arrow, standardizing coders, 
>>> portability
>>> for schemas, and presentations at Beam events.
>>> >>>
>>> >>> In consideration of Brian's contributions, the Beam PMC
>>> trusts him with the responsibilities of a Beam committer [1].
>>> >>>
>>> >>> Thank you, Brian, for your contributions and looking
>>> forward to many more!
>>> >>>
>>> >>> Kenn, on behalf of the Apache Beam PMC
>>> >>>
>>> >>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> This email may be confidential and privileged. If you
>>> received this communication by mistake, please don't forward it to 
>>> anyone
>>> else, please erase all copies and attachments, and please let me 
>>> know that
>>> it has gone to the wrong person.
>>> >>
>>> >> The above terms reflect a potential business arrangement, are
>>> provided solely as a basis for further discussion, and are not 
>>> intended to
>>> be and do not constitute a legally binding obligation. No legally 
>>> binding
>>> obligations will be created, implied, or inferred until an 
>>> agreement in
>>> final form is executed in writing by all parties involved.
>>>
>>
>>>

 --
 
 Ruoyun  Huang

Re: [ANNOUNCE] New committer: Alan Myrvold

[Alan Myrvold]

Thanks!! Looking forward to making more impact to Apache Beam

On Mon, Sep 30, 2019 at 10:56 AM Mikhail Gryzykhin 
wrote:

> Congratulations!
>
> On Mon, Sep 30, 2019 at 9:47 AM David Cavazos  wrote:
>
>> Congratulations Alan!
>>
>> On Mon, Sep 30, 2019 at 7:57 AM Connell O'Callaghan 
>> wrote:
>>
>>> Congratulations Alan - well done!!! Ahmet thank you for sharing this
>>> great news!!!
>>>
>>> On Mon, Sep 30, 2019 at 7:34 AM Łukasz Gajowy 
>>> wrote:
>>>
>>>> Congratulations :)
>>>>
>>>> pon., 30 wrz 2019 o 15:41 Reza Rokni  napisał(a):
>>>>
>>>>> Woohoo Congratulations!
>>>>>
>>>>> On Mon, 30 Sep 2019 at 21:06, Thomas Weise  wrote:
>>>>>
>>>>>> Congratulations, Alan!
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 30, 2019 at 4:47 AM Ismaël Mejía 
>>>>>> wrote:
>>>>>>
>>>>>>> Congrats Alan!
>>>>>>>
>>>>>>> On Mon, Sep 30, 2019, 11:20 AM Tanay Tummalapalli <
>>>>>>> ttanay...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Congratulations, Alan!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 30, 2019 at 1:03 PM Gleb Kanterov 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Congratulations!
>>>>>>>>>
>>>>>>>>> On Sat, Sep 28, 2019 at 12:07 AM Valentyn Tymofieiev <
>>>>>>>>> valen...@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Congratulations, Alan. Well deserved.
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 27, 2019 at 2:09 PM Chamikara Jayalath <
>>>>>>>>>> chamik...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Congrats Alan!!
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 27, 2019 at 1:49 PM Jan Lukavský 
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Congrats Alan!
>>>>>>>>>>>> On 9/27/19 10:22 PM, Mark Liu wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Congratulations Alan!!!
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 27, 2019 at 12:55 PM Ning Kang 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Congrats Alan!
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 27, 2019 at 12:02 PM Ankur Goenka <
>>>>>>>>>>>>> goe...@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Congratulations Alan!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 27, 2019 at 11:17 AM Yichi Zhang <
>>>>>>>>>>>>>> zyi...@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Congrats, Alan!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 27, 2019 at 10:26 AM Robin Qiu <
>>>>>>>>>>>>>>> robi...@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Congrats, Alan!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Sep 27, 2019 at 10:15 AM Hannah Jiang <
>>>>>>>>>>>>>>>> hannahji...@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Congrats Alan!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Sep 27, 2019 at 9:57 AM Ruoyun Huang <
>>>>>>>>>>>>>>>>> ruo...@goo

Re: [PROPOSAL] Preparing for Beam 2.16.0 release

[Alan Myrvold]

+1 Thanks for keeping to the schedule, Mark.

On Thu, Aug 29, 2019 at 6:21 PM jincheng sun 
wrote:

> Hi Mark,
>
> +1 and thank you for keeping the cadence!
>
> BTW I have mark the Fix Version for some of issues to 2.17, which can not
> be merged into 2.16.
>
> Best,
> Jincheng
>
> Mark Liu  于2019年8月29日周四 上午6:14写道：
>
>> Hi all,
>>
>> Beam 2.16 release branch cut is scheduled on Sep 11 according to the
>> release calendar [1]. I would like to volunteer myself to do this
>> release. The plan is to cut the branch on that date, and cherrypick
>> release-blocking fixes afterwards if any.
>>
>> If you have release blocking issues for 2.16 please mark their "Fix
>> Version" as 2.16.0 [2]. This tag is already created in JIRA in case you
>> would like to move any non-blocking issues to that version.
>>
>> Any thoughts, comments, objections?
>>
>> Regards.
>> Mark Liu
>>
>> [1]
>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
>> [2]
>> https://issues.apache.org/jira/browse/BEAM-8105?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.16.0
>>
>

Re: [ANNOUNCE] Beam 2.14.0 Released!

[Alan Myrvold]

Thanks for the release work, Anton!

On Fri, Aug 2, 2019 at 9:08 AM Hannah Jiang  wrote:

> Thanks Anton for all the work to release it.
>
> On Fri, Aug 2, 2019 at 7:12 AM Connell O'Callaghan 
> wrote:
>
>> Well done Anton and all involved!!!
>>
>> On Fri, Aug 2, 2019 at 06:56 Robert Bradshaw  wrote:
>>
>>> Lots of improvements all around. Thank you for pushing this through,
>>> Anton!
>>>
>>> On Fri, Aug 2, 2019 at 1:37 AM Chad Dombrova  wrote:
>>> >
>>> > Nice work all round!  I love the release blog format with the
>>> highlights and links to issues.
>>> >
>>> > -chad
>>> >
>>> >
>>> > On Thu, Aug 1, 2019 at 4:23 PM Anton Kedin  wrote:
>>> >>
>>> >> The Apache Beam team is pleased to announce the release of version
>>> 2.14.0.
>>> >>
>>> >> Apache Beam is an open source unified programming model to define and
>>> >> execute data processing pipelines, including ETL, batch and stream
>>> >> (continuous) processing. See https://beam.apache.org
>>> >>
>>> >> You can download the release here:
>>> >>
>>> >> https://beam.apache.org/get-started/downloads/
>>> >>
>>> >> This release includes bugfixes, features, and improvements detailed on
>>> >> the Beam blog:
>>> https://beam.apache.org/blog/2019/07/31/beam-2.14.0.html
>>> >>
>>> >> Thanks to everyone who contributed to this release, and we hope you
>>> enjoy
>>> >> using Beam 2.14.0.
>>> >>
>>> >> -- Anton Kedin, on behalf of The Apache Beam team
>>>
>>

Re: Proposal: Add permanent url to community metrics dashboard

[Alan Myrvold]

Are all of the CVE issues fixed at the version in use?
https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=grafana
XSS isn't much of a concern until there is a hostname associated.

On Wed, Jul 17, 2019 at 2:17 PM Pablo Estrada  wrote:

> I'd like to move this forward. Mikhail, would you be interested in filing
> an issue with Infra to see if it's possible? I can do it if you prefer.
>
> It seems that the concerns related to these dashboards showing up in
> search results have been addressed. Does the community have any other
> concern around this before we can move it forward?
> Best
> -P.
>
> On Wed, May 22, 2019 at 8:53 AM Kenneth Knowles  wrote:
>
>> I suggest asking infra about the best way to proceed, so that we don't
>> vote on something that doesn't work for them. This might be something handy
>> to spin up easily for any Apache project using similar tools.
>>
>> Kenn
>>
>> On Tue, May 21, 2019 at 1:02 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> Current http://104.154.241.245/robots.txt is already disallow all, so
>>> we are good here.
>>>
>>> On Tue, May 21, 2019 at 12:57 PM Ahmet Altay  wrote:
>>>
 If SSL is a concern that makes sense, I am not familiar with that
 enough to suggest whether another way to do this exists or not.

 It will be good to check that we can set robots.txt properly from the
 begging if we go down this path.

 On Mon, May 20, 2019 at 10:54 AM Mikhail Gryzykhin 
 wrote:

> @Ahmet Altay 
> Thank you for the comment.
>
> Point on search engines is really good. If that happens we can look
> into configuring robots.txt to notify search engines to ignore whole 
> domain.
> The link is a redirect to static IP. So it is still confusing.
>
> Having domain name will allow for getting SSL associated with it and
> will allow to keep same address even if IP changes (say we want to move to
> other hoster).
>

 I suppose short link will also allow us to change the host very similar
 to a domain name. That is a minor point anyway.


>
> Given two points above, I still consider that having explicit name
> will be beneficial. If there's some other way to get SSL cert and benefit
> of static name I'm eager to utilize it.
>
> On Mon, May 20, 2019 at 10:43 AM Ahmet Altay  wrote:
>
>> Hi Mikhail,
>>
>> Thank you for your work on this. I have some comments:
>>
>> - There is already a short link (
>> https://s.apache.org/beam-community-metrics). Would a link from
>> contributing to beam page (if there is not one already) sufficient> 
>> People
>> can bookmark the short link if they need to quickly access.
>> - Metrics is a developer facing tool. If it has its own subdomain and
>> start showing up in web search results, it will be a confusing landing 
>> page
>> for people simply searching for "beam metrics". I believe there is some
>> value in having a single domain and linking to various things from there.
>> This would be similar to how we link to jira, wiki, mailing list 
>> archives.
>>
>> Ahmet
>>
>> On Fri, May 17, 2019 at 9:26 PM Mikhail Gryzykhin <
>> gryzykhin.mikh...@gmail.com> wrote:
>>
>>> @Aizamat
>>> Code is not generalized and is project specific in some places. But
>>> it is small and pretty straightforward so can be ported easily. Whole 
>>> thing
>>> can be started locally with a single docker command, so it's easy to 
>>> try it
>>> out.
>>>
>>> On Fri, May 17, 2019, 19:33 Aizhamal Nurmamat kyzy <
>>> aizha...@google.com> wrote:
>>>
 Hi Mikhail,

 I think this dashboard is amazing, and would love to have an easy
 access to it. So here is my non binding +1.

 On the side note, how easy is to recreate it for other Apache
 projects? ;)

 Thanks,
 Aizhamal

 *From: *Mikhail Gryzykhin 
 *Date: *Fri, May 17, 2019 at 6:49 PM
 *To: *dev

 Hello everyone,
>
> Some time ago we started community metrics dashboard.
>  However we never
> had added a permanent URL for it. This is really inconvenient to use, 
> since
> only available way to access dashboard is by IP-address.
>
> In this tread I'd like to:
> 1. Vote to assign metrics.beam.apache.org to metrics dashboard (
> http://104.154.241.245).
> 2. Gather information on how to do it. I can assume only following
> steps so far: a) vote b) once vote is complete, contact Apache INFRA 
> to
> help with this.
>
> Regards,
> Mikhail.
>
>

Re: [PROPOSAL] Preparing for Beam 2.15.0 release

[Alan Myrvold]

+1 Thanks for keeping the release cadence going. I like to see regular
releases happening.

On Wed, Jul 17, 2019 at 11:01 AM Yifan Zou  wrote:

> Hello Beam community!
>
> Beam 2.15 release branch cut date is July 31 according to the release
> calendar [1]. I would like to volunteer myself to do this release. The plan
> is to cut the branch on that date, and cherrypick fixes if needed.
>
> If you have release blocking issues for 2.15 please mark their "Fix
> Version" as 2.15.0 [2]. Please use 2.16.0 release in JIRA in case you
> would like to move any non-blocking issues to that version.
>
> Any thoughts, comments, objections?
>
> Regards.
> Yifan Zou
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
> [2]
> https://issues.apache.org/jira/browse/BEAM-7730?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.15.0
>

Re: [ANNOUNCE] New committer: Robert Burke

[Alan Myrvold]

Congrats, Robert!

On Tue, Jul 16, 2019 at 11:46 AM Ismaël Mejía  wrote:

> Congrats Robert!
>
>
> On Tue, Jul 16, 2019 at 8:19 PM Yichi Zhang  wrote:
> >
> > Congratulations!
> >
> > On Tue, Jul 16, 2019 at 10:51 AM Holden Karau 
> wrote:
> >>
> >> Congratulations! :)
> >>
> >> On Tue, Jul 16, 2019 at 10:50 AM Mikhail Gryzykhin 
> wrote:
> >>>
> >>> Congratulations!
> >>>
> >>> On Tue, Jul 16, 2019 at 10:36 AM Ankur Goenka 
> wrote:
> 
>  Congratulations Robert!
> 
>  Go GO!
> 
>  On Tue, Jul 16, 2019 at 10:34 AM Rui Wang  wrote:
> >
> > Congrats!
> >
> >
> > -Rui
> >
> > On Tue, Jul 16, 2019 at 10:32 AM Udi Meiri  wrote:
> >>
> >> Congrats Robert B.!
> >>
> >> On Tue, Jul 16, 2019 at 10:23 AM Ahmet Altay 
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Robert Burke.
> >>>
> >>> Robert has been contributing to Beam and actively involved in the
> community for over a year. He has been actively working on Go SDK, helping
> users, and making it easier for others to contribute [1].
> >>>
> >>> In consideration of Robert's contributions, the Beam PMC trusts
> him with the responsibilities of a Beam committer [2].
> >>>
> >>> Thank you, Robert, for your contributions and looking forward to
> many more!
> >>>
> >>> Ahmet, on behalf of the Apache Beam PMC
> >>>
> >>> [1]
> https://lists.apache.org/thread.html/8f729da2d3009059d7a8b2d8624446be161700dcfa953939dd3530c6@%3Cdev.beam.apache.org%3E
> >>> [2]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
> >>
> >>
> >>
> >> --
> >> Twitter: https://twitter.com/holdenkarau
> >> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9
> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>

Re: Beam local website development environment

[Alan Myrvold]

You can serve (or test) the website locally, using
https://cwiki.apache.org/confluence/display/BEAM/Website+Tips

On Tue, Jun 11, 2019 at 3:27 PM Rakesh Kumar  wrote:

> Hi,
>
> I have been updating the documentation section of Beam website, mostly the
> code examples [1], & [2]. I want to check these changes locally before
> merging it to master just to make sure that changes will look good on the
> website. What is the best way of testing this in local?
>
> [1]: https://github.com/apache/beam/pull/8740
> [2]: https://github.com/apache/beam/pull/8803
>
> Thank you,
> Rakesh
>

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

[Alan Myrvold]

Congrats, Pablo!

*From: *Robin Qiu 
*Date: *Wed, May 15, 2019 at 10:44 AM
*To: * 

Congratulations, Pablo!!
>
> On Wed, May 15, 2019 at 10:43 AM Pablo Estrada  wrote:
>
>> Thanks everyone for the encouragement, and thanks to the PMC for the
>> recognition. I am honored and grateful. :)
>> Best
>> -P.
>>
>>
>> *From: *Kenneth Knowles 
>> *Date: *Tue, May 14, 2019, 10:25 PM
>> *To: *dev
>>
>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming Pablo Estrada
>>> to join the PMC.
>>>
>>> Pablo first picked up BEAM-722 in October of 2016 and has been a steady
>>> part of the Beam community since then. In addition to technical work on
>>> Beam Python & Java & runners, I would highlight how Pablo grows Beam's
>>> community by helping users, working on GSoC, giving talks at Beam Summits
>>> and other OSS conferences including Flink Forward, and holding training
>>> workshops. I cannot do justice to Pablo's contributions in a single
>>> paragraph.
>>>
>>> Thanks Pablo, for being a part of Beam.
>>>
>>> Kenn
>>>
>>

Re: All validates runner tests seems to be broken.

[Alan Myrvold]

Other ones are failing on the branch due to the 2.13.0 branch not having
https://github.com/apache/beam/pull/8194, and the seed job running from
master.

*From: *Michael Luckey 
*Date: *Tue, May 14, 2019 at 2:13 PM
*To: * 

Unfortunately, I missed the fact that seed job triggers automatically.
>
> Yes, you need to run the seed job on your branch to replace the old
> commands.
>
> We might consider resetting to legacy commands, i.e. revert ./test-infra
> folder. What do you think?
>
> On Tue, May 14, 2019 at 11:02 PM Andrew Pilloud 
> wrote:
>
>> So it sounds like a number of the failures are related to a single
>> jenkins config for all branches. This means you can't test the release
>> branch if the targets change after it is cut. One possibility: do "Run Seed
>> Job" on the release branch and kick off all the tests right after that
>> finishes. Then do "Run Seed Job" after they all start on head again to
>> restore the config.
>>
>> Andrew
>>
>> *From: *Alan Myrvold 
>> *Date: *Tue, May 14, 2019 at 1:59 PM
>> *To: * 
>>
>> That is a typo added in https://github.com/apache/beam/pull/8194
>>>
>>> https://github.com/apache/beam/commit/1e7ea0da5073566c3fa26dbc1105105fbe6043ae#diff-9591f0d06e82e711681fd77ed287578b
>>>
>>> *From: *Ankur Goenka 
>>> *Date: *Tue, May 14, 2019 at 1:43 PM
>>> *To: *dev
>>>
>>> Hi,
>>>>
>>>> Following tests seems to be broken because of "Project 'unners' not
>>>> found in root project 'beam'."
>>>> The command getting executed on Jenkins is
>>>> gradlew --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
>>>> -Dorg.gradle.jvmargs=-Xmx4g :unners:samza:validatesRunner
>>>> causing the failure.
>>>> The same is happening on the master
>>>> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_PR/19/console
>>>>
>>>> Are we making any changes to jenins parsing scripts?
>>>>
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_PR/20/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Apache Flink Runner ValidatesRunner Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_PR/63/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Apache Gearpump Runner ValidatesRunner Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_PR/16/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Apache Samza Runner ValidatesRunner Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_PR/17/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Apache Spark Runner ValidatesRunner Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_PR/73/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Google Cloud Dataflow Runner PortabilityApi ValidatesRunner Tests —
>>>> FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_PR/39/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Google Cloud Dataflow Runner Python ValidatesContainer Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/54/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Google Cloud Dataflow Runner Python ValidatesRunner Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_PR/79/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Google Cloud Dataflow Runner ValidatesRunner Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/59/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Java Flink PortableValidatesRunner Batch Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch_PR/50/>
>>>> [image: @asfgit] <https://github.com/asfgit>
>>>> Java Flink PortableValidatesRunner Streaming Tests — FAILURE
>>>> Details
>>>> <https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming_PR/77/>
>>>>
>>>> Thanks,
>>>> Ankur
>>>>
>>>

Re: All validates runner tests seems to be broken.

[Alan Myrvold]

That is a typo added in https://github.com/apache/beam/pull/8194
https://github.com/apache/beam/commit/1e7ea0da5073566c3fa26dbc1105105fbe6043ae#diff-9591f0d06e82e711681fd77ed287578b

*From: *Ankur Goenka 
*Date: *Tue, May 14, 2019 at 1:43 PM
*To: *dev

Hi,
>
> Following tests seems to be broken because of "Project 'unners' not found
> in root project 'beam'."
> The command getting executed on Jenkins is
> gradlew --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
> -Dorg.gradle.jvmargs=-Xmx4g :unners:samza:validatesRunner
> causing the failure.
> The same is happening on the master
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_PR/19/console
>
> Are we making any changes to jenins parsing scripts?
>
> Details
> 
> [image: @asfgit] 
> Apache Flink Runner ValidatesRunner Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Apache Gearpump Runner ValidatesRunner Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Apache Samza Runner ValidatesRunner Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Apache Spark Runner ValidatesRunner Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Google Cloud Dataflow Runner PortabilityApi ValidatesRunner Tests —
> FAILURE
> Details
> 
> [image: @asfgit] 
> Google Cloud Dataflow Runner Python ValidatesContainer Tests — FAILURE
> Details 
> [image: @asfgit] 
> Google Cloud Dataflow Runner Python ValidatesRunner Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Google Cloud Dataflow Runner ValidatesRunner Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Java Flink PortableValidatesRunner Batch Tests — FAILURE
> Details
> 
> [image: @asfgit] 
> Java Flink PortableValidatesRunner Streaming Tests — FAILURE
> Details
> 
>
> Thanks,
> Ankur
>

Re: Beam at Google Summer of Code 2019

[Alan Myrvold]

Welcome, Tanay! Nice proposal.

*From: *Kenneth Knowles 
*Date: *Mon, May 6, 2019 at 4:13 PM
*To: *dev
*Cc: * 

Nice! Welcome, Tanay!
>
> On Mon, May 6, 2019 at 3:56 PM Reza Rokni  wrote:
>
>> +1 Congratulations! Looking forward to using your work !
>>
>> *From: *Valentyn Tymofieiev 
>> *Date: *Tue, 7 May 2019 at 05:40
>> *To: *dev
>> *Cc: * 
>>
>> Congrats & good luck Tanay & Pablo!
>>>
>>> *From: *Connell O'Callaghan 
>>> *Date: *Mon, May 6, 2019 at 4:15 PM
>>> *To: * 
>>> *Cc: * 
>>>
>>> Well done Tanay - good luck with this project!!!

 +1 Pablo - thank you for this mentorship!!!

 On Mon, May 6, 2019 at 1:11 PM Chamikara Jayalath 
 wrote:

> Congrats Tanay and thanks Pablo for mentoring :)
>
> On Mon, May 6, 2019 at 11:53 AM Aizhamal Nurmamat kyzy <
> aizha...@google.com> wrote:
>
>> Congratulations Tanay!
>>
>> *From: *Pablo Estrada 
>> *Date: *Mon, May 6, 2019 at 11:34 AM
>> *To: *dev, 
>>
>> Hello all,
>>> it is my pleasure to share with everyone that Tanay Tummalapalli has
>>> been accepted as a GSoC student with Beam, to implement support for File
>>> Loads into BigQuery for streaming pipelines[1].
>>>
>>> Tanay wrote a very strong proposal, and showed understanding of the
>>> tricky streaming considerations that will play out in this project.
>>>
>>> I speak on behalf of everyone welcoming you Tanay, and we'll be
>>> happy to see your contributions to Beam. : )
>>> Best
>>> -P.
>>>
>>> [1]
>>> https://summerofcode.withgoogle.com/projects/?sp-search=Tanay#4999837794172928
>>>
>>
>>
>> --
>>
>> This email may be confidential and privileged. If you received this
>> communication by mistake, please don't forward it to anyone else, please
>> erase all copies and attachments, and please let me know that it has gone
>> to the wrong person.
>>
>> The above terms reflect a potential business arrangement, are provided
>> solely as a basis for further discussion, and are not intended to be and do
>> not constitute a legally binding obligation. No legally binding obligations
>> will be created, implied, or inferred until an agreement in final form is
>> executed in writing by all parties involved.
>>
>

Re: Updates on Beam Jenkins

[Alan Myrvold]

Thanks for this work, Yifan!

On Mon, Apr 29, 2019 at 8:14 AM Ismaël Mejía  wrote:

> Thanks Yifan for all your work. Sometimes the work on infrastructure
> is hidden, so it is great to acknowledge the importance of the
> improvements you and the others have done.
>
> On Mon, Apr 29, 2019 at 5:11 PM Lukasz Cwik  wrote:
> >
> > Thanks Yifan for driving this.
> >
> > On Mon, Apr 29, 2019 at 8:01 AM Yifan Zou  wrote:
> >>
> >> Hi all,
> >>
> >>
> >> We now fully switched the Jenkins to new agents. The old agents are
> deprecated and VMs will be deleted shortly to make more CPU available in
> the us-central1 for tests. Please let me know if you see anything abnormal
> on the Jenkins.
> >>
> >>
> >> Thanks to everyone who helped and contributed to making this migration
> success!
> >>
> >>
> >> Yifan Zou
> >>
> >>
> >>
> >> On Wed, Apr 10, 2019 at 9:55 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >>>
> >>> Thanks a lot Yifan! This work will likely be a great improvement in
> Beam developer velocity. INFRA team seems to be overbooked, and things like
> installing additional dependency took them significant time to be resolved
> in the past. By having more control and context over what needs to be
> installed on Jenkins workers, we should be able to move faster.
> >>>
> >>> On Tue, Apr 9, 2019 at 11:44 PM Connell O'Callaghan <
> conne...@google.com> wrote:
> 
>  Yifan thank you for this update and progressing this replacement!!!
> 
>  On Tue, Apr 9, 2019, 11:14 PM Yifan Zou  wrote:
> >
> > Thanks, Pablo. The new workers use custom image in the boot disk and
> it's easy to reboot without re-imaging. Means we will no longer need
> assistance from Infra to reconnect an offline node. Dockerizing the
> environment would be helpful to make changes on the environment, such as
> installing/updating a beam-required package.
> >
> >
> > On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada 
> wrote:
> >>
> >> Thanks for the updates Yifan. I am sure this process has been
> difficult, and I appreciate the good communication, and that this didn't
> really affect the workflow of anyone to validate the new setup for nodes.
> >>
> >> I imagine that once we move to dockerizing the testing environment,
> it will be much simpler to restart machines that are having trouble?
> >> Thanks again!
> >> -P.
> >>
> >> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou 
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I have some good news about our Jenkins nodes. We're now having 7
> new nodes online named as "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]",
> which substitute the old broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This
> helps to reduce the job waiting queue that let your tests getting executed
> quickly. They're starting up and accepting jobs. There is no action needed
> on your end.
> >>>
> >>> I'll keep the remaining old agents running for one more week just
> in case it won't fully block the development works if any unexpected issues
> on the new agents. Once the new agents are stable and verified, I'll move
> forward to connect the rest agents and remove old set. The playbook is also
> on the way.
> >>>
> >>> For more background and information of the Jenkins updates, please
> see:
> >>> One pager:
> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
> >>> Environment verification sheet:
> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
> >>> Previous Thread on dev@:
> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
> >>>
> >>> Thanks.
> >>>
> >>> Regards.
> >>> Yifan Zou
> >>>
> >>>
>

Re: [Discuss] Publishing pre-release artifacts to repositories

[Alan Myrvold]

Great idea. I like the RC candidates to follow as much as the release
artifact process as possible.

On Wed, Apr 24, 2019 at 3:27 PM Ahmet Altay  wrote:

> To clarify my proposal, I am proposing publishing to the production pypi
> repository with an rc tag in the version. And in turn allow users to depend
> on beam's rc version + all the other regular dependencies users would have
> directly from pypi.
>
> Publishing to test pypi repo would also be helpful if test pypi repo also
> mirrors other packages that exist in the production pypi repository.
>
> On Wed, Apr 24, 2019 at 3:12 PM Pablo Estrada  wrote:
>
>> I think this is a great idea. A way of doing it for python would be by
>> using the test repository for PyPi[1], and that way we would not have to do
>> an official PyPi release, but still would be able to install it with pip
>> (by passing an extra flag), and test.
>>
>> In fact, there are some Beam artifacts already in there[2]. At some point
>> I looked into this, but couldn't figure out who has access/the password for
>> it.
>>
>
> I also don't know who owns beam package in test pypi repo. Does
> anybody know?
>
>
>>
>> In short: +1, and I would suggest using the test PyPi repo to avoid
>> publishing to the main PyPi repo.
>> Best
>> -P.
>>
>> [1] https://test.pypi.org/
>> [2] https://test.pypi.org/project/apache-beam/
>>
>> On Wed, Apr 24, 2019 at 3:04 PM Ahmet Altay  wrote:
>>
>>> Hi all,
>>>
>>> What do you think about the idea of publishing pre-release artifacts as
>>> part of the RC emails?
>>>
>>> For Python this would translate into publishing the same artifacts from
>>> RC email with a version like "2.X.0rcY" to pypi. I do not know, but I am
>>> guessing we can do a similar thing with Maven central for Java artifacts as
>>> well.
>>>
>>> Advantages would be:
>>> - Allow end users to validate RCs for their own purposes using the same
>>> exact process they will normally use.
>>>  - Enable early-adaptors to start using RC releases early on in the
>>> release cycle if that is what they would like to do. This will in turn
>>> reduce time pressure on some releases. Especially for cases like someone
>>> needs a release to be finalized for an upcoming event.
>>>
>>> There will also be disadvantages, some I could think of:
>>> - Users could request support for RC artifacts. Hopefully in the form of
>>> feedback for us to improve the release. But it could also be in the form of
>>> folks using RC artifacts for production for a long time.
>>> - It will add toil to the current release process, there will be one
>>> more step for each RC. I think for python this will be a small step but
>>> nevertheless it will be additional work.
>>>
>>> For an example of this, you can take a look at tensorflow releases. For
>>> 1.13 there were 3 pre-releases [1].
>>>
>>> Ahmet
>>>
>>> [1] https://pypi.org/project/tensorflow/#history
>>>
>>

Re: [ANNOUNCE] New committer announcement: Yifan Zou

[Alan Myrvold]

Congrats, Yifan!!!

On Mon, Apr 22, 2019 at 10:24 AM Robin Qiu  wrote:

> Congratulations Yifan!
>
> On Mon, Apr 22, 2019 at 10:17 AM Chamikara Jayalath 
> wrote:
>
>> Congrats Yifan!
>>
>> On Mon, Apr 22, 2019 at 10:02 AM Maximilian Michels 
>> wrote:
>>
>>> Congrats! Great work.
>>>
>>> -Max
>>>
>>> On 22.04.19 19:00, Rui Wang wrote:
>>> > Congratulations! Thanks for your contribution!!
>>> >
>>> > -Rui
>>> >
>>> > On Mon, Apr 22, 2019 at 9:57 AM Ruoyun Huang >> > > wrote:
>>> >
>>> > Congratulations, Yifan!
>>> >
>>> > On Mon, Apr 22, 2019 at 9:48 AM Boyuan Zhang >> > > wrote:
>>> >
>>> > Congratulations, Yifan~
>>> >
>>> > On Mon, Apr 22, 2019 at 9:29 AM Connell O'Callaghan
>>> > mailto:conne...@google.com>> wrote:
>>> >
>>> > Well done Yifan!!!
>>> >
>>> > Thank you for sharing Kenn!!!
>>> >
>>> > On Mon, Apr 22, 2019 at 9:00 AM Ahmet Altay
>>> > mailto:al...@google.com>> wrote:
>>> >
>>> > Congratulations, Yifan!
>>> >
>>> > On Mon, Apr 22, 2019 at 8:46 AM Tim Robertson
>>> > >> > > wrote:
>>> >
>>> > Congratulations Yifan!
>>> >
>>> > On Mon, Apr 22, 2019 at 5:39 PM Cyrus Maden
>>> > mailto:cma...@google.com>>
>>> wrote:
>>> >
>>> > Congratulations Yifan!!
>>> >
>>> > On Mon, Apr 22, 2019 at 11:26 AM Kenneth
>>> Knowles
>>> > mailto:k...@apache.org>>
>>> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > Please join me and the rest of the Beam PMC
>>> > in welcoming a new committer: Yifan Zou.
>>> >
>>> > Yifan has been contributing to Beam since
>>> > early 2018. He has proposed 70+ pull
>>> > requests, adding dependency checking and
>>> > improving test infrastructure. But
>>> something
>>> > the numbers cannot show adequately is the
>>> > huge effort Yifan has put into working with
>>> > infra and keeping our Jenkins executors
>>> healthy.
>>> >
>>> > In consideration of Yian's contributions,
>>> > the Beam PMC trusts Yifan with the
>>> > responsibilities of a Beamcommitter[1].
>>> >
>>> > Thank you, Yifan, for your contributions.
>>> >
>>> > Kenn
>>> >
>>> > [1]
>>> >
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> > <
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > 
>>> > Ruoyun  Huang
>>> >
>>>
>>

Re: New contributor to Beam

[Alan Myrvold]

Welcome, Cyrus!

On Wed, Apr 17, 2019 at 12:49 PM Ahmet Altay  wrote:

> Welcome!
>
> On Wed, Apr 17, 2019 at 12:26 PM Rose Nguyen  wrote:
>
>> Welcome, Cyrus!!
>>
>> On Wed, Apr 17, 2019 at 11:58 AM Niklas Hansson <
>> niklas.sven.hans...@gmail.com> wrote:
>>
>>> Welcome :)
>>>
>>> Den ons 17 apr. 2019 kl 20:33 skrev Aizhamal Nurmamat kyzy <
>>> aizha...@google.com>:
>>>
 Welcome Cyrus! We'd love so much to have better docs for Beam [image:
 ][image: ][image: ] Thank you!


 On Wed, Apr 17, 2019 at 11:28 AM Joana Filipa Bernardo Carrasqueira <
 joanafil...@google.com> wrote:

> Welcome Cyrus!
>
> On Wed, Apr 17, 2019 at 11:18 AM Kyle Weaver 
> wrote:
>
>> Welcome!
>>
>> On Wed, Apr 17, 2019 at 10:32 AM Robert Burke 
>> wrote:
>>
>>> Welcome Cyrus! :D Yay better docs!
>>>
>>> On Wed, 17 Apr 2019 at 10:20, Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Welcome Cyrus!!!

 On Wed, Apr 17, 2019 at 10:11 AM Mikhail Gryzykhin <
 mig...@google.com> wrote:

> Welcome!
>
> --Mikhail
>
> On Wed, Apr 17, 2019 at 9:58 AM Melissa Pashniak <
> meliss...@google.com> wrote:
>
>>
>> Welcome Cyrus!
>>
>>
>> On Wed, Apr 17, 2019 at 7:31 AM Jean-Baptiste Onofré <
>> j...@nanthrax.net> wrote:
>>
>>> Welcome !
>>>
>>> Regards
>>> JB
>>>
>>> On 17/04/2019 16:05, Cyrus Maden wrote:
>>> > Hi all!
>>> >
>>> > My name's Cyrus and I'd like to start contributing to Beam.
>>> I'm a
>>> > technical writer so I'm particularly looking forward to
>>> contributing to
>>> > the Beam docs. Could someone add me as a contributor on JIRA
>>> so I can
>>> > create and assign tickets?
>>> >
>>> > My JIRA name is: *cyrusmaden*
>>> > *
>>> > *
>>> > Excited to be a part of this community and to work with ya'll!
>>> >
>>> > Best,
>>> > Cyrus
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>> --
>> Kyle Weaver | Software Engineer | github.com/ibzib |
>> kcwea...@google.com | +1650203
>>
>
>
>>
>> --
>> Rose Thị Nguyễn
>>
>

Re: Removing :beam-website:testWebsite from gradle build target

[Alan Myrvold]

+1 if possible, removing link checks would be nice too, if they are
unreliable and there is a way to disable them.

On Mon, Apr 1, 2019 at 10:33 AM Mikhail Gryzykhin  wrote:

> +1 on this. I'd prefer to have this as pre-commit only.
>
> On Mon, Apr 1, 2019 at 9:09 AM Andrew Pilloud  wrote:
>
>> +1 on this, particularly removing the dead link checker from default
>> tests. It is effectively testing that ~20 random websites are up. I wonder
>> if there is a way to limit it to locally testing links within the beam site?
>>
>> On Mon, Apr 1, 2019 at 3:54 AM Michael Luckey 
>> wrote:
>>
>>> Hi,
>>>
>>> after playing around with Gradle build for a while, I would like to
>>> suggest to remove ':beam-website:testWebsite target from Gradle's check
>>> task.
>>>
>>> Rationale:
>>> - the task seems to be very flaky. In fact, I always need to add '-x
>>> :beam-website:testWebsite' to my build [1]
>>> - task uses docker, which imho adds a (unnecessary) severe constraint on
>>> the build task. E.g. A part time user is unable to execute these tests in a
>>> docker environment
>>> - these tests are accessing production environment. So myself hitting
>>> the build several times an hour could be considered a DOS attack.
>>>
>>> Of course, these tests add lots of value and should definitely be
>>> executed, but wouldn't it be sufficient, to run this task only dedicated,
>>> i.e. by an explicit call to ':beam-website:testWebsite' o
>>> ':websitePreCommit'? Any thoughts?
>>>
>>> best,
>>>
>>> michel
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-6760
>>>
>>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Alan Myrvold]

congratulations, Mark!!!

On Mon, Mar 25, 2019 at 10:05 AM Ruoyun Huang  wrote:

> Congratulations Mark!
>
> On Mon, Mar 25, 2019 at 9:31 AM Udi Meiri  wrote:
>
>> Congrats Mark!
>>
>> On Mon, Mar 25, 2019 at 9:24 AM Ahmet Altay  wrote:
>>
>>> Congratulations, Mark! 
>>>
>>> On Mon, Mar 25, 2019 at 7:24 AM Tim Robertson 
>>> wrote:
>>>
 Congratulations Mark!


 On Mon, Mar 25, 2019 at 3:18 PM Michael Luckey 
 wrote:

> Nice! Congratulations, Mark.
>
> On Mon, Mar 25, 2019 at 2:42 PM Katarzyna Kucharczyk <
> ka.kucharc...@gmail.com> wrote:
>
>> Congratulations, Mark! 
>>
>> On Mon, Mar 25, 2019 at 11:24 AM Gleb Kanterov 
>> wrote:
>>
>>> Congratulations!
>>>
>>> On Mon, Mar 25, 2019 at 10:23 AM Łukasz Gajowy 
>>> wrote:
>>>
 Congrats! :)



 pon., 25 mar 2019 o 08:11 Aizhamal Nurmamat kyzy <
 aizha...@google.com> napisał(a):

> Congratulations, Mark!
>
> On Sun, Mar 24, 2019 at 23:18 Pablo Estrada 
> wrote:
>
>> Yeaah  Mark! : ) Congrats : D
>>
>> On Sun, Mar 24, 2019 at 10:32 PM Yifan Zou 
>> wrote:
>>
>>> Congratulations Mark!
>>>
>>> On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done congratulations Mark!!!

 On Sun, Mar 24, 2019 at 10:17 PM Robert Burke <
 rob...@frantil.com> wrote:

> Congratulations Mark! 
>
> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Congratulations, Mark!
>>
>> Thanks for your contributions, in particular for your efforts
>> to parallelize test execution for Python SDK and increase the 
>> speed of
>> Python precommit checks.
>>
>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a
>>> new committer: Mark Liu.
>>>
>>> Mark has been contributing to Beam since late 2016! He has
>>> proposed 100+ pull requests. Mark was instrumental in expanding 
>>> test and
>>> infrastructure coverage, especially for Python. In
>>> consideration of Mark's contributions, the Beam PMC trusts Mark 
>>> with the
>>> responsibilities of a Beam committer [1].
>>>
>>> Thank you, Mark, for your contributions.
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>> --
>
> *Aizhamal Nurmamat kyzy*
>
> Open Source Program Manager
>
> 646-355-9740 Mobile
>
> 601 North 34th Street, Seattle, WA 98103
>
>
>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: Selectively running tests?

[Alan Myrvold]

The includedRegions was set up as part of
https://issues.apache.org/jira/browse/BEAM-4445 and there are additional
paths added from
https://github.com/apache/beam/blob/6bb4b2332b11bd8295ac6965be8426b9c38fa454/.test-infra/jenkins/PrecommitJobBuilder.groovy#L65

Not sure why they are not working, but it would be good to get this going
again. Might have stopped at the same time Jenkins was updated.

On Mon, Mar 18, 2019 at 3:28 PM Pablo Estrada  wrote:

> We used to have tests run selectively depending on which directories were
> changes. I've just noticed that this is not the case anymore.
>
> Did we stop doing that? Or maybe the selector is faulty? Anyone know what
> happened here?
> Thanks!
> -P.
>

Re: [PROPOSAL] Prepare Beam 2.11.0 release

[Alan Myrvold]

Thanks for volunteering. Glad to see the schedule being kept.

On Fri, Feb 8, 2019 at 5:58 AM Maximilian Michels  wrote:

> +1 We should keep up the release cycle even though we are a bit late
> with 2.10.0
>
> On 08.02.19 02:10, Andrew Pilloud wrote:
> > +1 let's keep things going. Thanks for volunteering!
> >
> > Andrew
> >
> > On Thu, Feb 7, 2019, 4:49 PM Kenneth Knowles  > > wrote:
> >
> > I think this is a good idea. Even though 2.10.0 is still making it's
> > way out, there's still 6 weeks of changes between the cut dates,
> > lots of value. It is good to keep the rhythm and follow the calendar.
> >
> > Kenn
> >
> > On Thu, Feb 7, 2019, 15:52 Ahmet Altay  >  wrote:
> >
> > Hi all,
> >
> > Beam 2.11 release branch cut date is 2/13 according to the
> > release calendar [1]. I would like to volunteer myself to do
> > this release. I intend to cut the branch on the planned 2/13
> date.
> >
> > If you have releasing blocking issues for 2.11 please mark their
> > "Fix Version" as 2.11.0. (I also created a 2.12.0 release in
> > JIRA in case you would like to move any non-blocking issues to
> > that version.)
> >
> > What do you think?
> >
> > Ahmet
> >
> > [1]
> >
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com=America%2FLos_Angeles
> >
>

Re: Dealing with expensive jenkins + Dataflow jobs

[Alan Myrvold]

Agreeing with Robert about "what is it we're trying to test?". Would a
smaller performance test find the same issues, faster and more reliably?

We have seen issues with the apache-beam-testing project exceeding quota
during dataflow jobs, resulting in spurious failures during precommits and
postcommits. 32 workers per dataflow jobs sounds fine, provided there are
not too many concurrent dataflow jobs. Not all the tests have the number of
workers limited, so I've seen some with ~80 workers. For non-performance
tests, it would seem we should be able to drastically limit the number of
workers, which should provide more room for performance tests/

On Wed, Jan 23, 2019 at 7:10 AM Robert Bradshaw  wrote:

> I like the idea of creating separate project(s) for load tests so as
> to not compete with other tests and the standard development cycle.
>
> As for how many workers is too many, I would take the track "what is
> it we're trying to test?" Unless your stress-testing the shuffle
> itself, much of what Beam does is linearly parallizable with the
> number of machines. Of course one will still want to run over real,
> large data sets, but not every load test needs this every time. More
> interesting could be to try out running at 2x and 4x the data, with 2x
> and 4x the machines, and seeing where we fail to be linear.
>
> (As an aside, 4 hours x 10 workers seems like a lot for 23GB of
> data...or is it 230GB once you've fanned out?)
>
> On Wed, Jan 23, 2019 at 3:33 PM Łukasz Gajowy  wrote:
> >
> > Hi,
> >
> > pinging this thread (maybe some folks missed it). What do you think
> about those concerns/ideas?
> >
> > Łukasz
> >
> > pon., 14 sty 2019 o 17:11 Łukasz Gajowy  napisał(a):
> >>
> >> Hi all,
> >>
> >> one problem we need to solve while working with load tests we currently
> develop is that we don't really know how much GCP/Jenkins resources can we
> occupy. We did some initial testing with
> beam_Java_LoadTests_GroupByKey_Dataflow_Small[1] and it seems that for:
> >>
> >> - 1 000 000 000 (~ 23 GB) synthetic record
> >> - 10 fanouts
> >> - 10 dataflow workers (--maxNumWorkers)
> >>
> >> the total job time exceeds 4 hours. It seems too much for such a small
> load test. Additionally, we plan to add much bigger tests for other core
> operations too. The proposal [2] describes only few of them.
> >>
> >> The questions are:
> >> 1. how many workers can we assign to this job without starving the
> other jobs? Are 32 workers for a single Dataflow job fine? Would 64 workers
> for such job be fine either?
> >> 2. given the plans that we are going to add more and more load tests
> soon, do you think it is a good idea to create a separate GCP project +
> separate Jenkins workers for load testing purposes only? This would avoid
> starvation of critical tests (post commits, pre-commits, etc). Or maybe
> there is another solution that will bring such isolation? Is such isolation
> needed?
> >>
> >> Ad 2: Please note that we will also need to host Flink/Spark clusters
> later on GKE/Dataproc (not decided yet).
> >>
> >> [1]
> https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_Java_LoadTests_GroupByKey_Dataflow_Small_PR/
> >> [2] https://s.apache.org/load-test-basic-operations
> >>
> >>
> >> Thanks,
> >> Łukasz
> >>
>

Re: Master broken

[Alan Myrvold]

The Jenkins test does a fresh clone of the repo, without generating code
before the spotless test.

On Fri, Jan 18, 2019 at 11:41 AM Kenneth Knowles  wrote:

> Those are the paths that cause the Jenkins job to be run. It doesn't
> affect the Gradle task.
>
> Kenn
>
> On Fri, Jan 18, 2019 at 11:34 AM Reuven Lax  wrote:
>
>> FYI, Jenkins works because it explicitly specifies which paths to run
>> spotless on, as below.  As a result, Jenkins (correctly) does not run
>> spotless on generated src.
>>
>> PrecommitJobBuilder builder = new PrecommitJobBuilder(
>> scope: this,
>> nameBase: 'Spotless',
>> gradleTask: 'spotlessCheck',
>> triggerPathPatterns: [
>>   '^buildSrc/.*$',
>>   '^sdks/java/.*$',
>>   '^runners/.*$',
>>   '^examples/java/.*$',
>> ]
>> )
>>
>>
>> On Fri, Jan 18, 2019 at 10:08 AM Reuven Lax  wrote:
>>
>>> Thanks, working on a PR now to exclude generated code. I wonder if this
>>> is why spotless has always been so slow.
>>>
>>> On Fri, Jan 18, 2019 at 8:28 AM Ismaël Mejía  wrote:
>>>
 What command are you running to build?
 This issue was reported also by other users in the slack channel.
 Agree the fix should be trivial

 On Fri, Jan 18, 2019 at 5:13 PM Reuven Lax  wrote:
 >
 > Does this only happen on fresh clones? I created a fresh branch
 synced to origin/master, and I can't reproduce this still.
 >
 > If spotless is running against generated code, that seems like a bug
 in our spotless setup. Should be trivial to fix by creating a target block
 in our spotless config.
 >
 > Reuven
 >
 > On Fri, Jan 18, 2019 at 8:07 AM Ismaël Mejía 
 wrote:
 >>
 >> Just make a fresh clone and run `./gradlew check -p sdks/java/core`
 it
 >> should break. If you add 'spotlessApply' the build passes but this
 >> should not be the default, no?, the default is 'spotlessCheck'
 >>
 >> On Fri, Jan 18, 2019 at 5:02 PM Reuven Lax  wrote:
 >> >
 >> > I don't get those errors when I run spotlessApply, and I don't see
 those errors happening on Jenkins. Are you doing anything special to run
 spotless? In general, I don't think spotless was running on generated code
 before.
 >> >
 >> > On Fri, Jan 18, 2019 at 6:06 AM Ismaël Mejía 
 wrote:
 >> >>
 >> >> When running the build on master we got an error message.
 >> >> Looks related to the recent inclusion/generation of stuff with
 ANTLR.
 >> >> Can Reuven or someone else involved in this specific changes
 please
 >> >> take a look?
 >> >>
 >> >> > Task :beam-sdks-java-core:spotlessJava FAILED
 >> >>
 >> >> FAILURE: Build failed with an exception.
 >> >>
 >> >> * What went wrong:
 >> >> Execution failed for task ':beam-sdks-java-core:spotlessJava'.
 >> >> > The following files had format violations:
 >> >>
  
 sdks/java/core/build/generated-src/antlr/main/java/org/apache/beam/sdk/schemas/parser/generated/FieldSpecifierNotationParser.java
 >> >>   @@ -1,5 +1,3 @@
 >> >>
  
 -//·Generated·from·java/org/apache/beam/sdk/schemas/parser/generated/FieldSpecifierNotation.g4·by·ANTLR·4.7
 >> >>   -
 >> >>/*
 >> >>
 ·*·Licensed·to·the·Apache·Software·Foundation·(ASF)·under·one
 >> >>
 ·*·or·more·contributor·license·agreements.··See·the·NOTICE·file
 >> >>   @@ -19,609 +17,737 @@
 >> >>·*/
 >> >>package·org.apache.beam.sdk.schemas.parser.generated;
 >> >>
 >> >>   +import·java.util.List;
 >> >>   +import·org.antlr.v4.runtime.*;
 >> >>import·org.antlr.v4.runtime.atn.*;
 >> >>import·org.antlr.v4.runtime.dfa.DFA;
 >> >>   -import·org.antlr.v4.runtime.*;
 >> >>import·org.antlr.v4.runtime.misc.*;
 >> >>import·org.antlr.v4.runtime.tree.*;
 >> >>   -import·java.util.List;
 >> >>   -import·java.util.Iterator;
 >> >>   -import·java.util.ArrayList;
 >> >>
 >> >>
 @SuppressWarnings({"all",·"warnings",·"unchecked",·"unused",·"cast"})
 >> >>
 public·class·FieldSpecifierNotationParser·extends·Parser·{
 >> >>
  
 -\tstatic·{·RuntimeMetaData.checkVersion("4.7",·RuntimeMetaData.VERSION);·}
 >> >>   +··static·{
 >> >>
  +RuntimeMetaData.checkVersion("4.7",·RuntimeMetaData.VERSION);
 >> >>   +··}
 >> >>
 >> >>   -\tprotected·static·final·DFA[]·_decisionToDFA;
 >> >>
  -\tprotected·static·final·PredictionContextCache·_sharedContextCache·=
 >> >>   -\t\tnew·PredictionContextCache();
 >> >>   -\tpublic·static·final·int
 >> >>
  
 -\t\tT__0=1,·T__1=2,·T__2=3,·T__3=4,·T__4=5,·IDENTIFIER=6,·WILDCARD=7,·WS=8;
 >> >>   -\tpublic·static·final·int
 >> >>
  
 

Re: Proposal: Portability SDKHarness Docker Image Release with Beam Version Release.

[Alan Myrvold]

+1 This would be great. gcr.io seems like a good option for snapshots due
to the permissions from jenkins to upload and ability to keep snapshots
around.

On Wed, Jan 16, 2019 at 6:51 PM Ruoyun Huang  wrote:

> +1 This would be a great thing to have.
>
> On Wed, Jan 16, 2019 at 6:11 PM Ankur Goenka  wrote:
>
>> grc.io seems to be a good option. Given that we don't need the hosting
>> server name in the image name makes it easily changeable later.
>>
>> Docker container for Apache Flink is named "flink" and they have
>> different tags for different releases and configurations
>> https://hub.docker.com/_/flink .We can follow a similar model and can
>> name the image as "beam" (beam doesn't seem to be taken on docker hub) and
>> use tags to distinguish Java/Python/Go and versions etc.
>>
>> Tags will look like:
>> java-SNAPSHOT
>> java-2.10.1
>> python2-SNAPSHOT
>> python2-2.10.1
>> go-SNAPSHOT
>> go-2.10.1
>>
>>
>> On Wed, Jan 16, 2019 at 5:56 PM Ahmet Altay  wrote:
>>
>>> For snapshots, we could use gcr.io. Permission would not be a problem
>>> since Jenkins is already correctly setup. The cost will be covered under
>>> apache-beam-testing project. And since this is only for snapshots, it will
>>> be only for temporary artifacts not for release artifacts.
>>>
>>> On Wed, Jan 16, 2019 at 5:50 PM Valentyn Tymofieiev 
>>> wrote:
>>>
 +1, releasing containers is a useful process that we need to build in
 Beam and it is required for FnApi users. Among other reasons, having
 officially-released Beam SDK harness container images will make it easier
 for users to do simple customizations to  container images, as they will be
 able to use container image released by Beam as a base image.

 Good point about potential storage limitations on Bintray. With Beam
 Release cadence we may quickly exceed the 10 GB quota. It may also affect
 our decisions as to which images we want to release, for example: do we
 want to only release one container image with Python 3 interpreter, or do
 we want to release a container image for each Python 3 minor version that
 Beam is compatible with.

>>>
>>> Probably worth a separate discussion. I would favor first releasing a
>>> python 3 compatible version before figuring out how we would target
>>> multiple python 3 versions.
>>>
>>
>>>

 On Wed, Jan 16, 2019 at 5:48 PM Ankur Goenka  wrote:

>
>
> On Wed, Jan 16, 2019 at 5:37 PM Ahmet Altay  wrote:
>
>>
>>
>> On Wed, Jan 16, 2019 at 5:28 PM Ankur Goenka 
>> wrote:
>>
>>> - Could we start from snapshots first and then do it for releases?
>>> +1, releasing snapsots first makes sense to me.
>>> - For snapshots, do we need to clean old containers after a while?
>>> Otherwise I guess we will accumulate lots of containers.
>>> For snap shots we can maintain a single snapshot image from git HEAD
>>> daily. Docker has the internal image container id which changes 
>>> everytime
>>> an image is changed and pulls new images as needed.
>>>
>>
>> There is a potential use this may not work with. If a user picks up a
>> snaphsot build and want to use it until the next release arrives. I guess
>> in that case the user can copy the snapshotted container image and rely 
>> on
>> that.
>>
>>
> Yes, that should be reasonable.
>
>> - Do we also need additional code changes for snapshots and releases
>>> to default to these specific containers? There could be a version based
>>> mechanism to resolve the correct container to use.
>>> The current image defaults have username in it. We should be ok by
>>> just updating the default image url to published image url.
>>>
>>> We should also check for pricing and details about Apache-Bintray
>>> agreement before pushing images and changing defaults.
>>>
>>
>> There is information on bintray's pricing page about open source
>> projects [1]. I do not know if there is a special apache-bintray 
>> agreement
>> or not. If there is no special agreement there is a 10GB storage limit 
>> for
>> using bintray.
>>
> As each image can easily run into Gigs, 10GB might not be sufficient
> for future proofing.
> We can also register docker image to docker image registry and not
> have bintray in the name to later host images on a different vendor for
> future proofing.
>
>
>> [1] https://bintray.com/account/pricing?tab=account=pricing
>>
>>
>>>
>>> On Wed, Jan 16, 2019 at 5:11 PM Ahmet Altay 
>>> wrote:
>>>
 This sounds like a good idea. Some questions:

 - Could we start from snapshots first and then do it for releases?
 - For snapshots, do we need to clean old containers after a while?
 Otherwise I guess we will accumulate lots of containers.
 - Do we also need additional code changes 

Re: [RFC] I made a new tabbed Beam view in Jenkins

[Alan Myrvold]

The _Cron variants of pre-commits are run post-commit, to make it easier to
tell if the pre-commit is flaky or broken.

On Tue, Dec 18, 2018 at 1:40 PM Jason Kuster  wrote:

> This looks great! (also it looks like some precommits snuck into the
> postcommit view)
>
> On Tue, Dec 18, 2018 at 1:25 PM Alan Myrvold  wrote:
>
>> It does look much better!
>>
>> On Tue, Dec 18, 2018 at 1:10 PM Ahmet Altay  wrote:
>>
>>> I like this version, it looks cleaner than the current combined view.
>>>
>>> On Tue, Dec 18, 2018 at 12:53 PM Scott Wegner 
>>> wrote:
>>>
>>>> Very cool. I also didn't realize we had control over the Jenkins
>>>> "views".
>>>>
>>>> We currently lack a decent dashboard to monitor the build health across
>>>> Beam jenkins jobs and triage failures; this is a step in the right
>>>> direction.
>>>>
>>>> I haven't played with Jenkins views before, but it appears they can be
>>>> managed via the Job DSL similar to our job definitions [1]:
>>>>
>>>> > The DSL execution engine exposes several methods to create Jenkins
>>>> jobs, views, folders and config files. [..]
>>>>
>>>> It would be cool to integrate this into our job config in such a way
>>>> that we could automatically keep the views up-to-date as jobs are added or
>>>> renamed.
>>>>
>>>> [1] https://github.com/jenkinsci/job-dsl-plugin/wiki/Job-DSL-Commands
>>>>
>>>> On Tue, Dec 18, 2018 at 12:35 PM Anton Kedin  wrote:
>>>>
>>>>> This is really helpful, didn't realize it was possible. Categories and
>>>>> contents look reasonable. I think something like this definitely should be
>>>>> the top-level Beam view.
>>>>>
>>>>> Regards,
>>>>> Anton
>>>>>
>>>>> On Tue, Dec 18, 2018 at 12:05 PM Kenneth Knowles 
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I made a new view to split Beam builds into tabs:
>>>>>> https://builds.apache.org/view/A-D/view/Beam%20Nested/
>>>>>>
>>>>>>  - PostCommit tab includes PostCommit and "PreCommit_.*_Cron" because
>>>>>> these are actually post-commit jobs; it is a feature not a bug.
>>>>>>  - PreCommit tab includes jobs that have no meaningful history
>>>>>> because they are just against PRs, commits, phrase triggering
>>>>>>  - Inventory self-explanatory
>>>>>>  - PerformanceTests self-explanatory
>>>>>>  - All; I didn't want to keep making categories but just send this
>>>>>> for feedback
>>>>>>
>>>>>> WDYT about making this the top-level Beam view? (vs
>>>>>> https://builds.apache.org/view/A-D/view/Beam/)
>>>>>>
>>>>>> After that, maybe we could clean the categories so they fit into the
>>>>>> tabs more easily with fewer regexes (to make sure things don't get 
>>>>>> missed).
>>>>>> I have read also that if you use / instead of _ as a separator in a name
>>>>>> then Jenkins will display jobs as nested in folders automatically. Not 
>>>>>> sure
>>>>>> it actually results in a better view; haven't tried it.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>> Got feedback? tinyurl.com/swegner-feedback
>>>>
>>>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> <https://goto.google.com/jasonkuster-feedback>
>

Re: [RFC] I made a new tabbed Beam view in Jenkins

[Alan Myrvold]

It does look much better!

On Tue, Dec 18, 2018 at 1:10 PM Ahmet Altay  wrote:

> I like this version, it looks cleaner than the current combined view.
>
> On Tue, Dec 18, 2018 at 12:53 PM Scott Wegner  wrote:
>
>> Very cool. I also didn't realize we had control over the Jenkins "views".
>>
>> We currently lack a decent dashboard to monitor the build health across
>> Beam jenkins jobs and triage failures; this is a step in the right
>> direction.
>>
>> I haven't played with Jenkins views before, but it appears they can be
>> managed via the Job DSL similar to our job definitions [1]:
>>
>> > The DSL execution engine exposes several methods to create Jenkins
>> jobs, views, folders and config files. [..]
>>
>> It would be cool to integrate this into our job config in such a way that
>> we could automatically keep the views up-to-date as jobs are added or
>> renamed.
>>
>> [1] https://github.com/jenkinsci/job-dsl-plugin/wiki/Job-DSL-Commands
>>
>> On Tue, Dec 18, 2018 at 12:35 PM Anton Kedin  wrote:
>>
>>> This is really helpful, didn't realize it was possible. Categories and
>>> contents look reasonable. I think something like this definitely should be
>>> the top-level Beam view.
>>>
>>> Regards,
>>> Anton
>>>
>>> On Tue, Dec 18, 2018 at 12:05 PM Kenneth Knowles 
>>> wrote:
>>>
 Hi all,

 I made a new view to split Beam builds into tabs:
 https://builds.apache.org/view/A-D/view/Beam%20Nested/

  - PostCommit tab includes PostCommit and "PreCommit_.*_Cron" because
 these are actually post-commit jobs; it is a feature not a bug.
  - PreCommit tab includes jobs that have no meaningful history because
 they are just against PRs, commits, phrase triggering
  - Inventory self-explanatory
  - PerformanceTests self-explanatory
  - All; I didn't want to keep making categories but just send this for
 feedback

 WDYT about making this the top-level Beam view? (vs
 https://builds.apache.org/view/A-D/view/Beam/)

 After that, maybe we could clean the categories so they fit into the
 tabs more easily with fewer regexes (to make sure things don't get missed).
 I have read also that if you use / instead of _ as a separator in a name
 then Jenkins will display jobs as nested in folders automatically. Not sure
 it actually results in a better view; haven't tried it.

 Kenn

>>>
>>
>> --
>>
>>
>>
>>
>> Got feedback? tinyurl.com/swegner-feedback
>>
>

Re: INSTANCE_GROUP_MANAGERS quota limit exceeded

[Alan Myrvold]

The quota has been increased to 300.

On Fri, Nov 2, 2018 at 2:26 PM Lukasz Cwik  wrote:

> +1
>
> On Fri, Nov 2, 2018 at 1:54 PM Yifan Zou  wrote:
>
>> +1. I believe there were too many dataflow jobs running at that moment
>> that exceeded the compute engine quota of us-central1 ( the current quota
>> is 100).
>>
>> Yifan
>>
>> On Fri, Nov 2, 2018 at 1:49 PM Alan Myrvold  wrote:
>>
>>> Not sure, but I expect that each dataflow job needs a managed instance
>>> group on the apache-beam-testing project, so it would depend on how many
>>> dataflow jobs are being run in parallel.
>>> The quota for concurrent dataflow jobs is 300, the quota for managed
>>> instance groups is 100. It would make sense to increase the managed
>>> instance group quota or reduce the number of concurrent jobs.
>>>
>>> On Fri, Nov 2, 2018 at 1:26 PM Mikhail Gryzykhin 
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I was looking through (now 33) tickets open based on our post-commit
>>>> tests failures and noticed one interesting.
>>>>
>>>> Test failed due to INSTANCE_GROUP_MANAGERS quota exceeding.
>>>> BEAM-5938 <https://issues.apache.org/jira/browse/BEAM-5938> Thank you
>>>> Daniel for triaging failure.
>>>>
>>>> Even thought this seem to happen only once so far, it might become a
>>>> bigger issue since we are adding more tests.
>>>>
>>>> I've checked on the GCloud instance groups
>>>> <https://cloud.google.com/compute/docs/instance-groups/>, but it
>>>> didn't give me good insight into what might have triggered the quota issue.
>>>>
>>>> Does anyone know how we use instance group managers and triage whether
>>>> this might hit us once we increase amount of tests?
>>>>
>>>> Regards,
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>>

Re: INSTANCE_GROUP_MANAGERS quota limit exceeded

[Alan Myrvold]

Not sure, but I expect that each dataflow job needs a managed instance
group on the apache-beam-testing project, so it would depend on how many
dataflow jobs are being run in parallel.
The quota for concurrent dataflow jobs is 300, the quota for managed
instance groups is 100. It would make sense to increase the managed
instance group quota or reduce the number of concurrent jobs.

On Fri, Nov 2, 2018 at 1:26 PM Mikhail Gryzykhin  wrote:

> Hi everyone,
>
> I was looking through (now 33) tickets open based on our post-commit tests
> failures and noticed one interesting.
>
> Test failed due to INSTANCE_GROUP_MANAGERS quota exceeding.
> BEAM-5938  Thank you
> Daniel for triaging failure.
>
> Even thought this seem to happen only once so far, it might become a
> bigger issue since we are adding more tests.
>
> I've checked on the GCloud instance groups
> , but it didn't
> give me good insight into what might have triggered the quota issue.
>
> Does anyone know how we use instance group managers and triage whether
> this might hit us once we increase amount of tests?
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Re: Never get spotless errors with this one weird trick

[Alan Myrvold]

Thanks for the trick. I added it to
https://cwiki.apache.org/confluence/display/BEAM/Java+Tips

On Thu, Nov 1, 2018 at 2:26 PM Ankur Goenka  wrote:

> Thanks for sharing the trick.
>
>
> On Thu, Nov 1, 2018 at 9:30 AM Kenneth Knowles  wrote:
>
>> Hi all,
>>
>> Scott just separated the spotless check from the Java unit test precommit
>> job, so you get faster feedback on spotless errors.
>>
>> I wondered if there was a good place to just always reformat, and whether
>> it was fast enough to be OK. The answer is yes, and yes.
>>
>> You can set up a git precommit hook to always autoformat code, by putting
>> this in .git/hooks/pre-commit and setting the executable bit.
>>
>> #!/bin/sh
>> set -e
>> ./gradlew spotlessApply
>>
>> If you haven't used git hooks, the docs are here:
>> https://git-scm.com/docs/githooks. I'll call out that --no-verify will
>> skip it and `chmod u-x` will disable it.
>>
>> Then testing the time:
>>
>>  - From a fresh checkout ./gradlew spotlessJavaApply took 24s
>> configuration and 49s spotlessApply
>>  - Then I modified one file in nexmark, messed up the formatting, and
>> committed
>>  - The re-run took 1s in configuration and 4s in spotlessApply
>>
>> So this will add ~5s of waiting each time you `git commit`. You can
>> decide if it is worth it to you. If you are a "push a bunch of commits to
>> be squashed" GitHub user, you could amortize it by making it a pre-push
>> hook that adds a spotless commit (`git commit --fixup HEAD`).
>>
>> Kenn
>>
>

Re: Updated beam contribute page, development tips for languages in wiki

[Alan Myrvold]

I like that change, especially providing all the resources in one spot.
http://apache-beam-website-pull-requests.storage.googleapis.com/6912/contribute/get-help/index.html

On Thu, Nov 1, 2018 at 11:13 AM Kenneth Knowles  wrote:

> To make it concrete and not just critique, I put together
> https://github.com/apache/beam/pull/6912 for consideration.
>
> Kenn
>
> On Thu, Nov 1, 2018 at 10:48 AM Kenneth Knowles  wrote:
>
>> I can accept (but never love) a sidebar that isn't a site map.
>> Personally, I find them disorienting and they lower my confidence in the
>> quality and authoritativeness of the content. But I understand I am just
>> one person, composed of idiosyncracies.
>>
>> I actually think that the contents of Community > Contact Us are not
>> contextualized for an incoming contributor. How about "Contribute > Get
>> Help": a page more focused on contributors, clarifying the role of
>> different communication channels and the PMC/Committers roster for getting
>> a change build and incorporated. It could also subsume the off-site FAQ
>> link.
>>
>> Kenn
>>
>> On Thu, Nov 1, 2018 at 10:16 AM Scott Wegner  wrote:
>>
>>> In the case of "Contact Us", the redundancy is deliberate-- we received
>>> feedback on the Contribution Guide that the content is heavy and it wasn't
>>> obvious how to find help. We wanted the "Contact Us" link as visible as
>>> possible, so the proposed solution was to have it in both sidebars.
>>>
>>> In my opinion I don't think a strict content hierarchy is necessary in
>>> the sidebar, and we should instead focus on providing the right context.
>>> We'll hit additional cases like this as more content moves out to the wiki.
>>>
>>> Some additional details on the original feedback is in JIRA:
>>> https://issues.apache.org/jira/browse/BEAM-5735
>>>
>>> On Thu, Nov 1, 2018 at 10:02 AM Kenneth Knowles  wrote:
>>>
 And the site map structure actually has a couple more issues:

  - Contribute > Roadmap is the same as top-level Roadmap
  - Contribute > Contact Us is the same as Community > Contact Us.
  - Contribute > PMC and Committers is the same as Community > Team
  - Contribute > Policies > * is separate from Community > Policies, for
 good reasons but it looks weird

 I don't mean this only as a critique of the change, but also of the
 prior structure, which seemingly didn't work well.

 Sam/Thomas/Scott also - was it the case that while working through
 contributing it was unclear where these resources could be found? Is there
 a better division here? I think previously the Contribute and Community
 sections were one and the same, but we split them to put more heavyweight
 stuff under Contribute.

 Kenn

>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>> Got feedback? tinyurl.com/swegner-feedback
>>>
>>

Re: Updated beam contribute page, development tips for languages in wiki

[Alan Myrvold]

Great catch. I can update these.

On Thu, Nov 1, 2018 at 9:55 AM Kenneth Knowles  wrote:

> Another thing - for external links in the side menu we have elsewhere used
> an icon to indicate it is not part of our web page structure, which it
> being in the menu would otherwise imply. See the Community > Team link at
> https://beam.apache.org/community/contact-us/
>
> Two such links on the new "contribute" side menu are the FAQ (links to
> wiki) and "PMC and committers" which is the same link as Community > Team.
>
> Kenn
>
> On Thu, Nov 1, 2018 at 9:42 AM Kenneth Knowles  wrote:
>
>> I just walked through it, and this is really great. It really strikes a
>> good balance between detail, flexibility, and simplicity.
>>
>> I notice that it links to a design docs Google Drive folder, versus
>> https://beam.apache.org/contribute/design-documents/ or a wiki page.
>>
>> Kenn
>>
>> On Thu, Nov 1, 2018 at 9:21 AM Alan Myrvold  wrote:
>>
>>> Thanks to feedback from Sam when he got started with Beam, and helpful
>>> suggestions from Thomas and Scott, the beam contribute
>>> <https://beam.apache.org/contribute/> [1] page has been updated to
>>> hopefully make contributions easier, and the beam wiki
>>> <https://cwiki.apache.org/confluence/display/BEAM/Development+Environment+FAQ>
>>>  [2]
>>> has been updated with an FAQ and tips for Java, Python, Go, Gradle, Jenkins
>>> and website contributions.
>>>
>>> The tips pages are a little light, so if you have ideas for developers,
>>> please sign up for the wiki and ask for wiki access on dev@, and add in
>>> your favorite tips.
>>>
>>> Alan
>>>
>>> [1] https://beam.apache.org/contribute/
>>> [2]
>>> https://cwiki.apache.org/confluence/display/BEAM/Development+Environment+FAQ
>>>
>>

Re: [ANNOUNCE] New committer announcement, Euphoria edition

[Alan Myrvold]

Congrats! Euphoria for beam
 looks awesome.

On Thu, Nov 1, 2018 at 9:49 AM Ahmet Altay  wrote:

> Congratulations!
>
> On Thu, Nov 1, 2018 at 9:36 AM, Tim  wrote:
>
>> Congratulations and welcome!
>>
>> Tim
>>
>> On 1 Nov 2018, at 17:06, Matthias Baetens 
>> wrote:
>>
>> Congrats David!!!
>>
>> On Thu, Nov 1, 2018, 16:04 Kenneth Knowles  wrote:
>>
>>> Hi all,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new committer
>>> :
>>>
>>>  - David Morávek: of, but not limited to, the new Euphoria API
>>>
>>> Through his work with us merging the Euphoria API, community outreach,
>>> and other contributions to Beam, the PMC trusts David with the
>>> responsibilities of a Beam committer [1].
>>>
>>> Kenn
>>>
>>> [1] https://beam.apache.org/contribute/become-a-committer/
>>> #an-apache-beam-committer
>>>
>>> --
>>
>>
>>
>

Updated beam contribute page, development tips for languages in wiki

[Alan Myrvold]

Thanks to feedback from Sam when he got started with Beam, and helpful
suggestions from Thomas and Scott, the beam contribute
 [1] page has been updated to
hopefully make contributions easier, and the beam wiki

[2]
has been updated with an FAQ and tips for Java, Python, Go, Gradle, Jenkins
and website contributions.

The tips pages are a little light, so if you have ideas for developers,
please sign up for the wiki and ask for wiki access on dev@, and add in
your favorite tips.

Alan

[1] https://beam.apache.org/contribute/
[2]
https://cwiki.apache.org/confluence/display/BEAM/Development+Environment+FAQ

Edit access to the Apache Beam Confluence Wiki?

[Alan Myrvold]

Can I get edit access to the Apache Beam Confluence Wiki,
https://cwiki.apache.org/confluence/display/BEAM ?

I'd like to move some FAQ around contributing to the wiki.

Thanks
Alan

Re: Is it possible to be added to the apache beam testing group?

[Alan Myrvold]

I've added you to the apache-beam-jenkins account. We try to limit the
users with access and expect that the test results are debuggable.
Once you find the issue, if there are any testability improvements to make,
please let me know.

On Fri, Oct 26, 2018 at 1:59 PM Alex Amato  wrote:

> I was trying to rerun a test which failed precommit
> . I wanted to run this test
> locally
> (:beam-runners-google-cloud-dataflow-java-examples-streaming:preCommit)
>
> Luke mentioned that it would be easier to debug if I were added to the
> testing group. So that I can run test with the apache-beam-testing project.
> Then I could run these tests out of the box, instead of needing to change
> the GCP project and bucket every time I run these tests.
>
> If it's not too much trouble, it would help my productivity a little bit,
> if I could please be added to the testing group
>
> Thank you
> Alex
>
>
>

New Edit button on beam.apache.org pages

[Alan Myrvold]

To make small documentation changes easier, there is now an Edit button at
the top right of the pages on https://beam.apache.org. This button opens
the source .md file on the master branch of the beam repository in the
github web editor. After making changes you can create a pull request to
ask to have it merged.

Thanks to Scott for the suggestion to add this in [BEAM-4431]


Let me know if you run into any issues.

Alan

Re: python tests run as part of 'javaPreCommit'

[Alan Myrvold]

Good idea on having a flag to make the test run without GCP credentials.
I logged this as https://issues.apache.org/jira/browse/BEAM-5771 and can
look into this, unless someone else prefers.

On Tue, Oct 16, 2018 at 9:18 AM Scott Wegner  wrote:

> There was some discussion recently about ensuring anyone can easily run
> and reproduce precommit test results locally. The precommits run Dataflow
> jobs, which will fail if you don't have access to an Google Cloud project.
> One idea would be to add a flag to disable Google Cloud tests, i.e.
> ./gradlew :javaPreCommit -PdisableGcpTests
>
> On Tue, Oct 16, 2018 at 8:57 AM Kenneth Knowles  wrote:
>
>> Yes, it is exactly that. The :javaPreCommit is a deliberate attempt to
>> make a single task that runs all the tests that Jenkins runs, so it
>> includes some lightweight smoke tests on runners, including Google Cloud
>> Dataflow.
>>
>> With maven it was impossible to have a single mvn invocation that would
>> build what was necessary but only run the ITs so it was necessary to
>> conflate the two (we would use a bash Jenkins job with a few mvn
>> invocations in series, losing the Jenkins Maven Plugin integration).
>>
>> With gradle it is easy, so we can separate them and IMO should do so. It
>> does add a tiny bit of redundant build time.
>>
>> Kenn
>>
>> On Tue, Oct 16, 2018 at 8:36 AM Colm O hEigeartaigh 
>> wrote:
>>
>>> Thanks Kenn, rookie mistake on my part :-)
>>>
>>> A further question if I may - "./gradlew :javaPreCommit" is failing for
>>> me with:
>>>
>>> org.apache.beam.examples.WindowedWordCountIT >
>>> testWindowedWordCountInBatchDynamicSharding FAILED
>>> org.apache.beam.sdk.Pipeline$PipelineExecutionException at
>>> WindowedWordCountIT.java:188
>>>
>>> 4 tests completed, 4 failed
>>>
>>> > Task :beam-examples-java:directRunnerPreCommit FAILED
>>>
>>> Looking at the report I see:
>>>
>>> "Caused by: java.lang.RuntimeException: Unable to get application
>>> default credentials. Please see
>>> https://developers.google.com/accounts/docs/application-default-credentials
>>> for details on how to specify credentials. This version of the SDK is
>>> dependent on the gcloud core component version 2015.02.05 or newer to be
>>> able to get credentials from the currently authorized user via gcloud auth."
>>>
>>> It looks like some of the examples require google credentials to run
>>> properly?
>>>
>>> Colm.
>>>
>>> On Tue, Oct 16, 2018 at 4:07 PM Kenneth Knowles  wrote:
>>>
 One thing to clarify is that `:javaPreCommit` is a task and `build` is
 another task. There's so verb-object relationship in your commandline. So
 as written, you've asked for a whole-project `build`, which weirdly in
 Gradle means "build and test". Since it is one commandline, all the
 necessary steps for both tasks will be in one dependency graph so they
 won't be executed twice.

 Kenn

 On Tue, Oct 16, 2018 at 6:11 AM Colm O hEigeartaigh <
 cohei...@apache.org> wrote:

> Hi all,
>
> Just a quick question - I was wondering why the python tests/build run
> as part of the 'javaPreCommit' task?
>
> i.e. executing "./gradlew build :javaPreCommit" leads to python tests
> being run as well, which is not something you might expect from the name 
> of
> the task.
>
> Colm.
>
>
> --
> Colm O hEigeartaigh
>
> Talend Community Coder
> http://coders.talend.com
>

>>>
>>> --
>>> Colm O hEigeartaigh
>>>
>>> Talend Community Coder
>>> http://coders.talend.com
>>>
>>
>
> --
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
>

Re: Modular IO presentation at Apachecon

[Alan Myrvold]

Thanks for the slides.
Really enjoyed the talk in person, especially the concept that IO is a
transformation, and a source or sink are not special and the splittable
DoFn explanation.

On Wed, Sep 26, 2018 at 2:17 PM Ismaël Mejía  wrote:

> Hello, today Eugene and me did a talk about about modular APIs for IO
> at ApacheCon. This talk introduces some common patterns that we have
> found while creating IO connectors and also presents recent ideas like
> dynamic destinations, sequential writes among others using FileIO as a
> use case.
>
> In case you guys want to take a look, here is a copy of the slides, we
> will probably add this to the IO authoring documentation too.
>
> https://s.apache.org/beam-modular-io-talk
>

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

[Alan Myrvold]

Congrats, Kenn.

On Wed, Sep 19, 2018 at 1:08 PM Maximilian Michels  wrote:

> Congrats!
>
> On 19.09.18 22:07, Robin Qiu wrote:
> > Congratulations, Kenn!
> >
> > On Wed, Sep 19, 2018 at 1:05 PM Lukasz Cwik  > > wrote:
> >
> > Congrats Kenn.
> >
> > On Wed, Sep 19, 2018 at 12:54 PM Davor Bonaci  > > wrote:
> >
> > Hi everyone --
> > It is with great pleasure that I announce that at today's
> > meeting of the Foundation's Board of Directors, the Board has
> > appointed Kenneth Knowles as the second chair of the Apache Beam
> > project.
> >
> > Kenn has served on the PMC since its inception, and is very
> > active and effective in growing the community. His exemplary
> > posts have been cited in other projects. I'm super happy to have
> > Kenn accepted the nomination, and I'm confident that he'll serve
> > with distinction.
> >
> > As for myself, I'm not going anywhere. I'm still around and will
> > be as active as I have recently been. Thrilled to be able to
> > pass the baton to such a key member of this community and to
> > have less administrative work to do ;-).
> >
> > Please join me in welcoming Kenn to his new role, and I ask that
> > you support him as much as possible. As always, please let me
> > know if you have any questions.
> >
> > Davor
> >
>

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

[Alan Myrvold]

+1

On Fri, Sep 14, 2018 at 3:16 PM Boyuan Zhang  wrote:

> +1
>
> On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde  wrote:
>
>> +1
>>
>> On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay  wrote:
>>
>>> +1 (binding)
>>>
>>> On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik  wrote:
>>>
 +1 (binding)

 On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada 
 wrote:

> +1
>
> On Fri, Sep 14, 2018 at 2:32 PM Andrew Pilloud 
> wrote:
>
>> +1
>>
>> On Fri, Sep 14, 2018 at 2:31 PM Lukasz Cwik  wrote:
>>
>>> There was generally positive support and good feedback[1] but it was
>>> not unanimous. I wanted to bring the donation of the Dataflow worker 
>>> code
>>> base to Apache Beam master to a vote.
>>>
>>> +1: Support having the Dataflow worker code as part of Apache Beam
>>> master branch
>>> -1: Dataflow worker code should live elsewhere
>>>
>>> 1:
>>> https://lists.apache.org/thread.html/89efd3bc1d30f3d43d4b361a5ee05bd52778c9dc3f43ac72354c2bd9@%3Cdev.beam.apache.org%3E
>>>
>>
>>>

Re: Should we allow ValidatesRunner tests to have access to file systems?

[Alan Myrvold]

I think this should be an integration test if it requires more access than
the current ValidatesRunner tests.

Although the ValidatesRunner and integration tests are similar, the intent
is that the validates runner tests are smaller and more like component
tests, and there have been discusions on fusing the validates runner tests
into a smaller set of pipelines.

On Mon, Aug 27, 2018 at 11:27 AM Robin Qiu  wrote:

> Hello everyone,
>
> I am writing a test [1] for the support of @RequiresStableInput annotation
> in Java SDK [2]. For the test to work, I need to have a ParDo make some
> side effect (e.g. writing to a file system). However, ValidatesRunner tests
> in Beam currently cannot depend on external states (cannot write to file
> systems). So I am wondering if it is a good idea to allow ValidatesRunner
> tests to have access to file systems. This way we can create more flexible
> ValidatesRunner tests.
>
> I could make this test a integration test to get access to file systems
> (e.g. like WordCountIT.java [3]). But functionally I think this test should
> be a ValidatesRunner test, because it is testing the support of some SDK
> features on runners.
>
> So what do you think? Any suggestions or concerns are appreciated.
>
> Best,
> Robin
>
> [1] https://github.com/apache/beam/pull/6220
> [2]
> https://docs.google.com/document/d/117yRKbbcEdm3eIKB_26BHOJGmHSZl1YNoF0RqWGtqAM/edit#
> [3]
> https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/WordCountIT.java
>
>

Re: [Proposal] Track non-code contributions in Jira

[Alan Myrvold]

I like the idea of recognizing non-code contributions. These other efforts
have been very helpful.

On Thu, Aug 23, 2018 at 3:07 PM Griselda Cuevas  wrote:

> Hi Beam Community,
>
> I'd like to start tracking non-code contributions for Beam, specially
> around these six categories:
> 1) Project Management
> 2) Community Management
> 3) Advocacy
> 4) Events & Meetups
> 5) Documentation
> 6) Training
>
> The proposal would be to create six boards in Jira, one per proposed
> category, and as part of this initiative also clean the already existing
> "Project Management" component, i.e. making sure all issues there are still
> relevant.
>
> After this, I'd also create a landing page in the website that talks about
> all types of contributions to the project.
>
> The reason for doing this is mainly to give visibility to some of the
> great work our community does beyond code pushes in Github. Initiatives
> around Beam are starting to spark around the world, and it'd be great to
> become an Apache project recognized for our outstanding community
> recognition.
>
> What are your thoughts?
> G
>
> Gris
>

Re: Policy for Python ValidatesRunner vs IT tests?

[Alan Myrvold]

The ValidatesContainer test builds the python portable worker container
using docker, pushes it, then runs a dataflow pipeline using the container
with the flag --worker_harness_container_image. There is only one test that
does this, and it is also an 'IT' test.

On Tue, Aug 14, 2018 at 3:27 PM Pablo Estrada  wrote:

> Hello,
> In Python, we tag some test methods with @attr('ValidatesRunner') and
> @attr('IT'), which marks them to be run as pipeline tests.
>
> If I understand correctly:
> - ValidatesRunner tests are more like a component tests[1] as explained in
> Beam docs
> - IT tests are more like a E2E test[2] as explained in the docs. Is there
> an equivalent in Java?
> - Finally, there's ValidatesContainer tests. What are these for? What's
> the guidance for tagging our tests this way?
>
> Thanks!
> -P.
>
> [1] https://beam.apache.org/contribute/testing/#validatesrunner
> [2] https://beam.apache.org/contribute/testing/#e2e
> --
> Got feedback? go/pabloem-feedback
> 
>

Re: Java precommit and postcommit tests are failing

[Alan Myrvold]

There were 2 reasons for the failing java precommit; there was a consistent
compile break that will be fixed by
https://github.com/apache/beam/pull/6018 and
there was a change, presumably due to the jenkins change, in the number of
workers computed based on the number of virtual processors detected which
caused the -Xmx gradle jvm flag to decrease from 3g to 1g.

After adjusting maxWorkers (and the memory settings), the precommit passed
for pull/6018 at
https://builds.apache.org/job/beam_PreCommit_Java_Commit/517/

Setting reasonable values for maxWorkers and Xms/Xmx is being tested in
https://github.com/apache/beam/pull/6046



On Tue, Jul 24, 2018 at 9:21 AM Alan Myrvold  wrote:

> I can take a look at it today
>
> On Mon, Jul 23, 2018 at 10:32 PM Rui Wang  wrote:
>
>> It is at least frequent (and event consistent) based on the list of
>> failed PR which run java checks.
>>
>>
>> Also, this issue is being tracked by
>> https://issues.apache.org/jira/browse/BEAM-4847.
>>
>>
>> -Rui
>>
>> On Mon, Jul 23, 2018 at 9:55 PM Jean-Baptiste Onofré 
>> wrote:
>>
>>> Hi Rui
>>>
>>> I will. I guess it randomly happens, right ?
>>>
>>> Regards
>>> JB
>>> Le 24 juil. 2018, à 02:37, Rui Wang  a écrit:
>>>>
>>>> Hi community,
>>>>
>>>> Seems like both Java precommit and postcommit tests are failing due to
>>>> GC issues. I created a JIRA under test-failures component (
>>>> https://issues.apache.org/jira/browse/BEAM-4848). Could someone who is
>>>> familiar with Jenkins take a look at this?
>>>>
>>>>
>>>> Thanks,
>>>> Rui
>>>>
>>>

Re: Java precommit and postcommit tests are failing

[Alan Myrvold]

I can take a look at it today

On Mon, Jul 23, 2018 at 10:32 PM Rui Wang  wrote:

> It is at least frequent (and event consistent) based on the list of failed
> PR which run java checks.
>
>
> Also, this issue is being tracked by
> https://issues.apache.org/jira/browse/BEAM-4847.
>
>
> -Rui
>
> On Mon, Jul 23, 2018 at 9:55 PM Jean-Baptiste Onofré 
> wrote:
>
>> Hi Rui
>>
>> I will. I guess it randomly happens, right ?
>>
>> Regards
>> JB
>> Le 24 juil. 2018, à 02:37, Rui Wang  a écrit:
>>>
>>> Hi community,
>>>
>>> Seems like both Java precommit and postcommit tests are failing due to
>>> GC issues. I created a JIRA under test-failures component (
>>> https://issues.apache.org/jira/browse/BEAM-4848). Could someone who is
>>> familiar with Jenkins take a look at this?
>>>
>>>
>>> Thanks,
>>> Rui
>>>
>>

Re: [PROPOSAL] Prepare Beam 2.6.0 release

[Alan Myrvold]

+1 Thanks for volunteering, Pablo

On Wed, Jul 11, 2018 at 11:49 AM Jason Kuster 
wrote:

> +1 sounds great
>
> On Wed, Jul 11, 2018 at 11:06 AM Thomas Weise  wrote:
>
>> +1
>>
>> Thanks for volunteering, Pablo!
>>
>> On Mon, Jul 9, 2018 at 9:56 PM Jean-Baptiste Onofré 
>> wrote:
>>
>>> +1
>>>
>>> I planned to send the proposal as well ;)
>>>
>>> Regards
>>> JB
>>>
>>> On 09/07/2018 23:16, Pablo Estrada wrote:
>>> > Hello everyone!
>>> >
>>> > As per the previously agreed-upon schedule for Beam releases, the
>>> > process for the 2.6.0 Beam release should start on July 17th.
>>> >
>>> > I volunteer to perform this release.
>>> >
>>> > Here is the schedule that I have in mind:
>>> >
>>> > - We start triaging JIRA issues this week.
>>> > - I will cut a release branch on July 17.
>>> > - After July 17, any blockers will need to be cherry-picked into the
>>> > release branch.
>>> > - As soon as tests look good, and blockers have been addressed, I will
>>> > perform the other release tasks.
>>> >
>>> > Does that seem reasonable to the community?
>>> >
>>> > Best
>>> > -P.
>>> > --
>>> > Got feedback? go/pabloem-feedback
>>> 
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

Re: Broken seed job

[Alan Myrvold]

The outage likely fixed it.
I would have expected running the standalone job with the trigger phrase
should have fixed it. Did it not?

On Wed, Jul 11, 2018 at 8:45 AM Lukasz Cwik  wrote:

> I believe there was an outage a few hours ago.
>
> On Wed, Jul 11, 2018 at 8:36 AM Łukasz Gajowy 
> wrote:
>
>> It's totally fine. The problem is now gone even though I have taken no
>> action to fix it. I suppose Jenkins was restarted?
>>
>> BTW: Is it restarted only on demand by those who have access, or it's
>> done once in a while (periodically)?
>>
>> śr., 11 lip 2018 o 17:07 Lukasz Cwik  napisał(a):
>>
>>> Ah, sorry for my confusion.
>>>
>>> On Tue, Jul 10, 2018 at 4:59 PM Łukasz Gajowy 
>>> wrote:
>>>
 I didn't edit the "Standalone Seed job", only the "SeedJob". Now every
 time someone tries to run the seed job ("Run seed job") it results in an
 error even despite prior running the standalone job from master branch the
 way you described.



 śr., 11 lip 2018 o 01:25 Lukasz Cwik  napisał(a):

> job_seed_standalone should only be edited when we know that the
> regular seed job is in a healthy state.
>
> Note, that you can always recover back to what is checked in master by:
> 1) Creating an empty PR
> 2) Using the standalone seed job trigger phrase: "Run Standalone Seed
> Job"
>
> I kicked one off right now:
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob_Standalone/1289/
>
> On Tue, Jul 10, 2018 at 4:17 PM Łukasz Gajowy 
> wrote:
>
>> Hi,
>>
>> while working on Jenkins' seed job, some changes I introduced didn't
>> get reverted even after running the seed job again from the
>> master branch. This is why seed job is now failing. More details here [1]
>> and here [2].
>>
>> Since I can operate only with jobs phrase-triggered from GitHub's PR,
>> I think there's nothing more I can do than I already tried. In my
>> opinion, fixing this issue requires an aid of a person with some greater
>> access to Jenkins. Can someone help with that?
>>
>> Sorry for the inconvenience - I didn't expect that such situation can
>> occur. "job_seed_standalone" works fine and it can be used instead
>> (until the issue is fixed).
>>
>> [1] https://github.com/apache/beam/pull/5915
>> [2]
>> https://builds.apache.org/view/A-D/view/Beam/job/beam_SeedJob/2190/console
>>
>>
>> Best regards,
>> Łukasz
>>
>

Proposal: Apache Beam Contributor Metrics: Collection, Display, Actions

[Alan Myrvold]

I am proposing an html dashboard for some key contributor metrics,
initially the weekly passing rate of post-commit tests and the speed of
pre-commit tests, then expanding to code review and release metrics.

https://s.apache.org/beam-contributor-metrics

Comments welcome.

Alan

Re: Auto closing stale PRs label

[Alan Myrvold]

Glad to see it is working. The requests currently marked stale can be found
with https://github.com/apache/beam/labels/stale

On Tue, Jun 26, 2018 at 9:34 PM Rafael Fernandez 
wrote:

> Neat! Thanks for showing me where the options are.
>
> On Tue, Jun 26, 2018 at 7:24 PM Kenneth Knowles  wrote:
>
>> That's actually already how it works. We can configure how long it waits
>> after the message. Currently it is set for 60 day to stale and then 7 days
>> to close. You can see the options we've set up here; there may be more:
>> https://github.com/apache/beam/blob/master/.github/stale.yml
>>
>> Kenn
>>
>> On Tue, Jun 26, 2018 at 6:42 PM Rafael Fernandez 
>> wrote:
>>
>>> The new label makes sense to me, but Ismael: I want to make sure your
>>> concern is fully addressed. I see your point about making sure we are not
>>> shutting the door on a small fix that perhaps went unatended for benign
>>> reasons. Perhaps a step before closure is feasble? something like getting a
>>> nice message in the PR, "Ahoy! This PR hasn't moved in [X time]. If you're
>>> still working on it, can you comment? Otherwise, our highly sophisticated
>>> AI will declutter and close it in [Y days]".
>>>
>>> Thoughts?
>>>
>>>
>>> On Mon, Jun 25, 2018 at 8:23 AM Kenneth Knowles  wrote:
>>>
 Totally agree.

 By the way, these seem to be default labels for issue tracking. So I
 got rid of the ones that don't seem to make sense. Any committer can hack
 them I think. I just left "stale" for this purpose and "help wanted" since
 that makes sense on a PR. But probably we don't need any since we don't
 have a plan for them.

 Kenn

 On Mon, Jun 25, 2018 at 8:12 AM Ismaël Mejía  wrote:

> Thanks Kenn, much better.
>
> Yes closing stale PRs is worth, but our ultimate goal should be to get
> contributions in so we should keep in mind and try when it is worth to
> rescue fixes that can be lost  because of minor review issues or
> contributor inactivity.
>
> On Mon, Jun 25, 2018 at 4:23 PM Kenneth Knowles 
> wrote:
>
>> It is configured by just a file so alteration is very transparent. I
>> agree with your point about the label. I made a new one for it. Here:
>> https://github.com/apache/beam/pull/5750
>>
>> So far I have been satisfied that it close many _very_ stale PRs. I
>> have been watching it and didn't see any that seemed wrong.
>>
>> Kenn
>>
>>
>> On Mon, Jun 25, 2018 at 12:52 AM Ismaël Mejía 
>> wrote:
>>
>>> I saw some PRs auto closed recently and I was wondering if we could
>>> adjust the  label that is added to the autoclosed PRs, currently it
>>> is
>>> 'wontfix' but this label sends a fake (and negative) message. Can we
>>> parametrize the bot to put something closer to the intention like
>>> 'autoclosed'?
>>>
>>> Who can take care of this?
>>> Any other opinion/suggestion after these first days of the stale bot?
>>>
>>> I have the impression that the time between the staleness warning and
>>> the close is relatively short, of course PRs can be reopened but we
>>> (committers) should pay attention that a PR that is marked as stale
>>> is
>>> not stale because of unfinished reviews.
>>>
>>

Re: Going on leave for a bit

[Alan Myrvold]

Congrats, Kenn and have a great break.

On Tue, Jun 26, 2018 at 9:10 AM Alexey Romanenko 
wrote:

> Best wishes to you and your family, Kenneth!
>
> Alexey
>
> On 26 Jun 2018, at 15:45, Rafael Fernandez  wrote:
>
> Have a great time, Kenn!
>
> On Tue, Jun 26, 2018 at 5:49 AM Ismaël Mejía  wrote:
>
>> Enjoy your family time.
>>
>> Best wishes,
>> Ismael
>>
>>
>> On Tue, Jun 26, 2018 at 12:13 PM Pei HE  wrote:
>>
>>> (A late) Congrats for the newborn!
>>> --
>>> Pei
>>>
>>> On Tue, Jun 26, 2018 at 1:42 PM, Kenneth Knowles  wrote:
>>> > Hi friends,
>>> >
>>> > I think I did not mention on dev@ at the time, but my child #2
>>> arrived March
>>> > 14 (Pi day!) and I took some weeks off. Starting ~July 4 I will be
>>> taking a
>>> > more significant absence, until ~October 1, trying my best to be
>>> totally
>>> > offline.
>>> >
>>> > JFYI so that you know why JIRAs and PRs are not being addressed. I am
>>> also
>>> > unassigning my JIRAs so that I am not holding any mutexes, and I will
>>> close
>>> > PRs so they don't get stale.
>>> >
>>> > Any questions or pressing issues, I will be online this week and a
>>> little
>>> > bit next week.
>>> >
>>> > Kenn
>>>
>>
>

Re: [RESULT][VOTE] Apache Beam, version 2.5.0, release candidate #2

[Alan Myrvold]

It would be a more predictable cadence to have a consistent timing between
when the release branches are cut, and not when the release is published.

If 2.5.0 was cut on June 5, then 2.6.0 could be cut July 17?

On Mon, Jun 25, 2018 at 10:57 AM Ahmet Altay  wrote:

>
>
> On Mon, Jun 25, 2018 at 10:49 AM, Ahmet Altay  wrote:
>
>> JB, thank you for making this release happen.
>>
>> I noticed that python artifacts are not deployed to pypi yet. Would you
>> like me to do that?
>>
>> Thank you,
>> Ahmet
>>
>> On Sat, Jun 23, 2018 at 6:45 AM, Rafael Fernandez 
>> wrote:
>>
>>> Great news! Thanks so much to our Release Manager and everybody who
>>> helped iron out the wrinkles!
>>>
>>> If you haven't seen it already, look for the thread "[PROPOSAL] Add a
>>> blog post for Beam release 2.5.0
>>> ​" [1]​
>>> in dev@  - Alexey Romanenko has put together a very nice summary of all
>>> the good stuff in 2.5.0.
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/ae3284ca051b800b3edd73ad0f7f62344e26d3957b46794149bf1fb2@%3Cdev.beam.apache.org%3E
>>>
>>>
>>> "
>>>
>>> On Fri, Jun 22, 2018 at 8:33 PM Jean-Baptiste Onofré 
>>> wrote:
>>>
 I meant August (not July) for next release cycle.

 Regards
 JB

 On 23/06/2018 05:17, Jean-Baptiste Onofré wrote:
 > Hi all,
 >
 > I'm happy to announce that we have unanimously approved this release.
 >
 > There are 12 approving votes, 5 of which are binding:
 > * Ahmet Altay
 > * Jean-Baptiste Onofré
 > * Lukasz Cwik
 > * Reuven Lax
 > * Robert Bradshaw
 >
 > There are no disapproving votes.
 >
 > I'm finalizing the release.
 >
 > Thanks everyone!
 >
 > The 2.6.0 release process is expected to begin in 6 weeks. So we
 should
 > start the Jira triage on Saturday, 4th July and I would like to start
 > the release process on Tuesday 7th.

>>>
> My understanding was that there will be a release every 6 weeks. It seems
> like with this plan (to start release process on August 7), the
> understanding is that we will have 6 weeks between a release is out and the
> start of the next release process. My take is, we need to have a user
> centric view and a make promise to release every X weeks. If X=6 is not
> sustainable we can discuss. However, having X weeks in between a release
> cut and release start will result in unpredictable release dates.
>
> What do you think?
>
>
>> >
 > Regards
 > JB
 >
 >
 > On 17/06/2018 07:18, Jean-Baptiste Onofré wrote:
 >> Hi everyone,
 >>
 >> Please review and vote on the release candidate #2 for the version
 >> 2.5.0, as follows:
 >>
 >> [ ] +1, Approve the release
 >> [ ] -1, Do not approve the release (please provide specific comments)
 >>
 >> NB: this is the first release using Gradle, so don't be too harsh ;)
 A
 >> PR about the release guide will follow thanks to this release.
 >>
 >> The complete staging area is available for your review, which
 includes:
 >> * JIRA release notes [1],
 >> * the official Apache source release to be deployed to
 dist.apache.org
 >> [2], which is signed with the key with fingerprint C8282E76 [3],
 >> * all artifacts to be deployed to the Maven Central Repository [4],
 >> * source code tag "v2.5.0-RC2" [5],
 >> * website pull request listing the release and publishing the API
 >> reference manual [6].
 >> * Java artifacts were built with Gradle 4.7 (wrapper) and
 OpenJDK/Oracle
 >> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
 >> * Python artifacts are deployed along with the source release to the
 >> dist.apache.org [2].
 >>
 >> The vote will be open for at least 72 hours. It is adopted by
 majority
 >> approval, with at least 3 PMC affirmative votes.
 >>
 >> Thanks,
 >> JB
 >>
 >> [1]
 >>
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12342847
 >> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
 >> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
 >> [4]
 https://repository.apache.org/content/repositories/orgapachebeam-1043/
 >> [5] https://github.com/apache/beam/tree/v2.5.0-RC2
 >> [6] https://github.com/apache/beam-site/pull/463
 >>
 >

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com

>>>
>>
>

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

[Alan Myrvold]

+1
Thanks for removing the jar, Robert.

On Fri, Jun 22, 2018 at 11:48 AM Robert Bradshaw 
wrote:

> I was waiting for the gradlew jar file to be removed from the source
> release, but I figured it made more sense to just jump in and help
> out.
>
> $ git show-ref github/release-2.5.0
> 093ac64bc2d59f3d58c0f0e7f65f2a36eaf26a4a refs/remotes/github/release-2.5.0
>
> $ wget https://github.com/apache/beam/archive/release-2.5.0.zip -O
> apache-beam-2.5.0-source-release.zip
> $ zip -d apache-beam-2.5.0-source-release.zip
> beam-release-2.5.0/gradle/wrapper/gradle-wrapper.jar
> [signed and uploaded]
>
> I also uploaded the wheels to dist (which I worked on with Boyuan, and
> were verified by Yifan), putting all the python artifacts in a python
> sub-directory.
>
> I can now say +1 (binding) to this release.
> On Fri, Jun 22, 2018 at 9:47 AM Ahmet Altay  wrote:
> >
> > I already voted a +1 2 days ago.
> >
> > Again:
> >
> > +1 (binding)
> >
> > Thank you JB!
> >
> > On Fri, Jun 22, 2018 at 9:23 AM, Alan Myrvold 
> wrote:
> >>
> >> The
> https://dist.apache.org/repos/dist/dev/beam/2.5.0/apache-beam-2.5.0-source-release.zip
> still contains the beam-release-2.5.0/gradle/wrapper/gradle-wrapper.jar.
> Does that need to be removed before releasing it?
> >>
> >> On Fri, Jun 22, 2018 at 9:11 AM Reuven Lax  wrote:
> >>>
> >>> +1 (binding)
> >>>
> >>> On Fri, Jun 22, 2018 at 7:51 AM Jean-Baptiste Onofré 
> wrote:
> >>>>
> >>>> Gently reminder for the PMC member.
> >>>>
> >>>> Right now, if I'm not wrong, we only have two concrete binding votes
> >>>> (Luke and I).
> >>>>
> >>>> As reminder, to pass, the vote needs at least 3 binding votes (a
> binding
> >>>> vote is a vote from a PMC member).
> >>>>
> >>>> So, we would need an additional formal binding vote.
> >>>>
> >>>> NB: I prepared the apache-beam-wheels on my github and trigger a build
> >>>> on Travis. However, I have a build issue that I have to investigate.
> >>>>
> >>>> Thanks,
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 17/06/2018 07:18, Jean-Baptiste Onofré wrote:
> >>>> > Hi everyone,
> >>>> >
> >>>> > Please review and vote on the release candidate #2 for the version
> >>>> > 2.5.0, as follows:
> >>>> >
> >>>> > [ ] +1, Approve the release
> >>>> > [ ] -1, Do not approve the release (please provide specific
> comments)
> >>>> >
> >>>> > NB: this is the first release using Gradle, so don't be too harsh
> ;) A
> >>>> > PR about the release guide will follow thanks to this release.
> >>>> >
> >>>> > The complete staging area is available for your review, which
> includes:
> >>>> > * JIRA release notes [1],
> >>>> > * the official Apache source release to be deployed to
> dist.apache.org
> >>>> > [2], which is signed with the key with fingerprint C8282E76 [3],
> >>>> > * all artifacts to be deployed to the Maven Central Repository [4],
> >>>> > * source code tag "v2.5.0-RC2" [5],
> >>>> > * website pull request listing the release and publishing the API
> >>>> > reference manual [6].
> >>>> > * Java artifacts were built with Gradle 4.7 (wrapper) and
> OpenJDK/Oracle
> >>>> > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>>> > * Python artifacts are deployed along with the source release to the
> >>>> > dist.apache.org [2].
> >>>> >
> >>>> > The vote will be open for at least 72 hours. It is adopted by
> majority
> >>>> > approval, with at least 3 PMC affirmative votes.
> >>>> >
> >>>> > Thanks,
> >>>> > JB
> >>>> >
> >>>> > [1]
> >>>> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12342847
> >>>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1043/
> >>>> > [5] https://github.com/apache/beam/tree/v2.5.0-RC2
> >>>> > [6] https://github.com/apache/beam-site/pull/463
> >>>> >
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbono...@apache.org
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >
> >
>

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #2

[Alan Myrvold]

The
https://dist.apache.org/repos/dist/dev/beam/2.5.0/apache-beam-2.5.0-source-release.zip
still contains the beam-release-2.5.0/gradle/wrapper/gradle-wrapper.jar.
Does that need to be removed before releasing it?

On Fri, Jun 22, 2018 at 9:11 AM Reuven Lax  wrote:

> +1 (binding)
>
> On Fri, Jun 22, 2018 at 7:51 AM Jean-Baptiste Onofré 
> wrote:
>
>> Gently reminder for the PMC member.
>>
>> Right now, if I'm not wrong, we only have two concrete binding votes
>> (Luke and I).
>>
>> As reminder, to pass, the vote needs at least 3 binding votes (a binding
>> vote is a vote from a PMC member).
>>
>> So, we would need an additional formal binding vote.
>>
>> NB: I prepared the apache-beam-wheels on my github and trigger a build
>> on Travis. However, I have a build issue that I have to investigate.
>>
>> Thanks,
>> Regards
>> JB
>>
>> On 17/06/2018 07:18, Jean-Baptiste Onofré wrote:
>> > Hi everyone,
>> >
>> > Please review and vote on the release candidate #2 for the version
>> > 2.5.0, as follows:
>> >
>> > [ ] +1, Approve the release
>> > [ ] -1, Do not approve the release (please provide specific comments)
>> >
>> > NB: this is the first release using Gradle, so don't be too harsh ;) A
>> > PR about the release guide will follow thanks to this release.
>> >
>> > The complete staging area is available for your review, which includes:
>> > * JIRA release notes [1],
>> > * the official Apache source release to be deployed to dist.apache.org
>> > [2], which is signed with the key with fingerprint C8282E76 [3],
>> > * all artifacts to be deployed to the Maven Central Repository [4],
>> > * source code tag "v2.5.0-RC2" [5],
>> > * website pull request listing the release and publishing the API
>> > reference manual [6].
>> > * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>> > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>> > * Python artifacts are deployed along with the source release to the
>> > dist.apache.org [2].
>> >
>> > The vote will be open for at least 72 hours. It is adopted by majority
>> > approval, with at least 3 PMC affirmative votes.
>> >
>> > Thanks,
>> > JB
>> >
>> > [1]
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12342847
>> > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1043/
>> > [5] https://github.com/apache/beam/tree/v2.5.0-RC2
>> > [6] https://github.com/apache/beam-site/pull/463
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1

[Alan Myrvold]

+1 (non-binding)

tested some of the quickstarts

On Sun, Jun 10, 2018 at 1:39 AM Tim  wrote:

> Tested by our team:
> - mvn inclusion
> - Avro, ES, Hadoop IF IO
> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> - Reviewed release notes
>
> +1
>
> Thanks also to everyone who helped get over the gradle hurdle and in
> particular to JB.
>
> Tim
>
> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré  wrote:
> >
> > No problem Pablo.
> >
> > The vote period is a minimum, it can be extended as requested or if we
> > don't have the minimum of 3 binding votes.
> >
> > Regards
> > JB
> >
> >> On 09/06/2018 01:54, Pablo Estrada wrote:
> >> Hello all,
> >> I'd like to request an extension of the voting period until Monday
> >> evening (US time, so later in other geographical regions). This is
> >> because we were only now able to publish Dataflow Workers, and have not
> >> had the chance to run release validation tests on them. The extension
> >> will allow us to validate and vote by Monday.
> >> Is this acceptable to the community?
> >>
> >> Best
> >> -P.
> >>
> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> >> mailto:aromanenko@gmail.com>> wrote:
> >>
> >>Thank you JB for your work!
> >>
> >>I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
> >>HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
> >>
> >>WBR,
> >>Alexey
> >>
> >>
> >>>On 8 Jun 2018, at 10:00, Etienne Chauchot  >>>> wrote:
> >>>
> >>>I forgot to vote:
> >>>+1 (non binding).
> >>>What I tested:
> >>>- no functional or performance regression comparing to v2.4
> >>>- dependencies in the poms are ok
> >>>
> >>>Etienne
> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit
> :
> +1 (non-binding), mainstream usage is not broken by the pom
> changes and runtime has no known regression compared to the 2.4.0
> 
> (side note: kudo to JB for this build tool change release, I know
> how it can hurt ;))
> 
> Romain Manni-Bucau
> @rmannibucau  |  Blog
>  | Old Blog
>  | Github
>  | LinkedIn
>  | Book
> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> 
> 
> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> mailto:j...@nanthrax.net>> a écrit :
> >Thanks for the details Etienne !
> >
> >The good news is that the artifacts seem OK and the overall
> Nexmark
> >results are consistent with the 2.4.0 release ones.
> >
> >I'm starting a complete review using the beam-samples as well.
> >
> >Regards
> >JB
> >
> >>On 07/06/2018 16:14, Etienne Chauchot wrote:
> >> Hi,
> >> I've just run the nexmark queries on v2.5.0-RC1 tag
> >> What we can notice:
> >> - query 3 (exercises CoGroupByKey, state and timer) shows
> >different
> >> output with DR between batch and streaming and with the other
> >runners =>
> >> I compared with v2.4 there were still these differences but with
> >> different output size numbers
> >>
> >> - query 6 (exercises specialized combiner) shows different output
> >> between the runners => the correct output is 401. strange that
> >in batch
> >> mode some runners output les Sellers. I compared with v2.4
> >same output
> >>
> >> - response time of query 7 (exercices Max transform, fanout
> >and side
> >> input) is very slow on DR => I compared with v2.4 , comparable
> >execution
> >> times
> >>
> >> I'm not comparing q10 because it is a write to GCS so it is
> >very specific.
> >>
> >> => Basically no regression comparing to v2.4
> >>
> >> For the record here is the output (waiting for ongoing perfkit
> >integration):
> >>
> >>
> >> 1. DR batch
> >>
> >> Performance:
> >>
> >>
> >Conf  Runtime(sec)(Baseline)  Events(/sec)(Baseline)
>  Results(Baseline)
> >>
> >>
> >   5,8 17283,1
>   10
> >>
> >>
> >0001   3,2 31104,2
>92000
> >>
> >>
> >0002   1,2 82918,7
>  351
> >>
> >>
> >0003   2,2 46210,7
>  458
> >>
> >>
> >0004   1,2  8503,4
>   40
> >>
> >>
> >0005   4,0 25220,7
>   12
> >>
> >>
> >0006   0,9 11148,3
>  401
> >>
> >>
> >0007  13,2 

Re: Announcement & Proposal: HDFS tests on large cluster.

[Alan Myrvold]

Done. Changed the size of the io-datastores kubernetes cluster in
apache-beam-testing to 3 nodes.

On Thu, Jun 7, 2018 at 1:45 AM Kamil Szewczyk  wrote:

> Hi,
>
> the node pool size of io-datastores kubernetes cluster in
> apache-beam-testing project must be changed from 1 -> 3 (or other value).
> @Alan Myrvold was already helpful with kubernetes cluster settings so
> far, but I am not aware who made decisions regarding that as
> this will increase monthly billing.
>
> Kamil Szewczyk
>
> 2018-06-07 6:27 GMT+02:00 Kenneth Knowles :
>
>> This is rad. Another +1 from me for a bigger cluster. What do you need to
>> make that happen?
>>
>> Kenn
>>
>> On Wed, Jun 6, 2018 at 10:16 AM Pablo Estrada  wrote:
>>
>>> This is really cool!
>>>
>>> +1 for having a cluster with more than one machine run the test.
>>>
>>> -P.
>>>
>>> On Wed, Jun 6, 2018 at 9:57 AM Chamikara Jayalath 
>>> wrote:
>>>
>>>> On Wed, Jun 6, 2018 at 5:19 AM Łukasz Gajowy 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'd like to announce that thanks to Kamil Szewczyk, since this PR
>>>>> <https://github.com/apache/beam/pull/5441> we have 4 file-based HDFS
>>>>> tests run on a "Large HDFS Cluster"! More specifically I mean:
>>>>>
>>>>> - beam_PerformanceTests_Compressed_TextIOIT_HDFS
>>>>> - beam_PerformanceTests_Compressed_TextIOIT_HDFS
>>>>> - beam_PerformanceTests_AvroIOIT_HDFS
>>>>> - beam_PerformanceTests_XmlIOIT_HDFS
>>>>>
>>>>> The "Large HDFS Cluster" (in contrast to the small one, that is also
>>>>> available) consists of a master node and three data nodes all in separate
>>>>> pods. Thanks to that we can mimic more real-life scenarios on HDFS (3
>>>>> distributed nodes) and possibly run bigger tests so there's progress! :)
>>>>>
>>>>>
>>>> This is great. Also, looks like results are available in test
>>>> dashboard:
>>>> https://apache-beam-testing.appspot.com/explore?dashboard=5755685136498688
>>>> (BTW we should add information about dashboard to the testing doc:
>>>> https://beam.apache.org/contribute/testing/)
>>>>
>>>> I'm currently working on proper documentation for this so that everyone
>>>>> can use it in IOITs (stay tuned).
>>>>>
>>>>> Regarding the above, I'd like to propose scaling up the
>>>>> Kubernetes cluster. AFAIK, currently, it consists of 1 node. If we scale 
>>>>> it
>>>>> up to eg. 3 nodes, the HDFS' kubernetes pods will distribute themselves on
>>>>> different machines rather than one, making it an even more "real-life"
>>>>> scenario (possibly more efficient?). Moreover, other Performance Tests
>>>>> (such as JDBC or mongo) could use more space for their infrastructure as
>>>>> well. Scaling up the cluster could also turn out useful for some future
>>>>> efforts, like BEAM-4508[1] (adapting and running some old IOITs on
>>>>> Jenkins).
>>>>>
>>>>> WDYT? Are there any objections?
>>>>>
>>>> +1 for increasing the size of Kubernetes cluster.
>>>>
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/BEAM-4508
>>>>>
>>>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>>
>

Apache Beam Contribution Guide Improvements

[Alan Myrvold]

I've written up a brief document with ideas for Apache Beam contribution
guide improvements. I'm most interest in clarifying the goals of this
guide, including whether documenting on Windows is worth describing, and
what areas are missing.

Feedback welcome, especially comments / suggestions in the document,

https://docs.google.com/document/d/1zukoPXPgUq3Vli_rOJ0ykzK6NbR6g-FrgSHZjLd23bo/edit?usp=sharing

Alan Myrvold

Re: [VOTE] Code Review Response-time SLO

[Alan Myrvold]

Proposal 1: +1
Proposal 2: 0

Additional comments:
Does this apply to committers only, or are contributors encourage to
volunteer as an additional reviewer?
I like the idea of documenting who is keen to review which area, plus
reviewer availability.
>From a contributor standpoint, it is worth avoiding long delays; perhaps
having a dashboard or email of reviews that are awaiting feedback?
Will there be a document to propose tooling?
The 24 hours appears to be 7 days a week, 365/6 days a year? It would be
good to build in time for everyone to disconnect.

On Tue, Jun 5, 2018 at 1:07 PM Scott Wegner  wrote:

> Proposal 1: +0
> Proposal 2: +1
>
> Additional Comments:
> I like the idea of providing a clear expectation to contributors on how
> promptly they can expect code review feedback. From your proposal, my main
> feedback would to prefer a single goal instead of two. Having a single goal
> seems simpler when choosing a code reviewer ("PersonA might have more
> context on my change, but PersonB will give quicker feedback.."). Multiple
> tiers of commitment also feels like it implies level of commitment to the
> community ("PersonB must be more committed to Beam than PersonA because
> they agreed to faster code reviews").
>
> There is a balance to strike: we should provide prompt feedback and clear
> expectations for code contributors, but we shouldn't set unreasonable
> targets for code reviewers. I'd be interested in hearing from others
> whether 24-hours seems unreasonable.
>
> I think that we should also make it clear with any policy that opting-in
> is optional and not required to be part of the Beam community. There are
> many ways to contribute to Beam, and opting-in to reviewing code
> contributions is just one of them.
>
> On Mon, Jun 4, 2018 at 3:20 PM Huygaa Batsaikhan 
> wrote:
>
>> Proposal 1: +1
>> Proposal 2: +1
>> Additional Comments: This is an example vote
>>
>> On Mon, Jun 4, 2018 at 3:15 PM Huygaa Batsaikhan 
>> wrote:
>>
>>> A few months ago, Reuven sent out an email
>>> 
>>> about improvements to Beam's code review process. Because the email covered
>>> multiple issues, we did not really dig deep into each of them. One of the
>>> suggestions was to agree on a code review response turnaround time (SLO
>>> ). Here is the
>>> direct quote:
>>>
>>> It would be great if we could agree on a response-time SLA for Beam code
>>> reviews. The response might be “I am unable to do the review until next
>>> week,” however even that is better than getting no response.
>>>
>>>
>>> All the comments on the original thread supported having an agreed upon
>>> SLO. Therefore, I would like to discuss possible response-time SLO and
>>> finalize it within this thread. For the purpose of this discussion, let's
>>> put aside related topics such as the need of tooling support like PR
>>> dashboard or reviewer availability for future discussions.
>>>
>>> *My proposals*
>>>
>>> *Proposal 1*
>>> I propose having a *Default* review response time as *3 business days*.
>>> This aligns with the frequency we consider most developers are checking the
>>> dev list. My reasoning is, if one is checking the dev list, they could also
>>> check their PR review queue.
>>>
>>> *Proposal 2*
>>> I propose having an *Opt-in* review response time as *24 hours*.
>>> Contributors are happy when reviewers respond swiftly to their PRs.
>>> Specially, when we are making multiple small changes to Beam, waiting for
>>> even a few days is frustrating. I understand that not all the reviewers can
>>> review PRs daily. However, if some of us can incorporate half an hour of
>>> beam review to our schedule, it could improve contributors' experience
>>> drastically. Therefore, I suggest us having opt-in response time of 24
>>> hours. We can discuss how we can communicate this SLO to contributors and
>>> reviewers in a separate thread.
>>>
>>> Please vote on these 2 proposals and propose any other solutions using
>>> within this template:
>>>
>>> Template:
>>> Proposal 1: <+-1> 
>>> Proposal 2: <+-1> 
>>> Additional Comments: 
>>>
>>> Example answer:
>>> Proposal 1: +1 Great idea
>>> Proposal 2: +1
>>> Additional Comments: I have this idea foobar 
>>>
>>> Thank you,
>>> Huygaa
>>>
>>>

Re: [VOTE] Use probot/stale to automatically manage stale pull requests

[Alan Myrvold]

+1 (non-binding) I updated the pull request to be 60 days (instead of 90)
to match the contribute policy.

On Fri, Jun 1, 2018 at 9:21 AM Kenneth Knowles  wrote:

> Hi all,
>
> Following the discussion, please vote on the move to activate
> probot/stale [3] to notify authors of stale PRs per current policy and
> then close them after a 7 day grace period.
>
> For more details, see:
>
>  - our stale PR policy [1]
>  - the discussion thread [2]
>  - Probot stale [3]
>  - BEAM ticket summarizing discussion [4]
>  - INFRA ticket to activate probot/stale [5]
>  - Example PR that would activate it [6]
>
> Please vote:
> [ ] +1, Approve that we activate probot/stale
> [ ] -1, Do not approve (please provide specific comments)
>
> Kenn
>
> [1] https://beam.apache.org/contribute/#stale-pull-requests
> [2]
> https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E
> [3] https://github.com/probot/stale
> [4] https://issues.apache.org/jira/browse/BEAM-4423
> [5] https://issues.apache.org/jira/browse/INFRA-16589
> [6] https://github.com/apache/beam/pull/5532
>

Re: Closing (automatically?) inactive pull requests

[Alan Myrvold]

A vote makes sense.

The proposed config is at https://github.com/apache/beam/pull/5532/files
and will mark pull requests as stale after 90 days and close them 7 days
later.
Issues are not affected.

On Fri, Jun 1, 2018 at 9:14 AM Kenneth Knowles  wrote:

> Ismael had mentioned to vote on this, so let's do that now to make sure no
> one missed this discussion.
>
> On Thu, May 31, 2018 at 9:33 PM Alan Myrvold  wrote:
>
>> Thanks. I can look into adding the stale.yaml file for old pull requests/
>>
>> On Thu, May 31, 2018 at 8:07 PM Kenneth Knowles  wrote:
>>
>>> Update: you brought the information needed, and it is now enabled.
>>> Thanks for the follow-through!
>>>
>>> Since you dug into probot's details, I took the liberty of assigning
>>> BEAM-4423 to you, in case throwing together the needed configs is fresh in
>>> your mind and you are in the mood to continue. (if not, pass the hot
>>> potato, back to me is OK too :-)
>>>
>>> Kenn
>>>
>>> On Thu, May 31, 2018 at 4:50 PM Alan Myrvold 
>>> wrote:
>>>
>>>> INFRA-16589 got closed asking to clarify that the probot-stale app
>>>> would not have permissions to merge automatically.
>>>> From my reading of the permissions documentation, it would not. I added
>>>> a comment to INFRA-16589
>>>>
>>>> On Tue, May 29, 2018 at 10:05 AM Lukasz Cwik  wrote:
>>>>
>>>>> I opened up INFRA-16589
>>>>> <https://issues.apache.org/jira/browse/INFRA-16589> for Apache infra
>>>>> to install Stale but they denied a similar request INFRA-16394
>>>>> <https://issues.apache.org/jira/browse/INFRA-16394> because of
>>>>> permissions issues. I tried clarifying the permissions requested.
>>>>>
>>>>> On Tue, May 29, 2018 at 9:39 AM Scott Wegner 
>>>>> wrote:
>>>>>
>>>>>> +1. I've opened BEAM-4423 to capture the discussion here:
>>>>>> https://issues.apache.org/jira/browse/BEAM-4423
>>>>>>
>>>>>> On Thu, May 24, 2018 at 5:34 PM Chamikara Jayalath <
>>>>>> chamik...@google.com> wrote:
>>>>>>
>>>>>>> +1 for automatically closing. Currently, contribution guide mentions
>>>>>>> that pull requests without responses to actionable comments become stale
>>>>>>> (and closed) after two months but last I checked there were many pull
>>>>>>> requests that met this criteria but had not been closed after many 
>>>>>>> months.
>>>>>>> So seems like humans are reluctant to act on the established policy :)
>>>>>>>
>>>>>>> - Cham
>>>>>>>
>>>>>>> On Wed, May 23, 2018 at 11:25 AM Kenneth Knowles 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> That makes sense, to just focus on Beam's decision. It seems the
>>>>>>>> tool is already built. I thought we just had to deploy it, but maybe 
>>>>>>>> not
>>>>>>>> even that, if we can just activate it:
>>>>>>>> https://github.com/apps/stale
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Wed, May 23, 2018 at 9:31 AM Ismaël Mejía 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Given that reaching consensus in both communities seems like a
>>>>>>>>> harder task
>>>>>>>>> than just deciding our policy. in the Beam side Why don't we just
>>>>>>>>> go ahead
>>>>>>>>> and vote around this + build the tool, and if the Flink guys are
>>>>>>>>> interested
>>>>>>>>> they can take it, no?
>>>>>>>>>
>>>>>>>>> in the future we can share that code.
>>>>>>>>> On Wed, May 16, 2018 at 3:31 PM Piotr Nowojski <
>>>>>>>>> pi...@data-artisans.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> > The question is what would such tool offer on top of over a
>>>>>>>>> Github’s view
>>>>>>>>> of PR sorted by “least recently updated”:
>>>>>>>>>

Re: Documenting Github PR jenkins trigger phrases

[Alan Myrvold]

This is tracked by https://issues.apache.org/jira/browse/BEAM-3068

On Fri, Jun 1, 2018 at 7:38 AM Ismaël Mejía  wrote:

> Any progress on this ? Is there a JIRA to track it ?
> On Fri, May 11, 2018 at 6:02 PM Jean-Baptiste Onofré 
> wrote:
> >
> > Agree, I thought there's already a PR about that.
> >
> > Regards
> > JB
> >
> > On 11/05/2018 16:11, Alexey Romanenko wrote:
> > > +1 to add such reference guide of Jenkins commands to Testing guide. It
> > > should be extremely useful, especially for those who were not aware
> > > about this before.
> > >
> > > WBR,
> > > Alexey
> > >
> > >> On 11 May 2018, at 02:44, Ankur Goenka  > >> > wrote:
> > >>
> > >> In my experience affect of white space in commit is inconsistent for
> > >> certain commands they do matter while for others they don't.
> > >>
> > >> On Thu, May 10, 2018 at 5:43 PM Valentyn Tymofieiev
> > >> mailto:valen...@google.com>> wrote:
> > >>
> > >> +1 to writing a Beam Jenkins spellbook.
> > >> I have observed that Jenkins commands sometimes don't work for the
> > >> first time, why could this be? Do end of lines at the end of
> > >> command matter?
> > >>
> > >>
> > >> On Thu, May 10, 2018 at 1:24 PM, Andrew Pilloud
> > >> mailto:apill...@google.com>> wrote:
> > >>
> > >> It would be great to have the set of "Run {Java,Python,Go}
> > >> PreCommit" documented in the contributors guide as well. Those
> > >> match up to the jobs auto run on every PR and are the ones I
> > >> use most. There is no security, anyone can run them including
> > >> 'Run Seed Job'. That one seems like a good one to document in
> > >> testing, because it is the one that loads changes to the rest.
> > >>
> > >> Andrew
> > >>
> > >> On Thu, May 10, 2018 at 12:27 PM Huygaa Batsaikhan
> > >> mailto:bat...@google.com>> wrote:
> > >>
> > >> Hi devs,
> > >>
> > >> We can run various jenkins commands (precommit,
> > >> postcommit, performance tests) directly from Github Pull
> > >> Request UI by commenting phrases such as "retest this
> > >> please". Unfortunately, this tool is not documented. I am
> > >> adding a brief documentation in
> > >> https://beam.apache.org/contribute/testing/ and I need
> > >> some help.
> > >>
> > >>  1. What are the most common phrases used?
> > >>  2. Can anyone run these commands? Are there any
> > >> permission issues?
> > >>  3. Does it make sense to categorize the commands as
> > >> Performance tests, Precommit, Postcommit, and Release
> > >> Validation?
> > >>
> > >> Let me know what you think,
> > >>
> > >> Thanks,
> > >> Huygaa
> > >>
> > >>
> > >
>

Re: Closing (automatically?) inactive pull requests

[Alan Myrvold]

Thanks. I can look into adding the stale.yaml file for old pull requests/

On Thu, May 31, 2018 at 8:07 PM Kenneth Knowles  wrote:

> Update: you brought the information needed, and it is now enabled. Thanks
> for the follow-through!
>
> Since you dug into probot's details, I took the liberty of assigning
> BEAM-4423 to you, in case throwing together the needed configs is fresh in
> your mind and you are in the mood to continue. (if not, pass the hot
> potato, back to me is OK too :-)
>
> Kenn
>
> On Thu, May 31, 2018 at 4:50 PM Alan Myrvold  wrote:
>
>> INFRA-16589 got closed asking to clarify that the probot-stale app would
>> not have permissions to merge automatically.
>> From my reading of the permissions documentation, it would not. I added a
>> comment to INFRA-16589
>>
>> On Tue, May 29, 2018 at 10:05 AM Lukasz Cwik  wrote:
>>
>>> I opened up INFRA-16589
>>> <https://issues.apache.org/jira/browse/INFRA-16589> for Apache infra to
>>> install Stale but they denied a similar request INFRA-16394
>>> <https://issues.apache.org/jira/browse/INFRA-16394> because of
>>> permissions issues. I tried clarifying the permissions requested.
>>>
>>> On Tue, May 29, 2018 at 9:39 AM Scott Wegner  wrote:
>>>
>>>> +1. I've opened BEAM-4423 to capture the discussion here:
>>>> https://issues.apache.org/jira/browse/BEAM-4423
>>>>
>>>> On Thu, May 24, 2018 at 5:34 PM Chamikara Jayalath <
>>>> chamik...@google.com> wrote:
>>>>
>>>>> +1 for automatically closing. Currently, contribution guide mentions
>>>>> that pull requests without responses to actionable comments become stale
>>>>> (and closed) after two months but last I checked there were many pull
>>>>> requests that met this criteria but had not been closed after many months.
>>>>> So seems like humans are reluctant to act on the established policy :)
>>>>>
>>>>> - Cham
>>>>>
>>>>> On Wed, May 23, 2018 at 11:25 AM Kenneth Knowles 
>>>>> wrote:
>>>>>
>>>>>> That makes sense, to just focus on Beam's decision. It seems the tool
>>>>>> is already built. I thought we just had to deploy it, but maybe not even
>>>>>> that, if we can just activate it: https://github.com/apps/stale
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Wed, May 23, 2018 at 9:31 AM Ismaël Mejía 
>>>>>> wrote:
>>>>>>
>>>>>>> Given that reaching consensus in both communities seems like a
>>>>>>> harder task
>>>>>>> than just deciding our policy. in the Beam side Why don't we just go
>>>>>>> ahead
>>>>>>> and vote around this + build the tool, and if the Flink guys are
>>>>>>> interested
>>>>>>> they can take it, no?
>>>>>>>
>>>>>>> in the future we can share that code.
>>>>>>> On Wed, May 16, 2018 at 3:31 PM Piotr Nowojski <
>>>>>>> pi...@data-artisans.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > The question is what would such tool offer on top of over a
>>>>>>> Github’s view
>>>>>>> of PR sorted by “least recently updated”:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-asc
>>>>>>> <
>>>>>>> https://github.com/apache/flink/pulls?q=is:pr+is:open+sort:updated-asc
>>>>>>> >
>>>>>>>
>>>>>>> > ? Maybe it would be good enough to have a policy about waiting x
>>>>>>> months/weeks for a contributor to respond. If he doesn’t, we either
>>>>>>> take
>>>>>>> over PR we or close it. Having “clean” view of oldest PRs would be
>>>>>>> beneficial for reviewing PRs in ~FIFO order as part of community
>>>>>>> work.
>>>>>>>
>>>>>>> > Having
>>>>>>>
>>>>>>> > > On 16 May 2018, at 10:57, Fabian Hueske 
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > Hi,
>>>>>>> > >
>>>>>>> > > I

Re: [ANNOUNCEMENT] New committers, May 2018 edition!

[Alan Myrvold]

Congrats Gris+Pablo+Jason. Well deserved.

On Thu, May 31, 2018 at 9:15 PM Jason Kuster  wrote:

> Thank you to Davor and the PMC; I'm excited to be able to help Beam in
> this new capacity. Bring on the PRs. :D
>
> On Thu, May 31, 2018 at 8:55 PM Xin Wang  wrote:
>
>> Congrats!
>>
>> - Xin Wang
>>
>> 2018-06-01 11:52 GMT+08:00 Rui Wang :
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Thu, May 31, 2018 at 8:23 PM Jean-Baptiste Onofré 
>>> wrote:
>>>
 Congrats !

 Regards
 JB

 On 01/06/2018 04:08, Davor Bonaci wrote:
 > Please join me and the rest of Beam PMC in welcoming the following
 > contributors as our newest committers. They have significantly
 > contributed to the project in different ways, and we look forward to
 > many more contributions in the future.
 >
 > * Griselda Cuevas
 > * Pablo Estrada
 > * Jason Kuster
 >
 > (Apologizes for a delayed announcement, and the lack of the usual
 > paragraph summarizing individual contributions.)
 >
 > Congratulations to all three! Welcome!

>>>
>>
>>
>> --
>> Thanks,
>> Xin
>>
>
>
> --
> ---
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
> See something? Say something. go/jasonkuster-feedback
> 
>

Re: Closing (automatically?) inactive pull requests

[Alan Myrvold]

INFRA-16589 got closed asking to clarify that the probot-stale app would
not have permissions to merge automatically.
>From my reading of the permissions documentation, it would not. I added a
comment to INFRA-16589

On Tue, May 29, 2018 at 10:05 AM Lukasz Cwik  wrote:

> I opened up INFRA-16589
>  for Apache infra to
> install Stale but they denied a similar request INFRA-16394
>  because of
> permissions issues. I tried clarifying the permissions requested.
>
> On Tue, May 29, 2018 at 9:39 AM Scott Wegner  wrote:
>
>> +1. I've opened BEAM-4423 to capture the discussion here:
>> https://issues.apache.org/jira/browse/BEAM-4423
>>
>> On Thu, May 24, 2018 at 5:34 PM Chamikara Jayalath 
>> wrote:
>>
>>> +1 for automatically closing. Currently, contribution guide mentions
>>> that pull requests without responses to actionable comments become stale
>>> (and closed) after two months but last I checked there were many pull
>>> requests that met this criteria but had not been closed after many months.
>>> So seems like humans are reluctant to act on the established policy :)
>>>
>>> - Cham
>>>
>>> On Wed, May 23, 2018 at 11:25 AM Kenneth Knowles  wrote:
>>>
 That makes sense, to just focus on Beam's decision. It seems the tool
 is already built. I thought we just had to deploy it, but maybe not even
 that, if we can just activate it: https://github.com/apps/stale

 Kenn

 On Wed, May 23, 2018 at 9:31 AM Ismaël Mejía  wrote:

> Given that reaching consensus in both communities seems like a harder
> task
> than just deciding our policy. in the Beam side Why don't we just go
> ahead
> and vote around this + build the tool, and if the Flink guys are
> interested
> they can take it, no?
>
> in the future we can share that code.
> On Wed, May 16, 2018 at 3:31 PM Piotr Nowojski <
> pi...@data-artisans.com>
> wrote:
>
> > The question is what would such tool offer on top of over a Github’s
> view
> of PR sorted by “least recently updated”:
>
>
>
> https://github.com/apache/flink/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-asc
> <
> https://github.com/apache/flink/pulls?q=is:pr+is:open+sort:updated-asc
> >
>
> > ? Maybe it would be good enough to have a policy about waiting x
> months/weeks for a contributor to respond. If he doesn’t, we either
> take
> over PR we or close it. Having “clean” view of oldest PRs would be
> beneficial for reviewing PRs in ~FIFO order as part of community work.
>
> > Having
>
> > > On 16 May 2018, at 10:57, Fabian Hueske  wrote:
> > >
> > > Hi,
> > >
> > > I'm not objecting closing stale PRs.
> > > We have quite a few PRs with very little chance of being merged
> and I
> would
> > > certainly appreciate cleaning up those.
> > > However, I think we should not automate closing PRs for the
> reasons I
> gave
> > > before.
> > >
> > > A tool that reminds us of state PRs as proposed by Till seems like
> a
> good
> > > idea and might actually help to re-adjust priorities from time to
> time.
> > >
> > > @Yazdan: Right now there is no assignment happening. Committers
> decide
> > > which PRs they want to review and merge.
> > >
> > > Best, Fabian
> > >
> > > 2018-05-16 4:26 GMT+02:00 Yaz Sh :
> > >
> > >> I have questions in this regard (you guys might have addressed it
> in
> this
> > >> email chain):
> > >> how PRs get assigned to a reviewer apart of a contributor tag
> someone?
> > >> what if PR never gets a reviewer attention and it became
> in-active due
> to
> > >> long review respond? should Bot assign a reviewer to a PR based on
> > >> reviewers interest (i.e., defined via tags) and notify he/she if
> PR is
> > >> waiting for review?
> > >>
> > >>
> > >> Cheers,
> > >> /Yazdan
> > >>
> > >>
> > >>> On May 15, 2018, at 12:06 PM, Thomas Weise 
> wrote:
> > >>>
> > >>> I like Till's proposal to notify the participants on the PR to
> PTAL.
> But
> > >> I
> > >>> would also suggest to auto-close when no action is taken, with a
> friendly
> > >>> reminder that PRs can be reopened anytime.
> > >>>
> > >>> The current situation with 350 open PRs may send a signal to
> contributors
> > >>> that it may actually be too much hassle to get a change
> committed in
> > >> Flink.
> > >>> Since the count keeps on rising, this is also not a past problem.
> Pruning
> > >>> inactive PRs may help, that will also give a more accurate
> picture of
> the
> > >>> lack of review bandwidth, if that is the root cause.
> > >>>
> > >>> Thanks,
> > >>> Thomas
> > >>>
> > >>>

Re: Kubernetes cluster of apache-beam-testing project

[Alan Myrvold]

Updated the io-datastores kubernetes cluster in apache-beam-testing to
version 1.9.7-gke.1 for the master and version 1.9.7-gke.1 for the node
pool.

On Thu, May 24, 2018 at 10:31 AM Yifan Zou <yifan...@google.com> wrote:

> Seems like kubectl v1.9.0 was installed on those nodes. Here is the puppet
> script I found that used to download and install kubectl on beam-jenkins.
>
> https://github.com/apache/infrastructure-puppet/blob/95aed4f4f7cdce79a1f0fc8b1e581f13fedf63ff/modules/build_slaves/manifests/kube.pp
>
> On Thu, May 24, 2018 at 10:18 AM Alan Myrvold <amyrv...@google.com> wrote:
>
>> I can update the version of the  io-datastores cluster master and node
>> pools.
>> I don't have access to the kubectl version on the executors ... I can ask
>> around for that.
>>
>> On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk <szewi...@gmail.com>
>> wrote:
>>
>>> Dear Beam Devs,
>>>
>>> we are using kubernetes to on demand create/tear down resources for
>>> performance testing. It is done automatically by Jenkins jobs using
>>> PerfKit. On 20th of May, there was a blog post about security issues in
>>> kubernetes
>>> https://cloud.google.com/kubernetes-engine/docs/security-bulletins and
>>> we should upgrade it to the latest version of the main branch. As far as I
>>> know in apache-beam-testing, we are now using kubernetes cluster with
>>> engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
>>> We are missing some new cool features introduced in kubernetes 1.9 like
>>> StatefulSets which I would like to use to setup HDFS large cluster for
>>> performance testing, I have already submitted PR for that. My findings are
>>> described in this two JIRA issues:
>>>  - https://issues.apache.org/jira/browse/BEAM-4362
>>>  - https://issues.apache.org/jira/browse/BEAM-4390
>>>
>>> Who is in charge of Google Cloud Platform (have admin access) and can
>>> help with that?
>>>
>>>
>>>

Re: Kubernetes cluster of apache-beam-testing project

[Alan Myrvold]

I can update the version of the  io-datastores cluster master and node
pools.
I don't have access to the kubectl version on the executors ... I can ask
around for that.

On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk  wrote:

> Dear Beam Devs,
>
> we are using kubernetes to on demand create/tear down resources for
> performance testing. It is done automatically by Jenkins jobs using
> PerfKit. On 20th of May, there was a blog post about security issues in
> kubernetes
> https://cloud.google.com/kubernetes-engine/docs/security-bulletins and we
> should upgrade it to the latest version of the main branch. As far as I
> know in apache-beam-testing, we are now using kubernetes cluster with
> engine version 1.8, can we update it to version 1.9(1.9.7-gke.1)?
> We are missing some new cool features introduced in kubernetes 1.9 like
> StatefulSets which I would like to use to setup HDFS large cluster for
> performance testing, I have already submitted PR for that. My findings are
> described in this two JIRA issues:
>  - https://issues.apache.org/jira/browse/BEAM-4362
>  - https://issues.apache.org/jira/browse/BEAM-4390
>
> Who is in charge of Google Cloud Platform (have admin access) and can help
> with that?
>
>
>

Re: I'm back and ready to help grow our community!

[Alan Myrvold]

Congratulations on graduating!!! Glad you're back.

On Tue, May 22, 2018 at 3:01 AM Matthias Baetens 
wrote:

> Same here - shame on me. Congratulations on the graduation Gris, very
> happy to have you back!
>
> On Tue, 22 May 2018 at 09:19 Ismaël Mejía  wrote:
>
>> I missed somehow this email thread.
>> Congratulations Gris and welcome back!
>>
>> On Fri, May 18, 2018 at 5:34 AM Jesse Anderson > >
>> wrote:
>>
>> > Congrats!
>>
>> > On Thu, May 17, 2018, 6:44 PM Robert Burke  wrote:
>>
>> >> Congrats & welcome back!
>>
>> >> On Thu, May 17, 2018, 5:44 PM Huygaa Batsaikhan 
>> wrote:
>>
>> >>> Welcome back, Gris! Congratulations!
>>
>> >>> On Thu, May 17, 2018 at 4:24 PM Robert Bradshaw 
>> wrote:
>>
>>  Congratulations, Gris! And welcome back!
>>  On Thu, May 17, 2018 at 3:30 PM Robin Qiu 
>> wrote:
>>
>>  > Congratulations! Welcome back!
>>
>>  > On Thu, May 17, 2018 at 3:23 PM Reuven Lax 
>> wrote:
>>
>>  >> Congratulations! Good to see you back!
>>
>>  >> Reuven
>>
>>  >> On Thu, May 17, 2018 at 2:24 PM Griselda Cuevas 
>> wrote:
>>
>>  >>> Hi Everyone,
>>
>>
>>  >>> I was absent from the mailing list, slack channel and our Beam
>>  community for the past six weeks, the reason was that I took a leave
>> to
>>  focus on finishing my Masters Degree, which I finally did on May
>> 15th.
>>
>>
>>  >>> I graduated as a Masters of Engineering in Operations Research
>> with a
>>  concentration in Data Science from UC Berkeley. I'm glad to be part
>> of
>> this
>>  community and I'd like to share this accomplishment with you so I'm
>> adding
>>  two pictures of that day :)
>>
>>
>>  >>> Given that I've seen so many new folks around, I'd like to use
>> this
>>  opportunity to re-introduce myself. I'm Gris Cuevas and I work at
>> Google.
>>  Now that I'm back, I'll continue to work on supporting our community
>> in two
>>  main streams: Contribution Experience & Events, Meetups, and
>> Conferences.
>>
>>
>>  >>> It's good to be back and I look forward to collaborating with
>> you.
>>
>>
>>  >>> Cheers,
>>
>>  >>> Gris
>>
> --
>
>

Re: [VOTE] Go SDK

[Alan Myrvold]

+1 (non-binding)
Nice work!

On Tue, May 22, 2018 at 9:18 AM Pablo Estrada  wrote:

> +1 (binding)
> Very excited to see this!
>
> On Tue, May 22, 2018 at 9:09 AM Thomas Weise  wrote:
>
>> +1 and congrats!
>>
>>
>> On Tue, May 22, 2018 at 8:48 AM, Rafael Fernandez 
>> wrote:
>>
>>> +1 !
>>>
>>> On Tue, May 22, 2018 at 7:54 AM Lukasz Cwik  wrote:
>>>
 +1 (binding)

 On Tue, May 22, 2018 at 6:16 AM Robert Burke 
 wrote:

> +1 (non-binding)
>
> I'm looking forward to helping gophers solve their big data problems
> in their language of choice, and runner of choice!
>
> Next stop, a non-java portability runner?
>
> On Tue, May 22, 2018, 6:08 AM Kenneth Knowles  wrote:
>
>> +1 (binding)
>>
>> This is great. Feels like a phase change in the life of Apache Beam,
>> having three languages, with multiple portable runners on the horizon.
>>
>> Kenn
>>
>> On Tue, May 22, 2018 at 2:50 AM Ismaël Mejía 
>> wrote:
>>
>>> +1 (binding)
>>>
>>> Go SDK brings new language support for a community not well
>>> supported in
>>> the Big Data world the Go developers, so this is a great. Also the
>>> fact
>>> that this is the first SDK integrated with the portability work
>>> makes it an
>>> interesting project to learn lessons from for future languages.
>>>
>>> Now it is the time to start building a community around the Go SDK
>>> this is
>>> the most important task now, and the only way to do it is to have
>>> the SDK
>>> as an official part of Beam so +1.
>>>
>>> Congrats to Henning and all the other contributors for this important
>>> milestone.
>>> On Tue, May 22, 2018 at 10:21 AM Holden Karau 
>>> wrote:
>>>
>>> > +1 (non-binding), I've had a chance to work with the SDK and it's
>>> pretty
>>> neat to see Beam add support for a language before the most of the
>>> big data
>>> ecosystem.
>>>
>>> > On Mon, May 21, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>> j...@nanthrax.net>
>>> wrote:
>>>
>>> >> Hi Henning,
>>>
>>> >> SGA has been filed for the entire project during the incubation
>>> period.
>>>
>>> >> Here, we have to check if SGA/IP donation is clean for the Go SDK.
>>>
>>> >> We don't have a lot to do, just checked that we are clean on this
>>> front.
>>>
>>> >> Regards
>>> >> JB
>>>
>>> >> On 22/05/2018 06:42, Henning Rohde wrote:
>>>
>>> >>> Thanks everyone!
>>>
>>> >>> Davor -- regarding your two comments:
>>> >>> * Robert mentioned that "SGA should have probably already
>>> been
>>> filed" in the previous thread. I got the impression that nothing
>>> further
>>> was needed. I'll follow up.
>>> >>> * The standard Go tooling basically always pulls directly
>>> from
>>> github, so there is no real urgency here.
>>>
>>> >>> Thanks,
>>> >>>Henning
>>>
>>>
>>> >>> On Mon, May 21, 2018 at 9:30 PM Jean-Baptiste Onofré <
>>> j...@nanthrax.net
>>> > wrote:
>>>
>>> >>>  +1 (binding)
>>>
>>> >>>  I just want to check about SGA/IP/Headers.
>>>
>>> >>>  Thanks !
>>> >>>  Regards
>>> >>>  JB
>>>
>>> >>>  On 22/05/2018 03:02, Henning Rohde wrote:
>>> >>>   > Hi everyone,
>>> >>>   >
>>> >>>   > Now that the remaining issues have been resolved as
>>> discussed,
>>> >>>  I'd like
>>> >>>   > to propose a formal vote on accepting the Go SDK into
>>> master. The
>>> >>>  main
>>> >>>   > practical difference is that the Go SDK would be part of
>>> the
>>> >>>  Apache Beam
>>> >>>   > release going forward.
>>> >>>   >
>>> >>>   > Highlights of the Go SDK:
>>> >>>   >   * Go user experience with natively-typed DoFns with
>>> (simulated)
>>> >>>   > generic types
>>> >>>   >   * Covers most of the Beam model: ParDo, GBK, CoGBK,
>>> Flatten,
>>> >>>  Combine,
>>> >>>   > Windowing, ..
>>> >>>   >   * Includes several IO connectors: Datastore, BigQuery,
>>> PubSub,
>>> >>>   > extensible textio.
>>> >>>   >   * Supports the portability framework for both batch and
>>> streaming,
>>> >>>   > notably the upcoming portable Flink runner
>>> >>>   >   * Supports a direct runner for small batch workloads
>>> and
>>> testing.
>>> >>>   >   * Includes pre-commit tests and post-commit
>>> integration tests.
>>> >>>   >
>>> >>>   > And last but not least
>>> >>>   >   *  includes contributions from several 

Re: Jenkins Post Commit Status to Github

[Alan Myrvold]

I saw ~1-2 hour latency for the trigger happening yesterday. Not sure what
would cause it to stop or be slow.

On Wed, May 9, 2018 at 10:45 AM Lukasz Cwik  wrote:

> It seems as though precommits are no longer triggering and trigger
> requests like 'Run Java PreCommit' are no longer honored.
>
> On Wed, May 9, 2018 at 10:22 AM Andrew Pilloud 
> wrote:
>
>> I broke all the post commits with this. Sorry! It has been reverted. I'm
>> going to follow up with Apache Infra about getting the right credentials
>> configured on the Jenkins plugin.
>>
>> Andrew
>>
>> On Tue, May 8, 2018 at 1:38 PM Andrew Pilloud 
>> wrote:
>>
>>> Yep, mess with the groovy scripts appears to be the answer. We use
>>> different jenkins libraries to handle PRs vs pushes to master but I think I
>>> finally figured it out. Change is here:
>>> https://github.com/apache/beam/pull/5305
>>>
>>> Andrew
>>>
>>> On Mon, May 7, 2018 at 12:40 PM Kenneth Knowles  wrote:
>>>
 I think you want to mess with the groovy scripts in .test-infra/jenkins

 Kenn

 On Mon, May 7, 2018 at 11:12 AM Andrew Pilloud 
 wrote:

> The Github branches page shows the status of the latest commit on each
> branch and provides a set of links to the jobs run on that commit. But it
> doesn't appear Jenkins is publishing status from post commit jobs. This
> seems like a simple oversight that should be easy to fix. Could someone
> point me in the right direction to fix this?
>
> Andrew
>

Re: Beam parent SNAPSHOTs are not being built

[Alan Myrvold]

; is there a 
>>>>>> better
>>>>>> convention within Maven for importing a set property values?
>>>>>>
>>>>>> On Wed, Apr 25, 2018 at 4:37 PM Eric Beach <ebe...@google.com> wrote:
>>>>>>
>>>>>>> If I run $ mvn clean compile exec:java ... on my project, I get the
>>>>>>> following error stack trace:
>>>>>>>
>>>>>>> java.lang.reflect.InvocationTargetException
>>>>>>>
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>> Method)
>>>>>>>
>>>>>>> at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>
>>>>>>> at
>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
>>>>>>>
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>
>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>> com/google/api/services/clouddebugger/v2/CloudDebugger
>>>>>>>
>>>>>>> at java.lang.Class.getDeclaredMethods0(Native Method)
>>>>>>>
>>>>>>> at
>>>>>>> java.lang.Class.privateGetDeclaredMethods(Class.java:2703)
>>>>>>>
>>>>>>> at java.lang.Class.getDeclaredMethod(Class.java:2130)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:206)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
>>>>>>>
>>>>>>> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
>>>>>>>
>>>>>>> at
>>>>>>> com.google.cloud.pontem.CloudSpannerDatabaseBackup.main(CloudSpannerDatabaseBackup.java:359)
>>>>>>>
>>>>>>> ... 6 more
>>>>>>>
>>>>>>> Caused by: java.lang.ClassNotFoundException:
>>>>>>> com.google.api.services.clouddebugger.v2.CloudDebugger
>>>>>>>
>>>>>>> at
>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>
>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>
>>>>>>>
>>>>>>> When I dig deeper, the root problem seems to be that my project is
>>>>>>> pulling in v2-rev8-1.22.0 instead of v2-rev233-1.23.0 (changing
>>>>>>> commit here
>>>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036>
>>>>>>> ).
>>>>>>>
>>>>>>> In fact, all the version changes in this commit
>>>>>>> <https://github.com/apache/beam/commit/66fcf4bff19cda5831465ccedbbaa6fbaa648234#diff-600376dffeb79835ede4a0b285078036>
>>>>>>>  are missing.
>>>>>>>
>>>>>>> My project's pom.xml contains the following:
>>>>>>>
>>>>>>> 
>>>>>>>
>>>>>>> beam-examples-parent
>>>>>>>
>>>>>>> org.apache.beam
>>>>>>>
>>>>>>> 2.5.0-SNAPSHOT
>>>>>>>
>>>>>>> 
>>>>>>>
>>>>>>> 4.0.0
>>>>>>>
>>>>>>>
>>>>>>> com.google.cloud
>>>>>>>
>>&g

Re: Beam parent SNAPSHOTs are not being built

[Alan Myrvold]

The gradle build is currently only generating pom files when there is a jar
file.
What are these parent poms used for? Do all the parent poms need to be
generated or just the top level beam-parent?

On Wed, Apr 25, 2018 at 2:07 PM Chamikara Jayalath <chamik...@google.com>
wrote:

> Thanks. Should this be a 2.5.0 blocker ?
>
> On Wed, Apr 25, 2018 at 2:05 PM Alan Myrvold <amyrv...@google.com> wrote:
>
>> I think this corresponds with the move from maven to gradle snapshot
>> releases.
>> Issue logged as https://issues.apache.org/jira/browse/BEAM-4170
>>
>> On Wed, Apr 25, 2018 at 1:57 PM Chamikara Jayalath <chamik...@google.com>
>> wrote:
>>
>>> Seems like we are not building some of the SNAPSHOT artifacts since
>>> 4/9/2018.
>>> For example:
>>>
>>>
>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/
>>>
>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-examples-parent/2.5.0-SNAPSHOT/
>>>
>>> Users who depend on these artifacts are getting outdated dependencies.
>>>
>>> Any idea why ?
>>>
>>> Thanks,
>>> Cham
>>>
>>

Re: Beam parent SNAPSHOTs are not being built

[Alan Myrvold]

I think this corresponds with the move from maven to gradle snapshot
releases.
Issue logged as https://issues.apache.org/jira/browse/BEAM-4170

On Wed, Apr 25, 2018 at 1:57 PM Chamikara Jayalath 
wrote:

> Seems like we are not building some of the SNAPSHOT artifacts since
> 4/9/2018.
> For example:
>
>
> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-parent/2.5.0-SNAPSHOT/
>
> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-examples-parent/2.5.0-SNAPSHOT/
>
> Users who depend on these artifacts are getting outdated dependencies.
>
> Any idea why ?
>
> Thanks,
> Cham
>

Re: [PROPOSAL] Preparing 2.5.0 release next week

[Alan Myrvold]

We have been generating a maven project using the archetype from gradle
built snapshot releases and running the quickstarts using maven.

Are there other interesting tests to try that the quickstarts miss?

On Thu, Apr 12, 2018 at 9:21 AM Kenneth Knowles  wrote:

> Just because I have probably missed it in the flood of work/emails (yay!)
> have we been dry-running the user experience of a Maven-built project
> depending on Beam? If declaring a dependency on Beam doesn't work in some
> basic way, it seems like we would want to know that right away. Our release
> verification will definitely catch this, but perhaps we want to check it
> earlier.
>
> Kenn
>
> On Thu, Apr 12, 2018 at 9:11 AM Scott Wegner  wrote:
>
>> Romain, could you please open a JIRA describing the requirements for the
>> generated pom's? The gradle-generated pom's don't match the maven version
>> byte-for-byte, but I don't think that's a requirement.
>>
>> I and others are still hacking on the Gradle build, so it's possible we
>> could get the pom's ready within 2 weeks. It would be good to understand
>> the specific requirements for the generated pom's such that we don't break
>> consumers.
>>
>> On Thu, Apr 12, 2018 at 3:14 AM Etienne Chauchot 
>> wrote:
>>
>>> +1 to what Ahmet said,  I also think that it is important to take our
>>> time given that we're in the gradle migration process.
>>> +1 to what JB said also: try gradle and fall back to maven in case of
>>> troubles.
>>>
>>> Etienne
>>>
>>> Le mercredi 11 avril 2018 à 13:35 -0700, Ahmet Altay a écrit :
>>>
>>> +1 to delaying 2 weeks.
>>>
>>> I think it will be prudent to wait in this case. There is too much in
>>> flux with Gradle migration currently and based on Scott's latest update I
>>> think we will be in a more stable state in 2 weeks. Last Beam release date
>>> was 3/20 and our plan was to make a release every 6 weeks. Even if we start
>>> cutting a release in 2 weeks (~4/26), we will have more than a week to
>>> finish that release. Even if that is not enough time to finish a release it
>>> will put is in the proximity of the 5/5 target date. I prefer that option,
>>> to another potential release with Maven.
>>>
>>> On Wed, Apr 11, 2018 at 1:22 PM, Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
>>> Any hope the release is on central before the 30th?
>>>
>>>
>>> I do not think this was the plan to begin with. The time between the
>>> previous release and 4/30 is less than 6 weeks.
>>>
>>>
>>>
>>> Le 11 avr. 2018 22:02, "Robert Bradshaw"  a écrit :
>>>
>>> +1 to keeping up the regular release cycle, but I don't think it makes
>>> sense to cut until we're ready to actively work on the release. A cut date
>>> two weeks from now seems fine (unless someone else is volunteering to do it
>>> this time around).
>>>
>>> That being said, a dry run using gradle now may make a lot of sense,
>>> giving us a couple of weeks to iron out the kinks, if any.
>>>
>>> On Wed, Apr 11, 2018 at 12:58 PM Chamikara Jayalath <
>>> chamik...@google.com> wrote:
>>>
>>> Hi JB,
>>>
>>> Please note that I opened a blocker [1] (working on it) and looks like
>>> we have several other JIRAs marked for 2.5.0. So +1 for waiting for two
>>> weeks (or till JIRAs are resolved or moved off the list).
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-3991
>>> [2]
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%202.5.0%20ORDER%20BY%20priority%20DESC
>>>
>>> On Wed, Apr 11, 2018 at 12:28 PM Jean-Baptiste Onofré 
>>> wrote:
>>>
>>> Hi guys,
>>>
>>> Due to the last work, I think it makes sense to try a release using
>>> Gradle. If it doesn't work smoothly, we will rollback to Maven, and
>>> maybe in that case, we should ask ourselves if Gradle is really a good
>>> alternative for now.
>>>
>>> I'm in vacation tomorrow morning for 2 weeks. I can cut the release
>>> tomorrow end of the morning (my time). But in the case we need some
>>> additional PR merges or we have some work in progress, I'm proposing to
>>> postpone 2.5.0 release for the end of this month (in two weeks). If
>>> someone is volunteer to tackle this release, please, let us know.
>>>
>>> Regards
>>> JB
>>>
>>> On 06/04/2018 10:48, Jean-Baptiste Onofré wrote:
>>> > Hi guys,
>>> >
>>> > Apache Beam 2.4.0 has been released on March 20th.
>>> >
>>> > According to our cycle of release (roughly 6 weeks), we should think
>>> about 2.5.0.
>>> >
>>> > I'm volunteer to tackle this release.
>>> >
>>> > I'm proposing the following items:
>>> >
>>> > 1. We start the Jira triage now, up to Tuesday
>>> > 2. I would like to cut the release on Tuesday night (Europe time)
>>> > 2bis. I think it's wiser to still use Maven for this release. Do you
>>> think we
>>> > will be ready to try a release with Gradle ?
>>> >
>>> > After this release, I would 

Re: Build failed in Jenkins: beam_SeedJob #1480

[Alan Myrvold]

Typo in my pull request. Fixing it now.

On Wed, Apr 11, 2018 at 11:42 AM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See 
>
> --
> GitHub pull request #5101 of commit
> 0a27a08743d6547867df89f7cc76a343107f46d0, no merge conflicts.
> Setting status of 0a27a08743d6547867df89f7cc76a343107f46d0 to PENDING with
> url https://builds.apache.org/job/beam_SeedJob/1480/ and message: 'Build
> started sha1 is merged.'
> Using context: Jenkins: Seed Job
> [EnvInject] - Loading node environment variables.
> Building remotely on beam4 (beam) in workspace <
> https://builds.apache.org/job/beam_SeedJob/ws/>
>  > git rev-parse --is-inside-work-tree # timeout=10
> Fetching changes from the remote Git repository
>  > git config remote.origin.url https://github.com/apache/beam.git #
> timeout=10
> Fetching upstream changes from https://github.com/apache/beam.git
>  > git --version # timeout=10
>  > git fetch --tags --progress https://github.com/apache/beam.git
> +refs/heads/*:refs/remotes/origin/*
> +refs/pull/5101/*:refs/remotes/origin/pr/5101/*
>  > git rev-parse refs/remotes/origin/pr/5101/merge^{commit} # timeout=10
>  > git rev-parse refs/remotes/origin/origin/pr/5101/merge^{commit} #
> timeout=10
> Checking out Revision 8cf10fc1fc0064dc2264805ababb80584f5cf917
> (refs/remotes/origin/pr/5101/merge)
>  > git config core.sparsecheckout # timeout=10
>  > git checkout -f 8cf10fc1fc0064dc2264805ababb80584f5cf917
> Commit message: "Merge 0a27a08743d6547867df89f7cc76a343107f46d0 into
> fbfde99bc324725015e064fe140c5432dae00801"
> First time build. Skipping changelog.
> Cleaning workspace
>  > git rev-parse --verify HEAD # timeout=10
> Resetting working tree
>  > git reset --hard # timeout=10
>  > git clean -fdx # timeout=10
> Processing DSL script job_00_seed.groovy
> Processing DSL script job_beam_Inventory.groovy
> Processing DSL script job_beam_PerformanceTests_Dataflow.groovy
> Processing DSL script job_beam_PerformanceTests_FileBasedIO_IT.groovy
> Processing DSL script job_beam_PerformanceTests_FileBasedIO_IT_HDFS.groovy
> Processing DSL script job_beam_PerformanceTests_HadoopInputFormat.groovy
> Processing DSL script job_beam_PerformanceTests_JDBC.groovy
> Processing DSL script job_beam_PerformanceTests_MongoDBIO_IT.groovy
> Processing DSL script job_beam_PerformanceTests_Python.groovy
> Processing DSL script job_beam_PerformanceTests_Spark.groovy
> Processing DSL script job_beam_PostCommit_Go_GradleBuild.groovy
> ERROR: startup failed:
> job_beam_PostCommit_Go_GradleBuild.groovy: 24: expecting ''', found '\n' @
> line 24, column 57.
>tCommit tests against master.)
>  ^
>
> 1 error
>
>
>

Re: Python postcommit and precommit

[Alan Myrvold]

I think we should replace the shell script with a top level
pythonPostCommit gradle target, similar to the precomment.

On Mon, Apr 9, 2018 at 12:12 PM Lukasz Cwik  wrote:

> The shell scripts still exist instead of using Gradle. Migrating to Gradle
> as the build system hasn't addressed this (only change in the Gradle
> migration was an improvement where Gradle now creates a virtualenv
> automatically for building).
>
> Alan, any plans to integrate more closely with Gradle going forward
> instead of using shell scripts for task/input/output management?
>
> On Wed, Apr 4, 2018 at 2:11 PM Kenneth Knowles  wrote:
>
>> Was this resolved off list? I think it makes sense to have a
>> dependency-driven build tool as the entry point to these processes. So in
>> our case, Gradle. If setting it up in Gradle/Groovy is a pain, having it
>> shell out seems fine as an implementation detail, but you need to set up
>> inputs/outputs of the Gradle tasks properly.
>>
>> Kenn
>>
>> On Fri, Mar 30, 2018 at 3:30 PM Udi Meiri  wrote:
>>
>>> Hi,
>>>
>>> I noticed that Python precommit runs using this command:
>>>   mvn clean install -pl sdks/python -am -amd
>>> while postcommit invocation is simply a bash script:
>>>   bash sdks/python/run_postcommit.sh
>>>
>>> Both run unit tests via Tox, however since the runtime environment setup
>>> is configured in different files (pom.xml vs shell script), they don't
>>> always agree in their results (precommit is currently succeeded while
>>> postcommit is failing).
>>>
>>> So my naive question is: why does Python precommit run via Maven/Gradle?
>>> Could we not just use a script like run_postcommit.sh?
>>>
>>> (Side note: there's a lot of code/config duplication, such as: pypi
>>> package versions, *.c, *.so, etc. cleanup)
>>>
>>

Re: [ANNOUCEMENT] New Foundation members!

[Alan Myrvold]

Congratulations!

On Mon, Apr 2, 2018 at 9:14 AM Scott Wegner  wrote:

> Congrats!
>
> On Sat, Mar 31, 2018 at 12:18 PM Robert Burke  wrote:
>
>> Congratulations!
>>
>> On Sat, 31 Mar 2018 at 11:53 Ekrem Aksoy  wrote:
>>
>>> Congrats!
>>>
>>> On Sat, Mar 31, 2018 at 2:08 AM, Davor Bonaci  wrote:
>>>
 Now that this is public... please join me in welcoming three newly
 elected members of the Apache Software Foundation with ties to this
 community, who were elected during the most recent Members' Meeting.

 * Ismaël Mejía (Beam PMC)

 * Josh Wills (Crunch Chair; Beam, DataFu PMC)

 * Holden Karau (Spark, SystemML PMC; Mahout, Subversion committer; Beam
 contributor)

 These individuals demonstrated merit in Foundation's growth, evolution,
 and progress. They were recognized, nominated, and elected by existing
 membership for their significant impact to the Foundation as a whole, such
 as the roots of project-related and cross-project activities.

 As members, they now become legal owners and shareholders of the
 Foundation. They can vote for the Board, incubate new projects, nominate
 new members, participate in any PMC-private discussions, and contribute to
 any project.

 (For the Beam community, this election nearly doubles the number of
 Foundation members. The new members are joining Jean-Baptiste Onofré,
 Stephan Ewen, Romain Manni-Bucau and myself in this role.)

 I'm happy to be able to call all three of you my fellow members.
 Congratulations!

 Davor

>>>
>>> --
>
>
> Got feedback? http://go/swegner-feedback
>

Re: Build breaks on examples on jenkins with dataflow runner

[Alan Myrvold]

Thomas logged this issue as
https://issues.apache.org/jira/projects/BEAM/issues/BEAM-3964

On Wed, Mar 28, 2018 at 10:09 AM Alan Myrvold <amyrv...@google.com> wrote:

> I looked at one failure, and saw this error in the log. Could it be
> related to https://github.com/apache/beam/pull/4918 ?
>
>
> 2018-03-28 09:45:26.603 PDTThread 25 died.
> Expand all | Collapse all
> {
> insertId: "776545081849523:822129:0:57106"
> jsonPayload: {
> exception: "java.lang.NoSuchMethodError:
> org.apache.beam.sdk.metrics.MetricName.name()Ljava/lang/String; at
> com.google.cloud.dataflow.worker.MetricsToCounterUpdateConverter.structuredNameAndMetadata(MetricsToCounterUpdateConverter.java:99)
> at
> com.google.cloud.dataflow.worker.MetricsToCounterUpdateConverter.fromCounter(MetricsToCounterUpdateConverter.java:68)
> at
> com.google.cloud.dataflow.worker.BatchModeExecutionContext.lambda$null$1(BatchModeExecutionContext.java:463)
> at
> com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.Iterators$7.transform(Iterators.java:750)
> at
> com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
> at
> com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:66)
> at
> com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:388)
> at
> com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:472)
> at
> com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:669)
> at
> com.google.cloud.dataflow.worker.WorkItemStatusClient.populateCounterUpdates(WorkItemStatusClient.java:256)
> at
> com.google.cloud.dataflow.worker.WorkItemStatusClient.createStatusUpdate(WorkItemStatusClient.java:240)
> at
> com.google.cloud.dataflow.worker.WorkItemStatusClient.reportError(WorkItemStatusClient.java:94)
> at
> com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:358)
> at
> com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:284)
> at
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
> at
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
> at
> com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
> On Wed, Mar 28, 2018 at 6:32 AM Etienne Chauchot <echauc...@apache.org>
> wrote:
>
>> Hi all,
>> Please know that I have submitted a PR to skip IT tests from de PreCommit
>> java tests (1). They still run in PostCommit.
>> [1] https://github.com/apache/beam/pull/4967
>>
>> Etienne
>>
>> Le jeudi 22 mars 2018 à 11:54 +0100, Etienne Chauchot a écrit :
>>
>> Also, WDYT about running these tests as PostCommit instead of preCommit
>> as they are integration tests?
>>
>> Etienne
>>
>> Le jeudi 22 mars 2018 à 09:49 +0100, Etienne Chauchot a écrit :
>>
>> Hi all,
>> java PreCommit test fails on jenkins on the examples module
>> (woundCountIT). It gives incorrect signal on the build of PRs.
>> It seems to be related to communication issues with dataflow service
>>
>> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchStaticSharding
>> or
>> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchDynamicSharding
>>
>> A work item was attempted 4 times without success. Each time the worker 
>> eventually lost contact with the service. The work item was attempted on:
>>   testpipeline-jenkins-0321-03210922-9f05-harness-qxtj,
>>   testpipeline-jenkins-0321-03210922-9f05-harness-98n1,
>>   testpipeline-jenkins-0321-03210922-9f05-harness-47mf,
>>   testpipeline-jenkins-0321-03210922-9f05-harness-n1vb
>>
>>
>>
>> org.apache.beam.examples.WordCountIT.testE2EWordCount
>>
>> java.lang.RuntimeException: Workflow failed. Causes: The Dataflow appears to 
>> be stuck. You can get help with Cloud Dataflow at 
>> https://cloud.google.com/dataflow/suppo

Re: Build breaks on examples on jenkins with dataflow runner

[Alan Myrvold]

I looked at one failure, and saw this error in the log. Could it be related
to https://github.com/apache/beam/pull/4918 ?


2018-03-28 09:45:26.603 PDTThread 25 died.
Expand all | Collapse all
{
insertId: "776545081849523:822129:0:57106"
jsonPayload: {
exception: "java.lang.NoSuchMethodError:
org.apache.beam.sdk.metrics.MetricName.name()Ljava/lang/String; at
com.google.cloud.dataflow.worker.MetricsToCounterUpdateConverter.structuredNameAndMetadata(MetricsToCounterUpdateConverter.java:99)
at
com.google.cloud.dataflow.worker.MetricsToCounterUpdateConverter.fromCounter(MetricsToCounterUpdateConverter.java:68)
at
com.google.cloud.dataflow.worker.BatchModeExecutionContext.lambda$null$1(BatchModeExecutionContext.java:463)
at
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.Iterators$7.transform(Iterators.java:750)
at
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
at
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.MultitransformedIterator.next(MultitransformedIterator.java:66)
at
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:388)
at
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableCollection$ArrayBasedBuilder.addAll(ImmutableCollection.java:472)
at
com.google.cloud.dataflow.worker.repackaged.com.google.common.collect.ImmutableList$Builder.addAll(ImmutableList.java:669)
at
com.google.cloud.dataflow.worker.WorkItemStatusClient.populateCounterUpdates(WorkItemStatusClient.java:256)
at
com.google.cloud.dataflow.worker.WorkItemStatusClient.createStatusUpdate(WorkItemStatusClient.java:240)
at
com.google.cloud.dataflow.worker.WorkItemStatusClient.reportError(WorkItemStatusClient.java:94)
at
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:358)
at
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:284)
at
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)



On Wed, Mar 28, 2018 at 6:32 AM Etienne Chauchot 
wrote:

> Hi all,
> Please know that I have submitted a PR to skip IT tests from de PreCommit
> java tests (1). They still run in PostCommit.
> [1] https://github.com/apache/beam/pull/4967
>
> Etienne
>
> Le jeudi 22 mars 2018 à 11:54 +0100, Etienne Chauchot a écrit :
>
> Also, WDYT about running these tests as PostCommit instead of preCommit as
> they are integration tests?
>
> Etienne
>
> Le jeudi 22 mars 2018 à 09:49 +0100, Etienne Chauchot a écrit :
>
> Hi all,
> java PreCommit test fails on jenkins on the examples module
> (woundCountIT). It gives incorrect signal on the build of PRs.
> It seems to be related to communication issues with dataflow service
>
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchStaticSharding
> or
> org.apache.beam.examples.WindowedWordCountIT.testWindowedWordCountInBatchDynamicSharding
>
> A work item was attempted 4 times without success. Each time the worker 
> eventually lost contact with the service. The work item was attempted on:
>   testpipeline-jenkins-0321-03210922-9f05-harness-qxtj,
>   testpipeline-jenkins-0321-03210922-9f05-harness-98n1,
>   testpipeline-jenkins-0321-03210922-9f05-harness-47mf,
>   testpipeline-jenkins-0321-03210922-9f05-harness-n1vb
>
>
>
> org.apache.beam.examples.WordCountIT.testE2EWordCount
>
> java.lang.RuntimeException: Workflow failed. Causes: The Dataflow appears to 
> be stuck. You can get help with Cloud Dataflow at 
> https://cloud.google.com/dataflow/support.
>   at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:134)
>   at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:90)
>   at 
> org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:55)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297)
>   at org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185)
>   at 
> org.apache.beam.examples.WordCountIT.testE2EWordCount(WordCountIT.java:70)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> 

Re: Gradle status

[Alan Myrvold]

I think the investment in gradle is worthwhile, and incrementally we will
continue to make progress. From what I've seem, gradle is a good fit for
this project and a path to a faster, more reliable build system.

pull/4812  creates the release
artifacts, although it is not hooked up yet with authentication.

I expect to help make incremental progress over the next month converting
some of the integration tests, but welcome incremental improvements from
others.



On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau 
wrote:

>
>
> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik :
>
>> what do we do? "Gradle migration will happen incrementally."
>>
>> "last months prooved beam cant maintain 2 systems, easier with that
>> state is then to drop gradle since it is a 0 investment compared to the
>> opposite"
>> Its unfortunate that you feel this way but many people do not share your
>> opinion.
>>
>
> And a lot do so when a project is 50-50 it is time to act.
>
> Incrementally kind of means never (makes 4 months and nothing really
> changed in PRs and habits, gradle maintener(s) are still alone)
>
>
>>
>>
>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau 
>> wrote:
>>
>>> @Valentyn: concretely any user can PR and be part of that process so
>>> anyone can do it wrong (me first)
>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>> since it is a 0 investment compared to the opposite
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau  |  Blog
>>>  | Old Blog
>>>  | Github
>>>  | LinkedIn
>>>  | Book
>>> 
>>>
>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik :
>>>
 Romain, from the previous discussions several people agreed that
 running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
 worthwhile but there was nobody in the community with the time commitment
 to organize and run it so the status quo plan remained where the Gradle
 migration will happen incrementally.


 On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde 
 wrote:

> My understanding was the same as Ismaël's. I don't think breaking the
> build with a large known gaps (but not fully known cost) is practical.
> Also, most items in the jira are not even assigned yet.
>
>
> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
> rmannibu...@gmail.com> wrote:
>
>> Not really Ismaël, this thread was about to do it at once and have 1
>> day to fix it all.
>>
>> As mentionned at the very beginning nobody maintains the 2 system so
>> it must stop after months so either we drop maven or gradle *at once*
>> or we keep a state where each dev does what he wants and the build
>> system just doesn't work.
>>
>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía :
>>
>>> I don't think that removing all maven descriptors was the expected
>>> path, no ? Or even a good idea at this moment.
>>>
>>> I understood that what we were going to do was to replace
>>> incrementally the CI until we cover the whole maven functionality and
>>> then remove it, from looking at the JIRA ticket
>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>> from
>>> covering the complete maven functionality in particular for the
>>> release part that could be the biggest pain point.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>  wrote:
>>> > hey guys,
>>> >
>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
>>> on monday?
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles :
>>> >>
>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik 
>>> wrote:
>>> >>>
>>> >>> Based upon your description it seems as though you would rather
>>> have a
>>> >>> way to run existing postcommits without it impacting
>>> dashboard/health
>>> >>> stats/notifications/ (We have just run the PostCommits on
>>> PRs for
>>> >>> additional validation (like upgrading the Dataflow container
>>> image)).
>>> >>
>>> >>
>>> >> Yes, that is exactly what I have described.
>>> >>
>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>> for the
>>> >>> the Java PostCommit is the right way to go but I also don't have
>>> 

New beam contributor experience?

[Alan Myrvold]

There is a contribution guide at
https://beam.apache.org/contribute/contribution-guide/
Has anyone had challenges / pain points when getting started with new
contributions?
Any suggestions for making this better?

Alan

Re: [PROPOSITION] schedule some sanity tests on a daily basis

[Alan Myrvold]

Great ideas. I want to see a daily signal for anything that could prevent a
release from happening, and precommits that are fast and reliable for areas
that are commonly broken by code changes.

We are now running the java quickstarts daily on a cron schedule, using
direct, dataflow, and local spark and flink in
the beam_PostRelease_NightlySnapshot job, see
https://github.com/apache/beam/blob/master/release/build.gradle This should
provide a good signal for the examples integration tests against these
runners.

As Kenn noted, the java_maveninstall also runs lots of tests. It would be
good to be more clear and intentional about which tests run when, and to
consider implementing additional "always up" environments for use by the
tests.

Having the nexmark smoke tests run regularly and stored in a database would
really enhance our efforts, perhaps starting with directrunner for the
performance tests.

What area would have the most immediate impact? Nexmark smoke tests?




On Fri, Mar 9, 2018 at 12:57 PM Kenneth Knowles  wrote:

> On Fri, Mar 9, 2018 at 3:08 AM Etienne Chauchot 
> wrote:
>
>> Hi guys,
>>
>> I was looking at the various jenkins jobs and I wanted to submit a
>> proposition:
>>
>> - Validates runner tests: currently run at PostCommit for all the
>> runners. I think it is the quickest way to see
>> regressions. So keep it that way
>>
>
> We've also toyed with precommit for runners where it is fast.
>
>
>> - Integration tests: AFAIK we only run the ones in examples module and
>> only on demand. What about running all the IT (in
>> particular IO IT) as a cron job on a daily basis with direct runner?
>> Please note that it will require some always up
>> backend infrastructure.
>>
>
> I like this idea. We actually run more, but in postcommit. You can see the
> goal here:
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PostCommit_Java_MavenInstall.groovy#L47
>
> There's no infrastructure set up that I see. It is only DirectRunner and
> DataflowRunner currently, as they are "always up". But so could be local
> Flink and Spark. Do the ITs spin up local versions of what they are
> connecting to?
>
> If we have adequate resources, I also think ValidatesRunner on a real
> cluster would add value, once we have the cluster set up / tear down or
> "always up".
>
>
>> - Performance tests: what about running Nexmark SMOKE test suite in batch
>> and streaming modes with all the runners on a
>> daily basis and store the running times in a RRD database (to see
>> performance regressions)?
>
>
> I like this idea, too. I think we could do DirectRunner (and probably
> local Flink) as postcommit without being too expensive.
>
> Kenn
>
>
>
>> Please note that not all the
>> queries run in all the runners in all the modes right now. Also, we have
>> some streaming pipelines termination issues
>> (see https://issues.apache.org/jira/browse/BEAM-2847)
>>
>> I know that Stephen Sisk use to work on these topics. I also talked to
>> guys from Polidea. But As I understood, they
>> launch mainly integration tests on Dataflow runner.
>>
>> WDYT?
>>
>> Etienne
>>
>>
>>

Re: [VOTE] Release 2.4.0, release candidate #2

[Alan Myrvold]

The dataflow java worker version wasn't updated on the branch as in past
releases ... should it be?
https://issues.apache.org/jira/browse/BEAM-3815


On Thu, Mar 8, 2018 at 1:40 PM Romain Manni-Bucau 
wrote:

> Can still be provided as a generic one (like the an offset or key based
> one) but good enough for now, right, was just surprising to not see it when
> checking the breakage.
>
> Le 8 mars 2018 22:05, "Eugene Kirpichov"  a écrit :
>
> All SDF-related method annotations in DoFn are marked @Experimental. I
> guess that should apply to RestrictionTracker too, but I wouldn't be too
> worried about that, since it only makes sense to use in the context of
> those methods.
>
> On Thu, Mar 8, 2018 at 12:36 PM Romain Manni-Bucau 
> wrote:
>
>> Hmm, does sdf api misses some @Experimental then?
>>
>> To clarify: for waitUntilFinish I'm fine with the 2.4 as this but cant +1
>> or +0 since none of my tests pass reliably in current state without a retry
>> strategy making the call useless.
>>
>> Le 8 mars 2018 21:02, "Reuven Lax"  a écrit :
>>
>>> Does Nexmark use SerializableCoder?
>>>
>>>
>>> On Thu, Mar 8, 2018 at 10:42 AM Robert Bradshaw 
>>> wrote:
>>>
 I put the validation checklist spreadsheet is up at
 https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?ts=5a1c7310#gid=1663314475

 Regarding the direct runner regression on query 10, this is
 understandable given how mutation detection has been changed for
 serializable coders (and should be tracked, probably fixed by avoiding
 SerializableCoder). It should not affect other runners. Could you file a
 bug?

 Regarding waitUntilFinish, this is a bug but not a blocker--it's been
 this way since teardown was introduced. There are many nice-to-haves that
 one could merge from master to the release branch, but we've seen where
 that trend leads.

 Regarding the backwards incompatible changes in restriction tracker,
 this is (as I understand it) a change to the experimental SDF API. Eugene,
 do you want to comment on this?



 On Thu, Mar 8, 2018 at 2:07 AM Ismaël Mejía  wrote:

> I confirm that the new release fixes both problems reported previously:
>
> - python package name
> - nexmark query 10 mutability issue with the direct runner.
>
> One extra regression is that the the fix produced a way longer
> execution time on the query.
> Not sure if a blocker but worth tracking.
>
> Query 10 - Batch/Bounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   3.627609.1  1
>   2.4.0  30.8 3244.3  1
>
> Query 10 - Streaming/Unbounded
> Version  Runtime(sec)   Events(/sec)Results
>   2.3.0   6.315873.0  1
>   2.4.0 101.1  989.4  1
>
> On Thu, Mar 8, 2018 at 8:54 AM, Romain Manni-Bucau
>  wrote:
> > -1:
> > a) still consider waitUntilFinish broken and a big blocker
> > b) restrictiontracker api changed and is not backward compatible
> > (
> https://github.com/apache/beam/commit/e0034314ad196d2274cef9831ed63e090bf4d4c1#diff-098d7247eb1e9d9423bfa2ae2da38a9d
> )
> >
> > with workarounds and fixes for these two issues the other parts work
> (spark,
> > flink, direct runner, java core) on my projects
> >
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-08 6:26 GMT+01:00 Robert Bradshaw :
> >>
> >> Hi everyone,
> >>
> >> Please review and vote on the release candidate #2 for the version
> 2.4.0,
> >> as follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>
> >> The complete staging area is available for your review, which
> includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to
> dist.apache.org
> >> [2],
> >> which is signed with the key with fingerprint BDC9 89B0 1BD2 A463
> 6010
> >>A1CA 8F15 5E09 610D 69FB [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag "v2.4.0-RC2" [5],
> >> * website pull request listing the release and publishing the API
> >> reference
> >> manual [6].
> >> * Java artifacts were built with Maven 3.2.5 and OpenJDK 1.8.0_112.
> >> * Python artifact are deployed along with the source release to the
> >> dist.apache.org [2]. If I am able to figure out how to build the
> wheels, I
> >> will post them there as well.
> >>
> >> The vote will be open for at 

Re: [VOTE] Release 2.4.0, release candidate #1

[Alan Myrvold]

I don't see anything published to
https://repository.apache.org/content/repositories/orgapachebeam-1028/ ?


On Wed, Mar 7, 2018 at 9:26 AM Ahmet Altay  wrote:

> -1 for the same reason as Ismaël. Python version is not updated in the
> release branch [1].
>
> [1]
> https://github.com/apache/beam/blob/release-2.4.0/sdks/python/apache_beam/version.py#L21
>
> On Wed, Mar 7, 2018 at 8:39 AM, Jean-Baptiste Onofré 
> wrote:
>
>> No it's not (I'm testing the release right now), I just was curious and
>> noticed
>> the missing details ;)
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 03/07/2018 05:17 PM, Robert Bradshaw wrote:
>> > On Wed, Mar 7, 2018 at 12:50 AM Jean-Baptiste Onofré > > > wrote:
>> >
>> > For the record, the vote e-mail doesn't contain actual
>> MAVEN_VERSION and
>> > JDK_VERSION used to build.
>> >
>> >
>> > Sorry, it's Apache Maven 3.2.5 with Java version: 1.8.0_112. Hopefully
>> this
>> > isn't a deciding factor :).
>> >
>> >
>> > Regards
>> > JB
>> >
>> > On 03/07/2018 09:44 AM, Robert Bradshaw wrote:
>> > > Hi everyone,
>> > >
>> > > Please review and vote on the release candidate #1 for the
>> version 2.4.0,
>> > > as follows:
>> > > [ ] +1, Approve the release
>> > > [ ] -1, Do not approve the release (please provide specific
>> comments)
>> > >
>> > > The complete staging area is available for your review, which
>> includes:
>> > > * JIRA release notes [1],
>> > > * the official Apache source release to be deployed to
>> dist.apache.org
>> >  [2],
>> > > which is signed with the key with fingerprint BDC9 89B0 1BD2 A463
>> 6010
>> > >   A1CA 8F15 5E09 610D 69FB [3],
>> > > * all artifacts to be deployed to the Maven Central Repository
>> [4],
>> > > * source code tag "v2.4.0-RC1" [5],
>> > > * website pull request listing the release and publishing the API
>> reference
>> > > manual [6].
>> > > * Java artifacts were built with Maven MAVEN_VERSION and
>> OpenJDK/Oracle JDK
>> > > JDK_VERSION.
>> > > * Python artifacts are deployed along with the source release to
>> the
>> > > dist.apache.org  [2].
>> > >
>> > > The vote will be open for at least 72 hours. It is adopted by
>> majority
>> > > approval, with at least 3 PMC affirmative votes.
>> > >
>> > > Thanks,
>> > > - Robert
>> > >
>> > > [1]
>> > >
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12342682=12319527
>> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>> > > [3] https://dist.apache.org/repos/dist/dev/beam/KEYS
>> > > [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1028/
>> > > [5] https://github.com/apache/beam/tree/v2.4.0-RC1
>> > > [6] https://github.com/apache/beam-site/pull/398
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbono...@apache.org 
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

Re: Merging Python code? Help avoid Python 3 regressions with these two simple steps :)

[Alan Myrvold]

I ran "python3 --version" on each worker and all showed python 3.4.3. Is
that too old?


On Fri, Mar 2, 2018 at 10:04 AM Ahmet Altay  wrote:

> That is my understanding as well, it is requires attention from infra.
> Could anyone help with this? I know we worked with infra before, what is
> the best way to approach this?
>
> On Fri, Mar 2, 2018 at 9:50 AM, Holden Karau 
> wrote:
>
>> I agree, however I'm of the impression it's blocked on infra? (e.g. it's
>> important but out of my hands).
>>
>> On Mar 1, 2018 11:05 PM, "Ahmet Altay"  wrote:
>>
>>> I think we should prioritize the issue of installing Python 3 on the
>>> workers (https://issues.apache.org/jira/browse/BEAM-3671). I would
>>> appreciate if folks pay attention to these 2 steps but I am worried that it
>>> will be easily forgotten.
>>>
>>> On Thu, Mar 1, 2018 at 6:51 PM, Holden Karau 
>>> wrote:
>>>
 I may have watched too many buzzfeed videos this week but the steps are:
 1) git checkout the PR in question
 2) Run tox -e lint_py2,lint_py3

 This is important since Python 3 isn't installed on the Jenkins workers
 just yet and we have some tests to catch basic invalid Python 3 which we
 can slowly grow as we fix the issues and you can help us keep moving
 forward!

 If step 1 is too much work I like using the hub program I find it helps
 me with this part of my workflow in other projects. That being said you
 don't have to do this, we'll fix whatever errors come up, so if this is
 going to slow your workflow down or you otherwise don't like it feel free
 to pass along.

 --
 Twitter: https://twitter.com/holdenkarau

>>>
>>>
>

Re: Pain points in the 2.3.0 release / improvements to 2.4.0 release process?

[Alan Myrvold]

Thanks for the feedback.  Yifan and I have automated the Java quickstarts
for apex, direct, dataflow, flink local and spark [1] and automated python
release validation [2]. Yifan is working on fixing the Java mobile
archetype and validating the mobile example. The spark quickstart
automation came in late, but identified the [BEAM-3668] issue in RC2.

The Java quickstarts are now running daily with the snapshot release [3]
and are parameterized to make it easy to run with the next RC candidate.

Hope these help make the next release smoother. Open to ideas for other
areas to automate.

[1] https://github.com/apache/beam/tree/master/release/src/main/groovy
[2]
https://github.com/apache/beam/blob/master/release/src/main/groovy/run_release_candidate_python_validation.sh
[3]
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostRelease_NightlySnapshot/



On Wed, Feb 28, 2018 at 9:54 AM Lukasz Cwik <lc...@google.com> wrote:

> Validating the release by following quickstarts being a manual process I
> believe is still the largest pain point:
> * We missed that the archetypes were missing the mobile gaming examples.
> * The tcnative dependency conflict that we needed to cut RC2 for.
>
> Overall much smoother then the prior release but still a good amount of
> manual steps involved.
>
> On Tue, Feb 27, 2018 at 9:16 AM, Reuven Lax <re...@google.com> wrote:
>
>> Thanks for fixing the manual cleanup issue! This is something we kept
>> punting on in previous releases.
>>
>>
>> On Mon, Feb 26, 2018 at 7:14 PM Jean-Baptiste Onofré <j...@nanthrax.net>
>> wrote:
>>
>>> Hi Alan,
>>>
>>> Honestly,  it was an easy and smooth release, similar to other Apache
>>> project
>>> release.
>>>
>>> Maybe the points where we could a little bit improve are:
>>>
>>> 1. The website update was pretty long as @asfgit merge didn't work (due
>>> to a out
>>> of sync on the github mirror). I think the website publish PR can be
>>> automatized.
>>> 2. Upload to pylib required manual action (like renaming the files)
>>>
>>> Thanks to the change I did on the assembly, the artifacts don't require
>>> any
>>> manual cleanup (whereas it was the case in previous release).
>>>
>>> Regards
>>> JB
>>>
>>> On 02/26/2018 10:54 PM, Alan Myrvold wrote:
>>> > Is there a list of pain points during the 2.3.0 release and
>>> improvements that
>>> > can be made to the 2.4.0 release process?
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>

Pain points in the 2.3.0 release / improvements to 2.4.0 release process?

[Alan Myrvold]

Is there a list of pain points during the 2.3.0 release and improvements
that can be made to the 2.4.0 release process?

Re: [VOTE] Release 2.3.0, release candidate #3

[Alan Myrvold]

+1 Validated java quickstarts for direct, dataflow, apex, flink, and spark.

On Wed, Feb 14, 2018 at 9:21 AM, Lukasz Cwik  wrote:

> +1 (binding)
> Validated several quickstarts including the regression that I originally
> reported with Spark.
>
> On Wed, Feb 14, 2018 at 5:34 AM, Ismaël Mejía  wrote:
>
>> +1 (binding)
>>
>> Validated SHAs + tag vs source.zip file.
>> Run mvn clean install -Prelease OK
>> Validated that the 3 regressions reported for RC1 were fixed.
>> Run Nexmark on Direct/Flink runner on local mode, no regressions now.
>> Installed python version on virtualenv and run local wordcount with
>> success.
>> Checked that the hadoop-input-format artifact is in the extended staging
>> area.
>>
>> On Tue, Feb 13, 2018 at 5:41 PM, Jean-Baptiste Onofré 
>> wrote:
>> > +1 (binding)
>> >
>> > Tested the Spark runner (with wordcount example and beam samples)
>> > Tested the performance of the direct runner
>> >
>> > I just updated the spreadsheet.
>> >
>> > Regards
>> > JB
>> >
>> > On 02/11/2018 06:33 AM, Jean-Baptiste Onofré wrote:
>> >> Hi everyone,
>> >>
>> >> Please review and vote on the release candidate #3 for the version
>> 2.3.0, as
>> >> follows:
>> >>
>> >> [ ] +1, Approve the release
>> >> [ ] -1, Do not approve the release (please provide specific comments)
>> >>
>> >>
>> >> The complete staging area is available for your review, which includes:
>> >> * JIRA release notes [1],
>> >> * the official Apache source release to be deployed to dist.apache.org
>> [2],
>> >> which is signed with the key with fingerprint C8282E76 [3],
>> >> * all artifacts to be deployed to the Maven Central Repository [4],
>> >> * source code tag "v2.3.0-RC3" [5],
>> >> * website pull request listing the release and publishing the API
>> reference
>> >> manual [6].
>> >> * Java artifacts were built with Maven 3.3.9 and Oracle JDK 1.8.0_111.
>> >> * Python artifacts are deployed along with the source release to the
>> >> dist.apache.org [2].
>> >>
>> >> The vote will be open for at least 72 hours. It is adopted by majority
>> approval,
>> >> with at least 3 PMC affirmative votes.
>> >>
>> >> Thanks,
>> >> JB
>> >>
>> >> [1]
>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
>> ctId=12319527=12341608
>> >> [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
>> >> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> >> [4] https://repository.apache.org/content/repositories/orgapache
>> beam-1028/
>> >> [5] https://github.com/apache/beam/tree/v2.3.0-RC3
>> >> [6] https://github.com/apache/beam-site/pull/381
>> >>
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbono...@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>>
>
>

Re: [VOTE] Release 2.3.0, release candidate #1

[Alan Myrvold]

Yes, it is a dataflow step. Happy to test this again when they are
available.

On Wed, Jan 31, 2018 at 1:39 PM, Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:

> OK, I think I understood ;)
>
> So it's not "directly" related to Beam itself (it's more a Dataflow step
> to perform).
>
> @Alan, I think it's better to test first and then cast the vote. This kind
> of tests are valuable to validate the release and make sense. But vote
> should represent the state of the Beam release. So I think -1 vote is a bit
> too early before the test.
>
> Thanks !
> Regards
> JB
>
> On 31/01/2018 22:33, Reuven Lax wrote:
>
>> It's just a step that needs to be peformed before the new release works
>> on Dataflow. Alan is saying that we've been unable to validate Dataflow so
>> far, as worker images are not yet built. Hopefully they'll be built soon,
>> and we'll be able to validate.
>>
>> On Wed, Jan 31, 2018 at 1:31 PM, Jean-Baptiste Onofré <j...@nanthrax.net
>> <mailto:j...@nanthrax.net>> wrote:
>>
>> Hi Alan
>>
>> does it related to change in the codebase or in a dependency/related
>> project ?
>>
>> I mean: is it something we have to fix/change in Beam ?
>>
>> Just curious as I'm not sure what you mean by "worker images" ;)
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 31/01/2018 22:18, Alan Myrvold wrote:
>>
>> -1 (for now, hope to change this)
>>
>> Dataflow runner jobs are failing for me with 2.3.0 RC1, for both
>> Java and Python.
>>
>> This is not an issues with the 2.3.0 RC1 SDK, we (google) need
>> to release worker images.
>>
>> I have assigned these bugs to myself, and will be working to
>> help get these workers released.
>>
>> [BEAM-3584] Java dataflow job fails with 2.3.0 RC1, due to
>> missing worker image
>> [BEAM-3585] Python dataflow job fails with 2.3.0 RC1, due to
>> missing worker image
>>
>> On Wed, Jan 31, 2018 at 6:12 AM, Jean-Baptiste Onofré
>> <j...@nanthrax.net <mailto:j...@nanthrax.net>
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote:
>>
>>  Thanks Kenn,
>>
>>  I prepared the list of tasks I did. I will complete where
>> release is
>>  out.
>>
>>  Regards
>>  JB
>>
>>  On 01/31/2018 03:07 PM, Kenneth Knowles wrote:
>>  > I've cloned the release validation spreadsheet:
>>  >
>>  > https://s.apache.org/beam-2.3.0-release-validation
>> <https://s.apache.org/beam-2.3.0-release-validation>
>>  <https://s.apache.org/beam-2.3.0-release-validation
>> <https://s.apache.org/beam-2.3.0-release-validation>>
>>  >
>>  > If you plan to perform a manual validation task, please
>> sign up so multiple
>>  > people don't waste effort.
>>  >
>>  > Alan & JB, as far as your pairing up to automate more,
>> anything manual on this
>>  > sheet should be considered.
>>  >
>>  > Kenn
>>  >
>>  > On Wed, Jan 31, 2018 at 5:59 AM, Jean-Baptiste Onofré
>> <j...@nanthrax.net <mailto:j...@nanthrax.net>
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>
>>  > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>
>> <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>>> wrote:
>>  >
>>  > +1 (binding)
>>  >
>>  > Casting my own +1 ;)
>>  >
>>  > Regards
>>  > JB
>>  >
>>  > On 01/30/2018 09:04 AM, Jean-Baptiste Onofré wrote:
>>  > > Hi everyone,
>>  > >
>>  > > Please review and vote on the release candidate #1
>> for the version 2.3.0, as
>>  > > follows:
>>  > >
>>  > > [ ] +1, Approve the release
>>  > > [ ] -1, Do not approve the release (please provide
>>

Re: [VOTE] Release 2.3.0, release candidate #1

[Alan Myrvold]

-1 (for now, hope to change this)

Dataflow runner jobs are failing for me with 2.3.0 RC1, for both Java and
Python.

This is not an issues with the 2.3.0 RC1 SDK, we (google) need to release
worker images.

I have assigned these bugs to myself, and will be working to help get these
workers released.

[BEAM-3584] Java dataflow job fails with 2.3.0 RC1, due to missing worker
image
[BEAM-3585] Python dataflow job fails with 2.3.0 RC1, due to missing worker
image

On Wed, Jan 31, 2018 at 6:12 AM, Jean-Baptiste Onofré 
wrote:

> Thanks Kenn,
>
> I prepared the list of tasks I did. I will complete where release is out.
>
> Regards
> JB
>
> On 01/31/2018 03:07 PM, Kenneth Knowles wrote:
> > I've cloned the release validation spreadsheet:
> >
> > https://s.apache.org/beam-2.3.0-release-validation
> >
> > If you plan to perform a manual validation task, please sign up so
> multiple
> > people don't waste effort.
> >
> > Alan & JB, as far as your pairing up to automate more, anything manual
> on this
> > sheet should be considered.
> >
> > Kenn
> >
> > On Wed, Jan 31, 2018 at 5:59 AM, Jean-Baptiste Onofré  > > wrote:
> >
> > +1 (binding)
> >
> > Casting my own +1 ;)
> >
> > Regards
> > JB
> >
> > On 01/30/2018 09:04 AM, Jean-Baptiste Onofré wrote:
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the version
> 2.3.0, as
> > > follows:
> > >
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > >
> > >
> > > The complete staging area is available for your review, which
> includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to
> dist.apache.org
> >  [2],
> > > which is signed with the key with fingerprint C8282E76 [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "v2.3.0-RC1" [5],
> > > * website pull request listing the release and publishing the API
> reference
> > > manual [6].
> > > * Java artifacts were built with Maven 3.3.9 and Oracle JDK
> 1.8.0_111.
> > > * Python artifacts are deployed along with the source release to
> the
> > > dist.apache.org  [2].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > approval,
> > > with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > JB
> > >
> > > [1]
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527=12341608
> >  projectId=12319527=12341608>
> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.3.0/
> > 
> > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > 
> > > [4] https://repository.apache.org/content/repositories/
> orgapachebeam-1026/
> >  orgapachebeam-1026/>
> > > [5] https://github.com/apache/beam/tree/v2.3.0-RC1
> > 
> > > [6] https://github.com/apache/beam-site/pull/381
> > 
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [HEADS UP] Preparing Beam 2.3.0

[Alan Myrvold]

Great. I'm interested in any pain points in the process, and steps we can
automate. Can you keep a rough log of issues you run into?

On Mon, Jan 29, 2018 at 10:16 AM, Jean-Baptiste Onofré 
wrote:

> We are now ready for the release. I'm starting the process.
>
> Thanks again for your help and support !
>
> Regards
> JB
>
> On 01/29/2018 02:22 PM, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > new update (and I hope the last one ;)):
> >
> > - BEAM-3551 has been reviewed and merged. This Jira is fixed for 2.3.0.
> > - I just updated the PR for BEAM-793 (testing with write with backoff
> feature in
> > JdbcIO). I hope Eugene will give his green light to merge ;)
> > - we have a new Jira (BEAM-3559) that has been created this morning and
> targeted
> > to 2.3.0. I asked for detail in the Jira and no update yet. Can I have
> an update
> > on this one ? Else I will bump it to 2.4.0.
> >
> > I will cut the 2.3.0 release tonight my time then.
> >
> > Thanks,
> > Regards
> > JB
> >
> > On 01/29/2018 07:14 AM, Jean-Baptiste Onofré wrote:
> >> Hi,
> >>
> >> As BEAM-793 and BEAM-3551 are not release blocker, if I don't have
> update in the
> >> PRs in the coming couple of hours, I will bump these Jiras to 2.4.0 and
> start
> >> the release process.
> >>
> >> Thanks !
> >> Regards
> >> JB
> >>
> >> On 01/28/2018 08:29 PM, Jean-Baptiste Onofré wrote:
> >>> Hi guys,
> >>>
> >>> a new update about 2.3.0 release preparation. Only 2 Jira are pending
> (not
> >>> blocker for the release but good to have):
> >>>
> >>> - BEAM-793: I updated the PR (#4500), waiting Eugene's feedback.
> >>> - BEAM-3551: I created a PR (#4516), waiting Luke's feedback.
> >>>
> >>> Thanks to the timezone, I hope to have feedbacks soon.
> >>>
> >>> Anyway, these Jiras are not release blocker, so I will start the
> release process
> >>> tomorrow morning my time.
> >>>
> >>> By the way, I created 2.4.0 version in Jira.
> >>>
> >>> Thanks all for your help and support !
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 27/01/2018 13:00, Jean-Baptiste Onofré wrote:
>  Hi guys,
> 
>  we still have 7 Jira targeted to 2.3.0.
> 
>  For most of them, Ismaël and I are doing the PRs/fixes and we have
> review in
>  progress.
> 
>  I'm a little bit concerned by BEAM-3392: it's flagged as blocker but
> it's
>  related to a specific branch. Can you please provide an update asap ?
> 
>  However, I didn't have any update for BEAM-3087 (related to the Flink
> runner).
>  Without update soon, I will bump to 2.4.0.
> 
>  I would need a review on PR for BEAM-793 (PR #4500). To avoid to
> break anything
>  for existing user, I set the backoff strategy optional, the user has
> to
>  explicitly set to use it.
> 
>  I'm waiting a little more before cutting the release (especially for
> BEAM-3392
>  and BEAM-3087). However, I would like to cut the release asap.
> 
>  Thanks,
>  Regards
>  JB
> 
> 
>  On 01/23/2018 10:39 AM, Jean-Baptiste Onofré wrote:
> > Hi guys,
> >
> > Some days ago, I proposed to start Beam 2.3.0 around January 26th.
> So, we are
> > few days from this date.
> >
> > As a best effort, can you please in Jira flag the Jira with fix
> version 2.3.0
> > and blocker for the release. Then, I will know when I can start the
> release
> > process.
> >
> > Thanks !
> >
> > Regards
> > JB
> >
> 
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [HEADS UP] Preparing Beam 2.3.0

[Alan Myrvold]

Are there improvements to the release process to make the triaging of open
issues easier?

I see 30 unresolved issues with fixVersion 2.3.0 in
https://issues.apache.org/jira/browse/BEAM-3160?jql=project%20%3D%20BEAM%20AND%20status%20%3D%20Open%20AND%20fixVersion%20%3D%202.3.0%20ORDER%20BY%20updated%20DESC

The release process,
https://beam.apache.org/contribute/release-guide/#create-a-new-version-in-jira
indicates that all these issues will be moved to 2.4.0 if they are not
resolved.  Would it be better to unset the fixVersion for any issues that
aren't fixed instead of moving them to the next version?

On Tue, Jan 23, 2018 at 12:16 PM, Jean-Baptiste Onofré 
wrote:

> Awesome !
>
> Ready to review if you want !
>
> Regards
> JB
>
> On 23/01/2018 20:48, Eugene Kirpichov wrote:
>
>> Regarding Java 8, I'm about to send a large mechanical PR converting a
>> lot of code to use lambdas and better type inference.
>>
>> On Tue, Jan 23, 2018 at 11:37 AM Jean-Baptiste Onofré > > wrote:
>>
>> By the way, I would like to complete the Java 8 support (the subtasks
>> in
>> the main Java 8 Jira). So, let me try to move forward on this.
>>
>> Regards
>> JB
>>
>> On 23/01/2018 19:51, Reuven Lax wrote:
>>  > Sounds good. Also many things that are currently flagged as 2.3.0
>> don't
>>  > appear to be actual release blockers to me.
>>  >
>>  > On Tue, Jan 23, 2018 at 1:39 AM, Jean-Baptiste Onofré
>> 
>>  > >> wrote:
>>  >
>>  > Hi guys,
>>  >
>>  > Some days ago, I proposed to start Beam 2.3.0 around January
>> 26th.
>>  > So, we are
>>  > few days from this date.
>>  >
>>  > As a best effort, can you please in Jira flag the Jira with fix
>>  > version 2.3.0
>>  > and blocker for the release. Then, I will know when I can
>> start the
>>  > release process.
>>  >
>>  > Thanks !
>>  >
>>  > Regards
>>  > JB
>>  > --
>>  > Jean-Baptiste Onofré
>>  > jbono...@apache.org 
>> >
>>  > http://blog.nanthrax.net
>>  > Talend - http://www.talend.com
>>  >
>>  >
>>
>>

Re: Dataflow runner examples build fail

[Alan Myrvold]

The google cloud project apache-beam-testing was near quota. There were ~28
old dataflow jobs running, mostly python sdk. robertwb and I cancelled the
old jobs and will look why these got stuck.
After cancelling the old jobs, the apache-beam-testing project is no longer
near quota.

On Tue, Jan 9, 2018 at 2:20 AM, Łukasz Gajowy 
wrote:

> +1
>
> 2018-01-08 22:38 GMT+01:00 Daniel Oliveira :
>
>> +1
>>
>> On Mon, Jan 8, 2018 at 10:07 AM, Kenneth Knowles  wrote:
>>
>>> +1
>>>
>>> On Mon, Jan 8, 2018 at 9:33 AM, Henning Rohde 
>>> wrote:
>>>
 +1

 On Mon, Jan 8, 2018 at 1:32 AM, Ted Yu  wrote:

> +1
>
>  Original message 
> From: Jean-Baptiste Onofré 
> Date: 1/8/18 1:26 AM (GMT-08:00)
> To: dev@beam.apache.org
> Subject: Dataflow runner examples build fail
>
> Hi guys,
>
> The PRs and nightly builds are failing due to an issue with the
> dataflow
> platform: it seems we have a disk quota exceeded on the us-central1
> region.
>
> I would like to do a clean out and increase the quota a bit.
>
> Thoughts ?
>
> Thanks
> Regards
> JB
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>


>>>
>>
>

Re: [DISCUSS] Towards Beam 2.3.0

[Alan Myrvold]

+1 on having a release started in January and not holding on feature
requests.

I'd like to learn more about the release process in order to automate parts
of it. I'm not volunteering to do the release, but would like to pair with
whomever is doing the release to discuss pain points and help in any way I
can during the process. JB - be happy to pair with you over slack if you're
willing.

Alan

On Mon, Jan 8, 2018 at 10:01 AM, Robert Bradshaw 
wrote:

> +1 for starting the 2.3 ball rolling.
>
> In general, I'd like to avoid holding up releases for specific
> features/PRs. The is (one of the things) that holds up releases which
> then is a vicious cycle for more people wanting to make their feature
> a condition of the next release, etc. (Bugs, regressions, and
> backwards-incompatible refinements to new features are fair candidates
> as the need arises...)
>
> On Mon, Jan 8, 2018 at 7:04 AM, Jean-Baptiste Onofré 
> wrote:
> > Hi Romain,
> >
> > no problem: let's try a best effort and define the target version in the
> > Jira.
> >
> > Regards
> > JB
> >
> > On 01/08/2018 03:51 PM, Romain Manni-Bucau wrote:
> >>
> >> Hi JB,
> >>
> >> I'd like https://github.com/apache/beam/pull/4235 to be integrated if
> >> possible
> >>
> >> Also the JUnit 5 PR brings some light changes which can be worth the "3"
> >> digit upgrade so if anyone has some time to review it can be a good
> >> candidate too.
> >>
> >> Thanks for driving it
> >>
> >>
> >> Romain Manni-Bucau
> >> @rmannibucau  | Blog
> >>  | Old Blog
> >>  | Github  rmannibucau>
> >> | LinkedIn 
> >>
> >> 2018-01-08 15:37 GMT+01:00 Jean-Baptiste Onofré  >> >:
> >>
> >> Hi guys,
> >>
> >> In a previous discussion thread, we agreed that we should have a
> >> regular
> >> pace in term of releases.
> >>
> >> We released Beam 2.2.0 on the 16th of November '17, but the release
> >> takes a
> >> pretty long time.
> >>
> >> I think it's reasonable to think about Beam 2.3.0 in the coming
> weeks.
> >> I
> >> would like to propose target Beam 2.3.0 for end January/beginning of
> >> February.
> >>
> >> I'm volunteer to do this release.
> >>
> >> Thoughts ?
> >>
> >> Regards
> >> JB
> >> -- Jean-Baptiste Onofré
> >> jbono...@apache.org 
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >>
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>

Re: A personal update

[Alan Myrvold]

Welcome back! Look forward to your contributions.

On Wed, Dec 13, 2017 at 11:04 AM, Amit Sela  wrote:

> Congrats and welcome back! can't wait..
>
> On Wed, Dec 13, 2017 at 1:30 PM Mingmin Xu  wrote:
>
>> Welcome back and best wishes for your new phase!
>>
>> On Wed, Dec 13, 2017 at 10:05 AM, Raghu Angadi 
>> wrote:
>>
>>> Great to have you back Davor! New venture sounds like terrific news for
>>> Apache Beam. All the best.
>>>
>>> On Wed, Dec 13, 2017 at 9:33 AM, Thomas Groh  wrote:
>>>
 It's good to see you around. Welcome back.

 On Wed, Dec 13, 2017 at 8:43 AM, Chamikara Jayalath <
 chamik...@google.com> wrote:

> Welcome back :)
>
> - Cham
>
> On Wed, Dec 13, 2017 at 8:41 AM Jason Kuster 
> wrote:
>
>> Glad to have you back. :)
>>
>> On Wed, Dec 13, 2017 at 8:32 AM, Eugene Kirpichov <
>> kirpic...@google.com> wrote:
>>
>>> Happy to see you return, and thank you again for all you've done so
>>> far!
>>>
>>> On Wed, Dec 13, 2017 at 10:24 AM Aljoscha Krettek <
>>> aljos...@apache.org> wrote:
>>>
 Welcome back! :-)

 > On 13. Dec 2017, at 15:42, Ismaël Mejía 
 wrote:
 >
 > Hello Davor, great to know you are going to continue contributing
 to
 > the project. Welcome back and best of wishes for this new phase !
 >
 > On Wed, Dec 13, 2017 at 3:12 PM, Kenneth Knowles 
 wrote:
 >> Great to have you back!
 >>
 >> On Tue, Dec 12, 2017 at 11:20 PM, Robert Bradshaw <
 rober...@google.com>
 >> wrote:
 >>>
 >>> Great to hear from you again, and really happy you're sticking
 around!
 >>>
 >>> - Robert
 >>>
 >>>
 >>> On Tue, Dec 12, 2017 at 10:47 PM, Ahmet Altay 
 wrote:
  Welcome back! Looking forward to your contributions.
 
  Ahmet
 
  On Tue, Dec 12, 2017 at 10:05 PM, Jesse Anderson
  
  wrote:
 >
 > Congrats!
 >
 >
 > On Wed, Dec 13, 2017, 5:54 AM Jean-Baptiste Onofré <
 j...@nanthrax.net>
 > wrote:
 >>
 >> Hi Davor,
 >>
 >> welcome back !!
 >>
 >> It's really great to see you back active in the Beam
 community. We
 >> really
 >> need you !
 >>
 >> I'm so happy !
 >>
 >> Regards
 >> JB
 >>
 >> On 12/13/2017 05:51 AM, Davor Bonaci wrote:
 >>> My dear friends,
 >>> As many of you have noticed, I’ve been visibly absent from
 the
 >>> project
 >>> for a
 >>> little while. During this time, a great number of you kept
 reaching
 >>> out, and for
 >>> that I’m deeply humbled and grateful to each and every one
 of you.
 >>>
 >>> I needed some time for personal reflection, which led to a
 >>> transition
 >>> in my
 >>> professional life. As things have settled, I’m happy to
 again be
 >>> working among
 >>> all of you, as we propel this project forward. I plan to be
 active
 >>> in
 >>> the
 >>> future, but perhaps not quite full-time as I was before.
 >>>
 >>> In the near term, I’m working on getting the report to the
 Board
 >>> completed, as
 >>> well as framing the discussion about the project state and
 vision
 >>> going
 >>> forwards. Additionally, I’ll make sure that we foster
 healthy
 >>> community
 >>> culture
 >>> and operate in the Apache Way.
 >>>
 >>> For those who are curious, I’m happy to say that I’m
 starting a
 >>> company
 >>> building
 >>> products related to Beam, along with several other members
 of this
 >>> community and
 >>> authors of this technology. I’ll share more on this next
 year, but
 >>> until then if
 >>> you have a data processing problem or an Apache Beam
 question, I’d
 >>> love
 >>> to hear
 >>> from you ;-).
 >>>
 >>> Thanks -- and so happy to be back!
 >>>
 >>> Davor
 >>
 >> --
 >> Jean-Baptiste Onofré
 >> jbono...@apache.org