Re: [ANNOUNCE] New PMC Member: Chamikara Jayalath

[Yifan Zou]

Yay! Congratulations!

On Thu, Jan 21, 2021 at 3:18 PM Rui Wang  wrote:

> Congratulations, Cham!
>
> -Rui
>
> On Thu, Jan 21, 2021 at 3:15 PM Robert Bradshaw 
> wrote:
>
>> Congratulations, Cham!
>>
>> On Thu, Jan 21, 2021 at 3:13 PM Brian Hulette 
>> wrote:
>>
>>> Great news, congratulations Cham!
>>>
>>> On Thu, Jan 21, 2021 at 3:08 PM Robin Qiu  wrote:
>>>
 Congratulations, Cham!

 On Thu, Jan 21, 2021 at 3:05 PM Tyson Hamilton 
 wrote:

> Woo! Congrats Cham!
>
> On Thu, Jan 21, 2021 at 3:02 PM Robert Burke 
> wrote:
>
>> Congratulations! That's fantastic news.
>>
>> On Thu, Jan 21, 2021, 2:59 PM Reza Rokni  wrote:
>>
>>> Congratulations!
>>>
>>> On Fri, Jan 22, 2021 at 6:58 AM Ankur Goenka 
>>> wrote:
>>>
 Congrats Cham!

 On Thu, Jan 21, 2021 at 2:57 PM Ahmet Altay 
 wrote:

> Hi all,
>
> Please join me and the rest of Beam PMC in welcoming Chamikara
> Jayalath as our
> newest PMC member.
>
> Cham has been part of the Beam community from its early days and
> contributed to the project in significant ways, including 
> contributing new
> features and improvements especially related Beam IOs, advocating for
> users, and mentoring new community members.
>
> Congratulations Cham! And thanks for being a part of Beam!
>
> Ahmet
>

Re: [ANNOUNCE] New committer: Robin Qiu

[Yifan Zou]

Congratulations, Robin!

On Tue, May 19, 2020 at 10:53 AM Udi Meiri  wrote:

> Congratulations Robin!
>
> On Tue, May 19, 2020, 10:15 Valentyn Tymofieiev 
> wrote:
>
>> Congratulations, Robin!
>>
>> On Tue, May 19, 2020 at 9:10 AM Yichi Zhang  wrote:
>>
>>> Congrats Robin!
>>>
>>> On Tue, May 19, 2020 at 8:56 AM Kamil Wasilewski <
>>> kamil.wasilew...@polidea.com> wrote:
>>>
 Congrats!

 On Tue, May 19, 2020 at 5:33 PM Jan Lukavský  wrote:

> Congrats Robin!
> On 5/19/20 5:01 PM, Tyson Hamilton wrote:
>
> Congratulations!
>
> On Tue, May 19, 2020 at 6:10 AM Omar Ismail 
> wrote:
>
>> Congrats!
>>
>> On Tue, May 19, 2020 at 5:00 AM Gleb Kanterov 
>> wrote:
>>
>>> Congratulations!
>>>
>>> On Tue, May 19, 2020 at 7:31 AM Aizhamal Nurmamat kyzy <
>>> aizha...@apache.org> wrote:
>>>
 Congratulations, Robin! Thank you for your contributions!

 On Mon, May 18, 2020, 7:18 PM Boyuan Zhang 
 wrote:

> Congrats~~
>
> On Mon, May 18, 2020 at 7:17 PM Reza Rokni  wrote:
>
>> Congratulations!
>>
>> On Tue, May 19, 2020 at 10:06 AM Ahmet Altay 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Robin Qiu .
>>>
>>> Robin has been active in the community for close to 2 years,
>>> worked on HyperLogLog++ [1], SQL [2], improved documentation, and 
>>> helped
>>> with releases(*).
>>>
>>> In consideration of his contributions, the Beam PMC trusts him
>>> with the responsibilities of a Beam committer [3].
>>>
>>> Thank you for your contributions Robin!
>>>
>>> -Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://www.meetup.com/Zurich-Apache-Beam-Meetup/events/265529665/
>>> [2]
>>> https://www.meetup.com/Belgium-Apache-Beam-Meetup/events/264933301/
>>> [3] https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>> (*) And maybe he will be a release manager soon :)
>>>
>>> --
>>
>> Omar Ismail |  Technical Solutions Engineer |  omarism...@google.com
>>  |
>>
>

Re: CommunityMetrics precommit failing

[Yifan Zou]

Tracked in https://issues.apache.org/jira/browse/BEAM-7405

I've re-installed docker-credential-gcloud in beam1 to see if that works.


On Tue, Apr 7, 2020 at 2:40 PM Pablo Estrada  wrote:

> +Yifan Zou 
>
> On Tue, Apr 7, 2020 at 7:30 AM Kamil Wasilewski <
> kamil.wasilew...@polidea.com> wrote:
>
>> By the way, we have a JIRA for this:
>> https://issues.apache.org/jira/browse/BEAM-8409
>>
>> It would be great if someone with SSH access to the Jenkins workers took
>> care of that.
>>
>> On Tue, Apr 7, 2020 at 3:04 PM Michał Walenia 
>> wrote:
>>
>>> Hi all,
>>> I noticed that recently the community metrics precommit job started
>>> failing with
>>>
>>> dockerpycreds.errors.InitializationError: docker-credential-gcloud not 
>>> installed or not available in PATH
>>>
>>>
>>> error. Here's a link to a failing job:
>>>
>>> https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Phrase/65/console
>>>
>>>
>>> Does anyone know how to fix this? I think we need to patch the Jenkins 
>>> workers somehow.
>>>
>>>
>>> Have a good one!
>>>
>>> --
>>>
>>> Michał Walenia
>>> Polidea <https://www.polidea.com/> | Software Engineer
>>>
>>> M: +48 791 432 002 <+48791432002>
>>> E: michal.wale...@polidea.com
>>>
>>> Unique Tech
>>> Check out our projects! <https://www.polidea.com/our-work>
>>>
>>

Re: Unable to get java presubmit to run (not pass)

[Yifan Zou]

I tried 'Run Java PreCommit' in the PR and it created the job #1734 in
https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PreCommit_Java_Phrase/.
The job is in the waiting queue right now. But, I am not sure why the
`Retest this please` failed executing all necessary tests.

On Tue, Feb 11, 2020 at 9:48 AM Daniel Collins  wrote:

> Hello beam developers,
>
> I've been trying to rerun presubmits on
> https://github.com/apache/beam/pull/10476 quite a few times, but it keeps
> stalling out at the "Java ("Run Java PreCommit") Pending — Build triggered
> for merge commit." Is there currently a problem with the jenkins cluster?
>
> Thanks!
>

Re: Jenkins outage

[Yifan Zou]

Cleaned the beam7, it should be okay.

On Fri, Feb 7, 2020 at 9:42 AM Yifan Zou  wrote:

> I'll look into the 'no space' issue.
>
> On Fri, Feb 7, 2020 at 7:14 AM Ismaël Mejía  wrote:
>
>> mmm apache-beam-jenkins-14 also has issues:
>>
>> *16:08:47* ERROR: Error cloning remote repo 'origin'*16:08:47* 
>> hudson.plugins.git.GitException: Could not init 
>> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark_PR/src
>>
>> https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark_PR/6/console
>>
>>
>> On Fri, Feb 7, 2020 at 3:14 PM Ismaël Mejía  wrote:
>>
>>> I am getting "java.lang.IllegalStateException: java.io.IOException: No
>>> space left on device" on  apache-beam-jenkins-7
>>> Can somebody please clean the space. Thanks.
>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark_PR/26/
>>>
>>> On Fri, Feb 7, 2020 at 2:19 PM Ismaël Mejía  wrote:
>>>
>>>> Thanks for taking care of this issue with INFRA Michał.
>>>> Everything back to normal!
>>>>
>>>>
>>>> On Fri, Feb 7, 2020 at 11:34 AM Michał Walenia <
>>>> michal.wale...@polidea.com> wrote:
>>>>
>>>>> Everything looks fine now, the jobs are triggering correctly again
>>>>>
>>>>> On Fri, Feb 7, 2020 at 10:06 AM Michał Walenia <
>>>>> michal.wale...@polidea.com> wrote:
>>>>>
>>>>>> Hi there,
>>>>>> it seems that our Jenkins is experiencing some issues and the jobs
>>>>>> are getting stuck in the queue despite the executors being idle.
>>>>>> Here's the JIRA issue for this:
>>>>>> https://issues.apache.org/jira/browse/INFRA-19830
>>>>>> Let's hope it will be resolved soon.
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Michał Walenia
>>>>>> Polidea <https://www.polidea.com/> | Software Engineer
>>>>>>
>>>>>> M: +48 791 432 002 <+48791432002>
>>>>>> E: michal.wale...@polidea.com
>>>>>>
>>>>>> Unique Tech
>>>>>> Check out our projects! <https://www.polidea.com/our-work>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Michał Walenia
>>>>> Polidea <https://www.polidea.com/> | Software Engineer
>>>>>
>>>>> M: +48 791 432 002 <+48791432002>
>>>>> E: michal.wale...@polidea.com
>>>>>
>>>>> Unique Tech
>>>>> Check out our projects! <https://www.polidea.com/our-work>
>>>>>
>>>>

Re: Jenkins outage

[Yifan Zou]

I'll look into the 'no space' issue.

On Fri, Feb 7, 2020 at 7:14 AM Ismaël Mejía  wrote:

> mmm apache-beam-jenkins-14 also has issues:
>
> *16:08:47* ERROR: Error cloning remote repo 'origin'*16:08:47* 
> hudson.plugins.git.GitException: Could not init 
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Go_VR_Spark_PR/src
>
> https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark_PR/6/console
>
>
> On Fri, Feb 7, 2020 at 3:14 PM Ismaël Mejía  wrote:
>
>> I am getting "java.lang.IllegalStateException: java.io.IOException: No
>> space left on device" on  apache-beam-jenkins-7
>> Can somebody please clean the space. Thanks.
>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark_PR/26/
>>
>> On Fri, Feb 7, 2020 at 2:19 PM Ismaël Mejía  wrote:
>>
>>> Thanks for taking care of this issue with INFRA Michał.
>>> Everything back to normal!
>>>
>>>
>>> On Fri, Feb 7, 2020 at 11:34 AM Michał Walenia <
>>> michal.wale...@polidea.com> wrote:
>>>
 Everything looks fine now, the jobs are triggering correctly again

 On Fri, Feb 7, 2020 at 10:06 AM Michał Walenia <
 michal.wale...@polidea.com> wrote:

> Hi there,
> it seems that our Jenkins is experiencing some issues and the jobs are
> getting stuck in the queue despite the executors being idle.
> Here's the JIRA issue for this:
> https://issues.apache.org/jira/browse/INFRA-19830
> Let's hope it will be resolved soon.
>
> --
>
> Michał Walenia
> Polidea  | Software Engineer
>
> M: +48 791 432 002 <+48791432002>
> E: michal.wale...@polidea.com
>
> Unique Tech
> Check out our projects! 
>


 --

 Michał Walenia
 Polidea  | Software Engineer

 M: +48 791 432 002 <+48791432002>
 E: michal.wale...@polidea.com

 Unique Tech
 Check out our projects! 

>>>

Re: [ANNOUNCE] New committer: Hannah Jiang

[Yifan Zou]

Congratulations Hannah!!

On Tue, Jan 28, 2020 at 3:55 PM Boyuan Zhang  wrote:

> Thanks for all your contributions! Congratulations~
>
> On Tue, Jan 28, 2020 at 3:44 PM Pablo Estrada  wrote:
>
>> yoooho : D
>>
>> On Tue, Jan 28, 2020 at 3:21 PM Luke Cwik  wrote:
>>
>>> Hi everyone,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new
>>> committer: Hannah Jiang
>>>
>>> Hannah has contributed to Beam in many ways, including work on building
>>> and releasing the Apache Beam SDK containers.
>>>
>>> In consideration of their contributions, the Beam PMC trusts them with
>>> the responsibilities of a Beam committer[1].
>>>
>>> Thanks for your contributions Hannah!
>>>
>>> Luke, on behalf of the Apache Beam PMC.
>>>
>>> [1]
>>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>>>
>>

Re: Jenkins Master

[Yifan Zou]

Great! We'll follow up with Gavin to set up some nodes with the new master.

On Thu, Jan 23, 2020 at 11:29 AM Pablo Estrada  wrote:

> That makes sense Yifan. I think that's fine, to use two new instances.
>
> On Thu, Jan 23, 2020 at 11:28 AM Yifan Zou  wrote:
>
>> Thanks for updates on the Jenkins infrastructures!
>> The Beam nodes are currently connected via JNLP. Since all existing nodes
>> are heavy loaded, I'd prefer to have 1 or 2 temporal instances boot by the
>> most recent disk image for testing purposes.
>>
>> Thoughts? Objections?
>>
>> -yifan
>>
>>
>> On Thu, Jan 23, 2020 at 11:14 AM Ahmet Altay  wrote:
>>
>>> /cc +Alan Myrvold  +Yifan Zou 
>>>  +Dan Gazineu  -- cc'ing folks who worked on
>>> Jenkins related tasks before.
>>>
>>> On Thu, Jan 23, 2020 at 10:10 AM Pablo Estrada 
>>> wrote:
>>>
>>>> Removing private@, and adding dev@.
>>>>
>>>> ASF Infra has provisioned a new Jenkins master that we'll be able to
>>>> use with more independence. Now we can start by testing a couple workers on
>>>> it, and eventually moving all of our workers.
>>>>
>>>> Is anyone available to engage with Gavin in moving this forward?
>>>> Best
>>>> -P.
>>>>
>>>> On Mon, Jan 20, 2020 at 1:12 AM Gavin McDonald 
>>>> wrote:
>>>>
>>>>> Hi Beam PMC.
>>>>>
>>>>> A new Jenkins Master (connected to a Cloudbees Core Operations Center)
>>>>> has been
>>>>> created for your projects use. [1]
>>>>>
>>>>> It is ready to go, but needs your Beam nodes moving over from the
>>>>> current Jenkins instance.
>>>>>
>>>>> I'd like to move 1 or 2, let you guys test, and then on approval, move
>>>>> the rest.
>>>>>
>>>>> It looks like you are currently using JNLP to connect?
>>>>> I much prefer static IP and to connect via secure and more reliable
>>>>> SSH connection, would this be possible?
>>>>>
>>>>> Please pick 1 or two nodes to migrate over, or provide two new nodes ,
>>>>> to connect to your
>>>>> new master.
>>>>>
>>>>> Web access is via LDAP, and only Beam PMC/Committers have access to
>>>>> create jobs.
>>>>>
>>>>> Enjoy!
>>>>>
>>>>> I am not subscribed to your private list, so please ensure I am
>>>>> included in any reply, thanks.
>>>>>
>>>>> Gav... (ASF Infra.)
>>>>>
>>>>> [1] - https://ci-beam.apache.org/
>>>>>
>>>>>

Re: Jenkins Master

[Yifan Zou]

Thanks for updates on the Jenkins infrastructures!
The Beam nodes are currently connected via JNLP. Since all existing nodes
are heavy loaded, I'd prefer to have 1 or 2 temporal instances boot by the
most recent disk image for testing purposes.

Thoughts? Objections?

-yifan


On Thu, Jan 23, 2020 at 11:14 AM Ahmet Altay  wrote:

> /cc +Alan Myrvold  +Yifan Zou  +Dan
> Gazineu  -- cc'ing folks who worked on Jenkins
> related tasks before.
>
> On Thu, Jan 23, 2020 at 10:10 AM Pablo Estrada  wrote:
>
>> Removing private@, and adding dev@.
>>
>> ASF Infra has provisioned a new Jenkins master that we'll be able to use
>> with more independence. Now we can start by testing a couple workers on it,
>> and eventually moving all of our workers.
>>
>> Is anyone available to engage with Gavin in moving this forward?
>> Best
>> -P.
>>
>> On Mon, Jan 20, 2020 at 1:12 AM Gavin McDonald 
>> wrote:
>>
>>> Hi Beam PMC.
>>>
>>> A new Jenkins Master (connected to a Cloudbees Core Operations Center)
>>> has been
>>> created for your projects use. [1]
>>>
>>> It is ready to go, but needs your Beam nodes moving over from the
>>> current Jenkins instance.
>>>
>>> I'd like to move 1 or 2, let you guys test, and then on approval, move
>>> the rest.
>>>
>>> It looks like you are currently using JNLP to connect?
>>> I much prefer static IP and to connect via secure and more reliable SSH
>>> connection, would this be possible?
>>>
>>> Please pick 1 or two nodes to migrate over, or provide two new nodes ,
>>> to connect to your
>>> new master.
>>>
>>> Web access is via LDAP, and only Beam PMC/Committers have access to
>>> create jobs.
>>>
>>> Enjoy!
>>>
>>> I am not subscribed to your private list, so please ensure I am included
>>> in any reply, thanks.
>>>
>>> Gav... (ASF Infra.)
>>>
>>> [1] - https://ci-beam.apache.org/
>>>
>>>

Re: Is Jenkins stuck?

[Yifan Zou]

Seems like a global issue, the same was happening on kafka and Hive. ASF
Infra restarted the master and the jobs start executing now.
https://issues.apache.org/jira/browse/INFRA-19722
https://issues.apache.org/jira/browse/INFRA-19721



On Tue, Jan 14, 2020 at 12:54 PM Luke Cwik  wrote:

> Stuck for me as well. I tried rerunning them and it did work sometimes.
>
> On Tue, Jan 14, 2020 at 12:39 PM Andrew Pilloud 
> wrote:
>
>> I've noticed the same thing. The instances I've investigated have
>> actually ran the tests, but the status never got posted to github.
>>
>> On Tue, Jan 14, 2020 at 12:36 PM Boyuan Zhang  wrote:
>>
>>> Same here: https://github.com/apache/beam/pull/10375, at least has been
>>> pended for 15 hrs
>>>
>>> On Tue, Jan 14, 2020 at 12:31 PM Reuven Lax  wrote:
>>>
 For https://github.com/apache/beam/pull/10534 , all the tests have
 been yellow "pending" for the past 15 hours. The same was true before when
 I tried forcing then all to run again, but the new set of tests is also
 stuck. Does anyone know what's happening?

 Reuven

>>>

Re: Jenkins jobs not running for my PR 10438

[Yifan Zou]

done.

On Sun, Jan 12, 2020 at 6:27 PM Tomo Suzuki  wrote:

> Hi Beam committers,
>
> Four Jenkins jobs did not report back for this PR
> https://github.com/apache/beam/pull/10554 .
> Can somebody trigger them?
>
> On Fri, Jan 10, 2020 at 4:51 PM Andrew Pilloud 
> wrote:
> >
> > Done.
> >
> > On Fri, Jan 10, 2020 at 12:59 PM Tomo Suzuki  wrote:
> >>
> >> Hi Bean developers,
> >>
> >> I appreciate a committer can trigger precommit build for
> >> https://github.com/apache/beam/pull/10554.
> >>
> >> In addition to normal precommit checks, I want the followings:
> >> Run Java PostCommit
> >> Run Java HadoopFormatIO Performance Test
> >> Run BigQueryIO Streaming Performance Test Java
> >> Run Dataflow ValidatesRunner
> >> Run Spark ValidatesRunner
> >> Run SQL Postcommit
> >>
> >> Regards,
> >> Tomo
>
>
>
> --
> Regards,
> Tomo
>

Re: Jenkins jobs are not being displayed on GitHub

[Yifan Zou]

Glanced at the build workers. All jenkins executors are busy now. Lots of
jobs are in the waiting queue. And the recent completed jobs show a long
waiting time to get started, which is around 1h.
The Jenkins site also responses very slow.

On Tue, Dec 3, 2019 at 2:15 PM Kirill Kozlov 
wrote:

> Hello everyone!
>
> It looks like for the PRs created within the last 30 minutes status of
> Jenkins jobs is not being displayed.
> Seed job appears to be stuck [1]. #5293 is waiting for #5292 to finish,
> but #5293 shows that it is complete.
>
> [1] https://builds.apache.org/job/beam_SeedJob/
>
> --
> Kirill
>

Re: Failed retrieving service account

[Yifan Zou]

Hi,

I've looked into this issue and found that the default service account was
removed during the weekend for some reason log viewer
<https://pantheon.corp.google.com/logs/viewer?_ga=2.144495605.-1794192625.1569373214=apache-beam-testing=433637338589=0=false=2019-11-25T20:12:39.42800Z==true=2019-11-18T20:41:06.480Z=2019-11-25T20:41:06.480Z=P7D=gce_instance%2Finstance_id%2F8919707380078440409=2019-11-25T07:55:19.038646702Z=resource.type%3d%22service_account%22%0aresource.labels.email_id%3d%22844138762903-comp...@developer.gserviceaccount.com%22%0A%22DeleteServiceAccount%22=170pqn8e3ofe0=2019-11-25T07:55:19.038646702Z>
.
I restored the default service account. All workers should be backing to
normal and jobs start passing now.

-yifan

On Mon, Nov 25, 2019 at 11:17 AM Tomo Suzuki  wrote:

> Thank you for looking into this.
>
> On Mon, Nov 25, 2019 at 12:59 PM Yifan Zou  wrote:
>
>> Greetings,
>>
>> We're seeing some tests encountering permission issues such as *'Failed
>> to retrieve
>> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token
>> <http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token>
>> from the Google Compute Enginemetadata service. Status: 404
>> Response:\nb\'"The service account was not found.'*
>>
>> I am looking onto it. We might need to reboot some build workers to
>> restore the service account access. I'll try to make as little impact as
>> possible on current running jobs.
>>
>> -yifan
>>
>
>
> --
> Regards,
> Tomo
>

Failed retrieving service account

[Yifan Zou]

Greetings,

We're seeing some tests encountering permission issues such as *'Failed to
retrieve
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token

from the Google Compute Enginemetadata service. Status: 404
Response:\nb\'"The service account was not found.'*

I am looking onto it. We might need to reboot some build workers to restore
the service account access. I'll try to make as little impact as possible
on current running jobs.

-yifan

Re: [ANNOUNCE] New committer: Brian Hulette

[Yifan Zou]

Well deserved. Congratulations!

On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov 
wrote:

> Congratulations, Brian!
>
> On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:
>
>> Congrats Brian!
>>
>>
>> On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang  wrote:
>>
>>> Congrats Brian!
>>>
>>> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu  wrote:
>>>
 Congrats, Brian!

 On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
 danolive...@google.com> wrote:

> Congratulations Brian! It's well deserved.
>
> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
> aromanenko@gmail.com> wrote:
>
>> Congratulations, Brian!
>>
>> On 15 Nov 2019, at 18:27, Rui Wang  wrote:
>>
>> Congrats!
>>
>>
>> -Rui
>>
>> On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise  wrote:
>>
>>> Congratulations!
>>>
>>>
>>> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 Well done Brian!!!

 Kenn thank you for sharing

 On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden 
 wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía 
> wrote:
>
>> Congratulations Brian!
>> Happy to see this happening and eager to see more of your work!
>>
>> On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka 
>> wrote:
>> >
>> > Congrats Brian!
>> >
>> > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský 
>> wrote:
>> >>
>> >> Congrats Brian!
>> >>
>> >> On 11/15/19 9:58 AM, Reza Rokni wrote:
>> >>
>> >> Great news!
>> >>
>> >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov 
>> wrote:
>> >>>
>> >>> Congratulations!
>> >>>
>> >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>> 
>>  Congratulations, Brian!
>> 
>>  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
>> sunjincheng...@gmail.com> wrote:
>> >
>> > Congratulation Brian!
>> >
>> > Best,
>> > Jincheng
>> >
>> > Kyle Weaver  于2019年11月15日周五 上午7:19写道：
>> >>
>> >> Thanks for your contributions and congrats Brian!
>> >>
>> >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
>> k...@apache.org> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> Please join me and the rest of the Beam PMC in welcoming
>> a new committer: Brian Hulette
>> >>>
>> >>> Brian introduced himself to dev@ earlier this year and
>> has been contributing since then. His contributions to Beam include
>> explorations of integration with Arrow, standardizing coders, 
>> portability
>> for schemas, and presentations at Beam events.
>> >>>
>> >>> In consideration of Brian's contributions, the Beam PMC
>> trusts him with the responsibilities of a Beam committer [1].
>> >>>
>> >>> Thank you, Brian, for your contributions and looking
>> forward to many more!
>> >>>
>> >>> Kenn, on behalf of the Apache Beam PMC
>> >>>
>> >>> [1]
>> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> This email may be confidential and privileged. If you received
>> this communication by mistake, please don't forward it to anyone 
>> else,
>> please erase all copies and attachments, and please let me know that 
>> it has
>> gone to the wrong person.
>> >>
>> >> The above terms reflect a potential business arrangement, are
>> provided solely as a basis for further discussion, and are not 
>> intended to
>> be and do not constitute a legally binding obligation. No legally 
>> binding
>> obligations will be created, implied, or inferred until an agreement 
>> in
>> final form is executed in writing by all parties involved.
>>
>
>>
>>>
>>> --
>>> 
>>> Ruoyun  Huang
>>>
>>>

Re: Completeness of Beam Java Dependency Check Report

[Yifan Zou]

The dependency management tool is back. See the latest report
<https://builds.apache.org/job/beam_Dependency_Check/234/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html>
.

On Tue, Nov 12, 2019 at 9:51 AM Yifan Zou  wrote:

> Thanks Tomo. I'll follow up in JIRA.
>
> On Tue, Nov 12, 2019 at 9:44 AM Tomo Suzuki  wrote:
>
>> Yifan,
>> I created a ticket to track this finding:
>> https://issues.apache.org/jira/browse/BEAM-8621 .
>>
>>
>> On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki  wrote:
>>
>>> Kenn,
>>>
>>> Thank you for the analysis. Although Guava was randomly picked up, it's
>>> great learning for me to learn how you analyzed other modules using Guava.
>>>
>>> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles  wrote:
>>>
>>>> BeamModulePlugin just contains lists of versions to ease coordination
>>>> across Beam modules, but mostly does not create dependencies. Most of
>>>> Beam's modules only depend on a few things there. For example Guava is not
>>>> a core dependency, but here is where it is actually depended upon:
>>>>
>>>> $ find . -name build.gradle | xargs grep library.java.guava
>>>> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>>>> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile
>>>> library.java.guava
>>>> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
>>>> library.java.guava
>>>> ./sdks/java/io/kinesis/build.gradle:  testCompile
>>>> library.java.guava_testlib
>>>>
>>>> These results appear to be misleading. Grepping for 'import
>>>> com.google.common', I see this as the actual state of things:
>>>>
>>>>  - GCP connector does not appear to actually depend on Guava in compile
>>>> scope
>>>>  - The Beam SQL JDBC driver does not appear to actually depend on Guava
>>>> in compile scope
>>>>  - The Dataflow Java worker does depend on Guava at compile scope but
>>>> has incorrect dependencies (and it probably shouldn't)
>>>>  - KinesisIO does depend on Guava at compile scope but has incorrect
>>>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>>>> should be correctly declared)
>>>>  - ZetaSQL translator does depend on Guava at compile scope but has
>>>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>>>> should be correctly declared)
>>>>
>>>> We used to have an analysis that prevented this class of error.
>>>>
>>>> Once the errors are fixed, the guava_version is simply a version that
>>>> we have discovered that seems to work for both Kinesis and ZetaSQL,
>>>> libraries we do not control. Kinesis producer is built against 18.0.
>>>> Kinesis client against 26.0-jre. ZetaSQL against 26.0-android.
>>>>
>>>> (or maybe I messed up in my analysis)
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki 
>>>> wrote:
>>>>
>>>>>
>>>>> Chamikara and Yifan,
>>>>> Thank you for the responses! Looking forward to hearing the
>>>>> investigation result.
>>>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>>>> directory.
>>>>>
>>>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
>

Re: Completeness of Beam Java Dependency Check Report

[Yifan Zou]

Thanks Tomo. I'll follow up in JIRA.

On Tue, Nov 12, 2019 at 9:44 AM Tomo Suzuki  wrote:

> Yifan,
> I created a ticket to track this finding:
> https://issues.apache.org/jira/browse/BEAM-8621 .
>
>
> On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki  wrote:
>
>> Kenn,
>>
>> Thank you for the analysis. Although Guava was randomly picked up, it's
>> great learning for me to learn how you analyzed other modules using Guava.
>>
>> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles  wrote:
>>
>>> BeamModulePlugin just contains lists of versions to ease coordination
>>> across Beam modules, but mostly does not create dependencies. Most of
>>> Beam's modules only depend on a few things there. For example Guava is not
>>> a core dependency, but here is where it is actually depended upon:
>>>
>>> $ find . -name build.gradle | xargs grep library.java.guava
>>> ./sdks/java/core/build.gradle:  shadowTest library.java.guava_testlib
>>> ./sdks/java/extensions/sql/jdbc/build.gradle:  compile library.java.guava
>>> ./sdks/java/io/google-cloud-platform/build.gradle:  compile
>>> library.java.guava
>>> ./sdks/java/io/kinesis/build.gradle:  testCompile
>>> library.java.guava_testlib
>>>
>>> These results appear to be misleading. Grepping for 'import
>>> com.google.common', I see this as the actual state of things:
>>>
>>>  - GCP connector does not appear to actually depend on Guava in compile
>>> scope
>>>  - The Beam SQL JDBC driver does not appear to actually depend on Guava
>>> in compile scope
>>>  - The Dataflow Java worker does depend on Guava at compile scope but
>>> has incorrect dependencies (and it probably shouldn't)
>>>  - KinesisIO does depend on Guava at compile scope but has incorrect
>>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>>> should be correctly declared)
>>>  - ZetaSQL translator does depend on Guava at compile scope but has
>>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>>> should be correctly declared)
>>>
>>> We used to have an analysis that prevented this class of error.
>>>
>>> Once the errors are fixed, the guava_version is simply a version that we
>>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>>> against 26.0-jre. ZetaSQL against 26.0-android.
>>>
>>> (or maybe I messed up in my analysis)
>>>
>>> Kenn
>>>
>>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki  wrote:
>>>

 Chamikara and Yifan,
 Thank you for the responses! Looking forward to hearing the
 investigation result.
 In the meantime, I'll explore .test-infra/jenkins/dependency_check
 directory.


>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>

Re: Completeness of Beam Java Dependency Check Report

[Yifan Zou]

Hi,

Thanks for taking care of Beam dependencies. The guava was tracked in
BEAM-5559 . It was
filtered out by the tool because of the target version is x.y-jre.

On the other hand, I checked the logs of dependency job and found that the
high priority dependencies are way less than normal. I think the
gradle-versions-plugin sometimes couldn't get the versions which are
defined in BeamModulePlugin.groovy. Need more investigations on the
dependency management tool.

Yifan

On Mon, Nov 11, 2019 at 10:14 AM Tomo Suzuki  wrote:

> Hi Beam developers,
> (I'm thinking to contribute to upgrades of Java dependencies of Beam; I
> just read https://beam.apache.org/contribute/dependencies/)
>
> As per the weekly report, Apache Beam Java SDK only has 8 outdated
> dependencies based on the criteria. However, it seems many others are not
> up-to-date. For example Guava 20.0 used in Beam
> 
> is not the latest major release.
>
> Why do some outdated dependencies not appear in this report?
>
> Regards,
> Tomo
>
> On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>> High Priority Dependency Updates Of Beam Python SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> mock  2.0.0 3.0.5 2019-05-20 2019-05-20
>> BEAM-7369 
>> oauth2client  3.0.0 4.1.3
>> 2018-12-10 2018-12-10 BEAM-6089
>> 
>> Sphinx  1.8.5 2.2.1 2019-05-20
>> 2019-10-28 BEAM-7370  High
>> Priority Dependency Updates Of Beam Java SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
>> 
>> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
>> 
>> com.github.spotbugs:spotbugs
>>  3.1.12
>> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
>> 
>> com.github.spotbugs:spotbugs-annotations
>> 
>> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
>> 
>> javax.servlet:javax.servlet-api
>> 
>> 3.1.0 4.0.1 2013-04-25 2018-04-20 BEAM-5750
>> 
>> org.conscrypt:conscrypt-openjdk
>> 
>> 1.1.3 2.2.1 2018-06-04 2019-08-08 BEAM-5748
>> 
>> org.eclipse.jetty:jetty-server
>> 
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
>> 
>> org.eclipse.jetty:jetty-servlet
>> 
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
>> 
>> Gradle:  5.2.1 6.0 2019-08-19
>> 2019-11-11 BEAM-8002  A
>> dependency update is high priority if it satisfies one of following
>> criteria:
>>
>>- It has major versions update available, e.g.
>>org.assertj:assertj-core 2.5.0 -> 3.10.0;
>>
>>
>>- It is over 3 minor versions behind the latest version, e.g.
>>org.tukaani:xz 1.5 -> 1.8;
>>
>>
>>- The current version is behind the later version for over 180 days,
>>e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>>
>> In Beam, we make a best-effort attempt at keeping all dependencies
>> up-to-date. In the future, issues will be filed and tracked for these
>> automatically, but in the meantime you can search for existing issues or
>> open a new one. For more information: Beam Dependency Guide
>> 
>>
>
>
> --
> Regards,
> Tomo
>

Re: [ANNOUNCE] New committer: Alan Myrvold

[Yifan Zou]

Congratulations, Alan!

On Fri, Sep 27, 2019 at 9:18 AM Ahmet Altay  wrote:

> Hi,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Alan Myrvold
>
> Alan has been a long time Beam contributor. His contributions made Beam
> more productive and friendlier [1] for all contributors with significant
> improvements to Beam release process, automation, and infrastructure.
>
> In consideration of Alan's contributions, the Beam PMC trusts him
> with the responsibilities of a Beam committer [2].
>
> Thank you, Alan, for your contributions and looking forward to many more!
>
> Ahmet, on behalf of the Apache Beam PMC
>
> [1]
> https://beam-summit-na-2019.firebaseapp.com/schedule/2019-09-11?sessionId=1126
> [2] https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-
> committer
>

Re: [ANNOUNCE] New committer: Valentyn Tymofieiev

[Yifan Zou]

Congratulations, Valentyn! Well deserved!

On Mon, Aug 26, 2019 at 3:31 PM Aizhamal Nurmamat kyzy 
wrote:

> Congratulations! and thank you for your contributions, Valentyn!
>
> On Mon, Aug 26, 2019 at 3:26 PM Thomas Weise  wrote:
>
>> Congrats!
>>
>>
>> On Mon, Aug 26, 2019 at 3:22 PM Heejong Lee  wrote:
>>
>>> Congratulations! :)
>>>
>>> On Mon, Aug 26, 2019 at 2:44 PM Rui Wang  wrote:
>>>
 Congratulations!


 -Rui

 On Mon, Aug 26, 2019 at 2:36 PM Hannah Jiang 
 wrote:

> Congratulations Valentyn, well deserved!
>
> On Mon, Aug 26, 2019 at 2:34 PM Chamikara Jayalath <
> chamik...@google.com> wrote:
>
>> Congrats Valentyn!
>>
>> On Mon, Aug 26, 2019 at 2:32 PM Pablo Estrada 
>> wrote:
>>
>>> Thanks Valentyn!
>>>
>>> On Mon, Aug 26, 2019 at 2:29 PM Robin Qiu 
>>> wrote:
>>>
 Thank you Valentyn! Congratulations!

 On Mon, Aug 26, 2019 at 2:28 PM Robert Bradshaw <
 rober...@google.com> wrote:

> Hi,
>
> Please join me and the rest of the Beam PMC in welcoming a new
> committer: Valentyn Tymofieiev
>
> Valentyn has made numerous contributions to Beam over the last
> several
> years (including 100+ pull requests), most recently pushing through
> the effort to make Beam compatible with Python 3. He is also an
> active
> participant in design discussions on the list, participates in
> release
> candidate validation, and proactively helps keep our tests green.
>
> In consideration of Valentyn's contributions, the Beam PMC trusts
> him
> with the responsibilities of a Beam committer [1].
>
> Thank you, Valentyn, for your contributions and looking forward to
> many more!
>
> Robert, on behalf of the Apache Beam PMC
>
> [1]
> https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer
>

[ANNOUNCE] Beam 2.15.0 Released!

[Yifan Zou]

The Apache Beam team is pleased to announce the release of version 2.15.0.

Apache Beam is an open source unified programming model to define and
execute data processing pipelines, including ETL, batch and stream
(continuous) processing. See https://beam.apache.org

You can download the release here:

https://beam.apache.org/get-started/downloads/

This release includes bug fixes, features, and improvements detailed on
the Beam blog: https://beam.apache.org/blog/2019/08/22/beam-2.15.0.html

Thanks to everyone who contributed to this release, and we hope you enjoy
using Beam 2.15.0.

Yifan Zou

[RESULT] [VOTE] Release 2.15.0, release candidate #2

[Yifan Zou]

Hi all,

I'm happy to announce that we have unanimously approved this release.

There are 4 approving votes, 3 of which are binding (in order):
* Ahmet (al...@google.com);
* Pablo (pabl...@google.com);
* Lukasz (lc...@google.com);

There are no disapproving votes.

Thanks everyone!

Next step is to finalize the release (merge the docs/website/blog PRs,
publish artifacts). Please let me know if you have any questions.

Regards,
Yifan

Re: How to test a new precommit in PR?

[Yifan Zou]

Run the seed job then trigger your tests by using phrase.

On Tue, Aug 20, 2019 at 2:39 PM Rui Wang  wrote:

> Hi Community,
>
> I am trying to add a new precommit task (see [1] and [2]), and the PR is
> pending. Does anyone know how to test the added precommit directly in the
> PR before merging it?
>
>
>
> [1]:
> https://github.com/apache/beam/pull/9210/files#diff-d6dfd4f4d675cfe2d6f52ae6fea472d0
> [2]:
> https://github.com/apache/beam/pull/9210/files#diff-c197962302397baf3a4cc36463dce5ea
>
>
>
> -Rui
>

Re: [VOTE] Release 2.15.0, release candidate #2

[Yifan Zou]

Hi all,

This is a friendly reminder. Please help to review, verify and vote on the
release candidate #2 for the version 2.15.0.
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

I've verified Java quickstart & mobile games, and Python (both tar and
wheel) quickstart with Py27, 35, 36, 37. They worked well.
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804

Thanks.
Yifan



On Tue, Aug 20, 2019 at 9:33 AM Hannah Jiang  wrote:

> A side note about this test:
> Now we only have py2 and py35, so it only fails with py35. I am
> introducing minor versions, which will add py36 and py37, and all py3 are
> flaky.
> It's really difficult to pass Portable Precommit with minor versions, the
> chance of passing the test is around 15%.
>
> On Mon, Aug 19, 2019 at 5:27 PM Ahmet Altay  wrote:
>
>> Thank you. Unless there are any objects, let's continue with validating
>> RC2.
>>
>> On Mon, Aug 19, 2019 at 5:21 PM Kyle Weaver  wrote:
>>
>>> I'm not sure if it's worth blocking the release, since I can't reproduce
>>> the issue on my machine and a fix would be hard to verify.
>>>
>>> Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
>>>
>>>
>>> On Mon, Aug 19, 2019 at 4:56 PM Ahmet Altay  wrote:
>>>
>>>> Kyle, are you currently working on this to decide whether it is the
>>>> blocking case or not? Also is this affecting both release branch and master
>>>> branch?
>>>>
>>>> On Mon, Aug 19, 2019 at 4:49 PM Kyle Weaver 
>>>> wrote:
>>>>
>>>>> Re BEAM-7993: For some context, there are two possible causes here.
>>>>> The pessimistic take is that Dockerized SDK workers are taking forever to
>>>>> start. The optimistic take is that the Docker containers are just longer
>>>>> than normal (but not forever) to start on Jenkins, in which case this 
>>>>> issue
>>>>> is nothing new to this release. (The halting problem!) If it's the second,
>>>>> it's safe to wait to fix it in the next release.
>>>>>
>>>>> Kyle Weaver | Software Engineer | github.com/ibzib |
>>>>> kcwea...@google.com
>>>>>
>>>>>
>>>>> On Mon, Aug 19, 2019 at 4:42 PM Yifan Zou  wrote:
>>>>>
>>>>>> Mark and Kyle found a py35 portable test which is flaky:
>>>>>> https://issues.apache.org/jira/browse/BEAM-7993.
>>>>>> I plan to finalize the release this week. Would that be a blocker?
>>>>>> Could we include the fix in 2.16?
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 19, 2019 at 4:38 PM Yifan Zou 
>>>>>> wrote:
>>>>>>
>>>>>>> I've run most of validations and they're all good.
>>>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804
>>>>>>>
>>>>>>> On Mon, Aug 19, 2019 at 10:59 AM Hannah Jiang <
>>>>>>> hannahji...@google.com> wrote:
>>>>>>>
>>>>>>>> (resending it to dev@)
>>>>>>>> +1, I tested some test cases as well as customized test cases and
>>>>>>>> all looks good. I updated validation sheet.
>>>>>>>>
>>>>>>>> On Mon, Aug 19, 2019 at 10:40 AM Hannah Jiang <
>>>>>>>> hannahji...@google.com> wrote:
>>>>>>>>
>>>>>>>>> +1, I tested some test cases as well as customized test cases and
>>>>>>>>> all looks good. I updated validation sheet.
>>>>>>>>>
>>>>>>>>> On Mon, Aug 19, 2019 at 10:28 AM Ahmet Altay 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Please help with validation and voting on the RC2. Let's help
>>>>>>>>>> Yifan to finalize this release.
>>>>>>>>>>
>>>>>>>>>> Ahmet
>>>>>>>>>>
>>>>>>>>>> -- Forwarded message -
>>>>>>>>>> From: Ahmet Altay 
>>>>>>>>>>

Re: [VOTE] Release 2.15.0, release candidate #2

[Yifan Zou]

Mark and Kyle found a py35 portable test which is flaky:
https://issues.apache.org/jira/browse/BEAM-7993.
I plan to finalize the release this week. Would that be a blocker? Could we
include the fix in 2.16?

Thanks.


On Mon, Aug 19, 2019 at 4:38 PM Yifan Zou  wrote:

> I've run most of validations and they're all good.
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804
>
> On Mon, Aug 19, 2019 at 10:59 AM Hannah Jiang 
> wrote:
>
>> (resending it to dev@)
>> +1, I tested some test cases as well as customized test cases and all
>> looks good. I updated validation sheet.
>>
>> On Mon, Aug 19, 2019 at 10:40 AM Hannah Jiang 
>> wrote:
>>
>>> +1, I tested some test cases as well as customized test cases and all
>>> looks good. I updated validation sheet.
>>>
>>> On Mon, Aug 19, 2019 at 10:28 AM Ahmet Altay  wrote:
>>>
>>>> Hi all,
>>>>
>>>> Please help with validation and voting on the RC2. Let's help Yifan to
>>>> finalize this release.
>>>>
>>>> Ahmet
>>>>
>>>> -- Forwarded message -
>>>> From: Ahmet Altay 
>>>> Date: Mon, Aug 19, 2019 at 10:27 AM
>>>> Subject: Re: [VOTE] Release 2.15.0, release candidate #2
>>>> To: dev 
>>>>
>>>>
>>>> +1, verified a few python workloads and updated the validation sheet.
>>>>
>>>> Thank you Yifan!
>>>>
>>>> On Thu, Aug 15, 2019 at 12:49 AM Yifan Zou  wrote:
>>>>
>>>>> Hi everyone,
>>>>> Please review and vote on the release candidate #2 for the version
>>>>> 2.15.0, as follows:
>>>>> [ ] +1, Approve the release
>>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>>
>>>>>
>>>>> The complete staging area is available for your review, which includes:
>>>>> * JIRA release notes [1].
>>>>> * The official Apache source release to be deployed to dist.apache.org 
>>>>> [2],
>>>>> which is signed with the key with fingerprint
>>>>> AC9DB4F14CC90F37080F2C5B6D18F9A7F8DA25E1 [3].
>>>>> * All artifacts to be deployed to the Maven Central Repository [4].
>>>>> * Source code tag "v2.15.0-RC2" [5].
>>>>> * Website pull request listing the release [6], publishing the API
>>>>> reference manual [7], and the blog post [8].
>>>>> * Python artifacts are deployed along with the source release to the
>>>>> dist.apache.org [2].
>>>>> * Validation sheet with a tab for 2.15.0 release to help with
>>>>> validation [9].
>>>>>
>>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>
>>>>> Thanks,
>>>>> Yifan Zou
>>>>> [1]
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12345489
>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.15.0/
>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>>> [4]
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1082/
>>>>> <https://repository.apache.org/content/repositories/orgapachebeam-1082/>
>>>>> [5] https://github.com/apache/beam/tree/v2.15.0-RC2
>>>>> [6] https://github.com/apache/beam/pull/9341
>>>>> [7] https://github.com/apache/beam-site/pull/592
>>>>> [8] https://github.com/apache/beam/pull/9346
>>>>> [9]
>>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "datapls-plat-team" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to datapls-plat-team+unsubscr...@google.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/a/google.com/d/msgid/datapls-plat-team/CAO_Zxo1G89_ET0y3Gpkeu7v5PJJSZMK6n%2BXBG%2BSkkVeE-09Ptg%40mail.gmail.com
>>>> <https://groups.google.com/a/google.com/d/msgid/datapls-plat-team/CAO_Zxo1G89_ET0y3Gpkeu7v5PJJSZMK6n%2BXBG%2BSkkVeE-09Ptg%40mail.gmail.com?utm_medium=email_source=footer>
>>>> .
>>>>
>>>

Re: [VOTE] Release 2.15.0, release candidate #2

[Yifan Zou]

I've run most of validations and they're all good.
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804

On Mon, Aug 19, 2019 at 10:59 AM Hannah Jiang 
wrote:

> (resending it to dev@)
> +1, I tested some test cases as well as customized test cases and all
> looks good. I updated validation sheet.
>
> On Mon, Aug 19, 2019 at 10:40 AM Hannah Jiang 
> wrote:
>
>> +1, I tested some test cases as well as customized test cases and all
>> looks good. I updated validation sheet.
>>
>> On Mon, Aug 19, 2019 at 10:28 AM Ahmet Altay  wrote:
>>
>>> Hi all,
>>>
>>> Please help with validation and voting on the RC2. Let's help Yifan to
>>> finalize this release.
>>>
>>> Ahmet
>>>
>>> -- Forwarded message -
>>> From: Ahmet Altay 
>>> Date: Mon, Aug 19, 2019 at 10:27 AM
>>> Subject: Re: [VOTE] Release 2.15.0, release candidate #2
>>> To: dev 
>>>
>>>
>>> +1, verified a few python workloads and updated the validation sheet.
>>>
>>> Thank you Yifan!
>>>
>>> On Thu, Aug 15, 2019 at 12:49 AM Yifan Zou  wrote:
>>>
>>>> Hi everyone,
>>>> Please review and vote on the release candidate #2 for the version
>>>> 2.15.0, as follows:
>>>> [ ] +1, Approve the release
>>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>>
>>>>
>>>> The complete staging area is available for your review, which includes:
>>>> * JIRA release notes [1].
>>>> * The official Apache source release to be deployed to dist.apache.org [2],
>>>> which is signed with the key with fingerprint
>>>> AC9DB4F14CC90F37080F2C5B6D18F9A7F8DA25E1 [3].
>>>> * All artifacts to be deployed to the Maven Central Repository [4].
>>>> * Source code tag "v2.15.0-RC2" [5].
>>>> * Website pull request listing the release [6], publishing the API
>>>> reference manual [7], and the blog post [8].
>>>> * Python artifacts are deployed along with the source release to the
>>>> dist.apache.org [2].
>>>> * Validation sheet with a tab for 2.15.0 release to help with
>>>> validation [9].
>>>>
>>>> The vote will be open for at least 72 hours. It is adopted by majority
>>>> approval, with at least 3 PMC affirmative votes.
>>>>
>>>> Thanks,
>>>> Yifan Zou
>>>> [1]
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12345489
>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.15.0/
>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> [4]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1082/
>>>> <https://repository.apache.org/content/repositories/orgapachebeam-1082/>
>>>> [5] https://github.com/apache/beam/tree/v2.15.0-RC2
>>>> [6] https://github.com/apache/beam/pull/9341
>>>> [7] https://github.com/apache/beam-site/pull/592
>>>> [8] https://github.com/apache/beam/pull/9346
>>>> [9]
>>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "datapls-plat-team" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datapls-plat-team+unsubscr...@google.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/a/google.com/d/msgid/datapls-plat-team/CAO_Zxo1G89_ET0y3Gpkeu7v5PJJSZMK6n%2BXBG%2BSkkVeE-09Ptg%40mail.gmail.com
>>> <https://groups.google.com/a/google.com/d/msgid/datapls-plat-team/CAO_Zxo1G89_ET0y3Gpkeu7v5PJJSZMK6n%2BXBG%2BSkkVeE-09Ptg%40mail.gmail.com?utm_medium=email_source=footer>
>>> .
>>>
>>

Re: [VOTE] Release 2.15.0, release candidate #1

[Yifan Zou]

Sure. Would you please merge it to release-2.15.0, and I'll rebuild
artifacts and share the RC2?

On Wed, Aug 14, 2019 at 1:53 PM Chamikara Jayalath 
wrote:

> Can we rebuild to include the fix for
> https://issues.apache.org/jira/browse/BEAM-7866 which was merged
> yesterday ? (looks like fix version was changed to 2.16.0)
>
> Thanks,
> Cham
>
> On Wed, Aug 14, 2019 at 1:42 PM Yifan Zou  wrote:
>
>> Hi everyone,
>> Please review and vote on the release candidate #1 for the version
>> 2.15.0, as follows:
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1].
>> * The official Apache source release to be deployed to dist.apache.org
>> [2], which is signed with the key with fingerprint
>> AC9DB4F14CC90F37080F2C5B6D18F9A7F8DA25E1 [3].
>> * All artifacts to be deployed to the Maven Central Repository [4].
>> * Source code tag "v2.15.0-RC1" [5].
>> * Website pull request listing the release [6], publishing the API
>> reference manual [7].
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org [2].
>> * Validation sheet with a tab for 2.15.0 release to help with validation
>> [8].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> Yifan Zou
>>
>> [1]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12345489
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.15.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4]
>> https://repository.apache.org/content/repositories/orgapachebeam-1081/
>> [5] https://github.com/apache/beam/tree/v2.15.0-RC1
>> [6] https://github.com/apache/beam/pull/9341
>> [7] https://github.com/apache/beam-site/pull/592
>> [8]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804
>>
>

[VOTE] Release 2.15.0, release candidate #1

[Yifan Zou]

Hi everyone,
Please review and vote on the release candidate #1 for the version 2.15.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1].
* The official Apache source release to be deployed to dist.apache.org [2],
which is signed with the key with fingerprint
AC9DB4F14CC90F37080F2C5B6D18F9A7F8DA25E1 [3].
* All artifacts to be deployed to the Maven Central Repository [4].
* Source code tag "v2.15.0-RC1" [5].
* Website pull request listing the release [6], publishing the API
reference manual [7].
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
* Validation sheet with a tab for 2.15.0 release to help with validation
[8].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Yifan Zou

[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12345489
[2] https://dist.apache.org/repos/dist/dev/beam/2.15.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1081/
[5] https://github.com/apache/beam/tree/v2.15.0-RC1
[6] https://github.com/apache/beam/pull/9341
[7] https://github.com/apache/beam-site/pull/592
[8]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit?pli=1#gid=1036192804

Re: [Update] Beam 2.15 Release Progress

[Yifan Zou]

All blockers resolved and cherry-picks are done. Will start building the
release candidates.

On Wed, Aug 7, 2019 at 2:58 PM Yifan Zou  wrote:

> Thanks Udi.
>
> On Wed, Aug 7, 2019 at 2:58 PM Udi Meiri  wrote:
>
>> https://github.com/apache/beam/pull/9240 has been merged
>>
>> On Wed, Aug 7, 2019 at 12:33 PM Anton Kedin  wrote:
>>
>>> Perf regression is seemingly gone now. If this is caused by a PR we
>>> might want to find out which one and cherry-pick it into the release.
>>>
>>> Regards,
>>> Anton
>>>
>>> On Tue, Aug 6, 2019 at 4:52 PM Yifan Zou  wrote:
>>>
>>>> Hi,
>>>>
>>>> There is a perf regression on SQL Query3 on dataflow runner. This was
>>>> treated as a release blocker. We would appreciate if someone could look
>>>> into this issue?
>>>>
>>>> For more details, please see Anton's email [1] and JIRA [2].
>>>> [1]
>>>> https://lists.apache.org/thread.html/5441431cb2cf8fb445a2e30e6b2a8feb199d189755cf12b0c86fb1c8@%3Cdev.beam.apache.org%3E
>>>> [2] https://issues.apache.org/jira/browse/BEAM-7906
>>>>
>>>> Regards.
>>>> Yifan
>>>>
>>>>
>>>> On Mon, Aug 5, 2019 at 10:35 AM Yifan Zou  wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I've verified release branch, and all Pre/Post-commits passed. The
>>>>> next step would be verifying the javadoc.
>>>>> We still have a few blocking issues,
>>>>> https://issues.apache.org/jira/browse/BEAM-7880?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.15.0
>>>>> .
>>>>> Please ping me once the ticket got fixed, or update them to the next
>>>>> version to unblock the release. Thanks.
>>>>>
>>>>> Yifan
>>>>>
>>>>> On Wed, Jul 31, 2019 at 4:33 PM Yifan Zou  wrote:
>>>>>
>>>>>> Snapshots are published
>>>>>> http://repository.apache.org/content/groups/snapshots/org/apache/beam/
>>>>>> .
>>>>>>
>>>>>> On Wed, Jul 31, 2019 at 1:28 PM Yifan Zou 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The release branch is cut
>>>>>>> https://github.com/apache/beam/tree/release-2.15.0.
>>>>>>> The next step would be building snapshots and verify release branch.
>>>>>>>
>>>>>>> Regards.
>>>>>>> Yifan
>>>>>>>
>>>>>>

Re: [Update] Beam 2.15 Release Progress

[Yifan Zou]

Thanks Udi.

On Wed, Aug 7, 2019 at 2:58 PM Udi Meiri  wrote:

> https://github.com/apache/beam/pull/9240 has been merged
>
> On Wed, Aug 7, 2019 at 12:33 PM Anton Kedin  wrote:
>
>> Perf regression is seemingly gone now. If this is caused by a PR we might
>> want to find out which one and cherry-pick it into the release.
>>
>> Regards,
>> Anton
>>
>> On Tue, Aug 6, 2019 at 4:52 PM Yifan Zou  wrote:
>>
>>> Hi,
>>>
>>> There is a perf regression on SQL Query3 on dataflow runner. This was
>>> treated as a release blocker. We would appreciate if someone could look
>>> into this issue?
>>>
>>> For more details, please see Anton's email [1] and JIRA [2].
>>> [1]
>>> https://lists.apache.org/thread.html/5441431cb2cf8fb445a2e30e6b2a8feb199d189755cf12b0c86fb1c8@%3Cdev.beam.apache.org%3E
>>> [2] https://issues.apache.org/jira/browse/BEAM-7906
>>>
>>> Regards.
>>> Yifan
>>>
>>>
>>> On Mon, Aug 5, 2019 at 10:35 AM Yifan Zou  wrote:
>>>
>>>> Hi,
>>>>
>>>> I've verified release branch, and all Pre/Post-commits passed. The next
>>>> step would be verifying the javadoc.
>>>> We still have a few blocking issues,
>>>> https://issues.apache.org/jira/browse/BEAM-7880?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.15.0
>>>> .
>>>> Please ping me once the ticket got fixed, or update them to the next
>>>> version to unblock the release. Thanks.
>>>>
>>>> Yifan
>>>>
>>>> On Wed, Jul 31, 2019 at 4:33 PM Yifan Zou  wrote:
>>>>
>>>>> Snapshots are published
>>>>> http://repository.apache.org/content/groups/snapshots/org/apache/beam/
>>>>> .
>>>>>
>>>>> On Wed, Jul 31, 2019 at 1:28 PM Yifan Zou  wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The release branch is cut
>>>>>> https://github.com/apache/beam/tree/release-2.15.0.
>>>>>> The next step would be building snapshots and verify release branch.
>>>>>>
>>>>>> Regards.
>>>>>> Yifan
>>>>>>
>>>>>

Re: [Update] Beam 2.15 Release Progress

[Yifan Zou]

Hi,

There is a perf regression on SQL Query3 on dataflow runner. This was
treated as a release blocker. We would appreciate if someone could look
into this issue?

For more details, please see Anton's email [1] and JIRA [2].
[1]
https://lists.apache.org/thread.html/5441431cb2cf8fb445a2e30e6b2a8feb199d189755cf12b0c86fb1c8@%3Cdev.beam.apache.org%3E
[2] https://issues.apache.org/jira/browse/BEAM-7906

Regards.
Yifan


On Mon, Aug 5, 2019 at 10:35 AM Yifan Zou  wrote:

> Hi,
>
> I've verified release branch, and all Pre/Post-commits passed. The next
> step would be verifying the javadoc.
> We still have a few blocking issues,
> https://issues.apache.org/jira/browse/BEAM-7880?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.15.0
> .
> Please ping me once the ticket got fixed, or update them to the next
> version to unblock the release. Thanks.
>
> Yifan
>
> On Wed, Jul 31, 2019 at 4:33 PM Yifan Zou  wrote:
>
>> Snapshots are published
>> http://repository.apache.org/content/groups/snapshots/org/apache/beam/.
>>
>> On Wed, Jul 31, 2019 at 1:28 PM Yifan Zou  wrote:
>>
>>> Hi,
>>>
>>> The release branch is cut
>>> https://github.com/apache/beam/tree/release-2.15.0.
>>> The next step would be building snapshots and verify release branch.
>>>
>>> Regards.
>>> Yifan
>>>
>>

Re: [ANNOUNCE] New committer: Kyle Weaver

[Yifan Zou]

Congratulations Kyle!

On Tue, Aug 6, 2019 at 9:48 AM Anton Kedin  wrote:

> Congrats!
>
> On Tue, Aug 6, 2019, 9:37 AM Ankur Goenka  wrote:
>
>> Congratulations Kyle!
>>
>> On Tue, Aug 6, 2019 at 9:35 AM Ahmet Altay  wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new committer: 
>>> Kyle
>>> Weaver.
>>>
>>> Kyle has been contributing to Beam for a while now. And in that time
>>> period Kyle got the portable spark runner feature complete for batch
>>> processing. [1]
>>>
>>> In consideration of Kyle's contributions, the Beam PMC trusts him with
>>> the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Kyle, for your contributions and looking forward to many more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/c43678fc24c9a1dc9f48c51c51950aedcb9bc0fd3b633df16c3d595a@%3Cuser.beam.apache.org%3E
>>> [2] https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>

Re: [ANNOUNCE] New committer: Rui Wang

[Yifan Zou]

Congratulations Rui!

On Tue, Aug 6, 2019 at 9:47 AM Anton Kedin  wrote:

> Congrats!
>
> On Tue, Aug 6, 2019, 9:36 AM Ankur Goenka  wrote:
>
>> Congratulations Rui!
>> Well deserved 
>>
>> On Tue, Aug 6, 2019 at 9:35 AM Ahmet Altay  wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new committer: 
>>> Rui
>>> Wang.
>>>
>>> Rui has been an active contributor since May 2018. Rui has been very
>>> active in Beam SQL [1] and continues to help out on user@ and
>>> StackOverflow. Rui is one of the top answerers for apache-beam tag [2].
>>>
>>> In consideration of Rui's contributions, the Beam PMC trusts him with
>>> the responsibilities of a Beam committer [3].
>>>
>>> Thank you, Rui, for your contributions and looking forward to many more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1] https://github.com/apache/beam/pulls?q=is%3Apr+author%3Aamaliujia
>>> [2] https://stackoverflow.com/tags/apache-beam/topusers
>>> [3] https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>

Re: [Update] Beam 2.15 Release Progress

[Yifan Zou]

Hi,

I've verified release branch, and all Pre/Post-commits passed. The next
step would be verifying the javadoc.
We still have a few blocking issues,
https://issues.apache.org/jira/browse/BEAM-7880?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.15.0
.
Please ping me once the ticket got fixed, or update them to the next
version to unblock the release. Thanks.

Yifan

On Wed, Jul 31, 2019 at 4:33 PM Yifan Zou  wrote:

> Snapshots are published
> http://repository.apache.org/content/groups/snapshots/org/apache/beam/.
>
> On Wed, Jul 31, 2019 at 1:28 PM Yifan Zou  wrote:
>
>> Hi,
>>
>> The release branch is cut
>> https://github.com/apache/beam/tree/release-2.15.0.
>> The next step would be building snapshots and verify release branch.
>>
>> Regards.
>> Yifan
>>
>

Re: [Update] Beam 2.15 Release Progress

[Yifan Zou]

Snapshots are published
http://repository.apache.org/content/groups/snapshots/org/apache/beam/.

On Wed, Jul 31, 2019 at 1:28 PM Yifan Zou  wrote:

> Hi,
>
> The release branch is cut
> https://github.com/apache/beam/tree/release-2.15.0.
> The next step would be building snapshots and verify release branch.
>
> Regards.
> Yifan
>

[Update] Beam 2.15 Release Progress

[Yifan Zou]

Hi,

The release branch is cut https://github.com/apache/beam/tree/release-2.15.0
.
The next step would be building snapshots and verify release branch.

Regards.
Yifan

Re: [VOTE] Release 2.14.0, release candidate #1

[Yifan Zou]

AFAIK, there should not be any special prerequisites for this. Things the
script does including:
1. download the python rc in zip
2. start virtualenv and install the sdk.
3. verify hash.
4. config settings.xml and start a Java pubsub message injector.
5. run game examples and validate.

Could you double check if the sdk was installed properly (step 1&2)?

Yifan

On Fri, Jul 26, 2019 at 2:38 PM Anton Kedin  wrote:

> Validation script fails for me when I try to run [1] python leaderboard
> with direct runner:
>
> ```
> *
> * Running Python Leaderboard with DirectRunner
> *
> /usr/bin/python: No module named apache_beam.examples.complete.game
> ```
>
> If someone has more context, what are the prerequisites for this step? How
> does it look up the module?
>
> [1]
> https://github.com/apache/beam/blob/master/release/src/main/scripts/run_rc_validation.sh#L424
>
> Regards,
> Anton
>
> On Fri, Jul 26, 2019 at 10:23 AM Anton Kedin  wrote:
>
>> Cool, will make the post and will update the release guide as well then
>>
>> On Fri, Jul 26, 2019 at 10:20 AM Chad Dombrova  wrote:
>>
>>> I think the release guide needs to be updated to remove the optionality
 of blog creation and avoid confusion. Thanks for pointing that out.

>>>
>>> +1
>>>
>>>

Re: Jenkins nodes disconnected?

[Yifan Zou]

That was a known issue, BEAM-7650
. Basically, the disk was
full. We should either fix this problem in the python precommit, or as Udi
suggested, having a cron job to do the periodic disk space releases.
I'll try to restore those broken agents.

Thanks.

Yifan

On Tue, Jul 23, 2019 at 5:47 AM Łukasz Gajowy  wrote:

> Hi,
>
> I noticed that 5 Jenkins nodes are disconnected[1]. This results in a very
> long task queue and requires long waiting for a job to be completed. I'm
> currently waiting 42 minutes for seed job to be started (and still
> counting). Is anyone currently working on reconnecting the nodes? Why is
> this happening? Can I help in any way?
>
> [1] https://builds.apache.org/computer/
>

Re: [ANNOUNCE] New committer: Robert Burke

[Yifan Zou]

Congratulations!!

On Tue, Jul 16, 2019 at 10:34 AM Rui Wang  wrote:

> Congrats!
>
>
> -Rui
>
> On Tue, Jul 16, 2019 at 10:32 AM Udi Meiri  wrote:
>
>> Congrats Robert B.!
>>
>> On Tue, Jul 16, 2019 at 10:23 AM Ahmet Altay  wrote:
>>
>>> Hi,
>>>
>>> Please join me and the rest of the Beam PMC in welcoming a new committer: 
>>> Robert
>>> Burke.
>>>
>>> Robert has been contributing to Beam and actively involved in the
>>> community for over a year. He has been actively working on Go SDK, helping
>>> users, and making it easier for others to contribute [1].
>>>
>>> In consideration of Robert's contributions, the Beam PMC trusts him with
>>> the responsibilities of a Beam committer [2].
>>>
>>> Thank you, Robert, for your contributions and looking forward to many
>>> more!
>>>
>>> Ahmet, on behalf of the Apache Beam PMC
>>>
>>> [1]
>>> https://lists.apache.org/thread.html/8f729da2d3009059d7a8b2d8624446be161700dcfa953939dd3530c6@%3Cdev.beam.apache.org%3E
>>> [2] https://beam.apache.org/contribute/become-a-committer
>>> /#an-apache-beam-committer
>>>
>>

Re: python precommits failing at head

[Yifan Zou]

We saw similar timeouts of the python precommit and it usually break the
Jenkins build workers. I've run the precommit manually several times. It
sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build
VMs eventually ran out of memory (104G memory available in total) then
disconnected. Not sure what happened during that job.



On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli 
wrote:

> Yes. It passed on the second attempt.
>
> But, I'm yet to figure out why it hangs for ~1.5 hours.
>
> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar 
> wrote:
>
>>
>>
>> Even I am running into the same issue. Though my test passed but
>> somehow
>> the task didn't terminate, eventually the task was aborted.  I have already
>> tried a couple of times to retrigger tye python precommit but it failed
>> every time.
>>
>> @Tanay did it pass it for you?
>>
>>
>>
>>
>>
>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli 
>> wrote:
>>
>>> Thank You Valentyn!
>>>
>>> I'll retest it.
>>> Hopefully, it's a transient issue.
>>>
>>> Regards,
>>> - Tanay Tummalapalli
>>>
>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev 
>>> wrote:
>>>
 No, we did not reduce the timeout recently. Looking at console logs,
 nothing happened for an hour or so,

 *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :)
 06:57:50 *

 *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out 
 (after 120 minutes). Marking the build as aborted.


 However, we can also see in the logs that py36-cython suite never
 started, not sure way. I assume gradle waited for this suite to finish.
 Try "retest this please", hopefully this is a transient gradle issue. I
 did not observe it before.

 On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli 
 wrote:

> Hi Udi,
>
> I rebased another PR[1] onto the fix mentioned above. The lint error
> is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing
> because of a timeout at 120 minutes[2].
> The log says "Build timed out (after 120 minutes). Marking the build
> as aborted."
> Another PR's Python PreCommit job aborted with the same error[3].
>
> I found this issue - "[BEAM-3040] Python precommit timed out after 150
> minutes"[4].
> Was the timeout reduced recently?
>
> Regards,
> - Tanay Tummalapalli
>
> [1] https://github.com/apache/beam/pull/8871
> [2]
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull
>
> [3] https://github.com/apache/beam/pull/9050
> [4] https://issues.apache.org/jira/browse/BEAM-3040
>
> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri  wrote:
>
>> This is due to
>> https://github.com/apache/beam/pull/8969
>> and
>> https://github.com/apache/beam/pull/8934
>> being merged today.
>>
>> Fix is here: https://github.com/apache/beam/pull/9044
>>
>

Re: apache-beam-jenkins-15 out of disk

[Yifan Zou]

I reimaged the beam15. The worker is re-enabled. Let us know if anything
weird happens on any agent.

Thanks.
Yifan

On Mon, Jul 1, 2019 at 10:00 AM Yifan Zou  wrote:

> https://issues.apache.org/jira/browse/BEAM-7650 tracks the docker issue.
>
> On Sun, Jun 30, 2019 at 2:35 PM Mark Liu  wrote:
>
>> Thank you for triaging and working out a solution Yifan and Ankur.
>>
>> Ankur, from what you discovered, we should fix this race condition
>> otherwise same problem will happen in the future. Is there a jira tracking
>> this issue?
>>
>> On Fri, Jun 28, 2019 at 4:56 PM Yifan Zou  wrote:
>>
>>> Sorry for the inconvenience. I disabled the worker. I'll need more time
>>> to restore it.
>>>
>>> On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira 
>>> wrote:
>>>
>>>> Any updates to this issue today? It seems like this (or a similar bug)
>>>> is still happening across many Pre and Postcommits.
>>>>
>>>> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou  wrote:
>>>>
>>>>> I did the prune on beam15. The disk was free but all jobs fails with
>>>>> other weird problems. Looks like docker prune overkills, but I don't have
>>>>> evidence. Will look further in AM.
>>>>>
>>>>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri  wrote:
>>>>>
>>>>>> See how the hdfs IT already avoids tag collisions.
>>>>>>
>>>>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang  wrote:
>>>>>>
>>>>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>>>>> apart.
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This highlights the race condition caused by using single docker
>>>>>>>>> registry on a machine.
>>>>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>>>>> after another then the 2nd one will replace the 1st one and cause 
>>>>>>>>> flakyness.
>>>>>>>>>
>>>>>>>>> Is their a way to dynamically create and destroy docker repository
>>>>>>>>> on a machine and clean all the relevant data?
>>>>>>>>>
>>>>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The problem was because of the large quantity of stale docker
>>>>>>>>>> images generated by the Python portable tests and HDFS IT.
>>>>>>>>>>
>>>>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>>>>
>>>>>>>>>> TYPETOTAL   ACTIVE  SIZE
>>>>>>>>>>RECLAIMABLE
>>>>>>>>>> *Images  1039356
>>>>>>>>>> 424GB   384.2GB (90%)*
>>>>>>>>>> Containers  987 2
>>>>>>>>>> 2.042GB 2.041GB (99%)
>>>>>>>>>> Local Volumes   126 0
>>>>>>>>>> 392.8MB 392.8MB (100%)
>>>>>>>>>>
>>>>>>>>>> REPOSITORY
>>>>>>>>>> TAG IMAGE IDCREATED
>>>>>>>>>> SIZESHARED SIZE UNIQUE SIZE 
>>>>>>>>>> CONTAINERS
>>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>>>>  latest  ff1b949f444222 hours ago
>>>>>>>>>> 1.639GB
>>>>>>>>>>   922.3MB  716.9MB 0
>>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>>>>latest 

Re: apache-beam-jenkins-15 out of disk

[Yifan Zou]

https://issues.apache.org/jira/browse/BEAM-7650 tracks the docker issue.

On Sun, Jun 30, 2019 at 2:35 PM Mark Liu  wrote:

> Thank you for triaging and working out a solution Yifan and Ankur.
>
> Ankur, from what you discovered, we should fix this race condition
> otherwise same problem will happen in the future. Is there a jira tracking
> this issue?
>
> On Fri, Jun 28, 2019 at 4:56 PM Yifan Zou  wrote:
>
>> Sorry for the inconvenience. I disabled the worker. I'll need more time
>> to restore it.
>>
>> On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira 
>> wrote:
>>
>>> Any updates to this issue today? It seems like this (or a similar bug)
>>> is still happening across many Pre and Postcommits.
>>>
>>> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou  wrote:
>>>
>>>> I did the prune on beam15. The disk was free but all jobs fails with
>>>> other weird problems. Looks like docker prune overkills, but I don't have
>>>> evidence. Will look further in AM.
>>>>
>>>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri  wrote:
>>>>
>>>>> See how the hdfs IT already avoids tag collisions.
>>>>>
>>>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang  wrote:
>>>>>
>>>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>>>> apart.
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang 
>>>>>> wrote:
>>>>>>
>>>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> This highlights the race condition caused by using single docker
>>>>>>>> registry on a machine.
>>>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>>>> after another then the 2nd one will replace the 1st one and cause 
>>>>>>>> flakyness.
>>>>>>>>
>>>>>>>> Is their a way to dynamically create and destroy docker repository
>>>>>>>> on a machine and clean all the relevant data?
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The problem was because of the large quantity of stale docker
>>>>>>>>> images generated by the Python portable tests and HDFS IT.
>>>>>>>>>
>>>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>>>
>>>>>>>>> TYPETOTAL   ACTIVE  SIZE
>>>>>>>>>  RECLAIMABLE
>>>>>>>>> *Images  1039356 424GB
>>>>>>>>>   384.2GB (90%)*
>>>>>>>>> Containers  987 2
>>>>>>>>> 2.042GB 2.041GB (99%)
>>>>>>>>> Local Volumes   126 0
>>>>>>>>> 392.8MB 392.8MB (100%)
>>>>>>>>>
>>>>>>>>> REPOSITORY
>>>>>>>>> TAG IMAGE IDCREATED
>>>>>>>>> SIZESHARED SIZE UNIQUE SIZE 
>>>>>>>>> CONTAINERS
>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>>>  latest  ff1b949f444222 hours ago
>>>>>>>>> 1.639GB
>>>>>>>>>   922.3MB  716.9MB 0
>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>>>  latest  1dda7b9d974822 hours ago
>>>>>>>>> 1.624GB
>>>>>>>>>   913.7MB   710.3MB 0
>>>>>>>>> 
>>>>>>>>>  05458187a0e322 
>>>>>>>>> hours
>>>>>>>>> ago732.9MB 625.1MB107.8

Re: apache-beam-jenkins-15 out of disk

[Yifan Zou]

Sorry for the inconvenience. I disabled the worker. I'll need more time to
restore it.

On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira 
wrote:

> Any updates to this issue today? It seems like this (or a similar bug) is
> still happening across many Pre and Postcommits.
>
> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou  wrote:
>
>> I did the prune on beam15. The disk was free but all jobs fails with
>> other weird problems. Looks like docker prune overkills, but I don't have
>> evidence. Will look further in AM.
>>
>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri  wrote:
>>
>>> See how the hdfs IT already avoids tag collisions.
>>>
>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang  wrote:
>>>
>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>> apart.
>>>>
>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang  wrote:
>>>>
>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>
>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka 
>>>>> wrote:
>>>>>
>>>>>> This highlights the race condition caused by using single docker
>>>>>> registry on a machine.
>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>> after another then the 2nd one will replace the 1st one and cause 
>>>>>> flakyness.
>>>>>>
>>>>>> Is their a way to dynamically create and destroy docker repository on
>>>>>> a machine and clean all the relevant data?
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou 
>>>>>> wrote:
>>>>>>
>>>>>>> The problem was because of the large quantity of stale docker images
>>>>>>> generated by the Python portable tests and HDFS IT.
>>>>>>>
>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>
>>>>>>> TYPETOTAL   ACTIVE  SIZE
>>>>>>>RECLAIMABLE
>>>>>>> *Images  1039356 424GB
>>>>>>> 384.2GB (90%)*
>>>>>>> Containers  987 2   2.042GB
>>>>>>> 2.041GB (99%)
>>>>>>> Local Volumes   126 0   392.8MB
>>>>>>> 392.8MB (100%)
>>>>>>>
>>>>>>> REPOSITORY
>>>>>>>   TAG IMAGE IDCREATED
>>>>>>>   SIZESHARED SIZE UNIQUE SIZE CONTAINERS
>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>latest  ff1b949f444222 hours ago1.639GB
>>>>>>> 922.3MB  716.9MB 0
>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>latest  1dda7b9d974822 hours ago1.624GB
>>>>>>> 913.7MB   710.3MB 0
>>>>>>> 
>>>>>>>05458187a0e322 
>>>>>>> hours
>>>>>>> ago732.9MB 625.1MB107.8MB 4
>>>>>>> 
>>>>>>>896f35dd685f23 
>>>>>>> hours
>>>>>>> ago1.639GB 922.3MB   716.9MB
>>>>>>>  0
>>>>>>> 
>>>>>>>db4d24ca9f2b23 
>>>>>>> hours
>>>>>>> ago1.624GB 913.7MB  710.3MB >>>>>>> 0
>>>>>>> 
>>>>>>>   547df4d71c3123 hours
>>>>>>> ago732.9MB 625.1MB 107.8MB 4
>>>>>>> 
>>>>>>>   dd7d9582c3e023 hours
>>>>>>> ago1.639GB 922.3MB 716.9MB 0
>>>>>>> 
>>>>>>>

Re: apache-beam-jenkins-15 out of disk

[Yifan Zou]

I did the prune on beam15. The disk was free but all jobs fails with other
weird problems. Looks like docker prune overkills, but I don't have
evidence. Will look further in AM.

On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri  wrote:

> See how the hdfs IT already avoids tag collisions.
>
> On Thu, Jun 27, 2019, 20:42 Yichi Zhang  wrote:
>
>> for flakiness I guess a tag is needed to separate concurrent build apart.
>>
>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang  wrote:
>>
>>> maybe a cron job on jenkins node that does docker prune every day?
>>>
>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka  wrote:
>>>
>>>> This highlights the race condition caused by using single docker
>>>> registry on a machine.
>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>> after another then the 2nd one will replace the 1st one and cause 
>>>> flakyness.
>>>>
>>>> Is their a way to dynamically create and destroy docker repository on a
>>>> machine and clean all the relevant data?
>>>>
>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou  wrote:
>>>>
>>>>> The problem was because of the large quantity of stale docker images
>>>>> generated by the Python portable tests and HDFS IT.
>>>>>
>>>>> Dumping the docker disk usage gives me:
>>>>>
>>>>> TYPETOTAL   ACTIVE  SIZE
>>>>>  RECLAIMABLE
>>>>> *Images  1039356 424GB
>>>>>   384.2GB (90%)*
>>>>> Containers  987 2   2.042GB
>>>>>   2.041GB (99%)
>>>>> Local Volumes   126 0   392.8MB
>>>>>   392.8MB (100%)
>>>>>
>>>>> REPOSITORY
>>>>> TAG IMAGE IDCREATED
>>>>> SIZESHARED SIZE UNIQUE SIZE CONTAINERS
>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>  latest  ff1b949f444222 hours ago1.639GB
>>>>>   922.3MB  716.9MB 0
>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>  latest  1dda7b9d974822 hours ago1.624GB
>>>>>   913.7MB   710.3MB 0
>>>>> 
>>>>>  05458187a0e322 hours 
>>>>> ago
>>>>>732.9MB 625.1MB107.8MB 4
>>>>> 
>>>>>  896f35dd685f23 hours 
>>>>> ago
>>>>>1.639GB 922.3MB   716.9MB 0
>>>>> 
>>>>>  db4d24ca9f2b23 hours 
>>>>> ago
>>>>>1.624GB 913.7MB  710.3MB 0
>>>>> 
>>>>> 547df4d71c3123 hours ago
>>>>>732.9MB 625.1MB 107.8MB 4
>>>>> 
>>>>> dd7d9582c3e023 hours ago
>>>>>1.639GB 922.3MB 716.9MB 0
>>>>> 
>>>>> 664aae25523923 hours ago
>>>>>1.624GB 913.7MB 710.3MB 0
>>>>> 
>>>>>   b528fedf922823 hours
>>>>> ago732.9MB 625.1MB 107.8MB 4
>>>>> 
>>>>>   8e996f22435e25 hours
>>>>> ago1.624GB 913.7MB710.3MB 0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_testlatest
>>>>>24b73b3fec0625 hours ago1.305GB
>>>>> 965.7MB   339.5MB 0
>>>>> 
>>>>>   096325fb48de   25 hours 
>>>>> ago
>>>>>732.9MB 625.1MB107.8MB  2
>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>> 

Re: apache-beam-jenkins-15 out of disk

[Yifan Zou]

The problem was because of the large quantity of stale docker images
generated by the Python portable tests and HDFS IT.

Dumping the docker disk usage gives me:

TYPETOTAL   ACTIVE  SIZE
 RECLAIMABLE
*Images  1039356 424GB
  384.2GB (90%)*
Containers  987 2   2.042GB
2.041GB (99%)
Local Volumes   126 0   392.8MB
392.8MB (100%)

REPOSITORY
  TAG IMAGE IDCREATED SIZE
   SHARED SIZE UNIQUE SIZE CONTAINERS
jenkins-docker-apache.bintray.io/beam/python3
 latest  ff1b949f444222 hours ago1.639GB
  922.3MB  716.9MB 0
jenkins-docker-apache.bintray.io/beam/python
 latest  1dda7b9d974822 hours ago1.624GB
  913.7MB   710.3MB 0

   05458187a0e322 hours ago
 732.9MB 625.1MB107.8MB 4

   896f35dd685f23 hours ago
 1.639GB 922.3MB   716.9MB 0

   db4d24ca9f2b23 hours ago
 1.624GB 913.7MB  710.3MB 0

547df4d71c3123 hours ago
   732.9MB 625.1MB 107.8MB 4

dd7d9582c3e023 hours ago
   1.639GB 922.3MB 716.9MB 0

664aae25523923 hours ago
   1.624GB 913.7MB 710.3MB 0

b528fedf922823 hours ago
   732.9MB 625.1MB 107.8MB 4

8e996f22435e25 hours ago
   1.624GB 913.7MB710.3MB 0
hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_testlatest
 24b73b3fec0625 hours ago1.305GB 965.7MB
   339.5MB 0

096325fb48de   25 hours ago
 732.9MB 625.1MB107.8MB  2
jenkins-docker-apache.bintray.io/beam/java
 latest  c36d8ff2945d  25 hours ago685.6MB
625.1MB   60.52MB 0

11c86ebe025f26 hours ago
   1.639GB 922.3MB  716.9MB 0

2ecd69c89ec126 hours ago
   1.624GB 913.7MB 710.3MB 0
hdfs_it-jenkins-beam_postcommit_python_verify-8590_testlatest
   3d1d589d44fe2 days ago  1.305GB 965.7MB
 339.5MB 0
hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test  latest
   d1cc503ebe8e2 days ago  1.305GB 965.7MB
339.2MB 0
hdfs_it-jenkins-beam_postcommit_python_verify-8577_testlatest
   8582c6ca6e153 days ago  1.305GB 965.7MB
339.2MB 0
hdfs_it-jenkins-beam_postcommit_python_verify-8576_testlatest
   4591e09481703 days ago  1.305GB 965.7MB
339.2MB 0
hdfs_it-jenkins-beam_postcommit_python_verify-8575_testlatest
   ab181c49d56e4 days ago  1.305GB 965.7MB
339.2MB 0
hdfs_it-jenkins-beam_postcommit_python_verify-8573_testlatest
   2104ba0a6db74 days ago  1.305GB 965.7MB
339.2MB 0
...
<1000+ images>

I removed unused the images and the beam15 is back now.

Opened https://issues.apache.org/jira/browse/BEAM-7650.
Ankur, I assigned the issue to you. Feel free to reassign it if needed.

Thank you.
Yifan

On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou  wrote:

> Something were eating the disk. Disconnected the worker so jobs could be
> allocated to other nodes. Will look deeper.
> Filesystem  Size  Used  Avail Use% Mounted on
> /dev/sda1   485G  485G 96K 100%  /
>
>
> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou  wrote:
>
>> I'm on it.
>>
>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri  wrote:
>>
>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>
>>> Can someone investigate what's going on?
>>>
>>

Re: apache-beam-jenkins-15 out of disk

[Yifan Zou]

Something were eating the disk. Disconnected the worker so jobs could be
allocated to other nodes. Will look deeper.
Filesystem  Size  Used  Avail Use% Mounted on
/dev/sda1   485G  485G 96K 100%  /


On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou  wrote:

> I'm on it.
>
> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri  wrote:
>
>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>
>> Can someone investigate what's going on?
>>
>

Re: [ANNOUNCE] New PMC Member: Pablo Estrada

[Yifan Zou]

Congratulations, Pablo!

*From: *Maximilian Michels 
*Date: *Wed, May 15, 2019 at 2:06 AM
*To: * 

Congrats Pablo! Thank you for your help to grow the Beam community!
>
> On 15.05.19 10:33, Tim Robertson wrote:
> > Congratulations Pablo
> >
> > On Wed, May 15, 2019 at 10:22 AM Ismaël Mejía  > > wrote:
> >
> > Congrats Pablo, well deserved, nece to see your work recognized!
> >
> > On Wed, May 15, 2019 at 9:59 AM Pei HE  > > wrote:
> >  >
> >  > Congrats, Pablo!
> >  >
> >  > On Tue, May 14, 2019 at 11:41 PM Tanay Tummalapalli
> >  > mailto:ttanay.apa...@gmail.com>> wrote:
> >  > >
> >  > > Congratulations Pablo!
> >  > >
> >  > > On Wed, May 15, 2019, 12:08 Michael Luckey  > > wrote:
> >  > >>
> >  > >> Congrats, Pablo!
> >  > >>
> >  > >> On Wed, May 15, 2019 at 8:21 AM Connell O'Callaghan
> > mailto:conne...@google.com>> wrote:
> >  > >>>
> >  > >>> Awesome well done Pablo!!!
> >  > >>>
> >  > >>> Kenn thank you for sharing this great news with us!!!
> >  > >>>
> >  > >>> On Tue, May 14, 2019 at 11:01 PM Ahmet Altay
> > mailto:al...@google.com>> wrote:
> >  > 
> >  >  Congratulations!
> >  > 
> >  >  On Tue, May 14, 2019 at 9:11 PM Robert Burke
> > mailto:rob...@frantil.com>> wrote:
> >  > >
> >  > > Woohoo! Well deserved.
> >  > >
> >  > > On Tue, May 14, 2019, 8:34 PM Reuven Lax  > > wrote:
> >  > >>
> >  > >> Congratulations!
> >  > >>
> >  > >> From: Mikhail Gryzykhin  > >
> >  > >> Date: Tue, May 14, 2019 at 8:32 PM
> >  > >> To: mailto:dev@beam.apache.org>>
> >  > >>
> >  > >>> Congratulations Pablo!
> >  > >>>
> >  > >>> On Tue, May 14, 2019, 20:25 Kenneth Knowles
> > mailto:k...@apache.org>> wrote:
> >  > 
> >  >  Hi all,
> >  > 
> >  >  Please join me and the rest of the Beam PMC in welcoming
> > Pablo Estrada to join the PMC.
> >  > 
> >  >  Pablo first picked up BEAM-722 in October of 2016 and
> > has been a steady part of the Beam community since then. In addition
> > to technical work on Beam Python & Java & runners, I would highlight
> > how Pablo grows Beam's community by helping users, working on GSoC,
> > giving talks at Beam Summits and other OSS conferences including
> > Flink Forward, and holding training workshops. I cannot do justice
> > to Pablo's contributions in a single paragraph.
> >  > 
> >  >  Thanks Pablo, for being a part of Beam.
> >  > 
> >  >  Kenn
> >
>

Re: Jenkins commenting on PRs again

[Yifan Zou]

I've asked Infra and they are not sure as well.

*From: *Lukasz Cwik 
*Date: *Tue, May 14, 2019 at 10:09 AM
*To: *dev

I have seen this in the past, I don't remember how it was resolved.
>
> Kenn is specifically asking about seeing messages:
> asfgit  commented 5 minutes ago
> 
>
> SUCCESS
>
> --none--
>
> *From: *Kenneth Knowles 
> *Date: *Tue, May 14, 2019 at 9:58 AM
> *To: *dev
>
> Does anyone know of a change underway that could cause this, or should we
>> escalate to infra?
>>
>> https://github.com/apache/beam/pull/8576#issuecomment-492321226
>>
>> Kenn
>>
>

Re: Beam Dependency Check Report (2019-05-06)

[Yifan Zou]

The JIRA credentials are missing on the Jenkins. I'm on it.

*From: *Apache Jenkins Server 
*Date: *Mon, May 6, 2019 at 5:06 AM
*To: * 

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html'
> does not exist

Re: [ANNOUNCE] New committer announcement: Udi Meiri

[Yifan Zou]

Thanks Udi and congratulations!

On Fri, May 3, 2019 at 2:47 PM Robin Qiu  wrote:

> Congratulations Udi!!!
>
> *From: *Ruoyun Huang 
> *Date: *Fri, May 3, 2019 at 2:39 PM
> *To: * 
>
> Congratulations Udi!
>>
>> On Fri, May 3, 2019 at 2:30 PM Ahmet Altay  wrote:
>>
>>> Congratulations, Udi!
>>>
>>> *From: *Kyle Weaver 
>>> *Date: *Fri, May 3, 2019 at 2:11 PM
>>> *To: * 
>>>
>>> Congratulations Udi! I look forward to sending you all my reviews for
 the next month (just kidding :)

 Kyle Weaver | Software Engineer | github.com/ibzib |
 kcwea...@google.com | +1650203

 On Fri, May 3, 2019 at 1:52 PM Charles Chen  wrote:
 >
 > Thank you Udi!
 >
 > On Fri, May 3, 2019, 1:51 PM Aizhamal Nurmamat kyzy <
 aizha...@google.com> wrote:
 >>
 >> Congratulations, Udi! Thank you for all your contributions!!!
 >>
 >> From: Pablo Estrada 
 >> Date: Fri, May 3, 2019 at 1:45 PM
 >> To: dev
 >>
 >>> Thanks Udi and congrats!
 >>>
 >>> On Fri, May 3, 2019 at 1:44 PM Kenneth Knowles 
 wrote:
 
  Hi all,
 
  Please join me and the rest of the Beam PMC in welcoming a new
 committer: Udi Meiri.
 
  Udi has been contributing to Beam since late 2017, starting with
 HDFS support in the Python SDK and continuing with a ton of Python work. I
 also will highlight his work on community-building infrastructure,
 including documentation, experiments with ways to find reviewers for pull
 requests, gradle build work, analyzing and reducing build times.
 
  In consideration of Udi's contributions, the Beam PMC trusts Udi
 with the responsibilities of a Beam committer [1].
 
  Thank you, Udi, for your contributions.
 
  Kenn
 
  [1]
 https://beam.apache.org/contribute/become-a-committer/#an-apache-beam-committer

>>>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>

Re: Congrats to Beam's first 6 Google Open Source Peer Bonus recipients!

[Yifan Zou]

Congratulations! Well deserved!

On Thu, May 2, 2019 at 9:37 AM Rui Wang  wrote:

> Congratulations!
>
>
> -Rui
>
> On Thu, May 2, 2019 at 8:23 AM Michael Luckey  wrote:
>
>> Congrats! Well deserved!
>>
>> On Thu, May 2, 2019 at 3:29 PM Alexey Romanenko 
>> wrote:
>>
>>> Congrats!
>>>
>>> On 2 May 2019, at 10:06, Gleb Kanterov  wrote:
>>>
>>> Congratulations! Well deserved!
>>>
>>> On Thu, May 2, 2019 at 10:00 AM Ismaël Mejía  wrote:
>>>
 Congrats everyone !

 On Thu, May 2, 2019 at 9:14 AM Robert Bradshaw 
 wrote:

> Congratulation, and thanks for all the great contributions each one of
> you has made to Beam!
>
> On Thu, May 2, 2019 at 5:51 AM Ruoyun Huang  wrote:
>
>> Congratulations everyone!  Well deserved!
>>
>> On Wed, May 1, 2019 at 8:38 PM Kenneth Knowles 
>> wrote:
>>
>>> Congrats! All well deserved!
>>>
>>> Kenn
>>>
>>> On Wed, May 1, 2019 at 8:09 PM Reza Rokni  wrote:
>>>
 Congratulations!

 On Thu, 2 May 2019 at 10:53, Connell O'Callaghan <
 conne...@google.com> wrote:

> Well done - congratulations to you all!!! Rose thank you for
> sharing this news!!!
>
> On Wed, May 1, 2019 at 19:45 Rose Nguyen 
> wrote:
>
>> Matthias Baetens, Lukazs Gajowy, Suneel Marthi, Maximilian
>> Michels, Alex Van Boxel, and Thomas Weise:
>>
>> Thank you for your exceptional contributions to Apache
>> Beam. I'm looking forward to seeing this project grow and for more 
>> folks
>> to contribute and be recognized! Everyone can read more about this 
>> award on
>> the Google Open Source blog:
>> https://opensource.googleblog.com/2019/04/google-open-source-peer-bonus-winners.html
>>
>> Cheers,
>> --
>> Rose Thị Nguyễn
>>
>

 --
 This email may be confidential and privileged. If you received this
 communication by mistake, please don't forward it to anyone else, 
 please
 erase all copies and attachments, and please let me know that it has 
 gone
 to the wrong person.
 The above terms reflect a potential business arrangement, are
 provided solely as a basis for further discussion, and are not 
 intended to
 be and do not constitute a legally binding obligation. No legally 
 binding
 obligations will be created, implied, or inferred until an agreement in
 final form is executed in writing by all parties involved.

>>>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>
>>>
>>> --
>>> Cheers,
>>> Gleb
>>>
>>>
>>>

Re: Hello from Hannah Jiang

[Yifan Zou]

Welcome！

On Thu, Apr 25, 2019 at 7:34 PM Connell O'Callaghan 
wrote:

> Welcome Hannah!!!
>
> On Thu, Apr 25, 2019, 5:42 PM Reza Rokni  wrote:
>
>> Welcome!
>>
>> On Fri, 26 Apr 2019 at 04:36, Hannah Jiang 
>> wrote:
>>
>>> Thanks Cyrus!
>>>
>>> On Thu, Apr 25, 2019 at 1:34 PM Cyrus Maden  wrote:
>>>
 Welcome!!

 On Thu, Apr 25, 2019 at 4:30 PM Hannah Jiang 
 wrote:

> Thank you Robin!
>
> On Thu, Apr 25, 2019 at 1:27 PM Robin Qiu  wrote:
>
>> Welcome Hannah!
>>
>> On Thu, Apr 25, 2019 at 1:26 PM Hannah Jiang 
>> wrote:
>>
>>> Thanks Kenneth!
>>>
>>> On Thu, Apr 25, 2019 at 1:24 PM Kenneth Knowles 
>>> wrote:
>>>
 Welcome!

 On Thu, Apr 25, 2019 at 12:38 PM Matthias Baetens <
 baetensmatth...@gmail.com> wrote:

> Welcome to the community!
>
> On Thu, Apr 25, 2019, 18:55 Griselda Cuevas 
> wrote:
>
>> Welcome Hannah! - Very excited to see you in the Beam community
>> :)
>>
>> On Tue, 23 Apr 2019 at 12:59, Hannah Jiang <
>> hannahji...@google.com> wrote:
>>
>>> Hi everyone
>>>
>>> I joined Google recently and would work on Python portability
>>> part. I am happy to be part of the community. Looking forward to 
>>> working
>>> with all of you together.
>>>
>>> I have a minor request, can admin please give me access to JIRA?
>>>
>>> Thanks,
>>> Hannah
>>>
>>>
>>>
>>
>> --
>>
>> This email may be confidential and privileged. If you received this
>> communication by mistake, please don't forward it to anyone else, please
>> erase all copies and attachments, and please let me know that it has gone
>> to the wrong person.
>>
>> The above terms reflect a potential business arrangement, are provided
>> solely as a basis for further discussion, and are not intended to be and do
>> not constitute a legally binding obligation. No legally binding obligations
>> will be created, implied, or inferred until an agreement in final form is
>> executed in writing by all parties involved.
>>
>

Re: SNAPSHOTS have not been updated since february

[Yifan Zou]

We could send an email alert once the snapshots publishing fails.
Alternatively, we have a job_PostRelease_NightlySnapshot
<https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostRelease_NightlySnapshot.groovy>
to
validate the healthy of the snapshots. It pulls the latest snapshots and
run java examples against multiple runners. We may also be able to check
the publishing date of the recent snapshot and notify devs if it has not
been updated for a long time.

On Tue, Apr 23, 2019 at 3:25 AM Ismaël Mejía  wrote:

> Thanks a lot Yifan, just pulled them and everything working as expected +1
>
> I wonder still if we may need some extra verification task to detect
> when those have not been updated (which can happen for both INFRA or
> other reasons).
>
> On Tue, Apr 23, 2019 at 4:44 AM Yifan Zou  wrote:
> >
> > The Infra is reviewing the security on our new Jenkins agents. In the
> meanwhile, I assigned the publishing job to the 'ubuntu' labelled nodes as
> a workaround. The snapshots were successfully published in
> beam_Release_NightlySnapshot#421. I'll move the job assignment back once
> our nodes are fully setup.
> >
> > Thanks
> >
> > Yifan
> >
> >
> > On Fri, Apr 19, 2019 at 2:05 AM Ismaël Mejía  wrote:
> >>
> >> Thanks everyone for the quick answer and thanks Yifan for taking care.
> >>
> >> On Thu, Apr 18, 2019 at 7:15 PM Yifan Zou  wrote:
> >> >
> >> > The origin build nodes were updated in Jan 24 and the nexus
> credentials were removed from the filesystem because they are not supposed
> to be on external build nodes (nodes Infra does not own). We now need to
> set up the role account on the new Beam JNLP nodes. I am still contacting
> Infra to bring the snapshot back.
> >> >
> >> > Yifan
> >> >
> >> > On Thu, Apr 18, 2019 at 10:09 AM Lukasz Cwik 
> wrote:
> >> >>
> >> >> The permissions issue is that the credentials needed to publish to
> the maven repository are only deployed on machines managed by Apache Infra.
> Now that the machines have been given back to each project to manage Yifan
> was investigating some other way to get the permissions on to the machine.
> >> >>
> >> >> On Thu, Apr 18, 2019 at 10:06 AM Boyuan Zhang 
> wrote:
> >> >>>
> >> >>> There is a test target
> https://builds.apache.org/job/beam_Release_NightlySnapshot/ in beam,
> which builds and pushes snapshot to maven every day. Current failure is
> like, the jenkin machine cannot publish artifacts into maven owing to some
> weird permission issue. I think +Yifan Zou  is working on it actively.
> >> >>>
> >> >>> On Thu, Apr 18, 2019 at 9:44 AM Ismaël Mejía 
> wrote:
> >> >>>>
> >> >>>> And is there a way we can detect SNAPSHOTS not been published
> daily in
> >> >>>> the future?
> >> >>>>
> >> >>>> On Thu, Apr 18, 2019 at 6:37 PM Ismaël Mejía 
> wrote:
> >> >>>> >
> >> >>>> > Any progress on this?
> >> >>>> >
> >> >>>> > On Wed, Mar 27, 2019 at 5:38 AM Daniel Oliveira <
> danolive...@google.com> wrote:
> >> >>>> > >
> >> >>>> > > I made a bug for this specific issue (artifacts not publishing
> to the Apache Maven repo): https://issues.apache.org/jira/browse/BEAM-6919
> >> >>>> > >
> >> >>>> > > While I was gathering info for the bug report I also noticed
> +Yifan Zou has an experimental PR testing a fix:
> https://github.com/apache/beam/pull/8148
> >> >>>> > >
> >> >>>> > > On Tue, Mar 26, 2019 at 11:42 AM Boyuan Zhang <
> boyu...@google.com> wrote:
> >> >>>> > >>
> >> >>>> > >> +Daniel Oliveira
> >> >>>> > >>
> >> >>>> > >> On Tue, Mar 26, 2019 at 9:57 AM Boyuan Zhang <
> boyu...@google.com> wrote:
> >> >>>> > >>>
> >> >>>> > >>> Sorry for the typo. Ideally, the snapshot publish is
> independent from postrelease_snapshot.
> >> >>>> > >>>
> >> >>>> > >>> On Tue, Mar 26, 2019 at 9:55 AM Boyuan Zhang <
> boyu...@google.com> wrote:
> >> >>>> > >>>>
> >> >>>> > >>>> Hey,
> >> >>>> > >>>

Re: SNAPSHOTS have not been updated since february

[Yifan Zou]

The Infra is reviewing the security on our new Jenkins agents. In the
meanwhile, I assigned the publishing job to the 'ubuntu' labelled nodes as
a workaround. The snapshots were successfully published in
beam_Release_NightlySnapshot#421
<https://builds.apache.org/job/beam_Release_NightlySnapshot/421/>. I'll
move the job assignment back once our nodes are fully setup.

Thanks

Yifan


On Fri, Apr 19, 2019 at 2:05 AM Ismaël Mejía  wrote:

> Thanks everyone for the quick answer and thanks Yifan for taking care.
>
> On Thu, Apr 18, 2019 at 7:15 PM Yifan Zou  wrote:
> >
> > The origin build nodes were updated in Jan 24 and the nexus credentials
> were removed from the filesystem because they are not supposed to be on
> external build nodes (nodes Infra does not own). We now need to set up the
> role account on the new Beam JNLP nodes. I am still contacting Infra to
> bring the snapshot back.
> >
> > Yifan
> >
> > On Thu, Apr 18, 2019 at 10:09 AM Lukasz Cwik  wrote:
> >>
> >> The permissions issue is that the credentials needed to publish to the
> maven repository are only deployed on machines managed by Apache Infra. Now
> that the machines have been given back to each project to manage Yifan was
> investigating some other way to get the permissions on to the machine.
> >>
> >> On Thu, Apr 18, 2019 at 10:06 AM Boyuan Zhang 
> wrote:
> >>>
> >>> There is a test target
> https://builds.apache.org/job/beam_Release_NightlySnapshot/ in beam,
> which builds and pushes snapshot to maven every day. Current failure is
> like, the jenkin machine cannot publish artifacts into maven owing to some
> weird permission issue. I think +Yifan Zou  is working on it actively.
> >>>
> >>> On Thu, Apr 18, 2019 at 9:44 AM Ismaël Mejía 
> wrote:
> >>>>
> >>>> And is there a way we can detect SNAPSHOTS not been published daily in
> >>>> the future?
> >>>>
> >>>> On Thu, Apr 18, 2019 at 6:37 PM Ismaël Mejía 
> wrote:
> >>>> >
> >>>> > Any progress on this?
> >>>> >
> >>>> > On Wed, Mar 27, 2019 at 5:38 AM Daniel Oliveira <
> danolive...@google.com> wrote:
> >>>> > >
> >>>> > > I made a bug for this specific issue (artifacts not publishing to
> the Apache Maven repo): https://issues.apache.org/jira/browse/BEAM-6919
> >>>> > >
> >>>> > > While I was gathering info for the bug report I also noticed
> +Yifan Zou has an experimental PR testing a fix:
> https://github.com/apache/beam/pull/8148
> >>>> > >
> >>>> > > On Tue, Mar 26, 2019 at 11:42 AM Boyuan Zhang 
> wrote:
> >>>> > >>
> >>>> > >> +Daniel Oliveira
> >>>> > >>
> >>>> > >> On Tue, Mar 26, 2019 at 9:57 AM Boyuan Zhang 
> wrote:
> >>>> > >>>
> >>>> > >>> Sorry for the typo. Ideally, the snapshot publish is
> independent from postrelease_snapshot.
> >>>> > >>>
> >>>> > >>> On Tue, Mar 26, 2019 at 9:55 AM Boyuan Zhang <
> boyu...@google.com> wrote:
> >>>> > >>>>
> >>>> > >>>> Hey,
> >>>> > >>>>
> >>>> > >>>> I'm trying to publish the artifacts by commenting "Run Gradle
> Publish" in my PR, but there are several errors saying "cannot write
> artifacts into dir", anyone has idea on it? Ideally, the snapshot publish
> is dependent from postrelease_snapshot. The publish task is to build and
> publish artifacts and the postrelease_snapshot is to verify whether the
> snapshot works.
> >>>> > >>>>
> >>>> > >>>> On Tue, Mar 26, 2019 at 8:45 AM Ahmet Altay 
> wrote:
> >>>> > >>>>>
> >>>> > >>>>> I believe this is related to
> https://issues.apache.org/jira/browse/BEAM-6840 and +Boyuan Zhang has a
> fix in progress https://github.com/apache/beam/pull/8132
> >>>> > >>>>>
> >>>> > >>>>> On Tue, Mar 26, 2019 at 7:09 AM Ismaël Mejía <
> ieme...@gmail.com> wrote:
> >>>> > >>>>>>
> >>>> > >>>>>> I was trying to validate a fix on the Spark runner and
> realized that
> >>>> > >>>>>> Beam SNAPSHOTS have not been updated since February 24 !
> >>>> > >>>>>>
> >>>> > >>>>>>
> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.12.0-SNAPSHOT/
> >>>> > >>>>>>
> >>>> > >>>>>> Can somebody please take a look at why this is not been
> updated?
> >>>> > >>>>>>
> >>>> > >>>>>> Thanks,
> >>>> > >>>>>> Ismaël
>

Re: SNAPSHOTS have not been updated since february

[Yifan Zou]

The origin build nodes were updated in Jan 24 and the nexus credentials
were removed from the filesystem because they are not supposed to be on
external build nodes (nodes Infra does not own). We now need to set up the
role account on the new Beam JNLP nodes. I am still contacting Infra to
bring the snapshot back.

Yifan

On Thu, Apr 18, 2019 at 10:09 AM Lukasz Cwik  wrote:

> The permissions issue is that the credentials needed to publish to the
> maven repository are only deployed on machines managed by Apache Infra. Now
> that the machines have been given back to each project to manage Yifan was
> investigating some other way to get the permissions on to the machine.
>
> On Thu, Apr 18, 2019 at 10:06 AM Boyuan Zhang  wrote:
>
>> There is a test target
>> https://builds.apache.org/job/beam_Release_NightlySnapshot/ in beam,
>> which builds and pushes snapshot to maven every day. Current failure is
>> like, the jenkin machine cannot publish artifacts into maven owing to some
>> weird permission issue. I think +Yifan Zou   is
>> working on it actively.
>>
>> On Thu, Apr 18, 2019 at 9:44 AM Ismaël Mejía  wrote:
>>
>>> And is there a way we can detect SNAPSHOTS not been published daily in
>>> the future?
>>>
>>> On Thu, Apr 18, 2019 at 6:37 PM Ismaël Mejía  wrote:
>>> >
>>> > Any progress on this?
>>> >
>>> > On Wed, Mar 27, 2019 at 5:38 AM Daniel Oliveira <
>>> danolive...@google.com> wrote:
>>> > >
>>> > > I made a bug for this specific issue (artifacts not publishing to
>>> the Apache Maven repo): https://issues.apache.org/jira/browse/BEAM-6919
>>> > >
>>> > > While I was gathering info for the bug report I also noticed +Yifan
>>> Zou has an experimental PR testing a fix:
>>> https://github.com/apache/beam/pull/8148
>>> > >
>>> > > On Tue, Mar 26, 2019 at 11:42 AM Boyuan Zhang 
>>> wrote:
>>> > >>
>>> > >> +Daniel Oliveira
>>> > >>
>>> > >> On Tue, Mar 26, 2019 at 9:57 AM Boyuan Zhang 
>>> wrote:
>>> > >>>
>>> > >>> Sorry for the typo. Ideally, the snapshot publish is independent
>>> from postrelease_snapshot.
>>> > >>>
>>> > >>> On Tue, Mar 26, 2019 at 9:55 AM Boyuan Zhang 
>>> wrote:
>>> > >>>>
>>> > >>>> Hey,
>>> > >>>>
>>> > >>>> I'm trying to publish the artifacts by commenting "Run Gradle
>>> Publish" in my PR, but there are several errors saying "cannot write
>>> artifacts into dir", anyone has idea on it? Ideally, the snapshot publish
>>> is dependent from postrelease_snapshot. The publish task is to build and
>>> publish artifacts and the postrelease_snapshot is to verify whether the
>>> snapshot works.
>>> > >>>>
>>> > >>>> On Tue, Mar 26, 2019 at 8:45 AM Ahmet Altay 
>>> wrote:
>>> > >>>>>
>>> > >>>>> I believe this is related to
>>> https://issues.apache.org/jira/browse/BEAM-6840 and +Boyuan Zhang has a
>>> fix in progress https://github.com/apache/beam/pull/8132
>>> > >>>>>
>>> > >>>>> On Tue, Mar 26, 2019 at 7:09 AM Ismaël Mejía 
>>> wrote:
>>> > >>>>>>
>>> > >>>>>> I was trying to validate a fix on the Spark runner and realized
>>> that
>>> > >>>>>> Beam SNAPSHOTS have not been updated since February 24 !
>>> > >>>>>>
>>> > >>>>>>
>>> https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-core/2.12.0-SNAPSHOT/
>>> > >>>>>>
>>> > >>>>>> Can somebody please take a look at why this is not been updated?
>>> > >>>>>>
>>> > >>>>>> Thanks,
>>> > >>>>>> Ismaël
>>>
>>

Re: Insufficient CPU quota in apache-beam-testing causes test flakes

[Yifan Zou]

We recently created 16 compute instances for the Jenkins. Each one of them
has 16 CPUs, means they consume 256 CPU in total. I guess that is why the
CPU usage in us-central1 remains high. We're working on the migrating the
rest of old Jenkins agents, and the old instances will be removed once
finish. That should relieve the pain of quota.

Yifan

On Tue, Apr 16, 2019 at 1:58 PM Valentyn Tymofieiev 
wrote:

> FYI, I have recently observed a large amount of test failures in Beam test
> suites where Dataflow Jobs failed due to a lack of CPU quota in
> apache-beam-testing project.
>
> We have been adding new suites for Python 3.x versions, which may have
> contributed to this. problem.
>
> I have not investigated yet what consumes the quota yet, but the usage
> remains high.
>
> Possible mitigation options:
> - Increase quota.
> - Decrease per-suite parallelism [1]. Currently we may  run 1-8 tests from
> the same suite concurrently.
> - Audit usage, perhaps kill stale jobs or VMs.
>
> Ideas/opinions welcome.
>
> I opened https://issues.apache.org/jira/browse/BEAM-7085 to track this.
>
> [1]
> https://github.com/apache/beam/search?q=%22--processes%3D%22_q=%22--processes%3D%22
>

Re: Updates on Beam Jenkins

[Yifan Zou]

Thanks, Pablo. The new workers use custom image in the boot disk and it's
easy to reboot without re-imaging. Means we will no longer need assistance
from Infra to reconnect an offline node. Dockerizing the environment would
be helpful to make changes on the environment, such as installing/updating
a beam-required package.


On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada  wrote:

> Thanks for the updates Yifan. I am sure this process has been difficult,
> and I appreciate the good communication, and that this didn't really affect
> the workflow of anyone to validate the new setup for nodes.
>
> I imagine that once we move to dockerizing the testing environment, it
> will be much simpler to restart machines that are having trouble?
> Thanks again!
> -P.
>
> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>
>> Hello,
>>
>> I have some good news about our Jenkins nodes. We're now having 7 new
>> nodes <https://builds.apache.org/label/beam/> online named as
>> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
>> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
>> waiting queue that let your tests getting executed quickly. They're
>> starting up and accepting jobs. There is no action needed on your end.
>>
>> I'll keep the remaining old agents running for one more week just in case
>> it won't fully block the development works if any unexpected issues on the
>> new agents. Once the new agents are stable and verified, I'll move forward
>> to connect the rest agents and remove old set. The playbook is also on the
>> way.
>>
>> For more background and information of the Jenkins updates, please see:
>> One pager:
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>> Environment verification sheet:
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>> Previous Thread on dev@:
>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>
>> Thanks.
>>
>> Regards.
>> Yifan Zou
>>
>>
>>

Updates on Beam Jenkins

[Yifan Zou]

Hello,

I have some good news about our Jenkins nodes. We're now having 7 new nodes
<https://builds.apache.org/label/beam/> online named as
"apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
waiting queue that let your tests getting executed quickly. They're
starting up and accepting jobs. There is no action needed on your end.

I'll keep the remaining old agents running for one more week just in case
it won't fully block the development works if any unexpected issues on the
new agents. Once the new agents are stable and verified, I'll move forward
to connect the rest agents and remove old set. The playbook is also on the
way.

For more background and information of the Jenkins updates, please see:
One pager:
https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
Environment verification sheet:
https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
Previous Thread on dev@:
https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E

Thanks.

Regards.
Yifan Zou

Re: Changes in Beam Jenkins Agents

[Yifan Zou]

Thanks Kenn. Valentyn and I will verify this test.


On Fri, Apr 5, 2019 at 3:44 PM Kenneth Knowles  wrote:

> I was just looking to grab something, and everything is marked verified or
> not a blocker except "Beam PreCommit test with Cython for Py 3.6, Py 3.7"
>
> How does one invoke this?
>
> Kenn
>
> On Thu, Apr 4, 2019 at 10:04 AM Yifan Zou  wrote:
>
>> Great! Thank you, Lukasz!
>>
>> On Thu, Apr 4, 2019 at 3:10 AM Łukasz Gajowy 
>> wrote:
>>
>>> I verified load tests and IO performance tests jobs. Looking
>>> good. Thanks for doing this!
>>>
>>> Łukasz
>>>
>>>
>>>
>>> czw., 4 kwi 2019 o 03:35 Yifan Zou  napisał(a):
>>>
>>>> Hi,
>>>>
>>>> Our Jenkins are in a bad condition. 8 agents are down at this time, and
>>>> they are not going to be restored because of some bad errors happened on
>>>> the Puppet server due to the consistent re-provisioning. There are several
>>>> discussions in the recent weeks between the ASF Infra and us. In general,
>>>> the Infra team will move away from puppet third-party build nodes, and we
>>>> will need to manage the agents by ourselves.
>>>>
>>>> I wrote a one pager to describe the problem and the approach. We
>>>> created a new Jenkins node (
>>>> https://builds.apache.org/computer/beam17-jnlp/) for experimental
>>>> purpose. We're making efforts to verify the environment by running all beam
>>>> tests on it that make sure the required tools are installed properly. The
>>>> process is tracked in the attached spreadsheet. Any helps on verifying
>>>> tests on the new machine are appreciated! The instructions is on the top of
>>>> the spreadsheet.
>>>>
>>>> One pager:
>>>>
>>>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
>>>>
>>>> Verification tracking sheet:
>>>>
>>>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
>>>>
>>>> Thanks.
>>>>
>>>> Regards.
>>>> Yifan
>>>>
>>>

Re: Changes in Beam Jenkins Agents

[Yifan Zou]

Great! Thank you, Lukasz!

On Thu, Apr 4, 2019 at 3:10 AM Łukasz Gajowy 
wrote:

> I verified load tests and IO performance tests jobs. Looking good. Thanks
> for doing this!
>
> Łukasz
>
>
>
> czw., 4 kwi 2019 o 03:35 Yifan Zou  napisał(a):
>
>> Hi,
>>
>> Our Jenkins are in a bad condition. 8 agents are down at this time, and
>> they are not going to be restored because of some bad errors happened on
>> the Puppet server due to the consistent re-provisioning. There are several
>> discussions in the recent weeks between the ASF Infra and us. In general,
>> the Infra team will move away from puppet third-party build nodes, and we
>> will need to manage the agents by ourselves.
>>
>> I wrote a one pager to describe the problem and the approach. We created
>> a new Jenkins node (https://builds.apache.org/computer/beam17-jnlp/) for
>> experimental purpose. We're making efforts to verify the environment by
>> running all beam tests on it that make sure the required tools are
>> installed properly. The process is tracked in the attached spreadsheet. Any
>> helps on verifying tests on the new machine are appreciated! The
>> instructions is on the top of the spreadsheet.
>>
>> One pager:
>>
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys
>>
>> Verification tracking sheet:
>>
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0
>>
>> Thanks.
>>
>> Regards.
>> Yifan
>>
>

Changes in Beam Jenkins Agents

[Yifan Zou]

Hi,

Our Jenkins are in a bad condition. 8 agents are down at this time, and
they are not going to be restored because of some bad errors happened on
the Puppet server due to the consistent re-provisioning. There are several
discussions in the recent weeks between the ASF Infra and us. In general,
the Infra team will move away from puppet third-party build nodes, and we
will need to manage the agents by ourselves.

I wrote a one pager to describe the problem and the approach. We created a
new Jenkins node (https://builds.apache.org/computer/beam17-jnlp/) for
experimental purpose. We're making efforts to verify the environment by
running all beam tests on it that make sure the required tools are
installed properly. The process is tracked in the attached spreadsheet. Any
helps on verifying tests on the new machine are appreciated! The
instructions is on the top of the spreadsheet.

One pager:
https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#heading=h.lm27uybdtpys

Verification tracking sheet:
https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit?ts=5ca54b2d#gid=0

Thanks.

Regards.
Yifan

Re: Frequent failures on beam8

[Yifan Zou]

the beam8 is disabled by now.

On Mon, Mar 25, 2019 at 2:06 PM Mikhail Gryzykhin  wrote:

> Yifan is looking into this.
>
> On Mon, Mar 25, 2019 at 1:55 PM Boyuan Zhang  wrote:
>
>> Hey all,
>>
>> Could anyone help take a look at beam8
>> ? Seems like there are
>> many tests failed on beam8 owing to infra problems.
>>
>> Thanks!
>>
>

Re: [ANNOUNCE] New committer announcement: Mark Liu

[Yifan Zou]

Congratulations Mark!

On Sun, Mar 24, 2019 at 10:25 PM Connell O'Callaghan 
wrote:

> Well done congratulations Mark!!!
>
> On Sun, Mar 24, 2019 at 10:17 PM Robert Burke  wrote:
>
>> Congratulations Mark! 
>>
>> On Sun, Mar 24, 2019, 10:08 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Congratulations, Mark!
>>>
>>> Thanks for your contributions, in particular for your efforts to
>>> parallelize test execution for Python SDK and increase the speed of Python
>>> precommit checks.
>>>
>>> On Sun, Mar 24, 2019 at 9:40 PM Kenneth Knowles  wrote:
>>>
 Hi all,

 Please join me and the rest of the Beam PMC in welcoming a new
 committer: Mark Liu.

 Mark has been contributing to Beam since late 2016! He has proposed
 100+ pull requests. Mark was instrumental in expanding test and
 infrastructure coverage, especially for Python. In consideration of
 Mark's contributions, the Beam PMC trusts Mark with the responsibilities of
 a Beam committer [1].

 Thank you, Mark, for your contributions.

 Kenn

 [1] https://beam.apache.org/contribute/become-a-committer/#an-apache-
 beam-committer

>>>

Re: Python36/37 not installed on Beam2 and Beam12?

[Yifan Zou]

You could try to ping them in the slack channel
https://the-asf.slack.com/messages/  if it is really urgent.

On Wed, Mar 20, 2019 at 5:29 PM Mark Liu  wrote:

> Hi,
>
> I saw occasional py36 tox test failure in beam_PreCommit_Python
> and beam_Release_NightlySnapshot in cron job
>  as well
> as PR triggered job
> . The
> error is simple:
>
> ERROR: InterpreterNotFound: python3.6
>
> Turns out those failures only happened in Beam2 and Beam12. From console
> log of inventory jobs (beam2
>  and
> beam12 ),
> I found python3.6 and python3.7 interpreters are missing. This makes
> beam_PreCommit_Python_Cron
>  flaky
> recently and may fail any python build that runs on those two nodes.
>
> Infra team helped install Python3 on our Jenkins before, but they were
> slow for response on JIRA. What's the best way to have Infra team get
> involved to this problem?
>
> Thanks,
> Mark
>

Investigating Jenkins Agents OOM

[Yifan Zou]

Hi,

Some of you are aware of our Jenkins nodes fell in trouble very often.
We're now getting the beam4, 7 and 13 disconnected and jobs failed due to
the similar reason: OOM. I am asking ASF Infra to dump the usage of
resources (memory, disk, cpu...) from those machines before we reboot. In
the meantime, you may see longer waiting time on jobs due to the agent
reduction. We are sorry about it.

I've recorded some Jenkins console logs from those agents into a doc. Please
let us know if you have insights on these problems. Any helps
are appreciated.
https://docs.google.com/document/d/1OBmWumaJCuHPNMHM4-V6JWhYY2ZV_ETqUdFQG5A7R4E/edit?usp=sharing

Regards.
Yifan

Re: beam9 bad worker

[Yifan Zou]

FYI, the beam9 is off.

Re: When we use text to trigger jenkins check job, is there a way to
specify a server?
You cannot direct the job to a node via the github phrase. But, you could
simply specify the agent in the Jenkins DLS scripts by adding a parameter.
See the inventory jobs
<https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L41>
as examples.

On Mon, Mar 4, 2019 at 3:06 PM Ruoyun Huang  wrote:

> Thanks a lot folks. Looking forward to the fix.
>
> When we use text to trigger jenkins check job, is there are way to specify
> a server?
>
> On Mon, Mar 4, 2019 at 2:06 PM Yifan Zou  wrote:
>
>> I am looking into the error and will disconnect the beam9 to stop
>> breaking tests.
>>
>> On Mon, Mar 4, 2019 at 2:00 PM Pablo Estrada  wrote:
>>
>>> I've talked with Yifan, and I believe he's looking into it. : )
>>> Best
>>> -P.
>>>
>>> On Mon, Mar 4, 2019 at 1:55 PM Ankur Goenka  wrote:
>>>
>>>> Beam9 is failing all the scheduled jobs. Can we reboot the machine?
>>>>
>>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: beam9 bad worker

[Yifan Zou]

I am looking into the error and will disconnect the beam9 to stop breaking
tests.

On Mon, Mar 4, 2019 at 2:00 PM Pablo Estrada  wrote:

> I've talked with Yifan, and I believe he's looking into it. : )
> Best
> -P.
>
> On Mon, Mar 4, 2019 at 1:55 PM Ankur Goenka  wrote:
>
>> Beam9 is failing all the scheduled jobs. Can we reboot the machine?
>>
>

Re: FYI: beam11 bad worker

[Yifan Zou]

The beam11 is temporarily offline. Jobs won't be assigned to it at this
time. We'll reconfigure the VM and bring it back later.

On Mon, Feb 11, 2019 at 10:05 AM Mikhail Gryzykhin 
wrote:

> Hi everyone,
>
> Small update:
> We have a bad jenkins executor that fails all builds. You might experience
> pre/post commit failures.
>
> Yifan follows up on this.
>
> Regards,
> --Mikhail
>
> Have feedback ?
>

Re: Jenkins slowness

[Yifan Zou]

I saw many non-beam jobs are running on our nodes. It probably be one of
the reasons which caused long waiting time.
e.g. https://builds.apache.org/computer/beam13/builds

On Thu, Feb 7, 2019 at 12:24 PM Udi Meiri  wrote:

> There is also excessive python test logging tracked here:
> https://issues.apache.org/jira/browse/BEAM-6603
>
> On Thu, Feb 7, 2019, 12:23 Udi Meiri 
>> I suggest disabling Jacoco and re-enabling the build cache until we can
>> migrate to Gradle 5. I imagine the migration to v5 is not a simple change.
>> Meanwhile, I can't run postcommits on PRs on Jenkins (run seed job + run
>> postcommit).
>>
>> On Thu, Feb 7, 2019, 12:00 Chamikara Jayalath > wrote:
>>
>>> Seems like there was a spike for all build times yesterday probably
>>> added up to give slow Jenkins scheduling times for triggers. Also, seems
>>> like we had three spikes that are about a week apart recently.
>>>
>>>
>>> On Thu, Feb 7, 2019 at 11:46 AM Michael Luckey 
>>> wrote:
>>>
 What might have some influence is the implicit disabling of the build
 cache by activating Jacoco report. There seems to be a increase of
 beam_PreCommit_Java_Cron with
 https://builds.apache.org/job/beam_PreCommit_Java_Cron/914/ and
 looking into cacheable task there seems to be lots of work done now which
 previously was cacheable.

 Not sure, whether this is the culprit- or part of it -, but I d suggest
 to upgrade to gradle 5 pretty fast.

 On Thu, Feb 7, 2019 at 8:18 PM Udi Meiri  wrote:

> If anyone has done any investigation/is working on this please share.
>
> I'm investigating Jenkins slowness. I've noticed it happening since
> yesterday: precommits taking 3 hours to start, phrase commands similarly
> taking as much time to register.
>
> My current theory is that we have a job that's are taking much longer
> than usual to run.
>

Re: Beam Dependency Check Report (2018-06-13)

[Yifan Zou]

Hi,

You're looking at the old versions dependency bugs which were created
before Oct, 2018 (e.g BEAM-4904
<https://issues.apache.org/jira/browse/BEAM-4904>). Based on the discussion
[1]
<https://lists.apache.org/thread.html/28d3c349a5021c3598379b6f6b9210b4ef150a6235e55c0499250034@%3Cdev.beam.apache.org%3E>,
we modified the tool with the new Beam Dependency Policy
<https://beam.apache.org/contribute/dependencies/>, and closed the old bugs
(most of them were marked as won't fix, and they will never get updated).

The current dependency JIRA looks like this: BEAM-5549
<https://issues.apache.org/jira/browse/BEAM-5549>. The major changes
including [2] <https://issues.apache.org/jira/browse/BEAM-5339>:

1. A JIRA will be created if a dependency has more then 1 major version or
3 minor versions behind the latest version. Or, there is new version
available for more then a year that the dep didn't update in Beam.
2. A JIRA could be closed if the new version is not appropriate to be used
in Beam. In this case, the tool will stop checking updates on this dep
until the next major version available or after 3 months.
3. Stop specifying the target version number in the issue's title. This
ensures that only one JIRA would be opened for a dep that people can easily
track the update history.
4. Stop directly assigning bugs to a person. Instead, cc owners in the
descriptions.

Please use the new dependency JIRAs to track the updates. Thanks for taking
care of Beam dependencies and let me know if you have any questions and
concerns.

Regards.
Yifan

[1]:
https://lists.apache.org/thread.html/28d3c349a5021c3598379b6f6b9210b4ef150a6235e55c0499250034@%3Cdev.beam.apache.org%3E
[2]: https://issues.apache.org/jira/browse/BEAM-5339

On Mon, Jan 28, 2019 at 6:22 AM Ismaël Mejía  wrote:

> Hello,
>
> The dependency update report has been working fine. However I found some
> issues that I summarized in this issue.
> https://issues.apache.org/jira/browse/BEAM-6524
> Can Yifan or someone else that knows that area please take a look.
>
> Regards,
> Ismaël
>
>
> On Thu, Jun 14, 2018 at 11:37 PM Yifan Zou  wrote:
>
>> Thank you Paul for letting us know this issue. We will take care of it
>> when upgrading dependencies.
>>
>> On Thu, Jun 14, 2018 at 7:23 AM Paul Gerver  wrote:
>>
>>> I do have one request to be added to the Java SDK version updates:
>>> Beam-3831 [1]. The Google Core depends on the old org.json package which
>>> ASF discourages using because of the "Use only for good, not evil" clause.
>>>
>>> [1] https://issues.apache.org/jira/browse/BEAM-3831
>>>
>>> On Thu, Jun 14, 2018 at 3:03 AM Etienne Chauchot 
>>> wrote:
>>>
>>>> Thanks Yifan,
>>>>
>>>> This is great ! It would help us maintain Beam more easily and probably
>>>> help us fixing CVE as well.
>>>>
>>>> Etienne
>>>>
>>>> Le mercredi 13 juin 2018 à 07:45 -0700, Yifan Zou a écrit :
>>>>
>>>> Hi,
>>>>
>>>>
>>>> I want to follow up and explain this email.
>>>>
>>>>
>>>> This is a sample email that reports the results of Beam SDK dependency
>>>> check, which was proposed here
>>>> <https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#heading=h.u75g8bk11ngp>.
>>>> The goal is finding updates for all Beam Python & Java SDKs' dependencies
>>>> and prioritize them. The job will be auto triggered in Jenkins once a week
>>>> and generate a report. The report lists the high priority updates base on
>>>> the following criteria:
>>>>
>>>>
>>>> The dependency update is high priority if:
>>>>
>>>> 1. It has major versions update available;
>>>>
>>>>   e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0
>>>>
>>>>  2. or, it is over 3 minor versions behind the latest version;
>>>>
>>>>   e.g. org.tukaani:xz 1.5 -> 1.8
>>>>
>>>> 3. or, the current version is behind the later version for over 180
>>>> days.
>>>>
>>>>   e.g. com.google.auto.service:auto-service 2014-10-24 ->
>>>> 2017-12-11
>>>>
>>>>
>>>> This job helps Beam contributors to determine the dependency which is
>>>> far behind the latest released version. The next step would be automating
>>>> filing JIRA bugs for dep updates, group dependencies and identify owners to
>>>> take care of the upgrades follow Chamikara's proposal
>>>> <https:

Re: Our jenkins beam1 server is down

[Yifan Zou]

Looking. The following errors happened consistently.

Jan 23 16:05:55 apache-beam-jenkins-slave-group-51fn systemd[1]:
Started Session 72 of user jenkins.
Jan 23 16:06:03 apache-beam-jenkins-slave-group-51fn snmpd[16379]:
error on subcontainer 'ia_addr' insert (-1)
Jan 23 16:08:33 apache-beam-jenkins-slave-group-51fn snmpd[16379]:
message repeated 5 times: [ error on subcontainer 'ia_addr' insert
(-1)]


On Wed, Jan 23, 2019 at 7:19 AM Ismaël Mejía  wrote:

> Looks like beam9 is now gone.
>
> On Tue, Jan 22, 2019 at 8:57 PM Yifan Zou  wrote:
> >
> > The inventory test on the beam1 passed. The beam1 is back to normal.
> > https://builds.apache.org/job/beam_Inventory_beam1/303/
> >
> > On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou  wrote:
> >>
> >> Thanks for reporting the failures. Just disconnect and reconnect beam1.
> I am creating a PR that force run a job on that agent to verify.
> >>
> >> On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka 
> wrote:
> >>>
> >>> Beam 1 seems to be down again
> >>>
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
> >>>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
> >>>
> >>> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou 
> wrote:
> >>>>
> >>>> The beam1 and 14 are back and building.
> >>>>
> >>>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía 
> wrote:
> >>>>>
> >>>>> Thanks Yifan for taking care.
> >>>>>
> >>>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou 
> wrote:
> >>>>> >
> >>>>> > Yes, beam14 is offline as well. We're on it.
> >>>>> >
> >>>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang 
> wrote:
> >>>>> >>
> >>>>> >> With another try, succeeding on beam10.
> >>>>> >>
> >>>>> >> Thanks for the fix.
> >>>>> >>
> >>>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang 
> wrote:
> >>>>> >>>
> >>>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
> offline; cannot locate JDK 1.8 (latest)".
> >>>>> >>>
> >>>>> >>> Beam1 is not the only one broken?
> >>>>> >>>
> >>>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou 
> wrote:
> >>>>> >>>>
> >>>>> >>>> The beam1 was still accepting jobs and breaking them after
> reset this morning. We temporarily disconnect it so that jobs could be
> scheduled on healthy nodes. Infra is making efforts to fix beam1.
> >>>>> >>>>
> >>>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou 
> wrote:
> >>>>> >>>>>
> >>>>> >>>>> The VM instance was reset and Infra is trying to repuppetize
> it. https://issues.apache.org/jira/browse/INFRA-17672 is created to track
> this issue.
> >>>>> >>>>>
> >>>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu 
> wrote:
> >>>>> >>>>>>
> >>>>> >>>>>> Thanks you Yifan!
> >>>>> >>>>>>
> >>>>> >>>>>> Looks like following precommits are affected according to my
> PR:
> >>>>> >>>>>>
> >>>>> >>>>>> Java_Examples_Dataflow,
> >>>>> >>>>>> Portable_Python,
> >>>>> >>>>>> Website_Stage_GCS
> >>>>> >>>>>>
> >>>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou <
> yifan...@google.com> wrote:
> >>>>> >>>>>>>
> >>>>> >>>>>>> I am looking on it.
> >>>>> >>>>>>>
> >>>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía <
> ieme...@gmail.com> wrote:
> >>>>> >>>>>>>>
> >>>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling
> algorithm is
> >>>>> >>>>>>>> sending most builds to it so there are issues to validate
> some PRs.
> >>>>> >>>
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> --
> >>>>> >>> 
> >>>>> >>> Ruoyun  Huang
> >>>>> >>>
> >>>>> >>
> >>>>> >>
> >>>>> >> --
> >>>>> >> 
> >>>>> >> Ruoyun  Huang
> >>>>> >>
>

Re: Our jenkins beam1 server is down

[Yifan Zou]

The inventory test on the beam1 passed. The beam1 is back to normal.
https://builds.apache.org/job/beam_Inventory_beam1/303/

On Tue, Jan 22, 2019 at 11:41 AM Yifan Zou  wrote:

> Thanks for reporting the failures. Just disconnect and reconnect beam1. I
> am creating a PR that force run a job on that agent to verify.
>
> On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka  wrote:
>
>> Beam 1 seems to be down again
>>
>> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
>>
>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
>>
>> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou  wrote:
>>
>>> The beam1 and 14 are back and building.
>>>
>>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía  wrote:
>>>
>>>> Thanks Yifan for taking care.
>>>>
>>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou  wrote:
>>>> >
>>>> > Yes, beam14 is offline as well. We're on it.
>>>> >
>>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang 
>>>> wrote:
>>>> >>
>>>> >> With another try, succeeding on beam10.
>>>> >>
>>>> >> Thanks for the fix.
>>>> >>
>>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang 
>>>> wrote:
>>>> >>>
>>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
>>>> offline; cannot locate JDK 1.8 (latest)".
>>>> >>>
>>>> >>> Beam1 is not the only one broken?
>>>> >>>
>>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou 
>>>> wrote:
>>>> >>>>
>>>> >>>> The beam1 was still accepting jobs and breaking them after reset
>>>> this morning. We temporarily disconnect it so that jobs could be scheduled
>>>> on healthy nodes. Infra is making efforts to fix beam1.
>>>> >>>>
>>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou 
>>>> wrote:
>>>> >>>>>
>>>> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
>>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>>> this issue.
>>>> >>>>>
>>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu 
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Thanks you Yifan!
>>>> >>>>>>
>>>> >>>>>> Looks like following precommits are affected according to my PR:
>>>> >>>>>>
>>>> >>>>>> Java_Examples_Dataflow,
>>>> >>>>>> Portable_Python,
>>>> >>>>>> Website_Stage_GCS
>>>> >>>>>>
>>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou 
>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> I am looking on it.
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía 
>>>> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm
>>>> is
>>>> >>>>>>>> sending most builds to it so there are issues to validate some
>>>> PRs.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> 
>>>> >>> Ruoyun  Huang
>>>> >>>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> 
>>>> >> Ruoyun  Huang
>>>> >>
>>>>
>>>

Re: Our jenkins beam1 server is down

[Yifan Zou]

Thanks for reporting the failures. Just disconnect and reconnect beam1. I
am creating a PR that force run a job on that agent to verify.

On Tue, Jan 22, 2019 at 11:08 AM Ankur Goenka  wrote:

> Beam 1 seems to be down again
>
> https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/88/console
>
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/141/console
>
> On Tue, Jan 22, 2019 at 10:53 AM Yifan Zou  wrote:
>
>> The beam1 and 14 are back and building.
>>
>> On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía  wrote:
>>
>>> Thanks Yifan for taking care.
>>>
>>> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou  wrote:
>>> >
>>> > Yes, beam14 is offline as well. We're on it.
>>> >
>>> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang 
>>> wrote:
>>> >>
>>> >> With another try, succeeding on beam10.
>>> >>
>>> >> Thanks for the fix.
>>> >>
>>> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang 
>>> wrote:
>>> >>>
>>> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is
>>> offline; cannot locate JDK 1.8 (latest)".
>>> >>>
>>> >>> Beam1 is not the only one broken?
>>> >>>
>>> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou 
>>> wrote:
>>> >>>>
>>> >>>> The beam1 was still accepting jobs and breaking them after reset
>>> this morning. We temporarily disconnect it so that jobs could be scheduled
>>> on healthy nodes. Infra is making efforts to fix beam1.
>>> >>>>
>>> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou 
>>> wrote:
>>> >>>>>
>>> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>> this issue.
>>> >>>>>
>>> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu 
>>> wrote:
>>> >>>>>>
>>> >>>>>> Thanks you Yifan!
>>> >>>>>>
>>> >>>>>> Looks like following precommits are affected according to my PR:
>>> >>>>>>
>>> >>>>>> Java_Examples_Dataflow,
>>> >>>>>> Portable_Python,
>>> >>>>>> Website_Stage_GCS
>>> >>>>>>
>>> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou 
>>> wrote:
>>> >>>>>>>
>>> >>>>>>> I am looking on it.
>>> >>>>>>>
>>> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía 
>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>> >>>>>>>> sending most builds to it so there are issues to validate some
>>> PRs.
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> 
>>> >>> Ruoyun  Huang
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> 
>>> >> Ruoyun  Huang
>>> >>
>>>
>>

Re: Our jenkins beam1 server is down

[Yifan Zou]

The beam1 and 14 are back and building.

On Thu, Jan 17, 2019 at 7:04 AM Ismaël Mejía  wrote:

> Thanks Yifan for taking care.
>
> On Thu, Jan 17, 2019 at 1:24 AM Yifan Zou  wrote:
> >
> > Yes, beam14 is offline as well. We're on it.
> >
> > On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang  wrote:
> >>
> >> With another try, succeeding on beam10.
> >>
> >> Thanks for the fix.
> >>
> >> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang  wrote:
> >>>
> >>> Just did a rerun, got error saying "10:12:21 ERROR: beam14 is offline;
> cannot locate JDK 1.8 (latest)".
> >>>
> >>> Beam1 is not the only one broken?
> >>>
> >>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou  wrote:
> >>>>
> >>>> The beam1 was still accepting jobs and breaking them after reset this
> morning. We temporarily disconnect it so that jobs could be scheduled on
> healthy nodes. Infra is making efforts to fix beam1.
> >>>>
> >>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou 
> wrote:
> >>>>>
> >>>>> The VM instance was reset and Infra is trying to repuppetize it.
> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
> this issue.
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu 
> wrote:
> >>>>>>
> >>>>>> Thanks you Yifan!
> >>>>>>
> >>>>>> Looks like following precommits are affected according to my PR:
> >>>>>>
> >>>>>> Java_Examples_Dataflow,
> >>>>>> Portable_Python,
> >>>>>> Website_Stage_GCS
> >>>>>>
> >>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou 
> wrote:
> >>>>>>>
> >>>>>>> I am looking on it.
> >>>>>>>
> >>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía 
> wrote:
> >>>>>>>>
> >>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
> >>>>>>>> sending most builds to it so there are issues to validate some
> PRs.
> >>>
> >>>
> >>>
> >>> --
> >>> 
> >>> Ruoyun  Huang
> >>>
> >>
> >>
> >> --
> >> 
> >> Ruoyun  Huang
> >>
>

Re: Our jenkins beam1 server is down

[Yifan Zou]

Yes, beam14 is offline as well. We're on it.

On Wed, Jan 16, 2019 at 4:11 PM Ruoyun Huang  wrote:

> With another try, succeeding on beam10.
>
> Thanks for the fix.
>
> On Wed, Jan 16, 2019 at 3:53 PM Ruoyun Huang  wrote:
>
>> Just did a rerun, got error saying "*10:12:21* ERROR: beam14 is offline;
>> cannot locate JDK 1.8 (latest)".
>>
>> Beam1 is not the only one broken?
>>
>> On Wed, Jan 16, 2019 at 3:45 PM Yifan Zou  wrote:
>>
>>> The beam1 was still accepting jobs and breaking them after reset this
>>> morning. We temporarily disconnect it so that jobs could be scheduled on
>>> healthy nodes. Infra is making efforts to fix beam1.
>>>
>>> On Wed, Jan 16, 2019 at 11:15 AM Yifan Zou  wrote:
>>>
>>>> The VM instance was reset and Infra is trying to repuppetize it.
>>>> https://issues.apache.org/jira/browse/INFRA-17672 is created to track
>>>> this issue.
>>>>
>>>> On Wed, Jan 16, 2019 at 10:51 AM Mark Liu  wrote:
>>>>
>>>>> Thanks you Yifan!
>>>>>
>>>>> Looks like following precommits are affected according to my PR:
>>>>>
>>>>> Java_Examples_Dataflow,
>>>>> Portable_Python,
>>>>> Website_Stage_GCS
>>>>>
>>>>> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou  wrote:
>>>>>
>>>>>> I am looking on it.
>>>>>>
>>>>>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía 
>>>>>> wrote:
>>>>>>
>>>>>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>>>>>> sending most builds to it so there are issues to validate some PRs.
>>>>>>>
>>>>>>
>>
>> --
>> 
>> Ruoyun  Huang
>>
>>
>
> --
> 
> Ruoyun  Huang
>
>

Re: Our jenkins beam1 server is down

[Yifan Zou]

The VM instance was reset and Infra is trying to repuppetize it.
https://issues.apache.org/jira/browse/INFRA-17672 is created to track this
issue.

On Wed, Jan 16, 2019 at 10:51 AM Mark Liu  wrote:

> Thanks you Yifan!
>
> Looks like following precommits are affected according to my PR:
>
> Java_Examples_Dataflow,
> Portable_Python,
> Website_Stage_GCS
>
> On Wed, Jan 16, 2019 at 9:25 AM Yifan Zou  wrote:
>
>> I am looking on it.
>>
>> On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía  wrote:
>>
>>> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
>>> sending most builds to it so there are issues to validate some PRs.
>>>
>>

Re: Our jenkins beam1 server is down

[Yifan Zou]

I am looking on it.

On Wed, Jan 16, 2019 at 8:18 AM Ismaël Mejía  wrote:

> Can somebody PTAL. Sadly the poor jenkins shuffling algorithm is
> sending most builds to it so there are issues to validate some PRs.
>

Re: Beam snapshots broken

[Yifan Zou]

Beam9 is offline right now. But, the job also failed on beam4 and 13
with "Could
not determine the dependencies of task ':beam-sdks-python:test.".
Seems like the task dependency did not setup properly.



On Wed, Dec 12, 2018 at 2:03 AM Ismaël Mejía  wrote:

> You are right it seems that it was related to beam9 (wondering if it
> was bad luck that it was always assigned to beam9 or we can improve
> that poor balancing error).
> However it failed again today against beam13 maybe this time is just a
> build issue but seems related to python too.
>
> On Tue, Dec 11, 2018 at 7:33 PM Boyuan Zhang  wrote:
> >
> > Seems like, all failed jobs are not owing to the single task failure.
> There failed task were executed on beam9, which was rebooted yesterday
> because python tests failed continuously. +Yifan Zou may have more useful
> content here.
> >
> > On Tue, Dec 11, 2018 at 9:10 AM Ismaël Mejía  wrote:
> >>
> >> It seems that Beam snapshots are broken since Dec. 2
> >>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_Gradle_NightlySnapshot/
> >>
> >> It seems "The :beam-website:startDockerContainer task failed."
> >> Can somebody please take a look.
>

Re: INSTANCE_GROUP_MANAGERS quota limit exceeded

[Yifan Zou]

+1. I believe there were too many dataflow jobs running at that moment that
exceeded the compute engine quota of us-central1 ( the current quota is
100).

Yifan

On Fri, Nov 2, 2018 at 1:49 PM Alan Myrvold  wrote:

> Not sure, but I expect that each dataflow job needs a managed instance
> group on the apache-beam-testing project, so it would depend on how many
> dataflow jobs are being run in parallel.
> The quota for concurrent dataflow jobs is 300, the quota for managed
> instance groups is 100. It would make sense to increase the managed
> instance group quota or reduce the number of concurrent jobs.
>
> On Fri, Nov 2, 2018 at 1:26 PM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> I was looking through (now 33) tickets open based on our post-commit
>> tests failures and noticed one interesting.
>>
>> Test failed due to INSTANCE_GROUP_MANAGERS quota exceeding.
>> BEAM-5938  Thank you
>> Daniel for triaging failure.
>>
>> Even thought this seem to happen only once so far, it might become a
>> bigger issue since we are adding more tests.
>>
>> I've checked on the GCloud instance groups
>> , but it didn't
>> give me good insight into what might have triggered the quota issue.
>>
>> Does anyone know how we use instance group managers and triage whether
>> this might hit us once we increase amount of tests?
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback ?
>>
>

Re: Docker missing on Beam15

[Yifan Zou]

FYI, the docker was restarted on beam15.

On Tue, Oct 23, 2018 at 7:08 AM Thomas Weise  wrote:

> For the latter (createProcessWorker):
> https://github.com/apache/beam/pull/6793
>
>
> On Tue, Oct 23, 2018 at 6:47 AM Thomas Weise  wrote:
>
>> Thanks for taking a look Yifan. Yes, it appears this was an intermittent
>> issue.
>>
>> For beam_PostCommit_Python_VR_Flink we are left with:
>>
>> * beam15 docker errors
>> * segmentation faults
>> * "Execution failed for task ':beam-sdks-python:createProcessWorker'" -
>> which should not even execute since we are using Docker
>>
>>
>> On Mon, Oct 22, 2018 at 10:50 PM Yifan Zou  wrote:
>>
>>> I'm not able to reproduce that error in Beam6 (#459
>>> <https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/459/>,
>>> #460
>>> <https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/460/>),
>>> it probably due to some outage of Debian [1]. The image was successfully
>>> built, but the test failed in other reasons.
>>> And indeed, the beam_PostCommit_Python_VR_Flink is very flaky.
>>>
>>> Yifan
>>>
>>> [1] https://github.com/docker-library/python/issues/241
>>>
>>> On Mon, Oct 22, 2018 at 5:39 PM Thomas Weise  wrote:
>>>
>>>> Looks like we have more container build related errors.
>>>>
>>>> This is from beam6 -
>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/44/
>>>>
>>>> Reading package lists...
>>>> [91mW: The repository 'http://deb.debian.org/debian stretch Release'
>>>> does not have a Release file.
>>>>
>>>> W: The repository 'http://deb.debian.org/debian stretch-updates Release' 
>>>> does not have a Release file.
>>>> E: Failed to fetch 
>>>> http://deb.debian.org/debian/dists/stretch/main/binary-amd64/Packages  404 
>>>>  Not Found
>>>> E: Failed to fetch 
>>>> http://deb.debian.org/debian/dists/stretch-updates/main/binary-amd64/Packages
>>>>   404  Not Found
>>>> E: Some index files failed to download. They have been ignored, or old 
>>>> ones used instead.
>>>>
>>>>
>>>> On Mon, Oct 22, 2018 at 2:54 PM Ankur Goenka  wrote:
>>>>
>>>>> Thanks Yifan!
>>>>>
>>>>> On Mon, Oct 22, 2018 at 2:53 PM Yifan Zou  wrote:
>>>>>
>>>>>> So, looks like none of us have the permissions. I filed INFRA-17167
>>>>>> <https://issues.apache.org/jira/browse/INFRA-17167> to the Infra
>>>>>> team to restart the docker on the beam15.
>>>>>>
>>>>>> Thanks.
>>>>>> Yifan
>>>>>>
>>>>>> On Mon, Oct 22, 2018 at 9:20 AM Scott Wegner 
>>>>>> wrote:
>>>>>>
>>>>>>> I've seen the docker issue pop-up on website pre-commits as well:
>>>>>>> https://issues.apache.org/jira/browse/BEAM-5783. There were also on
>>>>>>> beam15.
>>>>>>>
>>>>>>> When I searched around the internet I found lots of instances of the
>>>>>>> same error; it seems to be some unreliability in the guts of Docker [1].
>>>>>>> Perhaps restarting the VM or docker daemon could help. Does anybody have
>>>>>>> permissions to log on and try it?
>>>>>>>
>>>>>>> [1] https://github.com/moby/moby/issues/31849#issuecomment-320236354
>>>>>>>
>>>>>>> On Sun, Oct 21, 2018 at 7:13 PM Thomas Weise  wrote:
>>>>>>>
>>>>>>>> There are two issues with
>>>>>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/
>>>>>>>> currently:
>>>>>>>>
>>>>>>>> 1) The mentioned issue with docker on beam15 - Jason, can you
>>>>>>>> possibly advise how to deal with it?
>>>>>>>>
>>>>>>>> 2) Frequent failure due to "Segmentation fault (core dumped)", as
>>>>>>>> exhibited by
>>>>>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/449/consoleText
>>>>>>>>
>>>>>>>> The Gradle scan is here:
>>>>>>>>
>>>>>>>>
>>>>>>>> https://sc

Re: Docker missing on Beam15

[Yifan Zou]

So, looks like none of us have the permissions. I filed INFRA-17167
<https://issues.apache.org/jira/browse/INFRA-17167> to the Infra team to
restart the docker on the beam15.

Thanks.
Yifan

On Mon, Oct 22, 2018 at 9:20 AM Scott Wegner  wrote:

> I've seen the docker issue pop-up on website pre-commits as well:
> https://issues.apache.org/jira/browse/BEAM-5783. There were also on
> beam15.
>
> When I searched around the internet I found lots of instances of the same
> error; it seems to be some unreliability in the guts of Docker [1]. Perhaps
> restarting the VM or docker daemon could help. Does anybody have
> permissions to log on and try it?
>
> [1] https://github.com/moby/moby/issues/31849#issuecomment-320236354
>
> On Sun, Oct 21, 2018 at 7:13 PM Thomas Weise  wrote:
>
>> There are two issues with
>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/ currently:
>>
>> 1) The mentioned issue with docker on beam15 - Jason, can you possibly
>> advise how to deal with it?
>>
>> 2) Frequent failure due to "Segmentation fault (core dumped)", as
>> exhibited by
>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/449/consoleText
>>
>> The Gradle scan is here:
>>
>>
>> https://scans.gradle.com/s/ebhxs4l65cow4/failure?openFailures=WzBd=WzEse31d#top=0
>>
>> There are multiple of those in sequence on beam13
>>
>> Some more comments: https://issues.apache.org/jira/browse/BEAM-5467
>>
>> Any help to further investigate or fix would be appreciated!
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> On Fri, Oct 19, 2018 at 4:51 PM Yifan Zou  wrote:
>>
>>> I got "Failed to restart docker.service: Interactive authentication
>>> required" while trying to restart the docker on beam15.
>>> Does anyone have the permission to do that? Or, we need to ask Apache
>>> Infra for help.
>>>
>>> Thanks.
>>> Yifan
>>>
>>> On Fri, Oct 19, 2018 at 2:51 PM Ankur Goenka  wrote:
>>>
>>>> Hi,
>>>>
>>>> Can we restart docker as it seems to have fixed the issue for others
>>>> https://github.com/moby/moby/issues/31849 ?
>>>>
>>>> Thanks,
>>>> Ankur
>>>>
>>>> On Fri, Oct 19, 2018 at 1:11 PM Yifan Zou  wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The docker has been installed on all Jenkins VMs. The image build
>>>>> process was interrupted by a grpc connection issue.
>>>>>
>>>>> *11:02:12* Starting process 'command 'docker''. Working directory: 
>>>>> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink/src/sdks/python/container/build/docker
>>>>>  Command: docker build --no-cache -t 
>>>>> jenkins-docker-apache.bintray.io/beam/python:latest .*11:02:12* 
>>>>> Successfully started process 'command 'docker''*11:02:12* Sending build 
>>>>> context to Docker daemon  17.65MB
>>>>> *11:02:12* Step 1/9 : FROM python:2-stretch*11:02:12*  ---> 
>>>>> 3c43a5d4034a*11:02:12* Step 2/9 : MAINTAINER "Apache Beam 
>>>>> "*11:02:12*  ---> Running in f86bad9aef9c*11:02:12*  
>>>>> ---> 610a5dec907e*11:02:12* Removing intermediate container 
>>>>> f86bad9aef9c*11:02:12* Step 3/9 : RUN apt-get update && apt-get 
>>>>> install -ylibsnappy-devlibyaml-dev&& rm -rf 
>>>>> /var/lib/apt/lists/**11:02:12*  ---> Running in 5e9b67be03f9*11:02:12* 
>>>>> grpc: the connection is unavailable
>>>>>
>>>>>
>>>>> - Yifan
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 19, 2018 at 12:45 PM Ankur Goenka 
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Flink Validates Runner test cases are failing on Beam 15 because
>>>>>> docker is not installed.
>>>>>> Failing tasks
>>>>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/buildTimeTrend
>>>>>> Can we install docker on all the machines as the Portable Validates
>>>>>> Runner tests need it.
>>>>>>
>>>>>> Thanks,
>>>>>> Ankur
>>>>>>
>>>>>
>
> --
>
>
>
>
> Got feedback? tinyurl.com/swegner-feedback
>

Re: Docker missing on Beam15

[Yifan Zou]

I got "Failed to restart docker.service: Interactive authentication required
" while trying to restart the docker on beam15.
Does anyone have the permission to do that? Or, we need to ask Apache Infra
for help.

Thanks.
Yifan

On Fri, Oct 19, 2018 at 2:51 PM Ankur Goenka  wrote:

> Hi,
>
> Can we restart docker as it seems to have fixed the issue for others
> https://github.com/moby/moby/issues/31849 ?
>
> Thanks,
> Ankur
>
> On Fri, Oct 19, 2018 at 1:11 PM Yifan Zou  wrote:
>
>> Hi,
>>
>> The docker has been installed on all Jenkins VMs. The image build process
>> was interrupted by a grpc connection issue.
>>
>> *11:02:12* Starting process 'command 'docker''. Working directory: 
>> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink/src/sdks/python/container/build/docker
>>  Command: docker build --no-cache -t 
>> jenkins-docker-apache.bintray.io/beam/python:latest .*11:02:12* Successfully 
>> started process 'command 'docker''*11:02:12* Sending build context to Docker 
>> daemon  17.65MB
>> *11:02:12* Step 1/9 : FROM python:2-stretch*11:02:12*  ---> 
>> 3c43a5d4034a*11:02:12* Step 2/9 : MAINTAINER "Apache Beam 
>> "*11:02:12*  ---> Running in f86bad9aef9c*11:02:12*  
>> ---> 610a5dec907e*11:02:12* Removing intermediate container 
>> f86bad9aef9c*11:02:12* Step 3/9 : RUN apt-get update && apt-get install 
>> -ylibsnappy-devlibyaml-dev&& rm -rf 
>> /var/lib/apt/lists/**11:02:12*  ---> Running in 5e9b67be03f9*11:02:12* grpc: 
>> the connection is unavailable
>>
>>
>> - Yifan
>>
>>
>>
>> On Fri, Oct 19, 2018 at 12:45 PM Ankur Goenka  wrote:
>>
>>> Hi,
>>>
>>> Flink Validates Runner test cases are failing on Beam 15 because docker
>>> is not installed.
>>> Failing tasks
>>> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/buildTimeTrend
>>> Can we install docker on all the machines as the Portable Validates
>>> Runner tests need it.
>>>
>>> Thanks,
>>> Ankur
>>>
>>

Re: Docker missing on Beam15

[Yifan Zou]

Hi,

The docker has been installed on all Jenkins VMs. The image build process
was interrupted by a grpc connection issue.

*11:02:12* Starting process 'command 'docker''. Working directory:
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink/src/sdks/python/container/build/docker
Command: docker build --no-cache -t
jenkins-docker-apache.bintray.io/beam/python:latest .*11:02:12*
Successfully started process 'command 'docker''*11:02:12* Sending
build context to Docker daemon  17.65MB
*11:02:12* Step 1/9 : FROM python:2-stretch*11:02:12*  --->
3c43a5d4034a*11:02:12* Step 2/9 : MAINTAINER "Apache Beam
"*11:02:12*  ---> Running in
f86bad9aef9c*11:02:12*  ---> 610a5dec907e*11:02:12* Removing
intermediate container f86bad9aef9c*11:02:12* Step 3/9 : RUN apt-get
update && apt-get install -ylibsnappy-dev
libyaml-dev&& rm -rf /var/lib/apt/lists/**11:02:12*  --->
Running in 5e9b67be03f9*11:02:12* grpc: the connection is unavailable


- Yifan



On Fri, Oct 19, 2018 at 12:45 PM Ankur Goenka  wrote:

> Hi,
>
> Flink Validates Runner test cases are failing on Beam 15 because docker is
> not installed.
> Failing tasks
> https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/buildTimeTrend
> Can we install docker on all the machines as the Portable Validates Runner
> tests need it.
>
> Thanks,
> Ankur
>

Re: Python PostCommit failures

[Yifan Zou]

The integration test has been fixed by #6567
<https://github.com/apache/beam/pull/6567>. The Python PostCommit Verify is
back to normal.
Thanks.

- Yifan

On Thu, Oct 4, 2018 at 12:22 AM Yifan Zou  wrote:

> We are seeing consecutive Python PostCommit
> <https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
> failures due to a broken IT test. I am looking into it (BEAM-5643
> <https://issues.apache.org/jira/browse/BEAM-5643>).
>
> Yifan
>

Python PostCommit failures

[Yifan Zou]

We are seeing consecutive Python PostCommit

failures due to a broken IT test. I am looking into it (BEAM-5643
).

Yifan

Re: Beam Dependency Check Report (2018-10-01)

[Yifan Zou]

The job recovered. The latest dependency report
<https://builds.apache.org/job/beam_Dependency_Check/lastSuccessfulBuild/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html>
was created and shared with dev@.

-Yifan

On Mon, Oct 1, 2018 at 10:50 AM Yifan Zou  wrote:

> The job failed. I am looking into this.
>
> On Mon, Oct 1, 2018 at 10:50 AM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>> Is this report supposed to be empty?
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Mon, Oct 1, 2018 at 5:07 AM Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>>
>>>

Re: Beam Dependency Check Report (2018-10-01)

[Yifan Zou]

The job failed. I am looking into this.

On Mon, Oct 1, 2018 at 10:50 AM Mikhail Gryzykhin  wrote:

> Hi everyone,
>
> Is this report supposed to be empty?
>
> Regards,
> --Mikhail
>
> Have feedback ?
>
>
> On Mon, Oct 1, 2018 at 5:07 AM Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
>
>>

Re: Cleanup Jenkins old jobs

[Yifan Zou]

+1 on this. There are also some jobs for test/experiment purpose which are
stale. Good to remove them that makes the UI neat.


On Mon, Sep 24, 2018 at 2:56 PM Andrew Pilloud  wrote:

> This sounds good to me. We have a significant number of jobs have been
> renamed. Having both the old and new jobs in the UI makes the correct job
> more difficult to find.
>
> Andrew
>
> On Mon, Sep 24, 2018 at 2:52 PM Ankur Goenka  wrote:
>
>> Hi,
>>
>> Jenkins UI has accumulated a lot of old jobs over time which are not in
>> use any more.
>> Shall we clean old jobs (Jobs which did not run in last 7 days) from the
>> jenkins UI for a cleaner view of valid jobs?
>> This is a low risk cleanup as Seed Job will recreate valid jobs if one
>> gets removed.
>>
>>
>> Thanks,
>> Ankur
>>
>

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

[Yifan Zou]

Congratulations Kenn!

On Wed, Sep 19, 2018 at 2:36 PM Robert Burke  wrote:

> Congrats Kenn! :D
>
> On Wed, Sep 19, 2018, 2:21 PM Ismaël Mejía  wrote:
>
>> Congratulations and welcome Kenn as new chair!
>> Thanks Davor for your hard work too.
>>
>> On Wed, Sep 19, 2018 at 11:14 PM Rui Wang  wrote:
>>
>>> Congrats!
>>>
>>> -Rui
>>>
>>> On Wed, Sep 19, 2018 at 2:12 PM Chamikara Jayalath 
>>> wrote:
>>>
 Congrats!

 On Wed, Sep 19, 2018 at 2:05 PM Ahmet Altay  wrote:

> Congratulations, Kenn! And thank you Davor.
>
> On Wed, Sep 19, 2018 at 1:44 PM, Anton Kedin  wrote:
>
>> Congrats!
>>
>> On Wed, Sep 19, 2018 at 1:36 PM Ankur Goenka 
>> wrote:
>>
>>> Congrats Kenn!
>>>
>>> On Wed, Sep 19, 2018 at 1:35 PM Amit Sela 
>>> wrote:
>>>
 Well deserved! Congrats Kenn.

 On Wed, Sep 19, 2018 at 4:25 PM Kai Jiang 
 wrote:

> Congrats, Kenn!
> ᐧ
>
> On Wed, Sep 19, 2018 at 1:23 PM Alan Myrvold 
> wrote:
>
>> Congrats, Kenn.
>>
>> On Wed, Sep 19, 2018 at 1:08 PM Maximilian Michels <
>> m...@apache.org> wrote:
>>
>>> Congrats!
>>>
>>> On 19.09.18 22:07, Robin Qiu wrote:
>>> > Congratulations, Kenn!
>>> >
>>> > On Wed, Sep 19, 2018 at 1:05 PM Lukasz Cwik >> > > wrote:
>>> >
>>> > Congrats Kenn.
>>> >
>>> > On Wed, Sep 19, 2018 at 12:54 PM Davor Bonaci <
>>> da...@apache.org
>>> > > wrote:
>>> >
>>> > Hi everyone --
>>> > It is with great pleasure that I announce that at
>>> today's
>>> > meeting of the Foundation's Board of Directors, the
>>> Board has
>>> > appointed Kenneth Knowles as the second chair of the
>>> Apache Beam
>>> > project.
>>> >
>>> > Kenn has served on the PMC since its inception, and is
>>> very
>>> > active and effective in growing the community. His
>>> exemplary
>>> > posts have been cited in other projects. I'm super
>>> happy to have
>>> > Kenn accepted the nomination, and I'm confident that
>>> he'll serve
>>> > with distinction.
>>> >
>>> > As for myself, I'm not going anywhere. I'm still
>>> around and will
>>> > be as active as I have recently been. Thrilled to be
>>> able to
>>> > pass the baton to such a key member of this community
>>> and to
>>> > have less administrative work to do ;-).
>>> >
>>> > Please join me in welcoming Kenn to his new role, and
>>> I ask that
>>> > you support him as much as possible. As always, please
>>> let me
>>> > know if you have any questions.
>>> >
>>> > Davor
>>> >
>>>
>>
>

Re: [VOTE] Donating the Dataflow Worker code to Apache Beam

[Yifan Zou]

+1

On Fri, Sep 14, 2018 at 4:20 PM David Morávek 
wrote:

> +1
>
>
>
> On 15 Sep 2018, at 00:59, Anton Kedin  wrote:
>
> +1
>
> On Fri, Sep 14, 2018 at 3:22 PM Alan Myrvold  wrote:
>
>> +1
>>
>> On Fri, Sep 14, 2018 at 3:16 PM Boyuan Zhang  wrote:
>>
>>> +1
>>>
>>> On Fri, Sep 14, 2018 at 3:15 PM Henning Rohde 
>>> wrote:
>>>
 +1

 On Fri, Sep 14, 2018 at 2:40 PM Ahmet Altay  wrote:

> +1 (binding)
>
> On Fri, Sep 14, 2018 at 2:35 PM, Lukasz Cwik  wrote:
>
>> +1 (binding)
>>
>> On Fri, Sep 14, 2018 at 2:34 PM Pablo Estrada 
>> wrote:
>>
>>> +1
>>>
>>> On Fri, Sep 14, 2018 at 2:32 PM Andrew Pilloud 
>>> wrote:
>>>
 +1

 On Fri, Sep 14, 2018 at 2:31 PM Lukasz Cwik 
 wrote:

> There was generally positive support and good feedback[1] but it
> was not unanimous. I wanted to bring the donation of the Dataflow 
> worker
> code base to Apache Beam master to a vote.
>
> +1: Support having the Dataflow worker code as part of Apache Beam
> master branch
> -1: Dataflow worker code should live elsewhere
>
> 1:
> https://lists.apache.org/thread.html/89efd3bc1d30f3d43d4b361a5ee05bd52778c9dc3f43ac72354c2bd9@%3Cdev.beam.apache.org%3E
>

>

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

[Yifan Zou]

Thanks all. I am struggling with the missing buildscan reports when running
jobs with containers. I believe it is a big disadvantage to use docker if
the buildscan doesn't show up. I will keep updating my progress in this
thread. In the meanwhile, any comments, suggestions and objections are
still welcome.

Regards.
Yifan

On Tue, Sep 11, 2018 at 6:08 AM Alexey Romanenko 
wrote:

> +1 Great feature that should help with complicated error cases.
>
> On 11 Sep 2018, at 03:39, Henning Rohde  wrote:
>
> +1 Nice proposal. It will help eradicate some of the inflexibility and
> frustrations with Jenkins.
>
> On Wed, Sep 5, 2018 at 2:30 PM Yifan Zou  wrote:
>
>> Thank you all for making comments on this and I apologize for the late
>> reply.
>>
>> To clarify the concerns of testing locally, it is still able to run tests
>> without Docker. One of the purposes of this is to create an identical
>> environment as we are running in Jenkins that would be helpful to reproduce
>> strange errors. Contributors could choose starting a container and run
>> tests in there, or just run tests directly.
>>
>>
>>
>> On Wed, Sep 5, 2018 at 6:37 AM Ismaël Mejía  wrote:
>>
>>> BIG +1, the previous work on having docker build images [1] had a
>>> similar goal (to have a reproducible build environment). But this is
>>> even better because we will guarantee the exact same environment in
>>> Jenkins as well as any further improvements. It is important to
>>> document the setup process as part of this (for future maintenance +
>>> local reproducibility).
>>>
>>> Just for clarification this is independent of running the tests
>>> locally without docker, it is more to improve the reproducibility of
>>> the environment we have on jenkins locally for example to address some
>>> weird Heissenbug.
>>>
>>> I just added BEAM-5311 to track the removal of the docker build images
>>> when this is ready (of course if there are no objections to this
>>> proposal).
>>>
>>> [1] https://beam.apache.org/contribute/docker-images/
>>> On Thu, Aug 30, 2018 at 3:58 PM Jean-Baptiste Onofré 
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > That's interesting, however, it's really important to still be able to
>>> > easily run test locally, without any VM/Docker required. It should be
>>> > activated by profile or so.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 27/08/2018 19:53, Yifan Zou wrote:
>>> > > Hi,
>>> > >
>>> > > I have a proposal for creating a reproducible environment for Jenkins
>>> > > tests by using docker container. The thing is, the environment
>>> > > configurations on Beam Jenkins slaves are sometimes different from
>>> > > developer's machines. Test failures on Jenkins may not be easy to
>>> > > reproduce locally. Also, it is not convenient for developers to add
>>> or
>>> > > modify underlying tools installed on Jenkins VMs, since they're
>>> managed
>>> > > by Apache Infra. This proposal is aimed to address those problems.
>>> > >
>>> > >
>>> https://docs.google.com/document/d/1y0YuQj_oZXC0uM5-gniG7r9-5gv2uiDhzbtgYYJW48c/edit#heading=h.bg2yi0wbhl9n
>>> > >
>>> > > Any comments are welcome. Thank you.
>>> > >
>>> > > Regards.
>>> > > Yifan
>>> > >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbono...@apache.org
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>>
>>
>

Re: [PROPOSAL] Beam jira bot exceptions

[Yifan Zou]

Thanks for raising this problem. There was a discussion about those issues
and we came up with new policies for the Jira bot. At this point, we could
close the JIRAs so that the bot won't create duplicate issues with the same
version.
https://lists.apache.org/list.html?dev@beam.apache.org:lte=1M:%5BDISCUSS%5D%20Versioning%2C%20Hadoop%20related%20dependencies%20and%20enterprise%20users


Regards.
Yifan

On Mon, Sep 10, 2018 at 6:14 AM Jean-Baptiste Onofré 
wrote:

> +1
>
> It would be great to be able to configure "known issues" ;)
>
> Regards
> JB
>
> On 10/09/2018 15:07, Etienne Chauchot wrote:
> > Hi,
> >
> > The bot for dependency checks opens tickets automatically when it sees
> > staled dependencies. But in some cases it is perfectly normal.
> > For example, in elasticsearchIO we have the core module (used by the
> > users) that is compatible with all the versions (for backward
> > compatibility and ease of use). But there is also one test module per
> > supported version (v2, v5, v6) that runs tests against an embedded
> > version of ES. These modules contain ES deps in v2, v5 and v6.
> > In such a case the bot should not open a ticket. Shall we configure it
> > with exceptions ?
> >
> > Note that for now I closed the related tickets.
> >
> > Best
> > Etienne
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

[Yifan Zou]

Thanks all for comments and suggestions. We want to close this thread and
start implementing the new policy based on the discussion:

1. Stop assigning JIRAs to the first person listed in the dependency owners
files <https://github.com/apache/beam/tree/master/ownership>. Instead, cc
people on the owner list.
2. We will be creating JIRAs for upgrading individual dependencies, not for
upgrading to specific versions of those dependencies. For example, if a
given dependency X is three minor versions or an year behind we will create
a JIRA for upgrading that. But the specific version to upgrade to has to be
determined by the Beam community. Beam community might choose to close a
JIRA if there are known issues with available recent releases. Tool will
reopen such a closed JIRA to inform owners if Beam is hitting the 'fixed
version' or 3 new versions of the dependency have been released since JIRA
was closed.

Thank you.

Regards.
Yifan

On Wed, Sep 5, 2018 at 2:14 PM Yifan Zou  wrote:

> +1 on the jira "fix version".
> The release frequency of dependencies are various, so that using new
> information such as versions from the Jira closing date to reopen the
> issues might not be very proper. We could check the fix versions first, and
> if specified, then reopen the issue in that version's release cycle; it
> not, follow Cham's proposal (2).
>
> On Wed, Sep 5, 2018 at 1:59 PM Chamikara Jayalath 
> wrote:
>
>>
>>
>> On Wed, Sep 5, 2018 at 12:50 PM Tim Robertson 
>> wrote:
>>
>>> Thank you Cham, and everyone for contributing
>>>
>>> Sorry for slow reply to a thread I started, but I've been swamped on non
>>> Beam projects.
>>>
>>> KafkaIO's policy of 'let the user decide exact version at runtime' has
>>>> been quite useful so far. How feasible is that for other connectors?
>>>
>>>
>>> I presume shimming might be needed in a few places but it's certainly
>>> something we might want to explore more. I'll look into KafkaIO.
>>>
>>> On Cham's proposal :
>>>
>>> (1) +0.5. We can always then opt to either assign or take ownership of
>>> an issue, although I am also happy to stick with the owners model - it
>>> prompted me to investigate and resulted in this thread.
>>>
>>> (2) I think this makes sense.
>>> A bot informing us that we're falling behind versions is immensely
>>> useful as long as we can link issues to others which might have a wider
>>> discussion (remember many dependencies need to be treated together such as
>>> "Support Hadoop 3.0.x" or "Support HBase 2.x"). Would it make sense to let
>>> owners use the Jira "fix versions" to put in future release to inform the
>>> bot when it should start alerting again?
>>>
>>
>> I think this makes sense. Setting a "fix version" will be specially
>> useful for dependency changes that result in API changes that have to be
>> postponed till next major version of Beam.
>>
>> On grouping, I believe we already group JIRAs into tasks and sub-tasks
>> based on group ids of dependencies. I suppose it will not be too hard to
>> close multiple sub-tasks with the same reasoning.
>>
>>
>>>
>>>
>>> On Wed, Sep 5, 2018 at 3:18 AM Yifan Zou  wrote:
>>>
>>>> Thanks Cham for putting this together. Also, after modifying the
>>>> dependency tool based on the policy above, we will close all existing JIRA
>>>> issues that prevent creating duplicate bugs and stop pushing assignees to
>>>> upgrade dependencies with old bugs.
>>>>
>>>> Please let us know if you have any comments on the revised policy in
>>>> Cham's email.
>>>>
>>>> Thanks all.
>>>>
>>>> Regards.
>>>> Yifan Zou
>>>>
>>>> On Tue, Sep 4, 2018 at 5:35 PM Chamikara Jayalath 
>>>> wrote:
>>>>
>>>>> Based on this email thread and offline feedback from several folks,
>>>>> current concerns regarding dependency upgrade policy and tooling seems to
>>>>> be following.
>>>>>
>>>>> (1) We have to be careful when upgrading dependencies. For example, we
>>>>> should not create JIRAs for upgrading to dependency versions that have
>>>>> known issues.
>>>>>
>>>>> (2) Dependency owners list can get stale. Somebody who is interested
>>>>> in upgrading a dependency today might not be interested in the same task 
>>>>> in
>>>>> six months. R

Re: [Proposal] Creating a reproducible environment for Beam Jenkins Tests

[Yifan Zou]

Thank you all for making comments on this and I apologize for the late
reply.

To clarify the concerns of testing locally, it is still able to run tests
without Docker. One of the purposes of this is to create an identical
environment as we are running in Jenkins that would be helpful to reproduce
strange errors. Contributors could choose starting a container and run
tests in there, or just run tests directly.



On Wed, Sep 5, 2018 at 6:37 AM Ismaël Mejía  wrote:

> BIG +1, the previous work on having docker build images [1] had a
> similar goal (to have a reproducible build environment). But this is
> even better because we will guarantee the exact same environment in
> Jenkins as well as any further improvements. It is important to
> document the setup process as part of this (for future maintenance +
> local reproducibility).
>
> Just for clarification this is independent of running the tests
> locally without docker, it is more to improve the reproducibility of
> the environment we have on jenkins locally for example to address some
> weird Heissenbug.
>
> I just added BEAM-5311 to track the removal of the docker build images
> when this is ready (of course if there are no objections to this
> proposal).
>
> [1] https://beam.apache.org/contribute/docker-images/
> On Thu, Aug 30, 2018 at 3:58 PM Jean-Baptiste Onofré 
> wrote:
> >
> > Hi,
> >
> > That's interesting, however, it's really important to still be able to
> > easily run test locally, without any VM/Docker required. It should be
> > activated by profile or so.
> >
> > Regards
> > JB
> >
> > On 27/08/2018 19:53, Yifan Zou wrote:
> > > Hi,
> > >
> > > I have a proposal for creating a reproducible environment for Jenkins
> > > tests by using docker container. The thing is, the environment
> > > configurations on Beam Jenkins slaves are sometimes different from
> > > developer's machines. Test failures on Jenkins may not be easy to
> > > reproduce locally. Also, it is not convenient for developers to add or
> > > modify underlying tools installed on Jenkins VMs, since they're managed
> > > by Apache Infra. This proposal is aimed to address those problems.
> > >
> > >
> https://docs.google.com/document/d/1y0YuQj_oZXC0uM5-gniG7r9-5gv2uiDhzbtgYYJW48c/edit#heading=h.bg2yi0wbhl9n
> > >
> > > Any comments are welcome. Thank you.
> > >
> > > Regards.
> > > Yifan
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

[Yifan Zou]

+1 on the jira "fix version".
The release frequency of dependencies are various, so that using new
information such as versions from the Jira closing date to reopen the
issues might not be very proper. We could check the fix versions first, and
if specified, then reopen the issue in that version's release cycle; it
not, follow Cham's proposal (2).

On Wed, Sep 5, 2018 at 1:59 PM Chamikara Jayalath 
wrote:

>
>
> On Wed, Sep 5, 2018 at 12:50 PM Tim Robertson 
> wrote:
>
>> Thank you Cham, and everyone for contributing
>>
>> Sorry for slow reply to a thread I started, but I've been swamped on non
>> Beam projects.
>>
>> KafkaIO's policy of 'let the user decide exact version at runtime' has
>>> been quite useful so far. How feasible is that for other connectors?
>>
>>
>> I presume shimming might be needed in a few places but it's certainly
>> something we might want to explore more. I'll look into KafkaIO.
>>
>> On Cham's proposal :
>>
>> (1) +0.5. We can always then opt to either assign or take ownership of an
>> issue, although I am also happy to stick with the owners model - it
>> prompted me to investigate and resulted in this thread.
>>
>> (2) I think this makes sense.
>> A bot informing us that we're falling behind versions is immensely useful
>> as long as we can link issues to others which might have a wider discussion
>> (remember many dependencies need to be treated together such as "Support
>> Hadoop 3.0.x" or "Support HBase 2.x"). Would it make sense to let owners
>> use the Jira "fix versions" to put in future release to inform the bot when
>> it should start alerting again?
>>
>
> I think this makes sense. Setting a "fix version" will be specially useful
> for dependency changes that result in API changes that have to be postponed
> till next major version of Beam.
>
> On grouping, I believe we already group JIRAs into tasks and sub-tasks
> based on group ids of dependencies. I suppose it will not be too hard to
> close multiple sub-tasks with the same reasoning.
>
>
>>
>>
>> On Wed, Sep 5, 2018 at 3:18 AM Yifan Zou  wrote:
>>
>>> Thanks Cham for putting this together. Also, after modifying the
>>> dependency tool based on the policy above, we will close all existing JIRA
>>> issues that prevent creating duplicate bugs and stop pushing assignees to
>>> upgrade dependencies with old bugs.
>>>
>>> Please let us know if you have any comments on the revised policy in
>>> Cham's email.
>>>
>>> Thanks all.
>>>
>>> Regards.
>>> Yifan Zou
>>>
>>> On Tue, Sep 4, 2018 at 5:35 PM Chamikara Jayalath 
>>> wrote:
>>>
>>>> Based on this email thread and offline feedback from several folks,
>>>> current concerns regarding dependency upgrade policy and tooling seems to
>>>> be following.
>>>>
>>>> (1) We have to be careful when upgrading dependencies. For example, we
>>>> should not create JIRAs for upgrading to dependency versions that have
>>>> known issues.
>>>>
>>>> (2) Dependency owners list can get stale. Somebody who is interested in
>>>> upgrading a dependency today might not be interested in the same task in
>>>> six months. Responsibility of upgrading a dependency should lie with the
>>>> community instead of pre-identified owner(s).
>>>>
>>>> On the other hand we do not want Beam to significantly fall behind when
>>>> it comes to dependencies. We should upgrade dependencies whenever it makes
>>>> sense. This allows us to offer a more up to date system and to makes things
>>>> easy for users that deploy Beam along with other systems.
>>>>
>>>> I discussed these issues with Yifan and we would like to suggest
>>>> following changes to current policy and tooling that might help alleviate
>>>> some of the concerns.
>>>>
>>>> (1) Instead of a dependency "owners" list we will be maintaining an
>>>> "interested parties" list. When we create a JIRA for a dependency we will
>>>> not assign it to an owner but rather we will CC all the folks that
>>>> mentioned that they will be interested in receiving updates related to that
>>>> dependency. Hope is that some of the interested parties will also put
>>>> forward the effort to upgrade dependencies they are interested in but the
>>>> responsibility of upgrading dependencies lie with the c

Re: [DISCUSS] Versioning, Hadoop related dependencies and enterprise users

[Yifan Zou]

Thanks Cham for putting this together. Also, after modifying the dependency
tool based on the policy above, we will close all existing JIRA issues that
prevent creating duplicate bugs and stop pushing assignees to upgrade
dependencies with old bugs.

Please let us know if you have any comments on the revised policy in Cham's
email.

Thanks all.

Regards.
Yifan Zou

On Tue, Sep 4, 2018 at 5:35 PM Chamikara Jayalath 
wrote:

> Based on this email thread and offline feedback from several folks,
> current concerns regarding dependency upgrade policy and tooling seems to
> be following.
>
> (1) We have to be careful when upgrading dependencies. For example, we
> should not create JIRAs for upgrading to dependency versions that have
> known issues.
>
> (2) Dependency owners list can get stale. Somebody who is interested in
> upgrading a dependency today might not be interested in the same task in
> six months. Responsibility of upgrading a dependency should lie with the
> community instead of pre-identified owner(s).
>
> On the other hand we do not want Beam to significantly fall behind when it
> comes to dependencies. We should upgrade dependencies whenever it makes
> sense. This allows us to offer a more up to date system and to makes things
> easy for users that deploy Beam along with other systems.
>
> I discussed these issues with Yifan and we would like to suggest following
> changes to current policy and tooling that might help alleviate some of the
> concerns.
>
> (1) Instead of a dependency "owners" list we will be maintaining an
> "interested parties" list. When we create a JIRA for a dependency we will
> not assign it to an owner but rather we will CC all the folks that
> mentioned that they will be interested in receiving updates related to that
> dependency. Hope is that some of the interested parties will also put
> forward the effort to upgrade dependencies they are interested in but the
> responsibility of upgrading dependencies lie with the community as a whole.
>
>  (2) We will be creating JIRAs for upgrading individual dependencies, not
> for upgrading to specific versions of those dependencies. For example, if a
> given dependency X is three minor versions or an year behind we will create
> a JIRA for upgrading that. But the specific version to upgrade to has to be
> determined by the Beam community. Beam community might choose to close a
> JIRA if there are known issues with available recent releases. Tool may
> reopen such a closed JIRA in the future if new information becomes
> available (for example, 3 new versions have been released since JIRA was
> closed).
>
> Thoughts ?
>
> Thanks,
> Cham
>
> On Tue, Aug 28, 2018 at 1:51 PM Chamikara Jayalath 
> wrote:
>
>>
>>
>> On Tue, Aug 28, 2018 at 12:05 PM Thomas Weise  wrote:
>>
>>> I think there is an invalid assumption being made in this discussion,
>>> which is that most projects comply with semantic versioning. The reality in
>>> the open source big data space is unfortunately quite different. Ismaël has
>>> well characterized the situation and HBase isn't an exception. Another
>>> indicator for the scale of problem is extensive amount of shading used in
>>> Beam and other projects. It wouldn't be necessary if semver compliance was
>>> something we can rely on.
>>>
>>> Our recent Flink upgrade broke user(s). And we noticed a backward
>>> incompatible Flink change that affected the portable Flink runner even
>>> between patches.
>>>
>>> Many projects (including Beam) guarantee compatibility only for a subset
>>> of public API. Sometimes a REST API is not covered, sometimes not strictly
>>> internal protocols change and so on, all of which can break users, despite
>>> the public API remaining "compatible". As much as I would love to rely on
>>> the version number to tell me wether an upgrade is safe or not, that's not
>>> practically possible.
>>>
>>> Furthermore, we need to proceed with caution forcing upgrades on users
>>> that host the target systems. To stay with the Flink example, moving Beam
>>> from 1.4 to 1.5 is actually a major change to some, because they now have
>>> to upgrade their Flink clusters/deployments to be able to use the new
>>> version of Beam.
>>>
>>> Upgrades need to be done with caution and may require extensive
>>> verification beyond what our automation provides. I think the Spark change
>>> from 1.x to 2.x and also the JDK 1.8 change were good examples, they
>>> provided the community a window to provide feedback and influence the
>>> change.
>>>

[Proposal] Creating a reproducible environment for Beam Jenkins Tests

[Yifan Zou]

Hi,

I have a proposal for creating a reproducible environment for Jenkins tests
by using docker container. The thing is, the environment configurations on
Beam Jenkins slaves are sometimes different from developer's machines. Test
failures on Jenkins may not be easy to reproduce locally. Also, it is not
convenient for developers to add or modify underlying tools installed on
Jenkins VMs, since they're managed by Apache Infra. This proposal is aimed
to address those problems.

https://docs.google.com/document/d/1y0YuQj_oZXC0uM5-gniG7r9-5gv2uiDhzbtgYYJW48c/edit#heading=h.bg2yi0wbhl9n

Any comments are welcome. Thank you.

Regards.
Yifan

Re: gradlew :rat broken

[Yifan Zou]

I've also encountered that problem before. The nosetests.xml doesn't have
the license specified and you have to removed it before running the :rat
task locally.

On Wed, Aug 15, 2018 at 4:19 PM Pablo Estrada  wrote:

> The files that are causing the failure seem to have been generated. They
> are not part of the checked-in source code:
> https://github.com/apache/beam/tree/master/sdks/python/container
>
> Your files are container/vendor/, and sdks/python/nosetests.xml (which
> is generated when running nosetests).
>
> On Wed, Aug 15, 2018 at 4:11 PM Udi Meiri  wrote:
>
>> Hi,
>> Whenever I run ../../gradlew :rat from the sdks/python directory I get
>> errors about unapproved licenses. Example:
>> https://scans.gradle.com/s/gdpcgbexwyrpm
>>
>> This didn't use to be the case a few weeks back.
>> Any ideas what could be causing this?
>>
> --
> Got feedback? go/pabloem-feedback
> 
>

Re: [VOTE] Community Examples Repository

[Yifan Zou]

3 for all reasons above.
Keeping the examples in the Beam repository is a straightforward way to let
the examples being visible and maintainable. And as Lukasz Cwik mentioned,
the Quickstart makes a clear and easy approach for users to generate the
archetype and run examples.
I did not see enough benefits to make tons of efforts splitting an example
repository. It make sense to keep consistent.

- Yifan

On Mon, Aug 13, 2018 at 10:21 AM Andrew Pilloud  wrote:

> +1 for 2, it would be nice to have a example project in it's own GitHub
> repo. Might not need to be an "official" repo thought. Could we provide
> links to community supplied examples?
>
> On Thu, Aug 9, 2018, 2:30 PM Robert Bradshaw  wrote:
>
>> (3)
>>
>> In particular, I see a lot of value for (quoting the proposal)
>>
>> """
>> Since then, there have been
>> numerous updates, increased Python parity, and new features that do
>> not have accompanying examples employing best practices and
>> demonstrating an end-to-end experience for new users. We would like to
>> leverage the existing examples by raising their visibility and
>> auditing them.
>> """
>>
>> and I think the situation would become *worse* on all these fronts
>> with a separate repo (as well as the other issues mentioned,
>> especially complexity). We should consider lowering the bar to liking
>> to user-maintained examples that don't merit being in the main repo,
>> as well as guidelines for adding examples in the main repo itself.
>> On Thu, Aug 9, 2018 at 1:44 PM Ismaël Mejía  wrote:
>> >
>> > 3 for all the reasons discussed above. I think there are better ways to
>> improve the status quo without the extra maintenance of having a new repo
>> for this.
>> >
>> > On Thu, Aug 9, 2018 at 7:00 PM Ahmet Altay  wrote:
>> >>
>> >> If we go forward with (3), could we actually update our documentation
>> on how we will support casual example contributions? I think we will need
>> to have information on how to add links to the new examples people want to
>> add to the set, what examples would be good additions to the Beam repo and
>> what examples would be better maintained somewhere else by their owners,
>> and what could they expect from our community when they work on such
>> examples.
>> >>
>> >> On Thu, Aug 9, 2018 at 9:41 AM, Mikhail Gryzykhin 
>> wrote:
>> >>>
>> >>> 3 (if contributors are up for voting) - We want to have beam
>> maintained examples in main repo. This will give good man to users and
>> allow us to test those easily with minimal maintenance.
>> >>>
>> >>> We can add links to opensource user repositories to our
>> documentation/wiki. This will be flexible enough to provide external
>> examples on one hand, and avoid responsibility of maintaining user code on
>> the other hand.
>> >>>
>> >>> --Mikhail
>> >>>
>> >>> Have feedback?
>> >>>
>> >>>
>> >>> On Thu, Aug 9, 2018 at 8:57 AM Rafael Fernandez 
>> wrote:
>> 
>>  Here is the Rose', David's, and Gris' proposal in text form, I hope
>>  the copy/paste helps:
>> 
>> 
>>  Apache Beam Examples Repository
>> 
>>  Authors: Rose Nguyen (rtngu...@google.com), David Cavazos
>>  (dcava...@google.com), Gris Cuevas (g...@apache.org)
>> 
>>  Status: Proposal
>>  Created: 2018-07-30
>>  Updated: 2018-07-30
>> 
>>  Summary
>> 
>>  The Apache Beam Community creates and contributes examples to the
>> core
>>  Apache Beam Github repository. We want to make the process easier and
>>  less dependent in the core repository by creating a separate repo,
>>  dedicated solely to Community examples, contribution guidelines and
>>  add the examples to the website.
>> 
>>  Background
>> 
>>  The original batch of examples on the Apache Beam GitHub repository
>>  was donated by Cloud Dataflow at the time of Java SDK 1.x to
>>  demonstrate the capability of this programming model. These initial
>>  examples were intended to demonstrate how a user can put together
>>  their code components and try out Beam. Since then, there have been
>>  numerous updates, increased Python parity, and new features that do
>>  not have accompanying examples employing best practices and
>>  demonstrating an end-to-end experience for new users. We would like
>> to
>>  leverage the existing examples by raising their visibility and
>>  auditing them. This is also an opportunity to establish
>>  contribution/maintenance guidelines for community contributions and
>> to
>>  start hosting the examples on the Beam site in an official
>> repository.
>>  Attracting and retaining new users necessitates updated, concrete
>>  examples that exhibit the range of capabilities of Beam.
>> 
>>  Proposed Tasks
>> 
>>  We would like to create a new GitHub Repository under the Apache
>>  Software Foundation Org page for Apache Beam Community Examples. This
>>  repo would be similar to apache/beam-site. The name 

Maintaining Beam Dependencies

[Yifan Zou]

Hi all,

As from today, we start filing JIRA issues automatically for tracking Beam
dependency updates based on the weekly report
<https://builds.apache.org/job/beam_Dependency_Check/lastSuccessfulBuild/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html>.
Issues will be created by the Beam JIRA Bot for each one of the dependency
listed in the report and assign to the owners. And the Java dep issues will
be grouped by their groupID that allows contributors easily see the upgrade
process of related packages. A sample issue is here: BEAM-4963
<https://issues.apache.org/jira/browse/BEAM-4963>

Keeping dependencies healthy is very important to make the perfect user
experience and let the development process being smooth. And maintaining on
our dependencies will be a continuous, incremental improvements that
requires the whole community to make efforts. So, please:

1. Taking care of the JIRA dependency upgrade requests assigned to you. If
the version is not good to upgrade for some reasons, you could close the
issue with 'won't fix', this will prevent the jira bot creating duplicate
issues.

2. If you are familiar with some packages which Beam depends on, you are
encouraged to take the ownership and help the community to look after
them. You can add your JIRA username in the dependency ownership files:
https://github.com/apache/beam/tree/master/ownership

3. Please also update the ownership files when introducing new dependencies
into Beam.

Please refer to the Dependencies Guide
<https://beam.apache.org/contribute/dependencies/> for more details.

Thank you very much.

Regards.
Yifan Zou