Re: Updates on Beam Jenkins

[Alan Myrvold]

Thanks for this work, Yifan!

On Mon, Apr 29, 2019 at 8:14 AM Ismaël Mejía  wrote:

> Thanks Yifan for all your work. Sometimes the work on infrastructure
> is hidden, so it is great to acknowledge the importance of the
> improvements you and the others have done.
>
> On Mon, Apr 29, 2019 at 5:11 PM Lukasz Cwik  wrote:
> >
> > Thanks Yifan for driving this.
> >
> > On Mon, Apr 29, 2019 at 8:01 AM Yifan Zou  wrote:
> >>
> >> Hi all,
> >>
> >>
> >> We now fully switched the Jenkins to new agents. The old agents are
> deprecated and VMs will be deleted shortly to make more CPU available in
> the us-central1 for tests. Please let me know if you see anything abnormal
> on the Jenkins.
> >>
> >>
> >> Thanks to everyone who helped and contributed to making this migration
> success!
> >>
> >>
> >> Yifan Zou
> >>
> >>
> >>
> >> On Wed, Apr 10, 2019 at 9:55 AM Valentyn Tymofieiev <
> valen...@google.com> wrote:
> >>>
> >>> Thanks a lot Yifan! This work will likely be a great improvement in
> Beam developer velocity. INFRA team seems to be overbooked, and things like
> installing additional dependency took them significant time to be resolved
> in the past. By having more control and context over what needs to be
> installed on Jenkins workers, we should be able to move faster.
> >>>
> >>> On Tue, Apr 9, 2019 at 11:44 PM Connell O'Callaghan <
> conne...@google.com> wrote:
> 
>  Yifan thank you for this update and progressing this replacement!!!
> 
>  On Tue, Apr 9, 2019, 11:14 PM Yifan Zou  wrote:
> >
> > Thanks, Pablo. The new workers use custom image in the boot disk and
> it's easy to reboot without re-imaging. Means we will no longer need
> assistance from Infra to reconnect an offline node. Dockerizing the
> environment would be helpful to make changes on the environment, such as
> installing/updating a beam-required package.
> >
> >
> > On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada 
> wrote:
> >>
> >> Thanks for the updates Yifan. I am sure this process has been
> difficult, and I appreciate the good communication, and that this didn't
> really affect the workflow of anyone to validate the new setup for nodes.
> >>
> >> I imagine that once we move to dockerizing the testing environment,
> it will be much simpler to restart machines that are having trouble?
> >> Thanks again!
> >> -P.
> >>
> >> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou 
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I have some good news about our Jenkins nodes. We're now having 7
> new nodes online named as "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]",
> which substitute the old broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This
> helps to reduce the job waiting queue that let your tests getting executed
> quickly. They're starting up and accepting jobs. There is no action needed
> on your end.
> >>>
> >>> I'll keep the remaining old agents running for one more week just
> in case it won't fully block the development works if any unexpected issues
> on the new agents. Once the new agents are stable and verified, I'll move
> forward to connect the rest agents and remove old set. The playbook is also
> on the way.
> >>>
> >>> For more background and information of the Jenkins updates, please
> see:
> >>> One pager:
> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
> >>> Environment verification sheet:
> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
> >>> Previous Thread on dev@:
> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
> >>>
> >>> Thanks.
> >>>
> >>> Regards.
> >>> Yifan Zou
> >>>
> >>>
>

Re: Updates on Beam Jenkins

[Ismaël Mejía]

Thanks Yifan for all your work. Sometimes the work on infrastructure
is hidden, so it is great to acknowledge the importance of the
improvements you and the others have done.

On Mon, Apr 29, 2019 at 5:11 PM Lukasz Cwik  wrote:
>
> Thanks Yifan for driving this.
>
> On Mon, Apr 29, 2019 at 8:01 AM Yifan Zou  wrote:
>>
>> Hi all,
>>
>>
>> We now fully switched the Jenkins to new agents. The old agents are 
>> deprecated and VMs will be deleted shortly to make more CPU available in the 
>> us-central1 for tests. Please let me know if you see anything abnormal on 
>> the Jenkins.
>>
>>
>> Thanks to everyone who helped and contributed to making this migration 
>> success!
>>
>>
>> Yifan Zou
>>
>>
>>
>> On Wed, Apr 10, 2019 at 9:55 AM Valentyn Tymofieiev  
>> wrote:
>>>
>>> Thanks a lot Yifan! This work will likely be a great improvement in Beam 
>>> developer velocity. INFRA team seems to be overbooked, and things like 
>>> installing additional dependency took them significant time to be resolved 
>>> in the past. By having more control and context over what needs to be 
>>> installed on Jenkins workers, we should be able to move faster.
>>>
>>> On Tue, Apr 9, 2019 at 11:44 PM Connell O'Callaghan  
>>> wrote:

 Yifan thank you for this update and progressing this replacement!!!

 On Tue, Apr 9, 2019, 11:14 PM Yifan Zou  wrote:
>
> Thanks, Pablo. The new workers use custom image in the boot disk and it's 
> easy to reboot without re-imaging. Means we will no longer need 
> assistance from Infra to reconnect an offline node. Dockerizing the 
> environment would be helpful to make changes on the environment, such as 
> installing/updating a beam-required package.
>
>
> On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada  wrote:
>>
>> Thanks for the updates Yifan. I am sure this process has been difficult, 
>> and I appreciate the good communication, and that this didn't really 
>> affect the workflow of anyone to validate the new setup for nodes.
>>
>> I imagine that once we move to dockerizing the testing environment, it 
>> will be much simpler to restart machines that are having trouble?
>> Thanks again!
>> -P.
>>
>> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>>>
>>> Hello,
>>>
>>> I have some good news about our Jenkins nodes. We're now having 7 new 
>>> nodes online named as "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", 
>>> which substitute the old broken agents "beam[1, 2, 4, 7, 8, 9, 12]". 
>>> This helps to reduce the job waiting queue that let your tests getting 
>>> executed quickly. They're starting up and accepting jobs. There is no 
>>> action needed on your end.
>>>
>>> I'll keep the remaining old agents running for one more week just in 
>>> case it won't fully block the development works if any unexpected 
>>> issues on the new agents. Once the new agents are stable and verified, 
>>> I'll move forward to connect the rest agents and remove old set. The 
>>> playbook is also on the way.
>>>
>>> For more background and information of the Jenkins updates, please see:
>>> One pager: 
>>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>>> Environment verification sheet: 
>>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>>> Previous Thread on dev@: 
>>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>>
>>> Thanks.
>>>
>>> Regards.
>>> Yifan Zou
>>>
>>>

Re: Updates on Beam Jenkins

[Lukasz Cwik]

Thanks Yifan for driving this.

On Mon, Apr 29, 2019 at 8:01 AM Yifan Zou  wrote:

> Hi all,
>
>
> We now fully switched the Jenkins to new agents. The old agents are
> deprecated and VMs will be deleted shortly to make more CPU available in
> the us-central1 for tests. Please let me know if you see anything abnormal
> on the Jenkins.
>
>
> Thanks to everyone who helped and contributed to making this migration
> success!
>
>
> Yifan Zou
>
>
> On Wed, Apr 10, 2019 at 9:55 AM Valentyn Tymofieiev 
> wrote:
>
>> Thanks a lot Yifan! This work will likely be a great improvement in Beam
>> developer velocity. INFRA team seems to be overbooked, and things like
>> installing additional dependency took them significant time to be resolved
>> in the past. By having more control and context over what needs to be
>> installed on Jenkins workers, we should be able to move faster.
>>
>> On Tue, Apr 9, 2019 at 11:44 PM Connell O'Callaghan 
>> wrote:
>>
>>> Yifan thank you for this update and progressing this replacement!!!
>>>
>>> On Tue, Apr 9, 2019, 11:14 PM Yifan Zou  wrote:
>>>
 Thanks, Pablo. The new workers use custom image in the boot disk and
 it's easy to reboot without re-imaging. Means we will no longer need
 assistance from Infra to reconnect an offline node. Dockerizing the
 environment would be helpful to make changes on the environment, such as
 installing/updating a beam-required package.


 On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada 
 wrote:

> Thanks for the updates Yifan. I am sure this process has been
> difficult, and I appreciate the good communication, and that this didn't
> really affect the workflow of anyone to validate the new setup for nodes.
>
> I imagine that once we move to dockerizing the testing environment, it
> will be much simpler to restart machines that are having trouble?
> Thanks again!
> -P.
>
> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>
>> Hello,
>>
>> I have some good news about our Jenkins nodes. We're now having 7 new
>> nodes  online named as
>> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
>> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
>> waiting queue that let your tests getting executed quickly. They're
>> starting up and accepting jobs. There is no action needed on your end.
>>
>> I'll keep the remaining old agents running for one more week just in
>> case it won't fully block the development works if any unexpected issues 
>> on
>> the new agents. Once the new agents are stable and verified, I'll move
>> forward to connect the rest agents and remove old set. The playbook is 
>> also
>> on the way.
>>
>> For more background and information of the Jenkins updates, please
>> see:
>> One pager:
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>> Environment verification sheet:
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>> Previous Thread on dev@:
>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>
>> Thanks.
>>
>> Regards.
>> Yifan Zou
>>
>>
>>

Re: Updates on Beam Jenkins

[Connell O'Callaghan]

Yifan thank you for this update and progressing this replacement!!!

On Tue, Apr 9, 2019, 11:14 PM Yifan Zou  wrote:

> Thanks, Pablo. The new workers use custom image in the boot disk and it's
> easy to reboot without re-imaging. Means we will no longer need assistance
> from Infra to reconnect an offline node. Dockerizing the environment would
> be helpful to make changes on the environment, such as installing/updating
> a beam-required package.
>
>
> On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada  wrote:
>
>> Thanks for the updates Yifan. I am sure this process has been difficult,
>> and I appreciate the good communication, and that this didn't really affect
>> the workflow of anyone to validate the new setup for nodes.
>>
>> I imagine that once we move to dockerizing the testing environment, it
>> will be much simpler to restart machines that are having trouble?
>> Thanks again!
>> -P.
>>
>> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>>
>>> Hello,
>>>
>>> I have some good news about our Jenkins nodes. We're now having 7 new
>>> nodes  online named as
>>> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
>>> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
>>> waiting queue that let your tests getting executed quickly. They're
>>> starting up and accepting jobs. There is no action needed on your end.
>>>
>>> I'll keep the remaining old agents running for one more week just in
>>> case it won't fully block the development works if any unexpected issues on
>>> the new agents. Once the new agents are stable and verified, I'll move
>>> forward to connect the rest agents and remove old set. The playbook is also
>>> on the way.
>>>
>>> For more background and information of the Jenkins updates, please see:
>>> One pager:
>>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>>> Environment verification sheet:
>>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>>> Previous Thread on dev@:
>>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>>
>>> Thanks.
>>>
>>> Regards.
>>> Yifan Zou
>>>
>>>
>>>

Re: Updates on Beam Jenkins

[Yifan Zou]

Thanks, Pablo. The new workers use custom image in the boot disk and it's
easy to reboot without re-imaging. Means we will no longer need assistance
from Infra to reconnect an offline node. Dockerizing the environment would
be helpful to make changes on the environment, such as installing/updating
a beam-required package.


On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada  wrote:

> Thanks for the updates Yifan. I am sure this process has been difficult,
> and I appreciate the good communication, and that this didn't really affect
> the workflow of anyone to validate the new setup for nodes.
>
> I imagine that once we move to dockerizing the testing environment, it
> will be much simpler to restart machines that are having trouble?
> Thanks again!
> -P.
>
> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>
>> Hello,
>>
>> I have some good news about our Jenkins nodes. We're now having 7 new
>> nodes  online named as
>> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
>> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
>> waiting queue that let your tests getting executed quickly. They're
>> starting up and accepting jobs. There is no action needed on your end.
>>
>> I'll keep the remaining old agents running for one more week just in case
>> it won't fully block the development works if any unexpected issues on the
>> new agents. Once the new agents are stable and verified, I'll move forward
>> to connect the rest agents and remove old set. The playbook is also on the
>> way.
>>
>> For more background and information of the Jenkins updates, please see:
>> One pager:
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>> Environment verification sheet:
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>> Previous Thread on dev@:
>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>
>> Thanks.
>>
>> Regards.
>> Yifan Zou
>>
>>
>>

Re: Updates on Beam Jenkins

[Lukasz Cwik]

This is great stuff, my last couple of PRs before this had several > 2hr
waits before the tests started up.

On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada  wrote:

> Thanks for the updates Yifan. I am sure this process has been difficult,
> and I appreciate the good communication, and that this didn't really affect
> the workflow of anyone to validate the new setup for nodes.
>
> I imagine that once we move to dockerizing the testing environment, it
> will be much simpler to restart machines that are having trouble?
> Thanks again!
> -P.
>
> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>
>> Hello,
>>
>> I have some good news about our Jenkins nodes. We're now having 7 new
>> nodes  online named as
>> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
>> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
>> waiting queue that let your tests getting executed quickly. They're
>> starting up and accepting jobs. There is no action needed on your end.
>>
>> I'll keep the remaining old agents running for one more week just in case
>> it won't fully block the development works if any unexpected issues on the
>> new agents. Once the new agents are stable and verified, I'll move forward
>> to connect the rest agents and remove old set. The playbook is also on the
>> way.
>>
>> For more background and information of the Jenkins updates, please see:
>> One pager:
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>> Environment verification sheet:
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>> Previous Thread on dev@:
>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>
>> Thanks.
>>
>> Regards.
>> Yifan Zou
>>
>>
>>

Re: Updates on Beam Jenkins

[Kenneth Knowles]

Yes, thanks Yifan! This is critical infrastructure that was in real trouble
without your work.

Kenn

On Tue, Apr 9, 2019 at 2:39 PM Pablo Estrada  wrote:

> Thanks for the updates Yifan. I am sure this process has been difficult,
> and I appreciate the good communication, and that this didn't really affect
> the workflow of anyone to validate the new setup for nodes.
>
> I imagine that once we move to dockerizing the testing environment, it
> will be much simpler to restart machines that are having trouble?
> Thanks again!
> -P.
>
> On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:
>
>> Hello,
>>
>> I have some good news about our Jenkins nodes. We're now having 7 new
>> nodes  online named as
>> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
>> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
>> waiting queue that let your tests getting executed quickly. They're
>> starting up and accepting jobs. There is no action needed on your end.
>>
>> I'll keep the remaining old agents running for one more week just in case
>> it won't fully block the development works if any unexpected issues on the
>> new agents. Once the new agents are stable and verified, I'll move forward
>> to connect the rest agents and remove old set. The playbook is also on the
>> way.
>>
>> For more background and information of the Jenkins updates, please see:
>> One pager:
>> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
>> Environment verification sheet:
>> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
>> Previous Thread on dev@:
>> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>>
>> Thanks.
>>
>> Regards.
>> Yifan Zou
>>
>>
>>

Re: Updates on Beam Jenkins

[Pablo Estrada]

Thanks for the updates Yifan. I am sure this process has been difficult,
and I appreciate the good communication, and that this didn't really affect
the workflow of anyone to validate the new setup for nodes.

I imagine that once we move to dockerizing the testing environment, it will
be much simpler to restart machines that are having trouble?
Thanks again!
-P.

On Tue, Apr 9, 2019 at 2:23 PM Yifan Zou  wrote:

> Hello,
>
> I have some good news about our Jenkins nodes. We're now having 7 new
> nodes  online named as
> "apache-beam-jenkins-[1, 2, 4, 7, 8, 9, 12]", which substitute the old
> broken agents "beam[1, 2, 4, 7, 8, 9, 12]". This helps to reduce the job
> waiting queue that let your tests getting executed quickly. They're
> starting up and accepting jobs. There is no action needed on your end.
>
> I'll keep the remaining old agents running for one more week just in case
> it won't fully block the development works if any unexpected issues on the
> new agents. Once the new agents are stable and verified, I'll move forward
> to connect the rest agents and remove old set. The playbook is also on the
> way.
>
> For more background and information of the Jenkins updates, please see:
> One pager:
> https://docs.google.com/document/d/1c38IPrF94PZC-ItGZgmAgAKrgmC1MGA6N6nkK0cL6L4/edit?ts=5ca54b3e#
> Environment verification sheet:
> https://docs.google.com/spreadsheets/d/1MDL6vy_0iaFSZeWQ-4JWKlRiZ5WFdDVjJh6Xvczgld0/edit#gid=0
> Previous Thread on dev@:
> https://lists.apache.org/thread.html/7b9863b241b37484f321d8812e2ad10d8f054ec720aec4b98efe0446@%3Cdev.beam.apache.org%3E
>
> Thanks.
>
> Regards.
> Yifan Zou
>
>
>