Re: Cloudbees Operations Center and Client Masters upgrade this weekend.

[Joan Touzet]

On 2020-04-05 2:22 p.m., Paul Davis wrote:

Howdy Gavin,

I've updated everything besides the two FreeBSD nodes that Joan
manages. Is there a trick to having Jenkins realize a node has
upgraded its slave.jar? Took me an extra twenty minutes fumbling
around before realizing that slave.jar is now actually agent.jar and
that even though Jenkins realized the jar had actually been updated. I
ended up realizing that I had to go through each node in the web UI to
re-enable each worker.

Joan, for the FreeBSD ones I'm not sure how you set up the run
scripts. For me it was just a `sudo sv restart jenkins` on each node
because I download the agent.jar as part of `/etc/sv/jenkins/run`.
Then it was just a random Googling till I happened to notice that was
enough to upgrade the jar and needed to click the button.


Yup, that was it here too, though I had to sv stop && killall java 
because it didn't seem to be stopping right.


Sorry for the delay, I've been ill this weekend and only got to this now.

-Joan



Paul

On Sun, Apr 5, 2020 at 11:32 AM Gavin McDonald  wrote:


Hi All,

This is now completed.

Overall time spent was around 3 hours on the Operations Center and 5
masters.

When the OC was being upgraded, there was no downtime for the masters.

Each master had downtime of around 20 minutes to 30 minutes, including
multiple restarts
, backups and plugin upgrades.

CouchDB folks - most of your nodes are connected via 'node connects to the
master' meaning that you'll
have to update each slave.jar manually. The preferred method is ssh i/o
non-blocking, all nodes connected
in this fashion all came back on their own without intervention.

Thanks all

Gavin McDonald (ASF Infra)


On Sat, Apr 4, 2020 at 3:20 PM Gavin McDonald  wrote:


Hi All,

I'll be upgrading the Cloudbees Core Operations Center and all attached
Masters on Sunday 5th April at around 10am UTC until completed.

I'll turn off new builds around 7am UTC.

I'm 'expecting' a couple of hours downtime assuming all goes well,
including upgrading all
associated plugins, performing backups etc. But, this is the first time
I'm upgrading this setup so there may be the odd gotcha that surprises me
who knows! - I'm as prepared as possible and I am not expecting any
problems.

Thanks

Gavin McDonald (ASF Infra)

Re: Cloudbees Operations Center and Client Masters upgrade this weekend.

[Paul Davis]

Howdy Gavin,

I've updated everything besides the two FreeBSD nodes that Joan
manages. Is there a trick to having Jenkins realize a node has
upgraded its slave.jar? Took me an extra twenty minutes fumbling
around before realizing that slave.jar is now actually agent.jar and
that even though Jenkins realized the jar had actually been updated. I
ended up realizing that I had to go through each node in the web UI to
re-enable each worker.

Joan, for the FreeBSD ones I'm not sure how you set up the run
scripts. For me it was just a `sudo sv restart jenkins` on each node
because I download the agent.jar as part of `/etc/sv/jenkins/run`.
Then it was just a random Googling till I happened to notice that was
enough to upgrade the jar and needed to click the button.

Paul

On Sun, Apr 5, 2020 at 11:32 AM Gavin McDonald  wrote:
>
> Hi All,
>
> This is now completed.
>
> Overall time spent was around 3 hours on the Operations Center and 5
> masters.
>
> When the OC was being upgraded, there was no downtime for the masters.
>
> Each master had downtime of around 20 minutes to 30 minutes, including
> multiple restarts
> , backups and plugin upgrades.
>
> CouchDB folks - most of your nodes are connected via 'node connects to the
> master' meaning that you'll
> have to update each slave.jar manually. The preferred method is ssh i/o
> non-blocking, all nodes connected
> in this fashion all came back on their own without intervention.
>
> Thanks all
>
> Gavin McDonald (ASF Infra)
>
>
> On Sat, Apr 4, 2020 at 3:20 PM Gavin McDonald  wrote:
>
> > Hi All,
> >
> > I'll be upgrading the Cloudbees Core Operations Center and all attached
> > Masters on Sunday 5th April at around 10am UTC until completed.
> >
> > I'll turn off new builds around 7am UTC.
> >
> > I'm 'expecting' a couple of hours downtime assuming all goes well,
> > including upgrading all
> > associated plugins, performing backups etc. But, this is the first time
> > I'm upgrading this setup so there may be the odd gotcha that surprises me
> > who knows! - I'm as prepared as possible and I am not expecting any
> > problems.
> >
> > Thanks
> >
> > Gavin McDonald (ASF Infra)
> >
> >

Re: Cloudbees Operations Center and Client Masters upgrade this weekend.

[Chris Lambertus]

HI folks, we’re doing one more restart to fix a URL redirection error. 
Everything will be back up in about 20 minutes.



> On Apr 5, 2020, at 9:32 AM, Gavin McDonald  wrote:
> 
> Hi All,
> 
> This is now completed.
> 
> Overall time spent was around 3 hours on the Operations Center and 5 masters.
> 
> When the OC was being upgraded, there was no downtime for the masters.
> 
> Each master had downtime of around 20 minutes to 30 minutes, including 
> multiple restarts 
> , backups and plugin upgrades.
> 
> CouchDB folks - most of your nodes are connected via 'node connects to the 
> master' meaning that you'll 
> have to update each slave.jar manually. The preferred method is ssh i/o 
> non-blocking, all nodes connected 
> in this fashion all came back on their own without intervention.
> 
> Thanks all
> 
> Gavin McDonald (ASF Infra)
> 
> 
> On Sat, Apr 4, 2020 at 3:20 PM Gavin McDonald  > wrote:
> Hi All,
> 
> I'll be upgrading the Cloudbees Core Operations Center and all attached 
> Masters on Sunday 5th April at around 10am UTC until completed.
> 
> I'll turn off new builds around 7am UTC.
> 
> I'm 'expecting' a couple of hours downtime assuming all goes well, including 
> upgrading all 
> associated plugins, performing backups etc. But, this is the first time I'm 
> upgrading this setup so there may be the odd gotcha that surprises me who 
> knows! - I'm as prepared as possible and I am not expecting any problems.
> 
> Thanks
> 
> Gavin McDonald (ASF Infra)
> 

Re: Cloudbees Operations Center and Client Masters upgrade this weekend.

[Gavin McDonald]

Hi All,

This is now completed.

Overall time spent was around 3 hours on the Operations Center and 5
masters.

When the OC was being upgraded, there was no downtime for the masters.

Each master had downtime of around 20 minutes to 30 minutes, including
multiple restarts
, backups and plugin upgrades.

CouchDB folks - most of your nodes are connected via 'node connects to the
master' meaning that you'll
have to update each slave.jar manually. The preferred method is ssh i/o
non-blocking, all nodes connected
in this fashion all came back on their own without intervention.

Thanks all

Gavin McDonald (ASF Infra)


On Sat, Apr 4, 2020 at 3:20 PM Gavin McDonald  wrote:

> Hi All,
>
> I'll be upgrading the Cloudbees Core Operations Center and all attached
> Masters on Sunday 5th April at around 10am UTC until completed.
>
> I'll turn off new builds around 7am UTC.
>
> I'm 'expecting' a couple of hours downtime assuming all goes well,
> including upgrading all
> associated plugins, performing backups etc. But, this is the first time
> I'm upgrading this setup so there may be the odd gotcha that surprises me
> who knows! - I'm as prepared as possible and I am not expecting any
> problems.
>
> Thanks
>
> Gavin McDonald (ASF Infra)
>
>