Re: [openstack-dev] [tripleo] what's up in upgrades?
Hi all, Since we decided to switch on OVB provisioner for the CI, the upgrade jobs is now working really better and consistently. Unfortunately, I can answer now if we will reach the timeout with those jobs because the current state of the upgrade (mitaka to newton) is currently broken due to a number of issues which we are working on. I'm making some cleaning in the (huge) review (https://review.openstack.org/#/c/323750) atm, but if all of you are ok, it would be great to merge this work, even if the upgrade (full overcloud) is not complete successfully. It would allow us, at least, to trigger the experimental pipeline and see what's going on with the upgrade. ( and example of the state of the full upgrade job: http://logs.openstack.org/50/323750/54/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-upgrades-nv/45916e6/console.html) On 09/09/2016 09:29 AM, mathieu bultel wrote: > On 09/09/2016 12:42 AM, Emilien Macchi wrote: >> On Thu, Sep 8, 2016 at 4:18 PM, David Moreau Simard wrote: >>> How long does the upgrade job take ? >> undercloud upgrade is a matter of 10 min max. >> overcloud upgrade will be much more, I don't have metrics right now. >> matbu maybe? > It really depend on the infra which run the upgrade. I don't know much > about the upstream infra but > on my local box, with a SSD, 8 cores and 32Go of RAM, It could take > around 1h30 2hours for a full upgrade. > On centos ci infra and with RDO, some jobs can takes 4hours or so ... > > I'm really curious to see how long a full upgrade will take with upstream. > > Right now, the full upgrade job didn't go far from the controller > upgrade (step 2). > AFAIK, the timeout in upstream is 3 hours minus 10 minutes ... > I think if we keep a 2 nodes deployment with only pacemaker, it would be > enough... I will keep you in touch of my progress here.. > > But, even if the jobs takes 2 or 3 hours to vote, I think it would be a > real huge gain for the tripleo work. >>> David Moreau Simard >>> Senior Software Engineer | Openstack RDO >>> >>> dmsimard = [irc, github, twitter] >>> >>> >>> On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi wrote: What's up in upgrades? 1) Undercloud: Mitaka to Newton We just approved a patch in openstack-infra/tripleo-ci that test undercloud upgrades. I don't think we should make it vote as for now it's quite experimental. Though I'm wondering if we should move it to the check pipeline as non-voting (currently in experimental queue). This is a first iteration and if you plan to upgrade your undercloud, you'll still have to do manual steps that we do in tripleo-ci. They are described here: https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 We need to decide where to put these bits: in tripleoclient? in instack-undercloud? Let's talk about it. 2) Overcloud: Mitaka to Newton matbu and myself are working together on a CI job that will test the upgrade of an undercloud + overcloud from Mitaka to Newton. Work is here: https://review.openstack.org/#/c/364859 and https://review.openstack.org/#/c/323750/ (both patches are going to merge together so we have one single patch for review). The idea is to use multinode job for now as a first iteration, with the simplest scenario possible so we can easily iterate later. 3) Overcloud: Newton to Newton I'm working on a simple patch that would test updates from Newton to Newton: https://review.openstack.org/#/c/351330/ like we had with OVB jobs but this time using multinode. Any feedback, help, is highly welcome. -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] what's up in upgrades?
On Fri, Sep 09, 2016 at 09:29:46AM +0200, mathieu bultel wrote: > On 09/09/2016 12:42 AM, Emilien Macchi wrote: > > On Thu, Sep 8, 2016 at 4:18 PM, David Moreau Simard wrote: > >> How long does the upgrade job take ? > > undercloud upgrade is a matter of 10 min max. > > overcloud upgrade will be much more, I don't have metrics right now. > > matbu maybe? > It really depend on the infra which run the upgrade. I don't know much > about the upstream infra but > on my local box, with a SSD, 8 cores and 32Go of RAM, It could take > around 1h30 2hours for a full upgrade. > On centos ci infra and with RDO, some jobs can takes 4hours or so ... > > I'm really curious to see how long a full upgrade will take with upstream. > > Right now, the full upgrade job didn't go far from the controller > upgrade (step 2). > AFAIK, the timeout in upstream is 3 hours minus 10 minutes ... > I think if we keep a 2 nodes deployment with only pacemaker, it would be > enough... I will keep you in touch of my progress here.. > > But, even if the jobs takes 2 or 3 hours to vote, I think it would be a > real huge gain for the tripleo work. Today we use 8vCPU 8GB RAM, 80GB HDD our nodepool nodes. Obviously how the job performs is subjective to which cloud it is one. With that said, it might be worth moving these tests into 3rd party CI where more powerful servers would live. Even though tripleo jobs upstream today support 3 hour 10 minutes as the time out, some of our longest running jobs. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] what's up in upgrades?
On 09/08/2016 01:27 PM, Emilien Macchi wrote: What's up in upgrades? 1) Undercloud: Mitaka to Newton We just approved a patch in openstack-infra/tripleo-ci that test undercloud upgrades. I don't think we should make it vote as for now it's quite experimental. Though I'm wondering if we should move it to the check pipeline as non-voting (currently in experimental queue). This is a first iteration and if you plan to upgrade your undercloud, you'll still have to do manual steps that we do in tripleo-ci. They are described here: https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 We need to decide where to put these bits: in tripleoclient? in instack-undercloud? Let's talk about it. I think we already did: https://specs.openstack.org/openstack/tripleo-specs/specs/newton/undercloud-upgrade.html There are details about where various bits of the implementation should live because it turns out that is pretty fundamental to the design of the upgrade process. 2) Overcloud: Mitaka to Newton matbu and myself are working together on a CI job that will test the upgrade of an undercloud + overcloud from Mitaka to Newton. Work is here: https://review.openstack.org/#/c/364859 and https://review.openstack.org/#/c/323750/ (both patches are going to merge together so we have one single patch for review). The idea is to use multinode job for now as a first iteration, with the simplest scenario possible so we can easily iterate later. 3) Overcloud: Newton to Newton I'm working on a simple patch that would test updates from Newton to Newton: https://review.openstack.org/#/c/351330/ like we had with OVB jobs but this time using multinode. Any feedback, help, is highly welcome. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] what's up in upgrades?
On 09/09/2016 12:42 AM, Emilien Macchi wrote: > On Thu, Sep 8, 2016 at 4:18 PM, David Moreau Simard wrote: >> How long does the upgrade job take ? > undercloud upgrade is a matter of 10 min max. > overcloud upgrade will be much more, I don't have metrics right now. > matbu maybe? It really depend on the infra which run the upgrade. I don't know much about the upstream infra but on my local box, with a SSD, 8 cores and 32Go of RAM, It could take around 1h30 2hours for a full upgrade. On centos ci infra and with RDO, some jobs can takes 4hours or so ... I'm really curious to see how long a full upgrade will take with upstream. Right now, the full upgrade job didn't go far from the controller upgrade (step 2). AFAIK, the timeout in upstream is 3 hours minus 10 minutes ... I think if we keep a 2 nodes deployment with only pacemaker, it would be enough... I will keep you in touch of my progress here.. But, even if the jobs takes 2 or 3 hours to vote, I think it would be a real huge gain for the tripleo work. >> David Moreau Simard >> Senior Software Engineer | Openstack RDO >> >> dmsimard = [irc, github, twitter] >> >> >> On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi wrote: >>> What's up in upgrades? >>> >>> 1) Undercloud: Mitaka to Newton >>> >>> We just approved a patch in openstack-infra/tripleo-ci that test >>> undercloud upgrades. >>> I don't think we should make it vote as for now it's quite >>> experimental. Though I'm wondering if we should move it to the check >>> pipeline as non-voting (currently in experimental queue). >>> >>> This is a first iteration and if you plan to upgrade your undercloud, >>> you'll still have to do manual steps that we do in tripleo-ci. They >>> are described here: >>> https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 >>> >>> We need to decide where to put these bits: in tripleoclient? in >>> instack-undercloud? Let's talk about it. >>> >>> >>> 2) Overcloud: Mitaka to Newton >>> >>> matbu and myself are working together on a CI job that will test the >>> upgrade of an undercloud + overcloud from Mitaka to Newton. >>> Work is here: https://review.openstack.org/#/c/364859 and >>> https://review.openstack.org/#/c/323750/ (both patches are going to >>> merge together so we have one single patch for review). >>> The idea is to use multinode job for now as a first iteration, with >>> the simplest scenario possible so we can easily iterate later. >>> >>> >>> 3) Overcloud: Newton to Newton >>> I'm working on a simple patch that would test updates from Newton to >>> Newton: https://review.openstack.org/#/c/351330/ like we had with OVB >>> jobs but this time using multinode. >>> >>> >>> Any feedback, help, is highly welcome. >>> -- >>> Emilien Macchi >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] what's up in upgrades?
On Thu, Sep 8, 2016 at 4:18 PM, David Moreau Simard wrote: > How long does the upgrade job take ? undercloud upgrade is a matter of 10 min max. overcloud upgrade will be much more, I don't have metrics right now. matbu maybe? > > David Moreau Simard > Senior Software Engineer | Openstack RDO > > dmsimard = [irc, github, twitter] > > > On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi wrote: >> What's up in upgrades? >> >> 1) Undercloud: Mitaka to Newton >> >> We just approved a patch in openstack-infra/tripleo-ci that test >> undercloud upgrades. >> I don't think we should make it vote as for now it's quite >> experimental. Though I'm wondering if we should move it to the check >> pipeline as non-voting (currently in experimental queue). >> >> This is a first iteration and if you plan to upgrade your undercloud, >> you'll still have to do manual steps that we do in tripleo-ci. They >> are described here: >> https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 >> >> We need to decide where to put these bits: in tripleoclient? in >> instack-undercloud? Let's talk about it. >> >> >> 2) Overcloud: Mitaka to Newton >> >> matbu and myself are working together on a CI job that will test the >> upgrade of an undercloud + overcloud from Mitaka to Newton. >> Work is here: https://review.openstack.org/#/c/364859 and >> https://review.openstack.org/#/c/323750/ (both patches are going to >> merge together so we have one single patch for review). >> The idea is to use multinode job for now as a first iteration, with >> the simplest scenario possible so we can easily iterate later. >> >> >> 3) Overcloud: Newton to Newton >> I'm working on a simple patch that would test updates from Newton to >> Newton: https://review.openstack.org/#/c/351330/ like we had with OVB >> jobs but this time using multinode. >> >> >> Any feedback, help, is highly welcome. >> -- >> Emilien Macchi >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] what's up in upgrades?
On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi wrote: > What's up in upgrades? > > 1) Undercloud: Mitaka to Newton > > We just approved a patch in openstack-infra/tripleo-ci that test > undercloud upgrades. > I don't think we should make it vote as for now it's quite > experimental. Though I'm wondering if we should move it to the check > pipeline as non-voting (currently in experimental queue). Done here: https://review.openstack.org/#/c/367641/ > This is a first iteration and if you plan to upgrade your undercloud, > you'll still have to do manual steps that we do in tripleo-ci. They > are described here: > https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 > > We need to decide where to put these bits: in tripleoclient? in > instack-undercloud? Let's talk about it. > > > 2) Overcloud: Mitaka to Newton > > matbu and myself are working together on a CI job that will test the > upgrade of an undercloud + overcloud from Mitaka to Newton. > Work is here: https://review.openstack.org/#/c/364859 and > https://review.openstack.org/#/c/323750/ (both patches are going to > merge together so we have one single patch for review). > The idea is to use multinode job for now as a first iteration, with > the simplest scenario possible so we can easily iterate later. > > > 3) Overcloud: Newton to Newton > I'm working on a simple patch that would test updates from Newton to > Newton: https://review.openstack.org/#/c/351330/ like we had with OVB > jobs but this time using multinode. Patch is merged, moving to check pipeline: https://review.openstack.org/#/c/367641/ -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] what's up in upgrades?
How long does the upgrade job take ? David Moreau Simard Senior Software Engineer | Openstack RDO dmsimard = [irc, github, twitter] On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi wrote: > What's up in upgrades? > > 1) Undercloud: Mitaka to Newton > > We just approved a patch in openstack-infra/tripleo-ci that test > undercloud upgrades. > I don't think we should make it vote as for now it's quite > experimental. Though I'm wondering if we should move it to the check > pipeline as non-voting (currently in experimental queue). > > This is a first iteration and if you plan to upgrade your undercloud, > you'll still have to do manual steps that we do in tripleo-ci. They > are described here: > https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 > > We need to decide where to put these bits: in tripleoclient? in > instack-undercloud? Let's talk about it. > > > 2) Overcloud: Mitaka to Newton > > matbu and myself are working together on a CI job that will test the > upgrade of an undercloud + overcloud from Mitaka to Newton. > Work is here: https://review.openstack.org/#/c/364859 and > https://review.openstack.org/#/c/323750/ (both patches are going to > merge together so we have one single patch for review). > The idea is to use multinode job for now as a first iteration, with > the simplest scenario possible so we can easily iterate later. > > > 3) Overcloud: Newton to Newton > I'm working on a simple patch that would test updates from Newton to > Newton: https://review.openstack.org/#/c/351330/ like we had with OVB > jobs but this time using multinode. > > > Any feedback, help, is highly welcome. > -- > Emilien Macchi > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo] what's up in upgrades?
What's up in upgrades? 1) Undercloud: Mitaka to Newton We just approved a patch in openstack-infra/tripleo-ci that test undercloud upgrades. I don't think we should make it vote as for now it's quite experimental. Though I'm wondering if we should move it to the check pipeline as non-voting (currently in experimental queue). This is a first iteration and if you plan to upgrade your undercloud, you'll still have to do manual steps that we do in tripleo-ci. They are described here: https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577 We need to decide where to put these bits: in tripleoclient? in instack-undercloud? Let's talk about it. 2) Overcloud: Mitaka to Newton matbu and myself are working together on a CI job that will test the upgrade of an undercloud + overcloud from Mitaka to Newton. Work is here: https://review.openstack.org/#/c/364859 and https://review.openstack.org/#/c/323750/ (both patches are going to merge together so we have one single patch for review). The idea is to use multinode job for now as a first iteration, with the simplest scenario possible so we can easily iterate later. 3) Overcloud: Newton to Newton I'm working on a simple patch that would test updates from Newton to Newton: https://review.openstack.org/#/c/351330/ like we had with OVB jobs but this time using multinode. Any feedback, help, is highly welcome. -- Emilien Macchi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
