Re: [openstack-dev] [tripleo] Proposing Lukas Bezdicka core on TripleO
+1 On Wed, Aug 01, 2018 at 01:31:40PM +0200, Giulio Fidente wrote: > Hi, > > I would like to propose Lukas Bezdicka core on TripleO. > > Lukas did a lot work in our tripleoclient, tripleo-common and > tripleo-heat-templates repos to make FFU possible. > > FFU, which is meant to permit upgrades from Newton to Queens, requires > in depth understanding of many TripleO components (for example Heat, > Mistral and the TripleO client) but also of specific TripleO features > which were added during the course of the three releases (for example > config-download and upgrade tasks). I believe his FFU work to have been > very challenging. > > Given his broad understanding, more recently Lukas started helping doing > reviews in other areas. > > I am so sure he'll be a great addition to our group that I am not even > looking for comments, just votes :D > -- > Giulio Fidente > GPG KEY: 08D733BA > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Stein blueprint - Plan to remove Keepalived support (replaced by Pacemaker)
On Wed, Jul 18, 2018 at 11:07:04AM -0400, Dan Prince wrote: > On Tue, 2018-07-17 at 22:00 +0200, Michele Baldessari wrote: > > Hi Jarda, > > > > thanks for these perspectives, this is very valuable! > > > > On Tue, Jul 17, 2018 at 06:01:21PM +0200, Jaromir Coufal wrote: > > > Not rooting for any approach here, just want to add a bit of > > > factors which might play a role when deciding which way to go: > > > > > > A) Performance matters, we should be improving simplicity and speed > > > of > > > deployments rather than making it heavier. If the deployment time > > > and > > > resource consumption is not significantly higher, I think it > > > doesn’t > > > cause an issue. But if there is a significant difference between > > > PCMK > > > and keepalived architecture, we would need to review that. > > > > +1 Should the pcmk take substantially more time then I agree, not > > worth > > defaulting to it. Worth also exploring how we could tweak things > > to make the setup of the cluster a bit faster (on a single node we > > can > > lower certain wait operations) but full agreement on this point. > > > > > B) Containerization of PCMK plans - eventually we would like to run > > > the whole undercloud/overcloud on minimal OS in containers to keep > > > improving the operations on the nodes (updates/upgrades/etc). If > > > because PCMK we would be forever stuck on BM, it would be a bit of > > > pita. As Michele said, maybe we can re-visit this. > > > > So I briefly discussed this in our team, and while it could be > > re-explored, we need to be very careful about the tradeoffs. > > This would be another layer which would bring quite a bit of > > complexity > > (pcs commands would have to be run inside a container, speed > > tradeoffs, > > more limited possibilities when it comes to upgrading/updating, etc.) > > > > > C) Unification of undercloud/overcloud is important for us, so +1 > > > to > > > whichever method is being used in both. But what I know, HA folks > > > went > > > to keepalived since it is simpler so would be good to keep in sync > > > (and good we have their presence here actually) :) > > > > Right so to be honest, the choice of keepalived on the undercloud for > > VIP predates me and I was not directly involved, so I lack the exact > > background for that choice (and I could not quickly reconstruct it > > from git > > history). But I think it is/was a reasonable choice for what it needs > > doing, although I probably would have picked just configuring the > > extra > > VIPs on the interfaces and have one service less to care about. > > +1 in general on the unification, with the caveats that have been > > discussed so far. > > I think it was more of that we wanted to use HAProxy for SSL > termination and keepalived is a simple enough way to set this up. > Instack-Undercloud has used HAProxy/keepalived for years in this > manner. > > I think this came up recently because downstream we did not have a > keepalived container. So it got a bit of spotlight on it as to why we > were using it. We do have a keepalived RPM and its worked as it has for > years already so as far as single node/undercloud setups go I think it > would continue to work fine. Kolla has had and supports the keepalived > container for awhile now as well. > > --- > > Comments on this thread seem to cover 2 main themes to me. > Simplification and the desire to use the same architecture as the > Overcloud (Pacemaker). And there is some competition between them. > > For simplification: If we can eliminate keepalived and still use > HAProxy (thus keeping the SSL termination features working) then I > think that would be worth trying. Specifically can we eliminate > Keepalived without swapping in Pacemaker? Michele: if you have ideas > here lets try them! I don't think it makes a lot of sense to just move to native IPs on interfaces just to remove keepalived. At least I don't see a good trade-off. If it has worked so far, I'd say let's just keep it (unless there are compelling arguments to remove it, of course) > With regards to Pacemaker I think we need to make an exception. It > seems way too heavy for single node setups and increases the complexity > there for very little benefit. > To me the shared architecture for > TripleO is the tools we use to setup services. By using t-h-t to drive > our setup of the Undercloud and All-In-One installers we are already > gaining a lot of benefit here. Pacemaker is we
Re: [openstack-dev] [tripleo] Stein blueprint - Plan to remove Keepalived support (replaced by Pacemaker)
Hi Jarda, thanks for these perspectives, this is very valuable! On Tue, Jul 17, 2018 at 06:01:21PM +0200, Jaromir Coufal wrote: > Not rooting for any approach here, just want to add a bit of factors which > might play a role when deciding which way to go: > > A) Performance matters, we should be improving simplicity and speed of > deployments rather than making it heavier. If the deployment time and > resource consumption is not significantly higher, I think it doesn’t > cause an issue. But if there is a significant difference between PCMK > and keepalived architecture, we would need to review that. +1 Should the pcmk take substantially more time then I agree, not worth defaulting to it. Worth also exploring how we could tweak things to make the setup of the cluster a bit faster (on a single node we can lower certain wait operations) but full agreement on this point. > B) Containerization of PCMK plans - eventually we would like to run > the whole undercloud/overcloud on minimal OS in containers to keep > improving the operations on the nodes (updates/upgrades/etc). If > because PCMK we would be forever stuck on BM, it would be a bit of > pita. As Michele said, maybe we can re-visit this. So I briefly discussed this in our team, and while it could be re-explored, we need to be very careful about the tradeoffs. This would be another layer which would bring quite a bit of complexity (pcs commands would have to be run inside a container, speed tradeoffs, more limited possibilities when it comes to upgrading/updating, etc.) > C) Unification of undercloud/overcloud is important for us, so +1 to > whichever method is being used in both. But what I know, HA folks went > to keepalived since it is simpler so would be good to keep in sync > (and good we have their presence here actually) :) Right so to be honest, the choice of keepalived on the undercloud for VIP predates me and I was not directly involved, so I lack the exact background for that choice (and I could not quickly reconstruct it from git history). But I think it is/was a reasonable choice for what it needs doing, although I probably would have picked just configuring the extra VIPs on the interfaces and have one service less to care about. +1 in general on the unification, with the caveats that have been discussed so far. > D) Undercloud HA is a nice have which I think we want to get to one > day, but it is not in as big demand as for example edge deployments, > BM provisioning with pure OS, or multiple envs managed by single > undercloud. So even though undercloud HA is important, it won’t bring > operators as many benefits as the previously mentioned improvements. > Let’s keep it in mind when we are considering the amount of work > needed for it. +100 > E) One of the use-cases we want to take into account is expanind a > single-node deployment (all-in-one) to 3 node HA controller. I think > it is important when evaluating PCMK/keepalived Right, so to be able to implement this, there is no way around having pacemaker (at least today until we have galera and rabbit). It still does not mean we have to default to it, but if you want to scale beyond one node, then there is no other option atm. > HTH It did, thanks! Michele > — Jarda > > > On Jul 17, 2018, at 05:04, Emilien Macchi wrote: > > > > Thanks everyone for the feedback, I've made a quick PoC: > > https://review.openstack.org/#/q/topic:bp/undercloud-pacemaker-default > > > > And I'm currently doing local testing. I'll publish results when progress > > is made, but I've made it so we have the choice to enable pacemaker > > (disabled by default), where keepalived would remain the default for now. > > > > On Mon, Jul 16, 2018 at 2:07 PM Michele Baldessari > > wrote: > > On Mon, Jul 16, 2018 at 11:48:51AM -0400, Emilien Macchi wrote: > > > On Mon, Jul 16, 2018 at 11:42 AM Dan Prince wrote: > > > [...] > > > > > > > The biggest downside IMO is the fact that our Pacemaker integration is > > > > not containerized. Nor are there any plans to finish the > > > > containerization of it. Pacemaker has to currently run on baremetal > > > > and this makes the installation of it for small dev/test setups a lot > > > > less desirable. It can launch containers just fine but the pacemaker > > > > installation itself is what concerns me for the long term. > > > > > > > > Until we have plans for containizing it I suppose I would rather see > > > > us keep keepalived as an option for these smaller setups. We can > > > > certainly change our default Undercloud to use Pacemaker (if we choose > > > > to do so). But having keepalived around for "lightweig
Re: [openstack-dev] [tripleo] Stein blueprint - Plan to remove Keepalived support (replaced by Pacemaker)
On Mon, Jul 16, 2018 at 11:48:51AM -0400, Emilien Macchi wrote: > On Mon, Jul 16, 2018 at 11:42 AM Dan Prince wrote: > [...] > > > The biggest downside IMO is the fact that our Pacemaker integration is > > not containerized. Nor are there any plans to finish the > > containerization of it. Pacemaker has to currently run on baremetal > > and this makes the installation of it for small dev/test setups a lot > > less desirable. It can launch containers just fine but the pacemaker > > installation itself is what concerns me for the long term. > > > > Until we have plans for containizing it I suppose I would rather see > > us keep keepalived as an option for these smaller setups. We can > > certainly change our default Undercloud to use Pacemaker (if we choose > > to do so). But having keepalived around for "lightweight" (zero or low > > footprint) installs that work is really quite desirable. > > > > That's a good point, and I agree with your proposal. > Michele, what's the long term plan regarding containerized pacemaker? Well, we kind of started evaluating it (there was definitely not enough time around pike/queens as we were busy landing the bundles code), then due to discussions around k8s it kind of got off our radar. We can at least resume the discussions around it and see how much effort it would be. I'll bring it up with my team and get back to you. cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Stein blueprint - Plan to remove Keepalived support (replaced by Pacemaker)
Hi Emilien, On Fri, Jul 13, 2018 at 02:33:02PM -0400, Emilien Macchi wrote: > We have been supporting both Keepalived and Pacemaker to handle VIP > management. Keepalived is actually the tool used by the undercloud > when SSL is enabled (for SSL termination). While Pacemaker is used on > the overcloud to handle VIPs but also services HA. > > I see some benefits at removing support for keepalived and deploying > Pacemaker by default: > - pacemaker can be deployed on one node (we actually do it in CI), so can > be deployed on the undercloud to handle VIPs and manage HA as well. > - it'll allow to extend undercloud & standalone use cases to support > multinode one day, with HA and SSL, like we already have on the overcloud. > - it removes the complexity of managing two tools so we'll potentially > removing code in TripleO. > - of course since pacemaker features from overcloud would be usable in > standalone environment, but also on the undercloud. > > There is probably some downside, the first one is I think Keepalived is > much more lightweight than Pacemaker, we probably need to run some > benchmark here and make sure we don't make the undercloud heavier than it > is now. Right, I think the service startup of pacemaker/corosync + starting the VIP will be a bit slower than a keepalived approach (seconds). But after that there should not be a lot of difference. Also, we're about to land proper support for updating pcmk resources so in the future managing them will also be a bit simpler than it is now. > I went ahead and created this blueprint for Stein: > https://blueprints.launchpad.net/tripleo/+spec/undercloud-pacemaker-default > I also plan to prototype some basic code soon and provide an upgrade path > if we accept this blueprint. I like the approach for the reasons you mention above. I'll be happy to chat about this at the PTG and help out in general with this. thanks, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Migration to Storyboard
On Mon, May 21, 2018 at 01:58:26PM -0700, Emilien Macchi wrote: > During the Storyboard session today: > https://etherpad.openstack.org/p/continuing-the-migration-lp-sb > > We mentioned that TripleO would continue to migrate during Rocky cycle. > Like Alex mentioned in this thread, we need to migrate the scripts used by > the CI squad so they work with SB. > Once this is done, we'll proceed to the full migration of all blueprints > and bugs into tripleo-common project in SB. > Projects like tripleo-validations, tripleo-ui (more?) who have 1:1 mapping > between their "name" and project repository could use a dedicated project > in SB, although we need to keep things simple for our users so they know > where to file a bug without confusion. > We hope to proceed during Rocky but it'll probably take some time to update > our scripts and documentation, also educate our community to use the tool, > so we expect the Stein cycle the first cycle where we actually consume SB. > > I really wanted to thank the SB team for their patience and help, TripleO > is big and this migration hasn't been easy but we'll make it :-) Having used storyboard for the first time today to file a bug^Wstory in heat, I'd like to raise a couple of concerns on this migration. And by all means, if I just missed to RTFM, feel free to point me in the right direction. 1. Searching for bugs in a specific project is *extremely* cumbersome and I am not even sure I got it right (first you need to put openstack/project in the search bar, wait and click it. Then you add the term you are looking for. I have genuinely no idea if I get all the issues I was looking for or not as it is not obvious on what fields this search is performed 2. Advanced search is either very well hidden or not existant yet? E.g. how do you search for bugs filed by someone or over a certain release, or just generally more complex searches which are super useful in order to avoid filing duplicate bugs. I think Zane's additional list also matches my experience very well: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131365.html So my take is that a migration atm is a bit premature and I would postpone it at least to Stein. Thanks, Michele > Thanks, > > On Tue, May 15, 2018 at 7:53 AM, Alex Schultz wrote: > > > Bumping this up so folks can review this. It was mentioned in this > > week's meeting that it would be a good idea for folks to take a look > > at Storyboard to get familiar with it. The upstream docs have been > > updated[0] to point to the differences when dealing with proposed > > patches. Please take some time to review this and raise any > > concerns/issues now. > > > > Thanks, > > -Alex > > > > [0] https://docs.openstack.org/infra/manual/developers.html# > > development-workflow > > > > On Wed, May 9, 2018 at 1:24 PM, Alex Schultz wrote: > > > Hello tripleo folks, > > > > > > So we've been experimenting with migrating some squads over to > > > storyboard[0] but this seems to be causing more issues than perhaps > > > it's worth. Since the upstream community would like to standardize on > > > Storyboard at some point, I would propose that we do a cut over of all > > > the tripleo bugs/blueprints from Launchpad to Storyboard. > > > > > > In the irc meeting this week[1], I asked that the tripleo-ci team make > > > sure the existing scripts that we use to monitor bugs for CI support > > > Storyboard. I would consider this a prerequisite for the migration. > > > I am thinking it would be beneficial to get this done before or as > > > close to M2. > > > > > > Thoughts, concerns, etc? > > > > > > Thanks, > > > -Alex > > > > > > [0] https://storyboard.openstack.org/#!/project_group/76 > > > [1] http://eavesdrop.openstack.org/meetings/tripleo/2018/ > > tripleo.2018-05-08-14.00.log.html#l-42 > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > -- > Emilien Macchi > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Proposing Alan Bishop tripleo core on storage bits
+1 On Wed, Jun 13, 2018 at 08:50:23AM -0700, Emilien Macchi wrote: > Alan Bishop has been highly involved in the Storage backends integration in > TripleO and Puppet modules, always here to update with new features, fix > (nasty and untestable third-party backends) bugs and manage all the > backports for stable releases: > https://review.openstack.org/#/q/owner:%22Alan+Bishop+%253Cabishop%2540redhat.com%253E%22 > > He's also well knowledgeable of how TripleO works and how containers are > integrated, I would like to propose him as core on TripleO projects for > patches related to storage things (Cinder, Glance, Swift, Manila, and > backends). > > Please vote -1/+1, > Thanks! > -- > Emilien Macchi > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Proposing Marius Cornea core on upgrade bits
+1 On Thu, Apr 19, 2018 at 10:01:50AM -0700, Emilien Macchi wrote: > Greetings, > > As you probably know mcornea on IRC, Marius Cornea has been contributing on > TripleO for a while, specially on the upgrade bits. > Part of the quality team, he's always testing real customer scenarios and > brings a lot of good feedback in his reviews, and quite often takes care of > fixing complex bugs when it comes to advanced upgrades scenarios. > He's very involved in tripleo-upgrade repository where he's already core, > but I think it's time to let him +2 on other tripleo repos for the patches > related to upgrades (we trust people's judgement for reviews). > > As usual, we'll vote! > > Thanks everyone for your feedback and thanks Marius for your hard work and > involvement in the project. > -- > Emilien Macchi > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Nominate chem and matbu for tripleo-core !
On Thu, Nov 09, 2017 at 10:48:03AM +0200, Marios Andreou wrote: > Hello fellow owls, > (appologies for duplicate, forgot to add the tripleo in subject so worried > it would be missed) > > I would like to nominate (and imo these are both long overdue already): > > Sofer Athlan Guyot (chem) and +1 > > Mathieu Bultel (matbu) +1 > > to tripleo-core. They have both made many many core contributions to the > upgrades & updates over the last 3 cycles touching many of the tripleo > repos (tripleo-heat-templates, tripleo-common, python-tripleoclient, > tripleo-ci, tripleo-docs and others tripleo-quickstart/extras too unless am > mistaken). > > IMO their efforts and contributions are invaluable for the upgrades squad > (and beyond - see openstack overcloud config download for example) and we > will be very lucky to have them as fully voting cores. > > Please vote with +1 or -1 for either or both chem and matbu - I'll keep it > open for a week as customary, thanks, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Proposing John Fulton core on TripleO
+1! On Wed, Nov 08, 2017 at 11:24:27PM +0100, Giulio Fidente wrote: > Hi, > > I would like to propose John Fulton core on TripleO. > > I think John did an awesome work during the Pike cycle around the > integration of ceph-ansible as a replacement for puppet-ceph, for the > deployment of Ceph in containers. > > I think John has good understanding of many different parts of TripleO > given that the ceph-ansible integration has been a complicated effort > involving changes in heat/tht/mistral workflows/ci and last but not > least, docs and he is more recently getting busier with reviews outside > his main comfort zone. > > I am sure John would be a great addition to the team and I welcome him > first to tune into radioparadise with the rest of us when joining #tripleo > > Feedback is welcomed! > -- > Giulio Fidente > GPG KEY: 08D733BA > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Nominate akrivoka for tripleo-validations core
+1 On Wed, Nov 08, 2017 at 12:11:40PM +0100, Jiří Stránský wrote: > +1! > > On 6.11.2017 15:32, Honza Pokorny wrote: > > Hello people, > > > > I would like to nominate Ana Krivokapić (akrivoka) for the core team for > > tripleo-validations. She has really stepped up her game on that project > > in terms of helpful reviews, and great patches. > > > > With Ana's help as a core, we can get more done, and innovate faster. > > > > If there are no objections within a week, we'll proceed with adding Ana > > to the team. > > > > Thanks > > > > Honza Pokorny > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Gate is broken - Do not approve any patch until further notice
On Thu, Aug 31, 2017 at 10:55:34AM +0200, Bogdan Dobrelya wrote:
> On 31.08.2017 10:33, Michele Baldessari wrote:
> > On Wed, Aug 30, 2017 at 11:31:14AM +0200, Bogdan Dobrelya wrote:
> >> On 30.08.2017 6:54, Emilien Macchi wrote:
> >>> On Tue, Aug 29, 2017 at 4:17 PM, Emilien Macchi
> >>> wrote:
> >>>> We are currently dealing with 4 issues and until they are fix, please
> >>>> do not approve any patch. We want to keep the gate clear to merge the
> >>>> fixes for the 4 problems first.
> >>>>
> >>>> 1) devstack-gate broke us because we use it as a library (bad)
> >>>> https://bugs.launchpad.net/tripleo/+bug/1713868
> >>>>
> >>>> 2) https://review.openstack.org/#/c/474578/ broke us and we're
> >>>> reverting it https://bugs.launchpad.net/tripleo/+bug/1713832
> >>>>
> >>>> 3) We shouldn't build images on multinode jobs
> >>>> https://bugs.launchpad.net/tripleo/+bug/1713167
> >>>>
> >>>> 4) We should use pip instead of git for delorean
> >>>> https://bugs.launchpad.net/tripleo/+bug/1708832
> >>>>
> >>>>
> >>>> Until further notice from Alex or myself, please do not approve any
> >>>> patch.
> >>>
> >>> The 4 problems have been mitigated.
> >>> You can now proceed to normal review.
> >>>
> >>> Please do not recheck a patch without an elastic-recheck comment, we
> >>> need to track all issues related to CI from now.
> >>> Paul Belanger has been doing extremely useful work to help us, now
> >>> let's use elastic-recheck more and stop blind rechecks.
> >>> All known issues are in http://status.openstack.org/elastic-recheck/
> >>> If one is missing, you're welcome to contribute by sending a patch to
> >>> elastic-recheck. Example with https://review.openstack.org/#/c/498954/
> >>
> >> That's a great example! Let me follow up on that and share my beginner's
> >> experience as well.
> >>
> >> Let's help with improving elastic-recheck queries to identify those
> >> unknown or new failures, this is really important. This also trains
> >> domain knowledge for particular areas, either openstack or *-infra, or
> >> tripleo specific.
> >>
> >> As beginners, we could start with watching for failing tripleo-ci
> >> periodic [0],[1] (available as RSS feeds) and gate jobs without e-r
> >> comments, also from that page [2].
> >>
> >> Then fetching the logs locally with tools like getthelogs [3], or
> >> looking into the logs.openstack.org directly, if advanced beginners wish
> >> so.
> >>
> >> Finally, identifying discovered (just do some grep, like I do with my
> >> tool [4]) errorish patterns and helping with root cause analysis. And,
> >> ideally, submitting new e-r queries (see also [5]) and corresponding lp
> >> bugs. And absolutely ideally, help with addressing those as well. This
> >> might be hard though as we may be not experts in some of the areas. Some
> >> of the error messages would literally mean nothing to us. For me, the
> >> most But as the best effort, we could invite the right persons to
> >> look into that, or at least ask folks on #tripleo or #openstack-infra.
> >>
> >> [0]
> >> http://status.openstack.org/openstack-health/#/g/project/openstack-infra~2Ftripleo-ci
> >> [1]
> >> http://status.openstack.org/openstack-health/#/g/project/openstack~2Ftripleo-quickstart
> >> [2] http://status.openstack.org/elastic-recheck/data/others.html
> >> [3] https://review.openstack.org/#/c/492178/
> >> [4]
> >> https://github.com/bogdando/fuel-log-parse/blob/master/fuel-log-parse.sh
> >> [5]
> >> https://docs.openstack.org/infra/elastic-recheck/readme.html#running-queries-locally
> >
> > Thanks Bogdan, this is very helpful. Do we have some docs/readme on [5].
> > It is failing here with a bunch of 404, so I presume I am missing a
> > proper elasticRecheck.conf file or some other settings?
> >
> > I was basically trying to validate https://review.openstack.org/#/c/499516/
> > before
> > submitting it.
>
> There is install docs [0] :)
> Although for my case, the following worked as well (given
> VENV=${HOME}/.virtualenvs):
>
> $ mkvirtualenv erqtest
> $ pip install -r requirements.txt
> $ python setup.py develop
> $ ${VENV}/erqtest/bin
Re: [openstack-dev] [tripleo] Gate is broken - Do not approve any patch until further notice
On Wed, Aug 30, 2017 at 11:31:14AM +0200, Bogdan Dobrelya wrote: > On 30.08.2017 6:54, Emilien Macchi wrote: > > On Tue, Aug 29, 2017 at 4:17 PM, Emilien Macchi wrote: > >> We are currently dealing with 4 issues and until they are fix, please > >> do not approve any patch. We want to keep the gate clear to merge the > >> fixes for the 4 problems first. > >> > >> 1) devstack-gate broke us because we use it as a library (bad) > >> https://bugs.launchpad.net/tripleo/+bug/1713868 > >> > >> 2) https://review.openstack.org/#/c/474578/ broke us and we're > >> reverting it https://bugs.launchpad.net/tripleo/+bug/1713832 > >> > >> 3) We shouldn't build images on multinode jobs > >> https://bugs.launchpad.net/tripleo/+bug/1713167 > >> > >> 4) We should use pip instead of git for delorean > >> https://bugs.launchpad.net/tripleo/+bug/1708832 > >> > >> > >> Until further notice from Alex or myself, please do not approve any patch. > > > > The 4 problems have been mitigated. > > You can now proceed to normal review. > > > > Please do not recheck a patch without an elastic-recheck comment, we > > need to track all issues related to CI from now. > > Paul Belanger has been doing extremely useful work to help us, now > > let's use elastic-recheck more and stop blind rechecks. > > All known issues are in http://status.openstack.org/elastic-recheck/ > > If one is missing, you're welcome to contribute by sending a patch to > > elastic-recheck. Example with https://review.openstack.org/#/c/498954/ > > That's a great example! Let me follow up on that and share my beginner's > experience as well. > > Let's help with improving elastic-recheck queries to identify those > unknown or new failures, this is really important. This also trains > domain knowledge for particular areas, either openstack or *-infra, or > tripleo specific. > > As beginners, we could start with watching for failing tripleo-ci > periodic [0],[1] (available as RSS feeds) and gate jobs without e-r > comments, also from that page [2]. > > Then fetching the logs locally with tools like getthelogs [3], or > looking into the logs.openstack.org directly, if advanced beginners wish so. > > Finally, identifying discovered (just do some grep, like I do with my > tool [4]) errorish patterns and helping with root cause analysis. And, > ideally, submitting new e-r queries (see also [5]) and corresponding lp > bugs. And absolutely ideally, help with addressing those as well. This > might be hard though as we may be not experts in some of the areas. Some > of the error messages would literally mean nothing to us. For me, the > most But as the best effort, we could invite the right persons to > look into that, or at least ask folks on #tripleo or #openstack-infra. > > [0] > http://status.openstack.org/openstack-health/#/g/project/openstack-infra~2Ftripleo-ci > [1] > http://status.openstack.org/openstack-health/#/g/project/openstack~2Ftripleo-quickstart > [2] http://status.openstack.org/elastic-recheck/data/others.html > [3] https://review.openstack.org/#/c/492178/ > [4] https://github.com/bogdando/fuel-log-parse/blob/master/fuel-log-parse.sh > [5] > https://docs.openstack.org/infra/elastic-recheck/readme.html#running-queries-locally Thanks Bogdan, this is very helpful. Do we have some docs/readme on [5]. It is failing here with a bunch of 404, so I presume I am missing a proper elasticRecheck.conf file or some other settings? I was basically trying to validate https://review.openstack.org/#/c/499516/ before submitting it. Thanks, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] weekly meetings on #tripleo
On Mon, Jul 10, 2017 at 11:36:03AM -0230, Brent Eagles wrote: > +1 for giving it a try. Agreed. > > On Wed, Jul 5, 2017 at 2:26 PM, Emilien Macchi wrote: > > > After reading http://lists.openstack.org/pipermail/openstack-dev/2017- > > June/118899.html > > - we might want to collect TripleO's community feedback on doing > > weekly meetings on #tripleo instead of #openstack-meeting-alt. > > > > I see some direct benefits: > > - if you come up late in meetings, you could easily read backlog in > > #tripleo > > - newcomers not aware about the meeting channel wouldn't have to search > > for it > > - meeting would maybe get more activity and we would expose the > > information more broadly > > > > Any feedback on this proposal is welcome before we make any change (or > > not). > > > > Thanks, > > -- > > Emilien Macchi > > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] proposing Alex Schultz tripleo-core in all projects
On Fri, Jul 07, 2017 at 10:39:25AM -0700, Emilien Macchi wrote: > Alex has demonstrated high technical and community skills in TripleO - > where he's already core on THT, instack-undercloud, and puppet-tripleo > - but also very involved in other repos. > I propose that we extend his core status to all TripleO projects and > of course trust him (like we trust all core members) to review patches > were we feel confortable with. > > He has shown an high interest in reviewed other TripleO projects and I > think he would be ready for this change. > As usual, this is an open proposal, any feedback is welcome. +1 -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Proposing Florian Fuchs for tripleo-validations core
+1 On Thu, Apr 06, 2017 at 11:53:04AM +0200, Martin André wrote: > Hellooo, > > I'd like to propose we extend Florian Fuchs +2 powers to the > tripleo-validations project. Florian is already core on tripleo-ui > (well, tripleo technically so this means there is no changes to make > to gerrit groups). > > Florian took over many of the stalled patches in tripleo-validations > and is now the principal contributor in the project [1]. He has built > a good expertise over the last months and I think it's time he has > officially the right to approve changes in tripleo-validations. > > Consider this my +1 vote. > > Martin > > [1] > http://stackalytics.com/?module=tripleo-validations&metric=patches&release=pike > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Propose Attila Darazs and Gabriele Cerami for tripleo-ci core
On Wed, Mar 15, 2017 at 11:44:22AM -0400, John Trowbridge wrote: > Both Attila and Gabriele have been rockstars with the work to transition > tripleo-ci to run via quickstart, and both have become extremely > knowledgeable about how tripleo-ci works during that process. They are > both very capable of providing thorough and thoughtful reviews of > tripleo-ci patches. > > On top of this Attila has greatly increased the communication from the > tripleo-ci squad as the liason, with weekly summary emails of our > meetings to this list. +1. Thanks to both! -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] propose Alex Schultz core on tripleo-heat-templates
On Mon, Mar 13, 2017 at 10:30:08AM -0400, Emilien Macchi wrote: > Hi, > > Alex is already core on instack-undercloud and puppet-tripleo. > His involvement and knowledge in TripleO Heat Templates has been very > appreciated over the last months and I think we can give him +2 on > this project. > > As usual, feel free to vote -1/+1 on this proposal. +1! -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Proposing Dmitry Tantsur and Alex Schultz as instack-undercloud cores
+1 On Tue, Jan 31, 2017 at 10:02:06AM -0600, Ben Nemec wrote: > In the spirit of all the core team changes, here are a couple more I'd like > to propose. > > Dmitry has been very helpful reviewing in instack-undercloud for a long time > so this is way overdue. I'm also going to propose that he be able to +2 > anything Ironic-related in TripleO since that is his primary area of > expertise. > > Alex has ramped up quickly on TripleO and has also been helping out with > instack-undercloud quite a bit. He's already core for the puppet modules, > and a lot of the changes to instack-undercloud these days are primarily in > the puppet manifest so it's not a huge stretch to add him. > > As usual, TripleO cores please vote and/or provide comments. Thanks. > > -Ben > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Update TripleO core members
On Mon, Jan 23, 2017 at 02:03:28PM -0500, Emilien Macchi wrote: > Greeting folks, > > I would like to propose some changes in our core members: > > - Remove Jay Dobies who has not been active in TripleO for a while > (thanks Jay for your hard work!). > - Add Flavio Percoco core on tripleo-common and tripleo-heat-templates > docker bits. > - Add Steve Backer on os-collect-config and also docker bits in > tripleo-common and tripleo-heat-templates. > > Indeed, both Flavio and Steve have been involved in deploying TripleO > in containers, their contributions are very valuable. I would like to > encourage them to keep doing more reviews in and out container bits. +1!! -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Proposing Sergey (Sagi) Shnaidman for core on tripleo-ci
On Tue, Jan 24, 2017 at 07:03:56PM +0200, Juan Antonio Osorio wrote: > Sagi (sshnaidm on IRC) has done significant work in TripleO CI (both > on the current CI solution and in getting tripleo-quickstart jobs for > it); So I would like to propose him as part of the TripleO CI core team. > > I think he'll make a great addition to the team and will help move CI > issues forward quicker. +1 Thanks, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Atlanta PTG
Hi Emilien, while not a design session per se, I would love to propose a short slot for TripleO CI Q&A, if we have some time left. In short, I'd like to be more useful around CI failures, but I lack the understanding of a few aspects of our current CI (promotion, when do images get built, etc.), that would benefit quite a bit from a short session where we have a few CI folks in the room that could answer questions or give some tips. I know of quite few other people that are in the same boat and maybe this will help a bit our current issue where only a few folks always chase CI issues. If there is consensus (and some CI folks willing to attend ;) and time for this, I'll be happy to organize this and prepare a bunch of questions ideas beforehand. Thoughts? Michele On Wed, Jan 04, 2017 at 07:26:52AM -0500, Emilien Macchi wrote: > I would like to bring this topic up on your inbox, so we can continue > to make progress on the agenda. Feel free to follow existing examples > in the etherpad and propose a design dession. > > Thanks, > > On Wed, Dec 21, 2016 at 9:06 AM, Emilien Macchi wrote: > > General infos about PTG: https://www.openstack.org/ptg/ > > > > Some useful informations about PTG/TripleO: > > > > * When? We have a room between Wednesday and Friday included. > > Important sessions will happen on Wednesday and Thursday. We'll > > probably have sessions on Friday, but it might be more hands-on and > > hackfest, where people can enjoy the day to work together. > > > > * Let's start to brainstorm our topics: > > https://etherpad.openstack.org/p/tripleo-ptg-pike > > Feel free to add any topic, as soon as you can. We need to know asap > > which sessions will be share with other projects (eg: tripleo/mistral, > > tripleo/ironic, tripleo/heat, etc). > > > > > > Please let us know any question or feedback, > > Looking forward to seeing you there! > > -- > > Emilien Macchi > > > > -- > Emilien Macchi > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Move redis out of Pacemaker
Hi Pradeep, On Mon, Dec 12, 2016 at 02:51:59PM +0100, Giulio Fidente wrote: > On 12/09/2016 04:49 PM, Pradeep Kilambi wrote: > >I would like to get some thoughts on $Subject. This came up when i was > >discussing the standalone roles for telemetry. Currently when we deploy > >redis in tripleo, its a pacemaker managed service. So if we were to > >deploy telemetry services on a dedicated node we could. But redis will > >have to be on a another node? (assuming we dont want to pull in > >pacemaker on to telemetry nodes). Ok so with the composable HA work [1] you should be able to split out the redis service on to dedicated nodes and these nodes can be either full pacemaker cluster members or only have the pacemaker-remote service. > currently redis instances are not configured as a redis cluster but use the > master/slave replication model instead and pacemaker is taking care of > electing/relocating the redis master as needed > > there shouldn't be any dependency on the redis profile for the telemetry > roles, they should instead just point at the redis_vip > > the redis_vip is always guaranteed (by haproxy) to point to the redis master > > >With most services moved out of pacemaker in Newton, I think its time to > >move redis as well? Are there any constraints in moving redis to be > >managed by systemd? Looking at how we do it, It should be easily movable > >to systemd? Can we consider doing this for Ocata? > > I think we could look at using the redis cluster which allows multiple > masters, but I am not sure this can happen in Ocata ... yet again, there > shouldn't be in the telemetry roles any dependency on redis itself > > if we were to use the cluster mode the only difference would probably be > that the redis_vip will start balancing requests across the nodes In general I am in favour to split out redis from pacemaker. There is the question that in theory we'd have two potentially separate quorums, but I think that with redis this should not be a big problem. Maybe let's start with a prototype and see how things look and iterate from there? I think it is a bit late for ocata, but we could at least start the work without changing the defaults (i.e. let the operator override the tripleo::service with a redis base profile instead of the pacemaker one) Does that make sense, Michele [1] https://review.openstack.org/#/q/topic:bp/composable-ha -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Proposing Alex Schultz core on puppet-tripleo
On Thu, Dec 01, 2016 at 05:26:31PM -0500, Emilien Macchi wrote: > Team, > > Alex Schultz (mwhahaha on IRC) has been active on TripleO since a few > months now. While he's very active in different areas of TripleO, his > reviews and contributions on puppet-tripleo have been very useful. > Alex is a Puppet guy and also the current PTL of Puppet OpenStack. I > think he perfectly understands how puppet-tripleo works. His > involvement in the project and contributions on puppet-tripleo deserve > that we allow him to +2 puppet-tripleo. > > Thanks Alex for your involvement and hard work in the project, this is > very appreciated! > > As usual, I'll let the team to vote about this proposal. +1 -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Proposing Julie Pichon for tripleo core
On Tue, Nov 22, 2016 at 05:01:49PM +, Dougal Matthews wrote: > Hi all, > > I would like to propose we add Julie (jpich) to the TripleO core team for > python-tripleoclient and tripleo-common. This nomination is based partially > on review stats[1] and also my experience with her reviews and > contributions. > > Julie has consistently provided thoughtful and detailed reviews since the > start of the Newton cycle. She has made a number of contributions which > improve the CLI and has been extremely helpful with other tasks that don't > often get enough attention (backports, bug triaging/reporting and improving > our processes[2]). > > I think she will be a valuable addition to the review team +1 -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] possible backports for stable/newton
Hi Brent ;) On Thu, Nov 03, 2016 at 01:20:12AM -0230, Brent Eagles wrote: > puppet-tripleo > > https://review.openstack.org/#/c/389583/ Set redis file descriptor limit > when run via pacemaker Yes, this one I will backport this week. > https://review.openstack.org/#/c/380414/ Only run ceilometer::db::sync on > bootstrap node > https://review.openstack.org/#/c/386042/ pacemaker/mysql: wait step 2 to > remove default accounts > > tripleo-heat-templates > > https://review.openstack.org/#/c/380979/ Change rabbitmq queues HA mode > from ha-all to ha-exactly We decided against it in the end. We merged it for Ocata and will implement it only there (i.e. we thought it was a bit too risky this close to the release) > https://review.openstack.org/#/c/381869/ Include redis/mongo hiera when > using pacemaker > https://review.openstack.org/#/c/387266/ Enable proxy headers parsing for > Neutron (not sure about this one... there are similar patches that landed > to stable/newton so maybe) > https://review.openstack.org/#/c/385058/ Remove duplicate metadata keys > from nova-api.yaml (probably not critical - the big related bug was the > worker count = 0 thing for nova) regards, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] Roles count and flavors inside Heat environment file
On Mon, Oct 03, 2016 at 02:23:08PM +0200, Marius Cornea wrote: > Hello everyone, > > In Newton we've deprecated the *-scale and *-flavor deploy command > arguments in favor of using Heat environment files. In the context of > testing the composable roles where the custom roles' node count and > flavor need to be passed inside an environment file I would like to > build the test plan by using an environment containing all nodes count > and flavors, including the preexisting roles. > > A deploy command example would look like: > > openstack overcloud deploy --stack cloudy --templates -e nodes.yaml > > where the nodes environment file contains something like: > > parameter_defaults: > ControllerCount: 3 > ComputeCount: 2 > CephStorageCount: 3 > ServiceApiCount: 3 > > OvercloudControlFlavor: controller > OvercloudComputeFlavor: compute > OvercloudCephStorageFlavor: ceph > OvercloudServiceApiFlavor: serviceapi > > > I would like to get some feedback about this approach. I think it's > better to keep all the roles count/flavors in the same place for > consistency reasons. I fully agree with this approach. I think it is also preferrable to have a single point where we define this. cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Making the nonha-multinode and undercloud jobs voting
Hi James, On Tue, Aug 09, 2016 at 01:43:58PM -0400, James Slagle wrote: > On Tue, Aug 9, 2016 at 1:30 PM, Michele Baldessari wrote: > > On Tue, Aug 09, 2016 at 01:14:24PM -0400, James Slagle wrote: > >> I've proposed: > >> https://review.openstack.org/353019 > >> > >> which makes gate-tripleo-ci-centos-7-nonha-multinode-nv and > >> gate-tripleo-ci-centos-7-undercloud-nv become voting jobs. > > > > definitely +1 for the gate-tripleo-ci-centos-7-undercloud-nv job. > > About the nonha job, I was actually wondering if we should still > > keep any non-ha templates/jobs around now that the New HA architecture > > has landed. I cannot think of any real usage and the NG HA stuff deploys > > fine on 1 controller as well so the "develop on a smaller machine" > > use-case is covered. > > > > Is there any reason/use-case to keep any non-ha templates/jobs around? > > I'd love to remove them, but maybe there are some uses I have not > > thought of ;) > > I personally agree and think that we should consolidate our > development and testing efforts onto the single NG pacemaker > architecture and use that for both for non-HA and HA. > > That being said, this needs to be driven via tripleo-heat-templates, > tripleoclient, etc, instead of from the tripleo-ci side. E.g., once > environments/puppet-pacemaker.yaml is the default environment in > tripleo-heat-templates, then tripleo-ci will be using it automatically > for nonha. Ack. Sounds like a plan. I shall look into it. Kind regards, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Making the nonha-multinode and undercloud jobs voting
Hi James, On Tue, Aug 09, 2016 at 01:14:24PM -0400, James Slagle wrote: > I've proposed: > https://review.openstack.org/353019 > > which makes gate-tripleo-ci-centos-7-nonha-multinode-nv and > gate-tripleo-ci-centos-7-undercloud-nv become voting jobs. definitely +1 for the gate-tripleo-ci-centos-7-undercloud-nv job. About the nonha job, I was actually wondering if we should still keep any non-ha templates/jobs around now that the New HA architecture has landed. I cannot think of any real usage and the NG HA stuff deploys fine on 1 controller as well so the "develop on a smaller machine" use-case is covered. Is there any reason/use-case to keep any non-ha templates/jobs around? I'd love to remove them, but maybe there are some uses I have not thought of ;) Thanks, Michele > I think these jobs have proven to be stable enough that we can promote > them to be voting. If you have concerns, please vote on the patch. The > nice thing about having these jobs voting is that jenkins will > actually vote -1 on TripleO patches when these jobs fail. -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo] HA NG status
Hi all, just wanted to give a short status on the work around the HA architecture described in the SPEC here: https://review.openstack.org/299628 (blueprint here [1]) The last chunk of the work is currently contained in these two reviews: https://review.openstack.org/#/c/342650/ - cinder volume constraint move https://review.openstack.org/#/c/314208/ - Main Tripleo Heat Templates changes The patches are very small at this point, because after a chat with Giulio we made sure that the initial big patches were split off in smaller, more easily consumable bits. We also made it so that even openstack-core is a profile that can be turned on and off. This work still depends on the Aodh profiles [2] work landing in master. Once Aodh is committed the change is rather trivial. Testing has been successful so far. Given that it can be rolled back with a simple tweak to environments/puppet-pacemaker.yaml we have a solid non-intrusive plan B, should we encounter any unforeseen major issues. For the long-term (so Ocata at least), we can work to remove all the tripleo::profile::pacemaker classes that won't be needed any longer. I think it makes sense to keep them around for at least one release. I hope we can land these changes sometime this week or beginning of the next (depending on how the aodh reviews go). In any case either Chris or I will be around for any question/issues. Thanks for reading so far ;) Michele [1] https://blueprints.launchpad.net/tripleo/+spec/ha-lightweight-architecture [2] https://review.openstack.org/#/c/333556/ -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] The Future of Service Orchestration with Puppet & Pacemaker
Hi, On Tue, Mar 29, 2016 at 03:48:29PM -0400, Emilien Macchi wrote: > Some TripleO folks are currently working hard on making automation for > upgrading TripleO between major OpenStack releases. > One of the biggest challenges is how to deal with Puppet who usually > notify Services when configuration change and Pacemaker who actually > manage the services. > > Currently, we do this horrible thing that is disabling service > management when Pacemaker manages the service. > It's terrible because Puppet wants to manage the service with > 'systemd' but we disable it to let Pacemaker deal with that. > The main limitation is that most of Puppet modules (specially in > OpenStack) know how to do orchestration for deployments & upgrades, > but we don't let it do because we disable service management. agreed. The split between managing some resources via puppet and some by hand with pcs commands is error-prone and cumbersome. > A few months ago, we initiated a discussion about writing a Puppet > Resource Service Provider in openstack/puppet-pacemaker that will deal > with service management. > What will it change? > * we will enable service management again in THT. > * Puppet won't use systemd to deal with services, but use 'pcs' tool. > * We'll take the benefit of puppet-* modules orchestration. Example: > this configuration change will restart this service (something we had > hard time to do before). > > We are currently working on it: > https://review.openstack.org/#/c/286124/ > > But this is work in progress: > * we need to implement the restart > * we need to create a special Puppet fact, that will make sure we only > run pcs on a single node (master by preference) > * investigate how to find who is master and stop hardcoding it in THT > (good for a bootstrap, but not good for upgrades). > > Here is the kind of scenario we expect at the end: > > Deployment of Neutron Server service > * openstack/puppet-neutron will configure neutron.conf > * openstack/puppet-neutron will notify neutron-server service > * THT is overriding the neutron-server service resource to use 'pcs' > provider, implemented in openstack/puppet-pacemaker > * once neutron.conf is configured on controllers, pcs will start > neutron-server, only from one node. > > Upgrade of Neutron Server service > * we change RabbitMQ parameters in neutron.conf with THT > * openstack/puppet-neutron will update neutron.conf and notify > neutron-server service > * 'pcs' provider will restart neutron-server from one node, after the > change in the neutron.conf > > That will be, in my opinion the most elegant way to make Puppet & > Pacemaker working together, any comment / feedback is highly welcome, > it will be a big change and we want to make it during Newton cycle. > Note: by designing this change, I found out we will also be able to > reduce the number of steps and also drop some bash code in the upgrade > scripts, which are both good news at making TripleO more consistent > and fast to deploy. While I haven't yet played with this changeset, my first question would be the following: In this pcs-provider scenario, do we care about the fact that if you stop 'resource A', also all the resourced that have 'A' as an ordering constraint will be stopped? Or should the users of the module care about this aspect? Thanks for working on this. I will try to look at the proposed changes this week and provide some feedback. cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo][fuel] puppet-pacemaker: collaboration is starting now
This is great news, thanks for working on this Emilien and everyone! On Tue, Mar 29, 2016 at 04:04:03PM -0400, Emilien Macchi wrote: > A few months ago, we moved redhat/puppet-pacemaker to > openstack/puppet-pacemaker for some reasons: > > * We wanted to take benefits from OpenStack Infra (Gerrit, Zuul, > Jenkins jobs) and improve testing coverage. > > Result: we succeed in here, changes in puppet-pacemaker no longer > break TripleO HA jobs, since we have the CI running for every patch. > Sofer is also doing an incredible work on testing at this time, with > beaker jobs and also make the module cleaner & testable. > > * TripleO is using this module and we saw an opportunity to share this > code with OpenStack community. > > Result: we recently had some conversations with Fuel folks on this ML > and on IRC, who also work on a puppet-pacemaker module, and they are > willing to merge both modules. > The collaboration is starting now: https://review.openstack.org/#/c/296440/ > > Some actions in progress: > * Move bits from fuel-infra/puppet-pacemaker to > openstack/puppet-pacemaker (see 296440) > * Adding Fuel CI running for patches in openstack/puppet-pacemaker > * Adding Beaker tests to run on Ubuntu > * Try to find an alternative to pcs for Ubuntu platform (pcs is not in > debian/ubuntu) > * Investigate if we can follow Fuel's moduel where XML is used instead of PCS. This is unlikely to fly for RHEL-based distros as pcs is the only officially supported way of interacting with a pacemaker cluster. I think in the short-term it makes sense to somehow have both mechanisms in place (is this even possible?) and then once 'pcs' hits the stable debian-derived distros to invest in that direction. > Some requirements: > * Work will be done by iterated and test driven, thanks to beaker > tests and Fuel CI / TripleO CI. > * We need to converge a maximum of resources, when we can, but still > keep a feature parity for both Fuel & TripleO installers. > > Feel free to jump in this work / conversation if you are involved in > TripleO / Fuel / or interested by this module, we're doing this the > open way. I am definitely interested in this work. I will start looking at the pcs provider changes and hopefully bring some feedback. If you do have any questions on the pacemaker side of things, please do reach out. cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo] [puppet] move puppet-pacemaker
On Wed, Mar 23, 2016 at 09:58:17AM +0100, Sofer Athlan-Guyot wrote: > > Can I change the interface of pcmk_resource? > > > > You have pcmk_constraint but I have pcmk_location/colocation/order > > separately. I can merge then into a single resource like you did > > or I can keep them separated. Or I can make both. Actually they are > > different enough to be separated. > > Yes, I think that you've got it right, and separating the resource seems > to be cleaner. Agreed. The resources have fairly different semantics that keeping them separate makes more sense. E.g. the following pattern cannot be expressed cleanly today with the openstack puppet-pacemaker module (at least a couple of months ago, haven't looked recently): pcs constraint location foores rule resource-discovery=exclusive score=0 nodetag eq foonode Not sure if the above would work with the fuel module. > > Will I have to develop 'pcs' style provider for every resource? Do we > > really need them? > > 'pcs' command is the way to go on rhel platform. It's the interface of > choice to the pacemaker interface. So, I would say yes. I can help > here. Also note that pcs did make it to debian unstable a few months ago, so in the mid-long term, this interface will be relevant for Debian-derived distros as well. cheers, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] Fencing configuration
Hi all,
currently in order to enable automatic fencing on the controllers we
need to pass something like the following yaml to an overcloud deploy:
"""
EnableFencing: true
FencingConfig:
{
"devices": [
{
"agent": "fence_xvm",
"host_mac": "52:54:00:2d:bb:38",
"params": {
"multicast_address": "225.0.0.12",
"port": "osp8-node1"
}
},
{
"agent": "fence_xvm",
"host_mac": "52:54:00:e9:f4:a8",
"params": {
"multicast_address": "225.0.0.12",
"port": "osp8-node2"
}
},
"""
the problem with this approach is two-fold:
1) The stonith resources will be named something like
"stonith-xvm-5254002dbb38".
This is rather suboptimal for a sysadmin as it is really important to
know which node is actually behind that stonith device without looking
at a db containing mac addresses<->node-name. Both for troubleshooting
purposes and for monitoring the health of the cluster.
2) While trying to build a template to configure instance HA, which also
requires the computing nodes to be fenced, the current implementation is
not really workable, because it assumes that each node has the pcs
command and it will basically check that the node where puppet runs
matches a macaddress in the FencingConfig table and will create the
stonith class in such a case. Compute nodes cannot invoke the pcs command
so the stonith devices for them need to be created on a controller and
they need the fencing information (node name + fencing info)
Jiri Stransky and I discussed this a bit and thought that it would be
best to bring it on the ML first to see if other people have opinions on
how to tackle this problem. Ideally we would have the FencingConfig info
above amended with the hostname of the node and then we could implement
the fencing for controllers + compute nodes in one of the steps in
overcloud_controller_pacemaker.pp.
Right now, one approach I am toying with is by tweaking
extraconfig/all_nodes/mac_hostname.yaml and then on a controller
parse the macaddresses + hostname and execute the pcs stonith commands
on a controller. It's quite hacky though, so am looking for other input
on this.
cheers,
Michele
--
Michele Baldessari
C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] Mitaka Milestone 2
Hi John, fully agreed on your proposal. On Mon, Jan 18, 2016 at 10:59:03AM -0500, John Trowbridge wrote: > I have been tracking the issues when deploying with current from all > other projects[2] (as opposed to using an old pinned delorean for > non-tripleo projects). This has been a rapidly moving target (that we > have yet to hit for Mitaka), since we are not doing this in CI (which I > totally get the reasons for). > > I think if we want to have TripleO working with the rest of OpenStack's > Mitaka milestone 2, we will need to prioritize resolving the outstanding > issues this week. I would love to see us not merge anything that is not > related to either adding some validation of the deployed overcloud or > fixing some issue related to deploying with delorean current for all > packages. I gave this a shot and after manually installing python-gnocchiclient in the undercloud (BZ https://bugzilla.redhat.com/show_bug.cgi?id=1300013) I got to the overcloud deployment. > Another benefit, would be that if we get RDO CI testing TripleO as part > of our delorean promotion process, TripleO will be able to use the > automated current-passed-ci link instead of the manual current-tripleo > link. It will then be much easier to trace issues close to when they are > introduced rather than having a huge number of commits to comb through, > with many issues happening concurrently. > > - trown > > [1] http://docs.openstack.org/releases/schedules/mitaka.html > [2] https://etherpad.openstack.org/p/delorean_master_current_issues I have added the issue I have seen on the etherpad with full details. hth, Michele -- Michele Baldessari C2A5 9DA3 9961 4FFB E01B D0BC DDD4 DCCB 7515 5C6D __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
