Re: [openstack-dev] [TripleO] pacemaker management tools
Hello, cibadmin sounds like a good idea, agree with this approach. Thanks for feedback. Jan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] pacemaker management tools
Hello, guy! I've just saw your thread and I have something to say about your topic. > What management tools are there? The old time pacemaker users of course will name crm shell first (crmsh package). It allows interacive configuration by enetering commands like this: > crm configure primitive test1 ocf:pacemaker:Dummy > crm configure primitive test2 ocf:pacemaker:Dummy > crm configure primitive test3 ocf:pacemaker:Dummy > crm configure colocatin test2_with_test1 100: test2 test1 > crm configure order test3_after_test2 200: test2 test3 > crm status Or by using editor: > crm configure edit There is also crm status and other usefull stuff. This days crm is not developed anymore and considered unsupported and depricated. Most distributions will not have it packaged at all. The newer configuration tools is pcs (pcs package). It allows intercative configuration and status monitoring too. > pcs resource create test1 ocf:pacemaker:Dummy > pcs resource create test2 ocf:pacemaker:Dummy > pcs resource create test3 ocf:pacemaker:Dummy > pcs constraint colocation add test2 test1 100 > pcs status But it lacks crm's interctive shell and very convinient editor featutres. Pcs should be included in all moders distributions. There are also basic tools written in C that come together with pacemaker itself and any sane distribution should include them. https://github.com/ClusterLabs/pacemaker/tree/master/tools Some of them can be very convinient even for intercative use. Both pcs and crm are just python wrappers that call basic C tools as a backend or parse XML cib. > What options do we have to generate and/or manage pacemaker configuration? In most cases it boils down to either adding and removing configuration elements during the installation or the runtime or to importing of the precreated configuration. YOur scripts can use crm/pcs to create primitives and constraints one by one as would human do. Or you can describe the entire configuration and then just import it. Of couse you can always write pcs/crm calls to a shell script and just run it. crm can even run batch chages in one transaction like this: config.crm: configure property stonith-enabled=false property no-quorum-policy=ignore primitive test1 ocf:pacemaker:Dummy primitive test2 ocf:pacemaker:Dummy primitive test3 ocf:pacemaker:Dummy colocation test2_with_test1 100: test2 test1 order test3_after_test2 200: test2 test3 commit then crm -f config.crm use --force if needed pcs has no single transaction update capabilities but you can use shell script and shadow/commit if you really want transaction The other solution would be to import precreated XML file as a patch: If we can somehow generate such a file it can be easily applied like this: cibadmin --patch --xml-file=patch.xml You can also use crm_diff to apply xml patches manually. I think, that for TripleO if you want import precreated configuration XML is a way to go. You will not depend on any python wrappers like pcs and crm and will be able to create any possible configuration. XML allows use of XSLT transformations if you are creative enough of can be just generated by template or written manually. 2014-06-20 0:03 GMT+04:00 Mike Scherbakov : > Anything we can take out of here for our HA fixes? May be we want to > participate in the thread? > > -- Forwarded message -- > From: Howley, Tom > Date: Wed, Jun 18, 2014 at 1:31 PM > Subject: Re: [openstack-dev] [TripleO] pacemaker management tools > To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > > > Jan/Adam, > > Is cibadmin available in the different distros? This can be used to update > the CIB based on XML description of full pacemaker config. I have used this > on ubuntu in the past and found it more reliable than using crm commands > for automated deployment/configuration of pacemaker clusters. It also has > patch facility, which I haven't used. > > I wouldn't have assumed that the pacemaker config needed to be a static > file baked into an image. If cibadmin is an option, the different elements > requiring pacemaker control could supply their relevant XML snippets (based > off config values supplied via heat) and a pacemaker/pacemaker-config > element could apply those XML configs to the running cluster (with checks > for resource naming clashes, etc.) Does that sound like a possible approach? > > Tom > > -Original Message----- > From: Jan Provaznik [mailto:jprov...@redhat.com] > Sent: 13 June 2014 13:53 > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [TripleO] pacemaker management tools > > On 06/
Re: [openstack-dev] [TripleO] pacemaker management tools
Jan/Adam, Is cibadmin available in the different distros? This can be used to update the CIB based on XML description of full pacemaker config. I have used this on ubuntu in the past and found it more reliable than using crm commands for automated deployment/configuration of pacemaker clusters. It also has patch facility, which I haven't used. I wouldn't have assumed that the pacemaker config needed to be a static file baked into an image. If cibadmin is an option, the different elements requiring pacemaker control could supply their relevant XML snippets (based off config values supplied via heat) and a pacemaker/pacemaker-config element could apply those XML configs to the running cluster (with checks for resource naming clashes, etc.) Does that sound like a possible approach? Tom -Original Message- From: Jan Provaznik [mailto:jprov...@redhat.com] Sent: 13 June 2014 13:53 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [TripleO] pacemaker management tools On 06/12/2014 09:37 PM, Adam Gandelman wrote: > It's been a while since I've used these tools and I'm not 100% surprised > they've fragmented once again. :) That said, does pcs support creating > the CIB configuration in bulk from a file? I know that crm shell would > let you dump the entire cluster config and restore from file. Unless > the CIB format has differs now, couldn't we just create the entire thing > first and use a single pcs or crm command to import it to the cluster, > rather than building each resource command-by-command? > That is an interesting idea. But I'm afraid that this can't be used in TripleO use-case. We would have to keep the whole cluster definition as a static file which would be included when building overcloud image. Keeping this static definition up-to-date sounds like a complex task. Also this would make impossible any customization based on used elements. For example if there are 2 elements which use pacemaker - neutron-l3-agent and ceilometer-agent-central, then I couldn't use them separately. Jan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] pacemaker management tools
On 06/12/2014 09:37 PM, Adam Gandelman wrote: It's been a while since I've used these tools and I'm not 100% surprised they've fragmented once again. :) That said, does pcs support creating the CIB configuration in bulk from a file? I know that crm shell would let you dump the entire cluster config and restore from file. Unless the CIB format has differs now, couldn't we just create the entire thing first and use a single pcs or crm command to import it to the cluster, rather than building each resource command-by-command? That is an interesting idea. But I'm afraid that this can't be used in TripleO use-case. We would have to keep the whole cluster definition as a static file which would be included when building overcloud image. Keeping this static definition up-to-date sounds like a complex task. Also this would make impossible any customization based on used elements. For example if there are 2 elements which use pacemaker - neutron-l3-agent and ceilometer-agent-central, then I couldn't use them separately. Jan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] pacemaker management tools
It's been a while since I've used these tools and I'm not 100% surprised they've fragmented once again. :) That said, does pcs support creating the CIB configuration in bulk from a file? I know that crm shell would let you dump the entire cluster config and restore from file. Unless the CIB format has differs now, couldn't we just create the entire thing first and use a single pcs or crm command to import it to the cluster, rather than building each resource command-by-command? Adam On Wed, Jun 11, 2014 at 4:28 AM, Jan Provazník wrote: > Hi, > ceilometer-agent-central element was added recently into overcloud image. > To be able scale out overcloud control nodes, we need HA for this central > agent. Currently central agent can not scale out (until [1] is done). For > now, the simplest way is add the central agent to Pacemaker, which is quite > simple. > > The issue is that distributions supported in TripleO provide different > tools for managing Pacemaker. Ubuntu/Debian provides crmsh, Fedora/RHEL > provides pcs, OpenSuse provides both. I didn't find packages for all our > distros for any of the tools. Also if there is a third-party repo providing > packages for various distros, adding dependency on an untrusted third-party > repo might be a problem for some users. > > Although it's a little bit annoying, I think we will end up with managing > commands for both config tools, a resource creation sample: > > if $USE_PCS;then > crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params > ip=192.168.122.120 cidr_netmask=32 op monitor interval=30s > else > pcs resource create ClusterIP IPaddr2 ip=192.168.0.120 cidr_netmask=32 > fi > > There are not many places where pacemaker configuration would be required, > so I think this is acceptable. Any other opinions? > > Jan > > > [1] https://blueprints.launchpad.net/ceilometer/+spec/central- > agent-improvement > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] pacemaker management tools
On 06/12/2014 12:17 AM, Gregory Haynes wrote: The issue is that distributions supported in TripleO provide different tools for managing Pacemaker. Ubuntu/Debian provides crmsh, Fedora/RHEL provides pcs, OpenSuse provides both. I didn't find packages for all our distros for any of the tools. Also if there is a third-party repo providing packages for various distros, adding dependency on an untrusted third-party repo might be a problem for some users. Although it's a little bit annoying, I think we will end up with managing commands for both config tools, a resource creation sample: if $USE_PCS;then crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.122.120 cidr_netmask=32 op monitor interval=30s else pcs resource create ClusterIP IPaddr2 ip=192.168.0.120 cidr_netmask=32 fi This seems like a reasonable solution if we can ensure that we have CI for both branches of the installation. This is a big issue with our current mariadb/percona installation and it sounds like were heading down the same path here. With mariadb/percona it's slightly different as these are separate elements and each of them works on all distros. With mariadb I was waiting until galera server is merged into Fedora (which already is), so it's a good time now to make it default for Fedora setup (a patch for this will be submitted soon). If we can make USE_PCS directly dependent on the installed distro that would be sufficient (CI for each distro would take care of the different branches) but this gets a bit more complicated if you want to split crm and pcs out into different elements (like in mariadb vs percona)... Yes, usage of pcs is directly dependent on distro. Pcs/crm wouldn't be splitted into separate elements, it's just about calling 2 different commands in an os-refresh-config script depending on distribution (so we are sure both are tested). Jan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [TripleO] pacemaker management tools
> The issue is that distributions supported in TripleO provide different > tools for managing Pacemaker. Ubuntu/Debian provides crmsh, Fedora/RHEL > provides pcs, OpenSuse provides both. I didn't find packages for all our > distros for any of the tools. Also if there is a third-party repo > providing packages for various distros, adding dependency on an > untrusted third-party repo might be a problem for some users. > > Although it's a little bit annoying, I think we will end up with > managing commands for both config tools, a resource creation sample: > > if $USE_PCS;then >crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params > ip=192.168.122.120 cidr_netmask=32 op monitor interval=30s > else >pcs resource create ClusterIP IPaddr2 ip=192.168.0.120 cidr_netmask=32 > fi This seems like a reasonable solution if we can ensure that we have CI for both branches of the installation. This is a big issue with our current mariadb/percona installation and it sounds like were heading down the same path here. If we can make USE_PCS directly dependent on the installed distro that would be sufficient (CI for each distro would take care of the different branches) but this gets a bit more complicated if you want to split crm and pcs out into different elements (like in mariadb vs percona)... -- Gregory Haynes g...@greghaynes.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [TripleO] pacemaker management tools
Hi, ceilometer-agent-central element was added recently into overcloud image. To be able scale out overcloud control nodes, we need HA for this central agent. Currently central agent can not scale out (until [1] is done). For now, the simplest way is add the central agent to Pacemaker, which is quite simple. The issue is that distributions supported in TripleO provide different tools for managing Pacemaker. Ubuntu/Debian provides crmsh, Fedora/RHEL provides pcs, OpenSuse provides both. I didn't find packages for all our distros for any of the tools. Also if there is a third-party repo providing packages for various distros, adding dependency on an untrusted third-party repo might be a problem for some users. Although it's a little bit annoying, I think we will end up with managing commands for both config tools, a resource creation sample: if $USE_PCS;then crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 params ip=192.168.122.120 cidr_netmask=32 op monitor interval=30s else pcs resource create ClusterIP IPaddr2 ip=192.168.0.120 cidr_netmask=32 fi There are not many places where pacemaker configuration would be required, so I think this is acceptable. Any other opinions? Jan [1] https://blueprints.launchpad.net/ceilometer/+spec/central-agent-improvement ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
