Re: [Pacemaker] cib: ERROR: send_ais_message: Not connected to AIS
On Mon, 14 Apr 2014 14:40:43 +1000 Andrew Beekhof and...@beekhof.net wrote: On 11 Apr 2014, at 10:54 pm, Marco Felettigh ma...@nucleus.it wrote: On Fri, 11 Apr 2014 17:17:57 +1000 Andrew Beekhof and...@beekhof.net wrote: On 8 Apr 2014, at 8:37 pm, ma...@nucleus.it wrote: On Tue, 8 Apr 2014 10:49:16 +1000 Andrew Beekhof and...@beekhof.net wrote: On 7 Apr 2014, at 8:46 pm, ma...@nucleus.it wrote: Hi, in a production environment with 2 nodes ( nodeA , nodeB ) we had an hardware failure so we restart the nodeB. After the restarted nodeB came up we restart corosync/pacemaker on it but for 2 days till now che corosync/pacemaker stuff is looping. crm_mon NodeA: Stack: openais Current DC: nodeA - partition with quorum Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 17 Resources configured. Online: [ nodeA ] OFFLINE: [ nodeB ] crm_mon NodeB: Stack: openais Current DC: NONE 2 Nodes configured, 2 expected votes 17 Resources configured. OFFLINE: [ nodeA nodeB ] This loop on nodeB reports: crmd: [7149]: debug: do_election_count_vote: Election 3 (owner: nodeA) lost: vote from nodeA (Age) So investigating around i found these message on nodeA: cib: [28755]: ERROR: send_ais_message: Not connected to AIS now this message is repeating for every operation. Is it a corosync problem or a cib/pacemaker one ? Any suggestion on what is happened ? For some reason the cib can't connect to corosync anymore. No software got upgraded recently? Are there any logs from corosync? Which distro is this? And why the start of a cluster node crasched the DC suff ? :( Bye Marco ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org Hi, the distro in an opensuse 11.1 and there is no updates also because the distro is out of maintenance. A good reason to be using SLES (or RHEL/CentOS). Better Gentoo ;) We are planning and upgrade but the interesting thing is to figure out the reasons of the problem. The log in attachment, thanks for the support There's nothing obvious in the logs. Just that as far as pacemaker could tell, corosync suddenly went away. Was the corosync process still running? Yes , corosync was still running . Stopping pacemaker and restarting it didnt help? At the end we restarted the two server and then start the corosync/pacemaker stuff. Thanks for the support Marco ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] cib: ERROR: send_ais_message: Not connected to AIS
On 11 Apr 2014, at 10:54 pm, Marco Felettigh ma...@nucleus.it wrote: On Fri, 11 Apr 2014 17:17:57 +1000 Andrew Beekhof and...@beekhof.net wrote: On 8 Apr 2014, at 8:37 pm, ma...@nucleus.it wrote: On Tue, 8 Apr 2014 10:49:16 +1000 Andrew Beekhof and...@beekhof.net wrote: On 7 Apr 2014, at 8:46 pm, ma...@nucleus.it wrote: Hi, in a production environment with 2 nodes ( nodeA , nodeB ) we had an hardware failure so we restart the nodeB. After the restarted nodeB came up we restart corosync/pacemaker on it but for 2 days till now che corosync/pacemaker stuff is looping. crm_mon NodeA: Stack: openais Current DC: nodeA - partition with quorum Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 17 Resources configured. Online: [ nodeA ] OFFLINE: [ nodeB ] crm_mon NodeB: Stack: openais Current DC: NONE 2 Nodes configured, 2 expected votes 17 Resources configured. OFFLINE: [ nodeA nodeB ] This loop on nodeB reports: crmd: [7149]: debug: do_election_count_vote: Election 3 (owner: nodeA) lost: vote from nodeA (Age) So investigating around i found these message on nodeA: cib: [28755]: ERROR: send_ais_message: Not connected to AIS now this message is repeating for every operation. Is it a corosync problem or a cib/pacemaker one ? Any suggestion on what is happened ? For some reason the cib can't connect to corosync anymore. No software got upgraded recently? Are there any logs from corosync? Which distro is this? And why the start of a cluster node crasched the DC suff ? :( Bye Marco ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org Hi, the distro in an opensuse 11.1 and there is no updates also because the distro is out of maintenance. A good reason to be using SLES (or RHEL/CentOS). Better Gentoo ;) We are planning and upgrade but the interesting thing is to figure out the reasons of the problem. The log in attachment, thanks for the support There's nothing obvious in the logs. Just that as far as pacemaker could tell, corosync suddenly went away. Was the corosync process still running? Yes , corosync was still running . Stopping pacemaker and restarting it didnt help? signature.asc Description: Message signed with OpenPGP using GPGMail ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] cib: ERROR: send_ais_message: Not connected to AIS
On Fri, 11 Apr 2014 17:17:57 +1000 Andrew Beekhof and...@beekhof.net wrote: On 8 Apr 2014, at 8:37 pm, ma...@nucleus.it wrote: On Tue, 8 Apr 2014 10:49:16 +1000 Andrew Beekhof and...@beekhof.net wrote: On 7 Apr 2014, at 8:46 pm, ma...@nucleus.it wrote: Hi, in a production environment with 2 nodes ( nodeA , nodeB ) we had an hardware failure so we restart the nodeB. After the restarted nodeB came up we restart corosync/pacemaker on it but for 2 days till now che corosync/pacemaker stuff is looping. crm_mon NodeA: Stack: openais Current DC: nodeA - partition with quorum Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 17 Resources configured. Online: [ nodeA ] OFFLINE: [ nodeB ] crm_mon NodeB: Stack: openais Current DC: NONE 2 Nodes configured, 2 expected votes 17 Resources configured. OFFLINE: [ nodeA nodeB ] This loop on nodeB reports: crmd: [7149]: debug: do_election_count_vote: Election 3 (owner: nodeA) lost: vote from nodeA (Age) So investigating around i found these message on nodeA: cib: [28755]: ERROR: send_ais_message: Not connected to AIS now this message is repeating for every operation. Is it a corosync problem or a cib/pacemaker one ? Any suggestion on what is happened ? For some reason the cib can't connect to corosync anymore. No software got upgraded recently? Are there any logs from corosync? Which distro is this? And why the start of a cluster node crasched the DC suff ? :( Bye Marco ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org Hi, the distro in an opensuse 11.1 and there is no updates also because the distro is out of maintenance. A good reason to be using SLES (or RHEL/CentOS). Better Gentoo ;) We are planning and upgrade but the interesting thing is to figure out the reasons of the problem. The log in attachment, thanks for the support There's nothing obvious in the logs. Just that as far as pacemaker could tell, corosync suddenly went away. Was the corosync process still running? Yes , corosync was still running . ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] cib: ERROR: send_ais_message: Not connected to AIS
Hi, in a production environment with 2 nodes ( nodeA , nodeB ) we had an hardware failure so we restart the nodeB. After the restarted nodeB came up we restart corosync/pacemaker on it but for 2 days till now che corosync/pacemaker stuff is looping. crm_mon NodeA: Stack: openais Current DC: nodeA - partition with quorum Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 17 Resources configured. Online: [ nodeA ] OFFLINE: [ nodeB ] crm_mon NodeB: Stack: openais Current DC: NONE 2 Nodes configured, 2 expected votes 17 Resources configured. OFFLINE: [ nodeA nodeB ] This loop on nodeB reports: crmd: [7149]: debug: do_election_count_vote: Election 3 (owner: nodeA) lost: vote from nodeA (Age) So investigating around i found these message on nodeA: cib: [28755]: ERROR: send_ais_message: Not connected to AIS now this message is repeating for every operation. Is it a corosync problem or a cib/pacemaker one ? Any suggestion on what is happened ? And why the start of a cluster node crasched the DC suff ? :( Bye Marco signature.asc Description: PGP signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] cib: ERROR: send_ais_message: Not connected to AIS
On 7 Apr 2014, at 8:46 pm, ma...@nucleus.it wrote: Hi, in a production environment with 2 nodes ( nodeA , nodeB ) we had an hardware failure so we restart the nodeB. After the restarted nodeB came up we restart corosync/pacemaker on it but for 2 days till now che corosync/pacemaker stuff is looping. crm_mon NodeA: Stack: openais Current DC: nodeA - partition with quorum Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3 2 Nodes configured, 2 expected votes 17 Resources configured. Online: [ nodeA ] OFFLINE: [ nodeB ] crm_mon NodeB: Stack: openais Current DC: NONE 2 Nodes configured, 2 expected votes 17 Resources configured. OFFLINE: [ nodeA nodeB ] This loop on nodeB reports: crmd: [7149]: debug: do_election_count_vote: Election 3 (owner: nodeA) lost: vote from nodeA (Age) So investigating around i found these message on nodeA: cib: [28755]: ERROR: send_ais_message: Not connected to AIS now this message is repeating for every operation. Is it a corosync problem or a cib/pacemaker one ? Any suggestion on what is happened ? For some reason the cib can't connect to corosync anymore. No software got upgraded recently? Are there any logs from corosync? Which distro is this? And why the start of a cluster node crasched the DC suff ? :( Bye Marco ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org signature.asc Description: Message signed with OpenPGP using GPGMail ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] cib connection error
Hi all, I use pacemaker 1.1.9 with corosync 2.3 both built from source. My OS is CentOS 6.4 x86_64 I have about 30 resources of one type managed by my own resource agent. It is nesessary for the resource agent to know utilization parameter of the configured resource. I query for this parameter by crm_resource utility in the start function of the RA. After I had implemented this feature, I got a lot of error's in my logs: Sep 23 19:19:47 iblade5 lrmd[7492]: notice: operation_finished: RESOURCE_start_0:8445:stderr [ Could not establish cib_rw connection: Resource temporarily unavailable (11) ] Sep 23 19:19:47 iblade5 lrmd[7492]: notice: operation_finished: RESOURCE_start_0:8445:stderr [ Error signing on to the CIB service: Transport endpoint is not connected ] So, only few resources (about 4 or 5), every time different, start correctly (crm_resource correctly returns the needed value during start action). And all other resources fail to start. I think there is a problem when many (20-30) resources start at the same time, and there are 20-30 queries to CIB from the resource agents How can I correct this ? Thank you in advance, Ivan Khalezov ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] cib connection error
On 24/09/2013, at 2:09 AM, Халезов Иван i.khale...@rts.ru wrote: Hi all, I use pacemaker 1.1.9 with corosync 2.3 both built from source. My OS is CentOS 6.4 x86_64 I have about 30 resources of one type managed by my own resource agent. It is nesessary for the resource agent to know utilization parameter of the configured resource. I query for this parameter by crm_resource utility in the start function of the RA. After I had implemented this feature, I got a lot of error's in my logs: Sep 23 19:19:47 iblade5 lrmd[7492]: notice: operation_finished: RESOURCE_start_0:8445:stderr [ Could not establish cib_rw connection: Resource temporarily unavailable (11) ] Sep 23 19:19:47 iblade5 lrmd[7492]: notice: operation_finished: RESOURCE_start_0:8445:stderr [ Error signing on to the CIB service: Transport endpoint is not connected ] So, only few resources (about 4 or 5), every time different, start correctly (crm_resource correctly returns the needed value during start action). And all other resources fail to start. I think there is a problem when many (20-30) resources start at the same time, and there are 20-30 queries to CIB from the resource agents How can I correct this ? I recall talking to NTT about this recently but I forget what they did to make progress. Perhaps you could look for $?=11 and try again. I _think_ there might have been a patch for libqb that resolved it. signature.asc Description: Message signed with OpenPGP using GPGMail ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
Hi, On Wed, 23 Jan 2013 18:52:20 +0100 Dejan Muhamedagic deja...@fastmail.fm wrote: nodes node id=35956928 uname=sipc2n2 Note sure if id can start with a digit. Corosync node id's are always digits-only. This should really work with versions = v1.2.4 Yeah… I have looked into the crmsh code and it has explicit support for node 'type' attribute in Pacemaker 1.1.8. For some reason this does not work for me on this cluster (no such problems on another cluster, which was not upgraded, but set up on Pacemaker 1.1 from the beginning). Which schema do you validate against? Look for the validate-with attribute of the cib element. validate-with=pacemaker-1.0 element name=node attribute name=idtext//attribute attribute name=unametext//attribute attribute name=type choice valuenormal/value valuemember/value valueping/value /choice /attribute So no, it is not optional here. But it is optional in the pacemaker-1.1 schema. So the problem is crmsh uses the wrong schema for the XML it generates… # cibadmin -Q | grep validate-with cib validate-with=pacemaker-1.0 crm_feature_set=3.0.6 have-quorum=1 admin_epoch=0 epoch=337 num_updates=158 cib-last-written=Wed Jan 23 15:23:22 2013 dc-uuid=19179712 So, the 'validate-with=pacemaker-1.0' comes from the current CIB. crmsh keeps that, but generates Pacemaker 1.1 XML, so the verification fails. I should probably upgrade my CIB somehow, but still it seems there is a bug in crmsh. Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
On Thu, 24 Jan 2013 09:04:14 +0100 Jacek Konieczny jaj...@jajcus.net wrote: I should probably upgrade my CIB somehow Indeed. 'cibadmin --upgrade --force' solved my problem. Thanks for all the hints. Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
On Thu, Jan 24, 2013 at 09:04:14AM +0100, Jacek Konieczny wrote: Hi, On Wed, 23 Jan 2013 18:52:20 +0100 Dejan Muhamedagic deja...@fastmail.fm wrote: nodes node id=35956928 uname=sipc2n2 Note sure if id can start with a digit. Corosync node id's are always digits-only. This should really work with versions = v1.2.4 Yeah… I have looked into the crmsh code and it has explicit support for node 'type' attribute in Pacemaker 1.1.8. For some reason this does not work for me on this cluster (no such problems on another cluster, which was not upgraded, but set up on Pacemaker 1.1 from the beginning). Which schema do you validate against? Look for the validate-with attribute of the cib element. validate-with=pacemaker-1.0 element name=node attribute name=idtext//attribute attribute name=unametext//attribute attribute name=type choice valuenormal/value valuemember/value valueping/value /choice /attribute So no, it is not optional here. But it is optional in the pacemaker-1.1 schema. So the problem is crmsh uses the wrong schema for the XML it generates… # cibadmin -Q | grep validate-with cib validate-with=pacemaker-1.0 crm_feature_set=3.0.6 have-quorum=1 admin_epoch=0 epoch=337 num_updates=158 cib-last-written=Wed Jan 23 15:23:22 2013 dc-uuid=19179712 So, the 'validate-with=pacemaker-1.0' comes from the current CIB. crmsh keeps that, but generates Pacemaker 1.1 XML, so the verification fails. I should probably upgrade my CIB somehow, but still it seems there is a bug in crmsh. crmsh relies in this case on the pacemaker version. It should check the schema, but currently the schema support is somewhat lacking, i.e. it's not possible to get information on whether this particular attribute is optional or not. Thanks, Dejan Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
On Thu, Jan 24, 2013 at 09:10:33AM +0100, Jacek Konieczny wrote: On Thu, 24 Jan 2013 09:04:14 +0100 Jacek Konieczny jaj...@jajcus.net wrote: I should probably upgrade my CIB somehow Indeed. 'cibadmin --upgrade --force' solved my problem. Thanks for all the hints. crm(live)configure# help upgrade If you get the `CIB not supported` error, which typically means that the current CIB version is coming from the older release, you may try to upgrade it to the latest revision. The command to perform the upgrade is: ... I knew it was somewhere. Thanks, Dejan Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] CIB verification failure with any change via crmsh
Hi, I have recently upgraded Pacemaker on one of my clusters from 1.0.something to 1.1.8 and installed crmsh to manage it as I used to. crmsh mostly works for me, until I try to change the configuration with 'crm configure'. Any, even trivial change shows verification errors and fails to commit: crm(live)configure# commit element instance_attributes: Relax-NG validity error : Expecting an element nvpair, got nothing element node: Relax-NG validity error : Expecting an element instance_attributes, got nothing element node: Relax-NG validity error : Element nodes has extra content: node element configuration: Relax-NG validity error : Invalid sequence in interleave element instance_attributes: Relax-NG validity error : Element node failed to validate attributes element cib: Relax-NG validity error : Element cib failed to validate content error: main: CIB did not pass DTD/schema validation Errors found during check: config not valid -V may provide more details Do you still want to commit? no It seems as crmsh fails to parse current configuration properly, as: crm configure save xml /tmp/saved.xml ; crm_verify -V --xml-file /tmp/saved.xml fails the same way: /tmp/saved.xml:19: element instance_attributes: Relax-NG validity error : Expecting an element nvpair, got nothing /tmp/saved.xml:18: element node: Relax-NG validity error : Expecting an element instance_attributes, got nothing /tmp/saved.xml:18: element node: Relax-NG validity error : Element nodes has extra content: node /tmp/saved.xml:3: element configuration: Relax-NG validity error : Invalid sequence in interleave /tmp/saved.xml:19: element instance_attributes: Relax-NG validity error : Element node failed to validate attributes /tmp/saved.xml:2: element cib: Relax-NG validity error : Element cib failed to validate content error: main: CIB did not pass DTD/schema validation Errors found during check: config not valid -V may provide more details But: cibadmin -Q /tmp/good.xml ; crm_verify --xml-file shows no error. Any ideas? Looking into the 'invalid' XML file gives me no hints, as the line 18 is the first node/ in: nodes node id=35956928 uname=sipc2n2 instance_attributes id=nodes-35956928 nvpair id=nodes-35956928-standby name=standby value=off/ /instance_attributes /node node id=19179712 uname=sipc2n1 instance_attributes id=nodes-19179712 nvpair id=nodes-19179712-standby name=standby value=off/ /instance_attributes /node /nodes which looks quite right too me. Oh… now I see the difference with the current cib. The node/ elements miss the type=normal attribute. After adding those to the crmsh-generated XML everything works. Then it is a crmsh bug, right? And the errors reported by crm_verify are very misleading. Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
On 2013-01-23T16:31:20, Jacek Konieczny jaj...@jajcus.net wrote: I have recently upgraded Pacemaker on one of my clusters from 1.0.something to 1.1.8 and installed crmsh to manage it as I used to. It'd be helpful if you mentioned which crmsh version you installed. The errors you get suggest you need to update it. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Experience is the name everyone gives to their mistakes. -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
On Wed, 23 Jan 2013 16:44:45 +0100 Lars Marowsky-Bree l...@suse.com wrote: On 2013-01-23T16:31:20, Jacek Konieczny jaj...@jajcus.net wrote: I have recently upgraded Pacemaker on one of my clusters from 1.0.something to 1.1.8 and installed crmsh to manage it as I used to. It'd be helpful if you mentioned which crmsh version you installed. The errors you get suggest you need to update it. You are right, I missed the information. It was crmsh 1.2.1 and the first thing I tried was an upgrade to 1.2.4, but this did not change a thing. So it is the same with crmsh 1.2.1 and crmsh 1.2.4. Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB verification failure with any change via crmsh
Hi, On Wed, Jan 23, 2013 at 04:31:20PM +0100, Jacek Konieczny wrote: Hi, I have recently upgraded Pacemaker on one of my clusters from 1.0.something to 1.1.8 and installed crmsh to manage it as I used to. crmsh mostly works for me, until I try to change the configuration with 'crm configure'. Any, even trivial change shows verification errors and fails to commit: crm(live)configure# commit element instance_attributes: Relax-NG validity error : Expecting an element nvpair, got nothing element node: Relax-NG validity error : Expecting an element instance_attributes, got nothing element node: Relax-NG validity error : Element nodes has extra content: node element configuration: Relax-NG validity error : Invalid sequence in interleave element instance_attributes: Relax-NG validity error : Element node failed to validate attributes element cib: Relax-NG validity error : Element cib failed to validate content error: main: CIB did not pass DTD/schema validation Errors found during check: config not valid -V may provide more details Do you still want to commit? no It seems as crmsh fails to parse current configuration properly, as: crm configure save xml /tmp/saved.xml ; crm_verify -V --xml-file /tmp/saved.xml fails the same way: /tmp/saved.xml:19: element instance_attributes: Relax-NG validity error : Expecting an element nvpair, got nothing /tmp/saved.xml:18: element node: Relax-NG validity error : Expecting an element instance_attributes, got nothing /tmp/saved.xml:18: element node: Relax-NG validity error : Element nodes has extra content: node /tmp/saved.xml:3: element configuration: Relax-NG validity error : Invalid sequence in interleave /tmp/saved.xml:19: element instance_attributes: Relax-NG validity error : Element node failed to validate attributes /tmp/saved.xml:2: element cib: Relax-NG validity error : Element cib failed to validate content error: main: CIB did not pass DTD/schema validation Errors found during check: config not valid -V may provide more details But: cibadmin -Q /tmp/good.xml ; crm_verify --xml-file shows no error. Any ideas? Looking into the 'invalid' XML file gives me no hints, as the line 18 is the first node/ in: nodes node id=35956928 uname=sipc2n2 Note sure if id can start with a digit. instance_attributes id=nodes-35956928 nvpair id=nodes-35956928-standby name=standby value=off/ /instance_attributes /node node id=19179712 uname=sipc2n1 instance_attributes id=nodes-19179712 nvpair id=nodes-19179712-standby name=standby value=off/ /instance_attributes /node /nodes which looks quite right too me. Oh… now I see the difference with the current cib. The node/ elements miss the type=normal attribute. After adding those to the crmsh-generated XML everything works. Then it is a crmsh bug, right? This should really work with versions = v1.2.4 Which schema do you validate against? Look for the validate-with attribute of the cib element. Does that schema support optional type attribute? Thanks, Dejan And the errors reported by crm_verify are very misleading. Greets, Jacek ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB not saved
Normally we log an error at startup if we can't write there... did this not happen? ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Ies, it happened. I saw a warning while writing CIB..but after I wrote in this mailing list :) Regards -- Fiorenza Meini Spazio Web S.r.l. V. Dante Alighieri, 10 - 13900 Biella Tel.: 015.2431982 - 015.9526066 Fax: 015.2522600 Reg. Imprese, CF e P.I.: 02414430021 Iscr. REA: BI - 188936 Iscr. CCIAA: Biella - 188936 Cap. Soc.: 30.000,00 Euro i.v. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] CIB not saved
Hi there, a strange thing happened to my two node cluster: I rebooted both machine at the same time, when s.o. went up again, no resources were configured anymore: as it was a fresh installation. Why ? It was explained to me that the configuration of resources managed by pacemaker should be in a file called cib.xml, but cannot find it in the system. Have I to specify any particular option in the configuration file? Thanks and regards -- Fiorenza Meini Spazio Web S.r.l. V. Dante Alighieri, 10 - 13900 Biella Tel.: 015.2431982 - 015.9526066 Fax: 015.2522600 Reg. Imprese, CF e P.I.: 02414430021 Iscr. REA: BI - 188936 Iscr. CCIAA: Biella - 188936 Cap. Soc.: 30.000,00 Euro i.v. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB not saved
On Thu, Mar 29, 2012 at 9:54 AM, Fiorenza Meini fme...@esseweb.eu wrote: Hi there, a strange thing happened to my two node cluster: I rebooted both machine at the same time, when s.o. went up again, no resources were configured anymore: as it was a fresh installation. Why ? It was explained to me that the configuration of resources managed by pacemaker should be in a file called cib.xml, but cannot find it in the system. Have I to specify any particular option in the configuration file? Normally you shouldn't worry about it. cib.xml is stored in /var/lib/heartbeat/crm/ or similar and the directory should have have hacluster:haclient permissions. What distro is it and how did you install it? Rasto -- Dipl.-Ing. Rastislav Levrinc rasto.levr...@gmail.com Linux Cluster Management Console http://lcmc.sf.net/ ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB not saved
Il 29/03/2012 10:12, Rasto Levrinc ha scritto: On Thu, Mar 29, 2012 at 9:54 AM, Fiorenza Meinifme...@esseweb.eu wrote: Hi there, a strange thing happened to my two node cluster: I rebooted both machine at the same time, when s.o. went up again, no resources were configured anymore: as it was a fresh installation. Why ? It was explained to me that the configuration of resources managed by pacemaker should be in a file called cib.xml, but cannot find it in the system. Have I to specify any particular option in the configuration file? Normally you shouldn't worry about it. cib.xml is stored in /var/lib/heartbeat/crm/ or similar and the directory should have have hacluster:haclient permissions. What distro is it and how did you install it? Rasto Thanks, it was a permission problems. Regards -- Fiorenza Meini Spazio Web S.r.l. V. Dante Alighieri, 10 - 13900 Biella Tel.: 015.2431982 - 015.9526066 Fax: 015.2522600 Reg. Imprese, CF e P.I.: 02414430021 Iscr. REA: BI - 188936 Iscr. CCIAA: Biella - 188936 Cap. Soc.: 30.000,00 Euro i.v. ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] CIB not saved
On Thu, Mar 29, 2012 at 8:45 PM, Fiorenza Meini fme...@esseweb.eu wrote: Il 29/03/2012 10:12, Rasto Levrinc ha scritto: On Thu, Mar 29, 2012 at 9:54 AM, Fiorenza Meinifme...@esseweb.eu wrote: Hi there, a strange thing happened to my two node cluster: I rebooted both machine at the same time, when s.o. went up again, no resources were configured anymore: as it was a fresh installation. Why ? It was explained to me that the configuration of resources managed by pacemaker should be in a file called cib.xml, but cannot find it in the system. Have I to specify any particular option in the configuration file? Normally you shouldn't worry about it. cib.xml is stored in /var/lib/heartbeat/crm/ or similar and the directory should have have hacluster:haclient permissions. What distro is it and how did you install it? Rasto Thanks, it was a permission problems. Normally we log an error at startup if we can't write there... did this not happen? ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] cib not connected
On Tue, Oct 25, 2011 at 4:08 AM, Proskurin Kirill k.prosku...@corp.mail.ru wrote: Hello. corosync-1.4.1 pacemaker-1.1.5 pacemaker runs with ver: 1 I run on strange problem. Hope someone can help me. I have 9 nodes cluster. All was fine till I need to reboot a node. After reboot it don`t want to come back to cluster with not in our membership error. I happens with other 2 nodes on this cluster. Network is fine. rm -rf /var/lib/heartbeat/crm/* not helps. I ask for help at IRC and we do this: I run one node with debug for few sec and I strace cib process. Both in links below. In debug logs we found cib not connected error but can`t understand reason of this. Debug logs: http://dl.dropbox.com/u/1932700/corosync.log.debug.gz cib strace: http://dl.dropbox.com/u/1932700/cib-starce.log.gz P.S. I have equal problem on other cluster and fix it with shutdown all nodes(corosync + pacemaker), rm -rf /var/lib/heartbeat/crm/* , startup all nodes. But it`s not really an option. :-) removing the contents of /var/lib/heartbeat/crm wont achieve much. its the restart thats getting corosync unstuck -- Best regards, Proskurin Kirill ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib
On Fri, Oct 1, 2010 at 3:45 PM, Shravan Mishra shravan.mis...@gmail.com wrote: Hi, Just a quick question, who generates the very first cib.xml when pacemaker processes are initialized? The cib Thanks Shravan On Thu, Sep 30, 2010 at 4:22 AM, Andrew Beekhof and...@beekhof.net wrote: On Tue, Sep 28, 2010 at 11:47 AM, Andrew Beekhof and...@beekhof.net wrote: On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra shravan.mis...@gmail.com wrote: Thanks Raoul for the response. Changing the permission to hacluster:haclient did stop that error. Now I'm hitting another problem whereby cib is failing to start Very strange logs. Which distribution is this? What does your corosync.conf look like? = Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has process list: 00110012 (1114130) Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has 1 quorum votes (was 0) Sep 27 00:16:29 corosync [pcmk ] info: send_member_notification: Sending membership update 100 to 0 children Sep 27 00:16:29 corosync [MAIN ] Completed service synchronization, ready to provide service. Sep 27 00:16:30 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14889, rc=127) Sep 27 00:16:30 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:30 corosync [pcmk ] info: spawn_child: Forked child 14896 for process cib crmd[14893]: 2010/09/27_00:16:30 WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry Sep 27 00:16:31 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14896, rc=127) Sep 27 00:16:31 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:31 corosync [pcmk ] info: spawn_child: Forked child 14901 for process cib Sep 27 00:16:32 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14901, rc=1 == I have attached the full logs. We are using corosync 1.2.8 and pacemaker 1.1.3. Thanks. Shravan On Sat, Sep 25, 2010 at 4:36 AM, Raoul Bhatia [IPAX] r.bha...@ipax.at wrote: On 24.09.2010 21:41, Shravan Mishra wrote: crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot change active directory to /var/lib/heartbeat/cores/hacluster: Permission denied (13) ls -ald /var/lib/heartbeat/cores/hacluster /var/lib/heartbeat/cores/ /var/lib/heartbeat/ /var/lib/ /var/ is haclient allowed to cd all the way into /var/lib/heartbeat/cores/hacluster ? cheers, ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib
Hi, Just a quick question, who generates the very first cib.xml when pacemaker processes are initialized? Thanks Shravan On Thu, Sep 30, 2010 at 4:22 AM, Andrew Beekhof and...@beekhof.net wrote: On Tue, Sep 28, 2010 at 11:47 AM, Andrew Beekhof and...@beekhof.net wrote: On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra shravan.mis...@gmail.com wrote: Thanks Raoul for the response. Changing the permission to hacluster:haclient did stop that error. Now I'm hitting another problem whereby cib is failing to start Very strange logs. Which distribution is this? What does your corosync.conf look like? = Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has process list: 00110012 (1114130) Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has 1 quorum votes (was 0) Sep 27 00:16:29 corosync [pcmk ] info: send_member_notification: Sending membership update 100 to 0 children Sep 27 00:16:29 corosync [MAIN ] Completed service synchronization, ready to provide service. Sep 27 00:16:30 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14889, rc=127) Sep 27 00:16:30 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:30 corosync [pcmk ] info: spawn_child: Forked child 14896 for process cib crmd[14893]: 2010/09/27_00:16:30 WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry Sep 27 00:16:31 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14896, rc=127) Sep 27 00:16:31 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:31 corosync [pcmk ] info: spawn_child: Forked child 14901 for process cib Sep 27 00:16:32 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14901, rc=1 == I have attached the full logs. We are using corosync 1.2.8 and pacemaker 1.1.3. Thanks. Shravan On Sat, Sep 25, 2010 at 4:36 AM, Raoul Bhatia [IPAX] r.bha...@ipax.at wrote: On 24.09.2010 21:41, Shravan Mishra wrote: crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot change active directory to /var/lib/heartbeat/cores/hacluster: Permission denied (13) ls -ald /var/lib/heartbeat/cores/hacluster /var/lib/heartbeat/cores/ /var/lib/heartbeat/ /var/lib/ /var/ is haclient allowed to cd all the way into /var/lib/heartbeat/cores/hacluster ? cheers, ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib
On Tue, Sep 28, 2010 at 11:47 AM, Andrew Beekhof and...@beekhof.net wrote: On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra shravan.mis...@gmail.com wrote: Thanks Raoul for the response. Changing the permission to hacluster:haclient did stop that error. Now I'm hitting another problem whereby cib is failing to start Very strange logs. Which distribution is this? What does your corosync.conf look like? = Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has process list: 00110012 (1114130) Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has 1 quorum votes (was 0) Sep 27 00:16:29 corosync [pcmk ] info: send_member_notification: Sending membership update 100 to 0 children Sep 27 00:16:29 corosync [MAIN ] Completed service synchronization, ready to provide service. Sep 27 00:16:30 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14889, rc=127) Sep 27 00:16:30 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:30 corosync [pcmk ] info: spawn_child: Forked child 14896 for process cib crmd[14893]: 2010/09/27_00:16:30 WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry Sep 27 00:16:31 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14896, rc=127) Sep 27 00:16:31 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:31 corosync [pcmk ] info: spawn_child: Forked child 14901 for process cib Sep 27 00:16:32 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14901, rc=1 == I have attached the full logs. We are using corosync 1.2.8 and pacemaker 1.1.3. Thanks. Shravan On Sat, Sep 25, 2010 at 4:36 AM, Raoul Bhatia [IPAX] r.bha...@ipax.at wrote: On 24.09.2010 21:41, Shravan Mishra wrote: crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot change active directory to /var/lib/heartbeat/cores/hacluster: Permission denied (13) ls -ald /var/lib/heartbeat/cores/hacluster /var/lib/heartbeat/cores/ /var/lib/heartbeat/ /var/lib/ /var/ is haclient allowed to cd all the way into /var/lib/heartbeat/cores/hacluster ? cheers, ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib
Hi,
I did a bt on the core, this is what I found:
==
Core was generated by `/usr/lib64/heartbeat/cib'.
Program terminated with signal 11, Segmentation fault.
[New process 12340]
#0 0x7f23acc553fa in strncmp () from /lib64/libc.so.6
(gdb) bt
#0 0x7f23acc553fa in strncmp () from /lib64/libc.so.6
#1 0x7f23acf87c39 in __xmlParserInputBufferCreateFilename () from
/usr/lib64/libxml2.so.2
#2 0x7f23acf6147b in xmlNewInputFromFile () from /usr/lib64/libxml2.so.2
#3 0x7f23acf641d4 in xmlCreateURLParserCtxt () from /usr/lib64/libxml2.so.2
#4 0x7f23acf78f3a in xmlReadFile () from /usr/lib64/libxml2.so.2
#5 0x7f23ad0167b1 in xmlRelaxNGParse () from /usr/lib64/libxml2.so.2
#6 0x7f23ae967321 in validate_with_relaxng (doc=0x626020, to_logs=1,
relaxng_file=0x7f23ae97ba10
/usr/share/pacemaker/pacemaker-1.2.rng) at xml.c:
#7 0x7f23ae967769 in validate_with (xml=0x6260d0, method=6,
to_logs=1) at xml.c:2287
#8 0x7f23ae967b9f in validate_xml (xml_blob=0x6260d0,
validation=0x626910 pacemaker-1.2,
to_logs=1) at xml.c:2373
#9 0x00405b23 in readCibXmlFile (dir=0x41b580
/var/lib/heartbeat/crm,
file=0x41c40a cib.xml, discard_status=1) at io.c:396
#10 0x00412285 in startCib (filename=0x41c40a cib.xml) at main.c:613
#11 0x00411309 in cib_init () at main.c:408
#12 0x0041064a in main (argc=1, argv=0x7fff942e0f58) at main.c:218
==
If it's a fresh install let's say then cib.xml will not exist.
Then why is it looking for this file on startup.
Sincerely
Shravan
On Tue, Sep 28, 2010 at 10:24 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Sorry forgot to attach my corosync.conf.
=
totem {
version: 2
# token: 3000
# token_retransmits_before_loss_const: 10
# join: 60
# consensus: 1500
# vsftype: none
# max_messages: 20
# clear_node_high_bit: yes
secauth: off
threads: 0
# rrp_mode: passive
interface {
ringnumber: 0
bindnetaddr: 192.168.2.0
#mcastaddr: 226.94.1.1
broadcast: yes
mcastport: 5405
}
# interface {
# ringnumber: 1
# bindnetaddr: 172.20.20.0
#mcastaddr: 226.94.1.1
# broadcast: yes
# mcastport: 5405
# }
}
logging {
fileline: off
to_stderr: yes
to_logfile: yes
to_syslog: yes
logfile: /tmp/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
service {
name: pacemaker
ver: 0
}
aisexec {
user:root
group: root
}
amf {
mode: disabled
}
=
On Tue, Sep 28, 2010 at 10:10 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Hi Andrew,
I'm attaching another log file as I reflashed my machine started
everything from scratch.
Looks like my old system got little messed up as I was trying to
install old HA libraries - corosyc/pacemaker that was initially
working for me.
Here are the details:
As of now I just want to see cib/attrd up so I have only one machine
where I want to see things in a sane state.
[r...@ha2 ~]# /usr/sbin/corosync -v
Corosync Cluster Engine, version '1.2.8' SVN revision '3035'
Copyright (c) 2006-2009 Red Hat, Inc.
[r...@ha2 ~]# /usr/lib64/heartbeat/crmd version
CRM Version: 1.1.2 (e0d731c2b1be446b27a73327a53067bf6230fb6a)
Pacemaker version is 1.1, the release based on the above output is
1.1.2 if I correctly understand.
This one is showing --
Sep 27 12:30:45 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib terminated with signal 11 (pid=9216, core=false)
Please find corosync logs attached.
Thanks
Shravan
On Tue, Sep 28, 2010 at 5:47 AM, Andrew Beekhof and...@beekhof.net wrote:
On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Thanks Raoul for the response.
Changing the permission to hacluster:haclient did stop that error.
Now I'm hitting another problem whereby cib is failing to start
Very strange logs.
Which distribution is this?
What does your corosync.conf look like?
=
Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node
ha2.itactics.com now has process list:
00110012 (1114130)
Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node
ha2.itactics.com now has 1 quorum votes (was 0)
Sep 27 00:16:29 corosync [pcmk ] info: send_member_notification:
Sending membership update 100 to 0 children
Sep 27 00:16:29 corosync [MAIN ] Completed service synchronization,
ready to provide service.
Sep 27 00:16:30 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib exited (pid=14889, rc=127)
Sep 27 00:16:30 corosync [pcmk ] notice: pcmk_wait_dispatch:
Respawning failed child process: cib
Re: [Pacemaker] cib
Some more info:
root 14170 14166 0 12:23 ?00:00:00 /usr/lib64/heartbeat/stonithd
nobody 14172 14166 0 12:23 ?00:00:00 /usr/lib64/heartbeat/lrmd
82 14173 14166 0 12:23 ?00:00:00 /usr/lib64/heartbeat/attrd
82 14174 14166 0 12:23 ?00:00:00 /usr/lib64/heartbeat/pengine
82 14175 14166 0 12:23 ?00:00:00 /usr/lib64/heartbeat/crmd
--lrmd is running as nobody when it should have been root.
I'm not sure why that would happen.
Thanks
Shravan
On Wed, Sep 29, 2010 at 10:29 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Hi,
I did a bt on the core, this is what I found:
==
Core was generated by `/usr/lib64/heartbeat/cib'.
Program terminated with signal 11, Segmentation fault.
[New process 12340]
#0 0x7f23acc553fa in strncmp () from /lib64/libc.so.6
(gdb) bt
#0 0x7f23acc553fa in strncmp () from /lib64/libc.so.6
#1 0x7f23acf87c39 in __xmlParserInputBufferCreateFilename () from
/usr/lib64/libxml2.so.2
#2 0x7f23acf6147b in xmlNewInputFromFile () from /usr/lib64/libxml2.so.2
#3 0x7f23acf641d4 in xmlCreateURLParserCtxt () from
/usr/lib64/libxml2.so.2
#4 0x7f23acf78f3a in xmlReadFile () from /usr/lib64/libxml2.so.2
#5 0x7f23ad0167b1 in xmlRelaxNGParse () from /usr/lib64/libxml2.so.2
#6 0x7f23ae967321 in validate_with_relaxng (doc=0x626020, to_logs=1,
relaxng_file=0x7f23ae97ba10
/usr/share/pacemaker/pacemaker-1.2.rng) at xml.c:
#7 0x7f23ae967769 in validate_with (xml=0x6260d0, method=6,
to_logs=1) at xml.c:2287
#8 0x7f23ae967b9f in validate_xml (xml_blob=0x6260d0,
validation=0x626910 pacemaker-1.2,
to_logs=1) at xml.c:2373
#9 0x00405b23 in readCibXmlFile (dir=0x41b580
/var/lib/heartbeat/crm,
file=0x41c40a cib.xml, discard_status=1) at io.c:396
#10 0x00412285 in startCib (filename=0x41c40a cib.xml) at main.c:613
#11 0x00411309 in cib_init () at main.c:408
#12 0x0041064a in main (argc=1, argv=0x7fff942e0f58) at main.c:218
==
If it's a fresh install let's say then cib.xml will not exist.
Then why is it looking for this file on startup.
Sincerely
Shravan
On Tue, Sep 28, 2010 at 10:24 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Sorry forgot to attach my corosync.conf.
=
totem {
version: 2
# token: 3000
# token_retransmits_before_loss_const: 10
# join: 60
# consensus: 1500
# vsftype: none
# max_messages: 20
# clear_node_high_bit: yes
secauth: off
threads: 0
# rrp_mode: passive
interface {
ringnumber: 0
bindnetaddr: 192.168.2.0
#mcastaddr: 226.94.1.1
broadcast: yes
mcastport: 5405
}
# interface {
# ringnumber: 1
# bindnetaddr: 172.20.20.0
#mcastaddr: 226.94.1.1
# broadcast: yes
# mcastport: 5405
# }
}
logging {
fileline: off
to_stderr: yes
to_logfile: yes
to_syslog: yes
logfile: /tmp/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
service {
name: pacemaker
ver: 0
}
aisexec {
user:root
group: root
}
amf {
mode: disabled
}
=
On Tue, Sep 28, 2010 at 10:10 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Hi Andrew,
I'm attaching another log file as I reflashed my machine started
everything from scratch.
Looks like my old system got little messed up as I was trying to
install old HA libraries - corosyc/pacemaker that was initially
working for me.
Here are the details:
As of now I just want to see cib/attrd up so I have only one machine
where I want to see things in a sane state.
[r...@ha2 ~]# /usr/sbin/corosync -v
Corosync Cluster Engine, version '1.2.8' SVN revision '3035'
Copyright (c) 2006-2009 Red Hat, Inc.
[r...@ha2 ~]# /usr/lib64/heartbeat/crmd version
CRM Version: 1.1.2 (e0d731c2b1be446b27a73327a53067bf6230fb6a)
Pacemaker version is 1.1, the release based on the above output is
1.1.2 if I correctly understand.
This one is showing --
Sep 27 12:30:45 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib terminated with signal 11 (pid=9216, core=false)
Please find corosync logs attached.
Thanks
Shravan
On Tue, Sep 28, 2010 at 5:47 AM, Andrew Beekhof and...@beekhof.net wrote:
On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Thanks Raoul for the response.
Changing the permission to hacluster:haclient did stop that error.
Now I'm hitting another problem whereby cib is failing to start
Very strange logs.
Which distribution is this?
What does your corosync.conf look like?
=
Sep 27 00:16:29 corosync [pcmk ] info:
Re: [Pacemaker] cib
On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra shravan.mis...@gmail.com wrote: Thanks Raoul for the response. Changing the permission to hacluster:haclient did stop that error. Now I'm hitting another problem whereby cib is failing to start Very strange logs. Which distribution is this? What does your corosync.conf look like? = Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has process list: 00110012 (1114130) Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node ha2.itactics.com now has 1 quorum votes (was 0) Sep 27 00:16:29 corosync [pcmk ] info: send_member_notification: Sending membership update 100 to 0 children Sep 27 00:16:29 corosync [MAIN ] Completed service synchronization, ready to provide service. Sep 27 00:16:30 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14889, rc=127) Sep 27 00:16:30 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:30 corosync [pcmk ] info: spawn_child: Forked child 14896 for process cib crmd[14893]: 2010/09/27_00:16:30 WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry Sep 27 00:16:31 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14896, rc=127) Sep 27 00:16:31 corosync [pcmk ] notice: pcmk_wait_dispatch: Respawning failed child process: cib Sep 27 00:16:31 corosync [pcmk ] info: spawn_child: Forked child 14901 for process cib Sep 27 00:16:32 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child process cib exited (pid=14901, rc=1 == I have attached the full logs. We are using corosync 1.2.8 and pacemaker 1.1.3. Thanks. Shravan On Sat, Sep 25, 2010 at 4:36 AM, Raoul Bhatia [IPAX] r.bha...@ipax.at wrote: On 24.09.2010 21:41, Shravan Mishra wrote: crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot change active directory to /var/lib/heartbeat/cores/hacluster: Permission denied (13) ls -ald /var/lib/heartbeat/cores/hacluster /var/lib/heartbeat/cores/ /var/lib/heartbeat/ /var/lib/ /var/ is haclient allowed to cd all the way into /var/lib/heartbeat/cores/hacluster ? cheers, ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib
Sorry forgot to attach my corosync.conf.
=
totem {
version: 2
# token: 3000
# token_retransmits_before_loss_const: 10
# join: 60
# consensus: 1500
# vsftype: none
# max_messages: 20
# clear_node_high_bit: yes
secauth: off
threads: 0
# rrp_mode: passive
interface {
ringnumber: 0
bindnetaddr: 192.168.2.0
#mcastaddr: 226.94.1.1
broadcast: yes
mcastport: 5405
}
# interface {
# ringnumber: 1
# bindnetaddr: 172.20.20.0
#mcastaddr: 226.94.1.1
# broadcast: yes
# mcastport: 5405
# }
}
logging {
fileline: off
to_stderr: yes
to_logfile: yes
to_syslog: yes
logfile: /tmp/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
service {
name: pacemaker
ver: 0
}
aisexec {
user:root
group: root
}
amf {
mode: disabled
}
=
On Tue, Sep 28, 2010 at 10:10 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Hi Andrew,
I'm attaching another log file as I reflashed my machine started
everything from scratch.
Looks like my old system got little messed up as I was trying to
install old HA libraries - corosyc/pacemaker that was initially
working for me.
Here are the details:
As of now I just want to see cib/attrd up so I have only one machine
where I want to see things in a sane state.
[r...@ha2 ~]# /usr/sbin/corosync -v
Corosync Cluster Engine, version '1.2.8' SVN revision '3035'
Copyright (c) 2006-2009 Red Hat, Inc.
[r...@ha2 ~]# /usr/lib64/heartbeat/crmd version
CRM Version: 1.1.2 (e0d731c2b1be446b27a73327a53067bf6230fb6a)
Pacemaker version is 1.1, the release based on the above output is
1.1.2 if I correctly understand.
This one is showing --
Sep 27 12:30:45 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib terminated with signal 11 (pid=9216, core=false)
Please find corosync logs attached.
Thanks
Shravan
On Tue, Sep 28, 2010 at 5:47 AM, Andrew Beekhof and...@beekhof.net wrote:
On Mon, Sep 27, 2010 at 6:26 AM, Shravan Mishra
shravan.mis...@gmail.com wrote:
Thanks Raoul for the response.
Changing the permission to hacluster:haclient did stop that error.
Now I'm hitting another problem whereby cib is failing to start
Very strange logs.
Which distribution is this?
What does your corosync.conf look like?
=
Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node
ha2.itactics.com now has process list:
00110012 (1114130)
Sep 27 00:16:29 corosync [pcmk ] info: update_member: Node
ha2.itactics.com now has 1 quorum votes (was 0)
Sep 27 00:16:29 corosync [pcmk ] info: send_member_notification:
Sending membership update 100 to 0 children
Sep 27 00:16:29 corosync [MAIN ] Completed service synchronization,
ready to provide service.
Sep 27 00:16:30 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib exited (pid=14889, rc=127)
Sep 27 00:16:30 corosync [pcmk ] notice: pcmk_wait_dispatch:
Respawning failed child process: cib
Sep 27 00:16:30 corosync [pcmk ] info: spawn_child: Forked child
14896 for process cib
crmd[14893]: 2010/09/27_00:16:30 WARN: do_cib_control: Couldn't
complete CIB registration 1 times... pause and retry
Sep 27 00:16:31 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib exited (pid=14896, rc=127)
Sep 27 00:16:31 corosync [pcmk ] notice: pcmk_wait_dispatch:
Respawning failed child process: cib
Sep 27 00:16:31 corosync [pcmk ] info: spawn_child: Forked child
14901 for process cib
Sep 27 00:16:32 corosync [pcmk ] ERROR: pcmk_wait_dispatch: Child
process cib exited (pid=14901, rc=1
==
I have attached the full logs.
We are using corosync 1.2.8 and pacemaker 1.1.3.
Thanks.
Shravan
On Sat, Sep 25, 2010 at 4:36 AM, Raoul Bhatia [IPAX] r.bha...@ipax.at
wrote:
On 24.09.2010 21:41, Shravan Mishra wrote:
crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot
change active directory to /var/lib/heartbeat/cores/hacluster:
Permission denied (13)
ls -ald /var/lib/heartbeat/cores/hacluster /var/lib/heartbeat/cores/
/var/lib/heartbeat/ /var/lib/ /var/
is haclient allowed to cd all the way into
/var/lib/heartbeat/cores/hacluster ?
cheers,
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs:
http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
Re: [Pacemaker] cib
On 24.09.2010 21:41, Shravan Mishra wrote: crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot change active directory to /var/lib/heartbeat/cores/hacluster: Permission denied (13) ls -ald /var/lib/heartbeat/cores/hacluster /var/lib/heartbeat/cores/ /var/lib/heartbeat/ /var/lib/ /var/ is haclient allowed to cd all the way into /var/lib/heartbeat/cores/hacluster ? cheers, ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
[Pacemaker] cib
Hi All, We recently upgraded to /usr/sbin/corosync -v Corosync Cluster Engine, version '1.2.1' SVN revision '2723:2724' Copyright (c) 2006-2009 Red Hat, Inc. In my logs I see the following lines: crmd[20612]: 2010/09/24_15:29:57 ERROR: crm_log_init_worker: Cannot change active directory to /var/lib/heartbeat/cores/hacluster: Permission denied (13) cib is not comming up. Others look ok. ps -ef | grep heart root 27797 27791 0 15:24 ?00:00:00 /usr/lib64/heartbeat/stonithd nobody 27799 27791 0 15:24 ?00:00:00 /usr/lib64/heartbeat/lrmd 82 27801 27791 0 15:24 ?00:00:00 /usr/lib64/heartbeat/pengine 82 29064 27791 0 15:39 ?00:00:00 /usr/lib64/heartbeat/crmd More info: getent group haclient haclient:x:101: getent passwd hacluster hacluster:x:82:101:cluster user:/var/lib/heartbeat/cores/hacluster:/sbin/nologin What could be the problem. Thanks Shravan ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib fails to start until host is rebooted
I spoke to Steve, and the only thing he could come up with was that the group might not be correct. When the cluster is in this state, please run: ps x -o pid,euser,ruser,egroup,rgroup,command And compare it to the normal output. Also, confirm that there is only one group named haclient, and one user named hacluster. On Tue, Sep 7, 2010 at 11:03 PM, Michael Smith msm...@cbnco.com wrote: Michael Smith wrote: On Mon, 6 Sep 2010, Andrew Beekhof wrote: Is /dev/shm full (or not mounted) by any chance? No - I tried clearing that out, too. And corosync is actually running? Yes, it's logging [IPC ] Invalid IPC credentials. when cib tries to connect. For what it's worth, I have the same problem after updating: cluster-glue-1.0.6-2.1 corosync-1.2.7-1.1 openais-1.1.3-1.1 pacemaker-1.1.2.1-5.1 Mike ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib fails to start until host is rebooted
Andrew Beekhof wrote: I spoke to Steve, and the only thing he could come up with was that the group might not be correct. When the cluster is in this state, please run: ps x -o pid,euser,ruser,egroup,rgroup,command And compare it to the normal output. Also, confirm that there is only one group named haclient, and one user named hacluster. Thanks, that was the right track. Looks like I fat-fingered a '9' in front of the '0' in root's gid in /etc/passwd: root:x:0:90:root:/root:/bin/bash gid 90 happens to be owned by haclient. With root's gid fixed, everything works as expected. Mike ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib fails to start until host is rebooted
On Thu, Sep 2, 2010 at 2:18 PM, Michael Smith msm...@cbnco.com wrote: On Thu, 2 Sep 2010, Andrew Beekhof wrote: On Mon, Aug 30, 2010 at 10:04 PM, Michael Smith msm...@cbnco.com wrote: Hi, I have a pacemaker/corosync setup on a bunch of fully patched SLES11 SP1 systems. On one of the systems, if I /etc/init.d/openais stop, then /etc/init.d/openais start, pacemaker fails to come up: Is /dev/shm full (or not mounted) by any chance? No - I tried clearing that out, too. And corosync is actually running? ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib fails to start until host is rebooted
On Mon, 6 Sep 2010, Andrew Beekhof wrote: Is /dev/shm full (or not mounted) by any chance? No - I tried clearing that out, too. And corosync is actually running? Yes, it's logging [IPC ] Invalid IPC credentials. when cib tries to connect. Mike ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib fails to start until host is rebooted
On Mon, Aug 30, 2010 at 10:04 PM, Michael Smith msm...@cbnco.com wrote: Hi, I have a pacemaker/corosync setup on a bunch of fully patched SLES11 SP1 systems. On one of the systems, if I /etc/init.d/openais stop, then /etc/init.d/openais start, pacemaker fails to come up: Is /dev/shm full (or not mounted) by any chance? Aug 30 15:48:09 xen-test1 cib: [5858]: info: crm_cluster_connect: Connecting to OpenAIS Aug 30 15:48:09 xen-test1 cib: [5858]: info: init_ais_connection: Creating connection to our AIS plugin Aug 30 15:48:10 xen-test1 corosync[5851]: [IPC ] Invalid IPC credentials. Aug 30 15:48:10 xen-test1 cib: [5858]: info: init_ais_connection: Connection to our AIS plugin (9) failed: unknown (100) Aug 30 15:48:10 xen-test1 cib: [5858]: CRIT: cib_init: Cannot sign in to the cluster... terminating I've tried rm /var/run/crm/*, but it doesn't help; the only fix is to reboot. I have an strace -f of /etc/init.d/openais start, if that would help. cluster-glue-1.0.5-0.5.1 corosync-1.2.1-0.5.1 libpacemaker3-1.1.2-0.2.1 libcorosync4-1.2.1-0.5.1 libopenais3-1.1.2-0.5.19 pacemaker-1.1.2-0.2.1 openais-1.1.2-0.5.19 Thanks, Mike ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] cib fails to start until host is rebooted
On Thu, 2 Sep 2010, Andrew Beekhof wrote: On Mon, Aug 30, 2010 at 10:04 PM, Michael Smith msm...@cbnco.com wrote: Hi, I have a pacemaker/corosync setup on a bunch of fully patched SLES11 SP1 systems. On one of the systems, if I /etc/init.d/openais stop, then /etc/init.d/openais start, pacemaker fails to come up: Is /dev/shm full (or not mounted) by any chance? No - I tried clearing that out, too. Thanks, Mike ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
[Pacemaker] cib fails to start until host is rebooted
Hi, I have a pacemaker/corosync setup on a bunch of fully patched SLES11 SP1 systems. On one of the systems, if I /etc/init.d/openais stop, then /etc/init.d/openais start, pacemaker fails to come up: Aug 30 15:48:09 xen-test1 cib: [5858]: info: crm_cluster_connect: Connecting to OpenAIS Aug 30 15:48:09 xen-test1 cib: [5858]: info: init_ais_connection: Creating connection to our AIS plugin Aug 30 15:48:10 xen-test1 corosync[5851]: [IPC ] Invalid IPC credentials. Aug 30 15:48:10 xen-test1 cib: [5858]: info: init_ais_connection: Connection to our AIS plugin (9) failed: unknown (100) Aug 30 15:48:10 xen-test1 cib: [5858]: CRIT: cib_init: Cannot sign in to the cluster... terminating I've tried rm /var/run/crm/*, but it doesn't help; the only fix is to reboot. I have an strace -f of /etc/init.d/openais start, if that would help. cluster-glue-1.0.5-0.5.1 corosync-1.2.1-0.5.1 libpacemaker3-1.1.2-0.2.1 libcorosync4-1.2.1-0.5.1 libopenais3-1.1.2-0.5.19 pacemaker-1.1.2-0.2.1 openais-1.1.2-0.5.19 Thanks, Mike ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
Lars Ellenberg wrote: On Thu, Apr 01, 2010 at 08:27:02AM -0600, Alan Robertson wrote: Lars Ellenberg wrote: On Thu, Apr 01, 2010 at 12:12:47AM -0600, Alan Robertson wrote: OK Since there was no ssh-as-root between the cluster nodes, I didn't send all the logs along from every node in the cluster - and it didn't occur to me to look at all of them. However, the problem has gotten curioser and curioser - because ALL the nodes in the cluster reported the same problem at the same time... That makes it a lot less likely to be a race condition with the disk writing infrastructure... I've attached the relevant lines from the various machines - slightly processed (date stamp format changed and a few other minor things). Let me know if you want me to send all the system logs along... There should be core files. You should be able to get some interessting information out there, especially the_cib and digest at the point of abort(). Also, for my reference - what method are you using to compute the digest of the file? That is, what command should I execute to get the same results? It's an md5sum over the xml tree -- not over the formated ascii buffer, though, so md5sum cib.xml won't do. I think it is the same as echo $(perl -pe 's/^\s*(.*?)\s*\z/$1/g' cib.whatever) | md5sum But there is cibadmin --md5-sum -x cib.xml, to use the exact same code path. This is a change from how it used to be (the last time I looked - at least according to my not-always-reliable memory). Thanks for the update. 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:624 : retrieveCib(tmp1, tmp2, FALSE) != NULL So it did not verify right after it was written. Can you reproduce? I have no idea. I didn't do anything much. Hopefully the test suite does a lot more strenuous things... The core files may actually contains some hints, so have a look there. None of them verified. All the nodes in the cluster failed the test at the same time - and now I have no official CIBs on disk - on any cluster nodes... I sent Andrew all the CIBs, and all the core Well, Andrew is on vacation right now... you will have noticed. files, and basically everything under /var/lib/heartbeat/ from one machine. They're from the latest official release - so the binaries that match them are readily available. The strange thing is that your corrupt cib.uHFtAW contains a status/ thing. it should not. No other cib*.raw or cib.xml does. Because status/ is explicitly filtered out in write_cib_contents: free_xml_from_parent(the_cib, cib_status_root); before write_xml_file(the_cib, tmp1, FALSE), so that should never have made it in there. Something is very wrong somewhere... Did you manage to get two status sections in there, somehow? You tried anything funky with the cib as last action before this failed? Not that I recall... Do it again, with higher log level. Sorry, no time right now to rebuild your exact thing with your exact gcc and stuff to look at your core file. You can just download the RPM and extract the objects. That's what I used. -- Alan Robertson al...@unix.sh Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions. - William Wilberforce ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
On Fri, Apr 02, 2010 at 08:16:32AM -0600, Alan Robertson wrote: Do it again, with higher log level. Sorry, no time right now to rebuild your exact thing with your exact gcc and stuff to look at your core file. You can just download the RPM and extract the objects. That's what I used. core files generated on a rhel box are not particularly easy to use on a debian box... so I did not do much beyond strings on it. anyways, I suggest that you hit (some variant of) http://markmail.org/message/exsz6rf7vhjntqgu there is also a patch: http://markmail.org/message/utjcety2tiu6zaer and the upstream commit of it http://git.gnome.org/browse/libxml2/commit/?id=c4ba8a42214c9f1cc16da14f29d63db2d0cec55a if this guess turns out to be true: congrats, you have found a bug in libxml2 that has been fixed for over two years! enterprisey ;-) -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
OK Since there was no ssh-as-root between the cluster nodes, I didn't send all the logs along from every node in the cluster - and it didn't occur to me to look at all of them. However, the problem has gotten curioser and curioser - because ALL the nodes in the cluster reported the same problem at the same time... That makes it a lot less likely to be a race condition with the disk writing infrastructure... I've attached the relevant lines from the various machines - slightly processed (date stamp format changed and a few other minor things). Let me know if you want me to send all the system logs along... Alan Robertson wrote: Hi, I've run into what looks at first blush to be a CIB bug in writing to disk. The key messages from this incident are these: Mar 31 19:02:52 vhost0384 cib: [13294]: ERROR: validate_cib_digest: Digest comparision failed: expected 316049fa7ee8d2e107573ce7cded07cf (/var/lib/heartbeat/crm/cib.GUdD9T), calculated 0bac3440f5c42f0f37d22ea7dfe433e8 Mar 31 19:02:52 vhost0384 cib: [13294]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.uHFtAW failed! Configuration contents ignored! Mar 31 19:02:52 vhost0384 cib: [13294]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected Mar 31 19:02:52 vhost0384 cib: [13294]: WARN: retrieveCib: Continuing but /var/lib/heartbeat/crm/cib.uHFtAW will NOT used. I did not make manual changes on a running CIB. I was using the cluster shell at the time. The CIB it is complaining about appears to be an intact, valid CIB with contents approximately like they should have been at the time. By the way, I have a report from another IBMer that they have seen systems that stop writing to their local CIBs. I'll contact him. Here are some relevant facts: These machines are virtual guests in a cloud somewhere - operations have somewhat unpredictable latency. But, nothing too egregious was happening at the time or Heartbeat would have bitched. I was doing some testing at the time. I was putting on and taking off constraints using the cluster shell migrate and unmigrate operations. Given that the file looks intact, and I know how the CIB is written to disk (since I originally wrote that code), I wonder if it isn't a versioning issue / race condition. That is, the code for writing to disk does NOT guarantee when it gets done (assuming you're still using it). It would be easy to do a checksum on the wrong version compared to the version you thought it should be (or before it completed). Andrew: You should have already received all the relevant logs to you on a separate email. Also, for my reference - what method are you using to compute the digest of the file? That is, what command should I execute to get the same results? -- Alan Robertson al...@unix.sh Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions. - William Wilberforce 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:624 : retrieveCib(tmp1, tmp2, FALSE) != NULL 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.uHFtAW failed! Configuration contents ignored! 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: validate_cib_digest: Digest comparision failed: expected 316049fa7ee8d2e107573ce7cded07cf (/var/lib/heartbeat/crm/cib.GUdD9T), calculated 0bac3440f5c42f0f37d22ea7dfe433e8 2010/03/31_19:02:52 vhost0384 [6297]: ERROR: cib_diskwrite_complete: Disabling disk writes after write failure 2010/03/31_19:02:52 vhost0384 [6297]: ERROR: cib_diskwrite_complete: Disk write failed: status=134, signo=6, exitcode=0 2010/03/31_19:02:52 vhost0384 [6297]: ERROR: Managed write_cib_contents process 13294 dumped core 2010/03/31_19:02:53 vhost0150 [15083]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:624 : retrieveCib(tmp1, tmp2, FALSE) != NULL 2010/03/31_19:02:53 vhost0150 [15083]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.n66oB0 failed! Configuration contents ignored! 2010/03/31_19:02:53 vhost0150 [15083]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected 2010/03/31_19:02:53 vhost0150 [15083]: ERROR: validate_cib_digest: Digest comparision failed: expected 316049fa7ee8d2e107573ce7cded07cf (/var/lib/heartbeat/crm/cib.UJSSzR), calculated 0bac3440f5c42f0f37d22ea7dfe433e8 2010/03/31_19:02:53 vhost0150 [2564]: ERROR: cib_diskwrite_complete: Disabling disk writes
Re: [Pacemaker] CIB write-to-disk bug?
On Thu, Apr 01, 2010 at 12:12:47AM -0600, Alan Robertson wrote: OK Since there was no ssh-as-root between the cluster nodes, I didn't send all the logs along from every node in the cluster - and it didn't occur to me to look at all of them. However, the problem has gotten curioser and curioser - because ALL the nodes in the cluster reported the same problem at the same time... That makes it a lot less likely to be a race condition with the disk writing infrastructure... I've attached the relevant lines from the various machines - slightly processed (date stamp format changed and a few other minor things). Let me know if you want me to send all the system logs along... There should be core files. You should be able to get some interessting information out there, especially the_cib and digest at the point of abort(). I did not make manual changes on a running CIB. I was using the cluster shell at the time. The CIB it is complaining about appears to be an intact, valid CIB with contents approximately like they should have been at the time. By the way, I have a report from another IBMer that they have seen systems that stop writing to their local CIBs. I'll contact him. Here are some relevant facts: These machines are virtual guests in a cloud somewhere - operations have somewhat unpredictable latency. But, nothing too egregious was happening at the time or Heartbeat would have bitched. I was doing some testing at the time. I was putting on and taking off constraints using the cluster shell migrate and unmigrate operations. Given that the file looks intact, and I know how the CIB is written to disk (since I originally wrote that code), I wonder if it isn't a versioning issue / race condition. That is, the code for writing to disk does NOT guarantee when it gets done (assuming you're still using it). It would be easy to do a checksum on the wrong version compared to the version you thought it should be (or before it completed). Andrew: You should have already received all the relevant logs to you on a separate email. Also, for my reference - what method are you using to compute the digest of the file? That is, what command should I execute to get the same results? It's an md5sum over the xml tree -- not over the formated ascii buffer, though, so md5sum cib.xml won't do. I think it is the same as echo $(perl -pe 's/^\s*(.*?)\s*\z/$1/g' cib.whatever) | md5sum But there is cibadmin --md5-sum -x cib.xml, to use the exact same code path. 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:624 : retrieveCib(tmp1, tmp2, FALSE) != NULL So it did not verify right after it was written. Can you reproduce? The core files may actually contains some hints, so have a look there. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
Lars Ellenberg wrote: On Thu, Apr 01, 2010 at 12:12:47AM -0600, Alan Robertson wrote: OK Since there was no ssh-as-root between the cluster nodes, I didn't send all the logs along from every node in the cluster - and it didn't occur to me to look at all of them. However, the problem has gotten curioser and curioser - because ALL the nodes in the cluster reported the same problem at the same time... That makes it a lot less likely to be a race condition with the disk writing infrastructure... I've attached the relevant lines from the various machines - slightly processed (date stamp format changed and a few other minor things). Let me know if you want me to send all the system logs along... There should be core files. You should be able to get some interessting information out there, especially the_cib and digest at the point of abort(). Also, for my reference - what method are you using to compute the digest of the file? That is, what command should I execute to get the same results? It's an md5sum over the xml tree -- not over the formated ascii buffer, though, so md5sum cib.xml won't do. I think it is the same as echo $(perl -pe 's/^\s*(.*?)\s*\z/$1/g' cib.whatever) | md5sum But there is cibadmin --md5-sum -x cib.xml, to use the exact same code path. This is a change from how it used to be (the last time I looked - at least according to my not-always-reliable memory). Thanks for the update. 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:624 : retrieveCib(tmp1, tmp2, FALSE) != NULL So it did not verify right after it was written. Can you reproduce? I have no idea. I didn't do anything much. Hopefully the test suite does a lot more strenuous things... The core files may actually contains some hints, so have a look there. None of them verified. All the nodes in the cluster failed the test at the same time - and now I have no official CIBs on disk - on any cluster nodes... I sent Andrew all the CIBs, and all the core files, and basically everything under /var/lib/heartbeat/ from one machine. They're from the latest official release - so the binaries that match them are readily available. Thanks Lars! -- Alan Robertson al...@unix.sh Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions. - William Wilberforce ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
On 2010-04-01 16:27, Alan Robertson wrote: None of them verified. All the nodes in the cluster failed the test at the same time - and now I have no official CIBs on disk - on any cluster nodes... I sent Andrew all the CIBs, and all the core files, and basically everything under /var/lib/heartbeat/ from one machine. They're from the latest official release - so the binaries that match them are readily available. Any particular reason to not create an hb_report tarball and attach that to a bug report in the LF bugzilla? Cheers, Florian signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
Florian Haas wrote: On 2010-04-01 16:27, Alan Robertson wrote: None of them verified. All the nodes in the cluster failed the test at the same time - and now I have no official CIBs on disk - on any cluster nodes... I sent Andrew all the CIBs, and all the core files, and basically everything under /var/lib/heartbeat/ from one machine. They're from the latest official release - so the binaries that match them are readily available. Any particular reason to not create an hb_report tarball and attach that to a bug report in the LF bugzilla? I did create the tarball - and a second one with all the CIBs, core files, and so on. I just didn't create a bug report. This looks like the same bugzilla that Heartbeat uses. Is that right? I was kind of hoping someone would have an easy answer ;-). -- Alan Robertson al...@unix.sh Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions. - William Wilberforce ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] CIB write-to-disk bug?
On Thu, Apr 01, 2010 at 08:27:02AM -0600, Alan Robertson wrote: Lars Ellenberg wrote: On Thu, Apr 01, 2010 at 12:12:47AM -0600, Alan Robertson wrote: OK Since there was no ssh-as-root between the cluster nodes, I didn't send all the logs along from every node in the cluster - and it didn't occur to me to look at all of them. However, the problem has gotten curioser and curioser - because ALL the nodes in the cluster reported the same problem at the same time... That makes it a lot less likely to be a race condition with the disk writing infrastructure... I've attached the relevant lines from the various machines - slightly processed (date stamp format changed and a few other minor things). Let me know if you want me to send all the system logs along... There should be core files. You should be able to get some interessting information out there, especially the_cib and digest at the point of abort(). Also, for my reference - what method are you using to compute the digest of the file? That is, what command should I execute to get the same results? It's an md5sum over the xml tree -- not over the formated ascii buffer, though, so md5sum cib.xml won't do. I think it is the same as echo $(perl -pe 's/^\s*(.*?)\s*\z/$1/g' cib.whatever) | md5sum But there is cibadmin --md5-sum -x cib.xml, to use the exact same code path. This is a change from how it used to be (the last time I looked - at least according to my not-always-reliable memory). Thanks for the update. 2010/03/31_19:02:52 vhost0384 [13294]: ERROR: crm_abort: write_cib_contents: Triggered fatal assert at io.c:624 : retrieveCib(tmp1, tmp2, FALSE) != NULL So it did not verify right after it was written. Can you reproduce? I have no idea. I didn't do anything much. Hopefully the test suite does a lot more strenuous things... The core files may actually contains some hints, so have a look there. None of them verified. All the nodes in the cluster failed the test at the same time - and now I have no official CIBs on disk - on any cluster nodes... I sent Andrew all the CIBs, and all the core Well, Andrew is on vacation right now... you will have noticed. files, and basically everything under /var/lib/heartbeat/ from one machine. They're from the latest official release - so the binaries that match them are readily available. The strange thing is that your corrupt cib.uHFtAW contains a status/ thing. it should not. No other cib*.raw or cib.xml does. Because status/ is explicitly filtered out in write_cib_contents: free_xml_from_parent(the_cib, cib_status_root); before write_xml_file(the_cib, tmp1, FALSE), so that should never have made it in there. Something is very wrong somewhere... Did you manage to get two status sections in there, somehow? You tried anything funky with the cib as last action before this failed? Do it again, with higher log level. Sorry, no time right now to rebuild your exact thing with your exact gcc and stuff to look at your core file. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
[Pacemaker] CIB write-to-disk bug?
Hi, I've run into what looks at first blush to be a CIB bug in writing to disk. The key messages from this incident are these: Mar 31 19:02:52 vhost0384 cib: [13294]: ERROR: validate_cib_digest: Digest comparision failed: expected 316049fa7ee8d2e107573ce7cded07cf (/var/lib/heartbeat/crm/cib.GUdD9T), calculated 0bac3440f5c42f0f37d22ea7dfe433e8 Mar 31 19:02:52 vhost0384 cib: [13294]: ERROR: retrieveCib: Checksum of /var/lib/heartbeat/crm/cib.uHFtAW failed! Configuration contents ignored! Mar 31 19:02:52 vhost0384 cib: [13294]: ERROR: retrieveCib: Usually this is caused by manual changes, please refer to http://clusterlabs.org/wiki/FAQ#cib_changes_detected Mar 31 19:02:52 vhost0384 cib: [13294]: WARN: retrieveCib: Continuing but /var/lib/heartbeat/crm/cib.uHFtAW will NOT used. I did not make manual changes on a running CIB. I was using the cluster shell at the time. The CIB it is complaining about appears to be an intact, valid CIB with contents approximately like they should have been at the time. By the way, I have a report from another IBMer that they have seen systems that stop writing to their local CIBs. I'll contact him. Here are some relevant facts: These machines are virtual guests in a cloud somewhere - operations have somewhat unpredictable latency. But, nothing too egregious was happening at the time or Heartbeat would have bitched. I was doing some testing at the time. I was putting on and taking off constraints using the cluster shell migrate and unmigrate operations. Given that the file looks intact, and I know how the CIB is written to disk (since I originally wrote that code), I wonder if it isn't a versioning issue / race condition. That is, the code for writing to disk does NOT guarantee when it gets done (assuming you're still using it). It would be easy to do a checksum on the wrong version compared to the version you thought it should be (or before it completed). Andrew: You should have already received all the relevant logs to you on a separate email. Also, for my reference - what method are you using to compute the digest of the file? That is, what command should I execute to get the same results? -- Alan Robertson al...@unix.sh Openness is the foundation and preservative of friendship... Let me claim from you at all times your undisguised opinions. - William Wilberforce ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib and attrd processes segfault
Please - enable coredumps (set ulimit -c unlimited at the top of the corosync init file) - use hb_report to create a support tarball covering the problem - attach the tarball to a new bug: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker Thats the minimum we'd need to be able to assist. Ok thank you very much ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker -- All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy. All work and no play makes Jack... ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
And you'll also want this patch for the crmd
diff -r 4619c842d58c crmd/callbacks.c
--- a/crmd/callbacks.c Fri May 22 16:52:14 2009 +0200
+++ b/crmd/callbacks.c Fri May 22 21:34:12 2009 +0200
@@ -179,7 +179,6 @@ crmd_ha_msg_callback(HA_Message *hamsg,
} else {
crmd_ha_msg_filter(msg);
- return;
}
bail:
On Wed, May 20, 2009 at 2:47 PM, Nikola Ciprich extmaill...@linuxbox.cz wrote:
On Wed, May 20, 2009 at 02:02:52PM +0200, Andrew Beekhof wrote:
Ah, well that was pretty obvious.
/me humbly apologizes for such a stupid error.
Hi and thanks! no problem
(It wasn't caught by my own valgrind testing because this function is
specific to heartbeat based clusters)
don't worry, I'm doing a lots of testing for you ;)
I've already compiled it an deployed on testing machines,
memory usage seems to be pretty low. I'll report
few days later if everything is OK.
thanks a lot once more!
nik
Try this:
diff -r ea5d0b58c0be cib/callbacks.c
--- a/cib/callbacks.c Wed May 20 11:56:39 2009 +0200
+++ b/cib/callbacks.c Wed May 20 14:01:30 2009 +0200
@@ -1064,6 +1064,7 @@ cib_ha_peer_callback(HA_Message * msg, v
{
xmlNode *xml = convert_ha_message(NULL, msg, __FUNCTION__);
cib_peer_callback(xml, private_data);
+ free_xml(xml);
}
void
On Tue, May 19, 2009 at 8:24 PM, Andrew Beekhof and...@beekhof.net wrote:
I'll take a look at the valgrind data. Thanks!
On Tue, May 19, 2009 at 6:39 PM, Nikola Ciprich extmaill...@linuxbox.cz
wrote:
Hello,
sorry to bother again. I've discovered why valgrind didn't
find anything. It is important to stop the process in order to
have valgrind finish the analysis. And it seems that there
really are leaks not only in cib, but also in attrd and crmd.
I just had a slight look into the code reported by valgrind
as problematic and though I would certainly need to examine
it much more to understand it properly, I think there are
leaks. I'm attaching the valgrind reports, In case You would be
interested in examining them.
If I could provide any help, I'll be more than happy.
(well, I guess I could of course help by sending patches :) but I'm
afraid this will take me a lot of time, I can try though).
with best regards
nik
Not really. Sorry :(
--
-
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-
___
Pacemaker mailing list
Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
Ah, well that was pretty obvious.
/me humbly apologizes for such a stupid error.
(It wasn't caught by my own valgrind testing because this function is
specific to heartbeat based clusters)
Try this:
diff -r ea5d0b58c0be cib/callbacks.c
--- a/cib/callbacks.c Wed May 20 11:56:39 2009 +0200
+++ b/cib/callbacks.c Wed May 20 14:01:30 2009 +0200
@@ -1064,6 +1064,7 @@ cib_ha_peer_callback(HA_Message * msg, v
{
xmlNode *xml = convert_ha_message(NULL, msg, __FUNCTION__);
cib_peer_callback(xml, private_data);
+free_xml(xml);
}
void
On Tue, May 19, 2009 at 8:24 PM, Andrew Beekhof and...@beekhof.net wrote:
I'll take a look at the valgrind data. Thanks!
On Tue, May 19, 2009 at 6:39 PM, Nikola Ciprich extmaill...@linuxbox.cz
wrote:
Hello,
sorry to bother again. I've discovered why valgrind didn't
find anything. It is important to stop the process in order to
have valgrind finish the analysis. And it seems that there
really are leaks not only in cib, but also in attrd and crmd.
I just had a slight look into the code reported by valgrind
as problematic and though I would certainly need to examine
it much more to understand it properly, I think there are
leaks. I'm attaching the valgrind reports, In case You would be
interested in examining them.
If I could provide any help, I'll be more than happy.
(well, I guess I could of course help by sending patches :) but I'm
afraid this will take me a lot of time, I can try though).
with best regards
nik
Not really. Sorry :(
--
-
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-
___
Pacemaker mailing list
Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
On Wed, May 20, 2009 at 02:02:52PM +0200, Andrew Beekhof wrote:
Ah, well that was pretty obvious.
/me humbly apologizes for such a stupid error.
Hi and thanks! no problem
(It wasn't caught by my own valgrind testing because this function is
specific to heartbeat based clusters)
don't worry, I'm doing a lots of testing for you ;)
I've already compiled it an deployed on testing machines,
memory usage seems to be pretty low. I'll report
few days later if everything is OK.
thanks a lot once more!
nik
Try this:
diff -r ea5d0b58c0be cib/callbacks.c
--- a/cib/callbacks.c Wed May 20 11:56:39 2009 +0200
+++ b/cib/callbacks.c Wed May 20 14:01:30 2009 +0200
@@ -1064,6 +1064,7 @@ cib_ha_peer_callback(HA_Message * msg, v
{
xmlNode *xml = convert_ha_message(NULL, msg, __FUNCTION__);
cib_peer_callback(xml, private_data);
+free_xml(xml);
}
void
On Tue, May 19, 2009 at 8:24 PM, Andrew Beekhof and...@beekhof.net wrote:
I'll take a look at the valgrind data. Thanks!
On Tue, May 19, 2009 at 6:39 PM, Nikola Ciprich extmaill...@linuxbox.cz
wrote:
Hello,
sorry to bother again. I've discovered why valgrind didn't
find anything. It is important to stop the process in order to
have valgrind finish the analysis. And it seems that there
really are leaks not only in cib, but also in attrd and crmd.
I just had a slight look into the code reported by valgrind
as problematic and though I would certainly need to examine
it much more to understand it properly, I think there are
leaks. I'm attaching the valgrind reports, In case You would be
interested in examining them.
If I could provide any help, I'll be more than happy.
(well, I guess I could of course help by sending patches :) but I'm
afraid this will take me a lot of time, I can try though).
with best regards
nik
Not really. Sorry :(
--
-
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: ser...@linuxbox.cz
-
___
Pacemaker mailing list
Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
On Sat, May 16, 2009 at 10:33 PM, Nikola Ciprich
extmaill...@linuxbox.cz wrote:
Hi guys,
I was able to enable valgrind on our production cluster today,
but unfortunately only on the secondary node, I'll be allowed to enable
it on primary node hopefully during next weekend.
Unfortunately it seems that valgrind probably won't be of much help here.
I've got some output from it, but it's only few warnings and it seems
that growing memory consumption is not really caused by leak, but (maybe)
only by some growing memory structure. I'm doing one not very nice thing
in my cluster which might be the culprit:
I'm monitoring some service by a cron script and periodically changing
related resource score by the following command:
cibadmin -U -o constraints -X
rsc_location id=divacard0-master-loc rsc=ms-divacard0
rule id=divacard0-master-rule-${host} score=${score}
role=Master
expression id=divacard0-master-exp-${host} attribute=#uname
operation=eq value=${host}/
/rule
/rsc_location
Is it possible that this could be causing cib growing memory consumption?
Anything is possible, but it would be unlikely.
There's nothing special about that command that would make only it leak.
Anyways, I'm attaching valgrind output for cib process:
==14779== My PID = 14779, parent PID = 14766. Prog and args are:
==14779== /usr/lib64/heartbeat/cib
==14779==
Can this help?
Not really. Sorry :(
___
Pacemaker mailing list
Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
Hi, Dejan, thanks a lot, I compiled Your version, but crmd with shipped pacemaker keeps segfaulting with it, and unable to rebuild pacemaker with this heartbeat to get the -debug package. compilation fails with: plugin.c: In function 'check_message_sanity': plugin.c:1190: warning: format '%d' expects type 'int', but argument 10 has type 'long unsigned int' plugin.c:1190: warning: format '%d' expects type 'int', but argument 10 has type 'long unsigned int' gmake[2]: *** [plugin.lo] Error 1 gmake[2]: Leaving directory `/home/src/redhat/BUILD/pacemaker/lib/ais' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/home/src/redhat/BUILD/pacemaker/lib' make: *** [all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.81431 (%build) Could You please send me only the related patch, so I could try compiling latest stable version? I don't see it in the mercurial... Andrew thanks for Your patches as well, I'll try them, but honestly I'm a bit confused, first patch is for heartbeat, right? and the second one for pacemaker? It doesn't seem to apply either to -tip, or to 1.0.3... BR nik -- - Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
On Thu, May 14, 2009 at 3:58 PM, Nikola Ciprich extmaill...@linuxbox.cz wrote: Hi, Dejan, thanks a lot, I compiled Your version, but crmd with shipped pacemaker keeps segfaulting with it, and unable to rebuild pacemaker with this heartbeat to get the -debug package. compilation fails with: plugin.c: In function 'check_message_sanity': plugin.c:1190: warning: format '%d' expects type 'int', but argument 10 has type 'long unsigned int' plugin.c:1190: warning: format '%d' expects type 'int', but argument 10 has type 'long unsigned int' gmake[2]: *** [plugin.lo] Error 1 gmake[2]: Leaving directory `/home/src/redhat/BUILD/pacemaker/lib/ais' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/home/src/redhat/BUILD/pacemaker/lib' make: *** [all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.81431 (%build) Could You please send me only the related patch, so I could try compiling latest stable version? I don't see it in the mercurial... When you configure pacemaker, just add the --without-ais option. Andrew thanks for Your patches as well, I'll try them, but honestly I'm a bit confused, first patch is for heartbeat, right? actually, you probably dont need the second one. i think its in 1.0 already. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] cib still leaks in pacemaker-1.0.3
Hi guys, sooo I've got valgrind grinding:) I had some trouble getting the latest stuff working, so I used heartbeat-2.99.2 with Dejan's (fixed) patch and --enable-valgrind --with-valgrind-log=--log-file=/tmp/crm-%p.valgrind and recompiled pacemaker-1.0.3 (withount openais as Andrew suggested). now enabling valgrind works! Unfortulately I don't see the leaks on my testing machine, so I'll have to try it directly on production one. Hopefully I'll have some time for playing Tomorrow or during the weekend, so I'll report ASAP. thanks a lot for all Your help! best regards nik On Thu, May 14, 2009 at 04:12:52PM +0200, Andrew Beekhof wrote: On Thu, May 14, 2009 at 3:58 PM, Nikola Ciprich extmaill...@linuxbox.cz wrote: Hi, Dejan, thanks a lot, I compiled Your version, but crmd with shipped pacemaker keeps segfaulting with it, and unable to rebuild pacemaker with this heartbeat to get the -debug package. compilation fails with: plugin.c: In function 'check_message_sanity': plugin.c:1190: warning: format '%d' expects type 'int', but argument 10 has type 'long unsigned int' plugin.c:1190: warning: format '%d' expects type 'int', but argument 10 has type 'long unsigned int' gmake[2]: *** [plugin.lo] Error 1 gmake[2]: Leaving directory `/home/src/redhat/BUILD/pacemaker/lib/ais' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/home/src/redhat/BUILD/pacemaker/lib' make: *** [all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.81431 (%build) Could You please send me only the related patch, so I could try compiling latest stable version? I don't see it in the mercurial... When you configure pacemaker, just add the --without-ais option. Andrew thanks for Your patches as well, I'll try them, but honestly I'm a bit confused, first patch is for heartbeat, right? actually, you probably dont need the second one. i think its in 1.0 already. -- - Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax:+420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: ser...@linuxbox.cz - ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
