Ah, finally got it.
Core was generated by `/usr/libexec/pacemaker/crmd'.
Program terminated with signal 6, Aborted.
#0 0x00007f81896ac8a5 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install
libtool-ltdl-2.2.6-15.5.el6.x86_64
(gdb) bt
#0 0x00007f81896ac8a5 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f81896ae085 in abort () at abort.c:92
#2 0x00007f818bb8a56b in crm_abort (file=0x7f818bba9d58 "xml.c",
function=0x7f818bbab6b4 "string2xml", line=650,
assert_condition=0x7f818bbaa01a "String parsing error", do_core=<value
optimized out>, do_fork=<value optimized out>) at utils.c:1073
#3 0x00007f818bb933af in string2xml (
input=0x1e745f8 "<lrmd_notify lrmd_origin=\"send_cmd_complete_notify\"
lrmd_timeout=\"30000\" lrmd_rsc_interval=\"15000\"
lrmd_rsc_start_delay=\"15000\" lrmd_exec_rc=\"0\" lrmd_exec_op_status=\"1\"
lrmd_callid=\"2747\" lrmd_rsc_del"...) at xml.c:650
#4 0x00007f818b76a2fc in lrmd_ipc_dispatch (buffer=<value optimized out>,
length=<value optimized out>, userdata=0x1e72910) at lrmd_client.c:310
#5 0x00007f818bba2e90 in mainloop_gio_callback (gio=<value optimized out>,
condition=G_IO_IN, data=0x1e73be0) at mainloop.c:585
#6 0x00007f8188fbbf0e in g_main_dispatch (context=0x1d4f120) at gmain.c:1960
#7 IA__g_main_context_dispatch (context=0x1d4f120) at gmain.c:2513
#8 0x00007f8188fbf938 in g_main_context_iterate (context=0x1d4f120, block=1,
dispatch=1, self=<value optimized out>) at gmain.c:2591
#9 0x00007f8188fbfd55 in IA__g_main_loop_run (loop=0x1e734a0) at gmain.c:2799
#10 0x00000000004052ce in crmd_init () at main.c:154
#11 0x00000000004055cc in main (argc=1, argv=0x7fffe77a4f88) at main.c:120
Xavier Lashmar
Analyste de Systèmes | Systems Analyst
Service étudiants, service de l'informatique et des communications/Student
services, computing and communications services.
1 Nicholas Street (810)
Ottawa ON K1N 7B7
Tél. | Tel. 613-562-5800 (2120)
-----Original Message-----
From: Andrew Beekhof [mailto:and...@beekhof.net]
Sent: Wednesday, May 1, 2013 7:07 PM
To: The Pacemaker cluster resource manager
Subject: Re: [Pacemaker] Pacemaker core dumps
On 01/05/2013, at 11:36 PM, Xavier Lashmar <xlash...@uottawa.ca> wrote:
> I'm not sure if anyone has run into this issue but I can't seem to
> find a debuginfo package for one of the libraries for CentOS 6.3 with
> Kernel 2.6.32-279.9.1el6.x86_64 : libtool-ltdl
>
> Here's what I get so far from the core dump, but I think it's incomplete:
>
> ...
> ...
> ...
> Reading symbols from /lib64/libfreebl3.so...
> warning: the debug information found in
> "/usr/lib/debug//lib64/libfreebl3.so.debug" does not match
> "/lib64/libfreebl3.so" (CRC mismatch).
>
> warning: the debug information found in
> "/usr/lib/debug/lib64/libfreebl3.so.debug" does not match
> "/lib64/libfreebl3.so" (CRC mismatch).
>
> Missing separate debuginfo for /lib64/libfreebl3.so
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/68/195872ecfb188389d29aaf01031a976fd18168.deb
> ug
> (no debugging symbols found)...done.
> Loaded symbols for /lib64/libfreebl3.so Reading symbols from
> /lib64/libnss_files-2.12.so...Reading symbols from
> /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libnss_files-2.12.so Core was generated by
> `/usr/libexec/pacemaker/crmd'.
> Program terminated with signal 6, Aborted.
> #0 0x00007f81896ac8a5 in raise (sig=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> Missing separate debuginfos, use: debuginfo-install
> libtool-ltdl-2.2.6-15.5.el6.x86_64
>
> Any info about either finding the right debuginfo files, or about the error
> itself would be greatly appreciated.
The libtool parts aren't so interesting.
Were there no other frames? (lines starting with # and a number)
>
> Xavier Lashmar
> Analyste de Systèmes | Systems Analyst Service étudiants, service de
> l'informatique et des communications/Student services, computing and
> communications services.
> 1 Nicholas Street (810)
> Ottawa ON K1N 7B7
> Tél. | Tel. 613-562-5800 (2120)
>
>
>
>
> -----Original Message-----
> From: Andrew Beekhof [mailto:and...@beekhof.net]
> Sent: Monday, April 29, 2013 11:00 PM
> To: The Pacemaker cluster resource manager
> Subject: Re: [Pacemaker] Pacemaker core dumps
>
>
> On 30/04/2013, at 1:32 AM, Xavier Lashmar <xlash...@uottawa.ca> wrote:
>
>> Hello Andrew,
>>
>> Thanks for your help. We've upgrade to pacemaker 1.1.9 and still have the
>> same issue.
>
> Thats a disappointing but useful data point.
>
>>
>> We are trying to get the core information but we are missing some debuginfo
>> files which we are trying to get our hands on. I'll try to forward this
>> information soon.
>
> Great
>
>>
>> Is there something we need to do to the CIB when we upgrade?
>
> No, anything that needs to happen will be done under the hood.
>
>>
>>
>> Xavier Lashmar
>> Analyste de Systèmes | Systems Analyst Service étudiants, service de
>> l'informatique et des communications/Student services, computing and
>> communications services.
>> 1 Nicholas Street (810)
>> Ottawa ON K1N 7B7
>> Tél. | Tel. 613-562-5800 (2120)
>>
>>
>>
>> -----Original Message-----
>> From: Andrew Beekhof [mailto:and...@beekhof.net]
>> Sent: Thursday, April 25, 2013 8:15 PM
>> To: The Pacemaker cluster resource manager
>> Subject: Re: [Pacemaker] Pacemaker core dumps
>>
>>
>> On 26/04/2013, at 10:06 AM, Andrew Beekhof <and...@beekhof.net> wrote:
>>
>>>
>>> On 25/04/2013, at 11:59 PM, Xavier Lashmar <xlash...@uottawa.ca> wrote:
>>>
>>>> Following further investigation, we were able to determine that upgrading
>>>> both nodes (in a two node cluster) from Pacemaker 1.1.7-6 to Pacemaker
>>>> 1.1.8-7 (CentOS 6.3 or Centos 6.4) caused these errors to begin happening:
>>>
>>> Would you be able to try the 1.1.9 packages from
>>> http://www.clusterlabs.org/rpm-next to see if they are also affected?
>>>
>>>>
>>>> We were able to replicate the initiation of the errors by upgrading
>>>> another cluster in the same manner. This other cluster is now
>>>> experiencing the same core-dumping and errors as the previous cluster:
>>>>
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity:
>>>> line 1: parser error : invalid character in attribute value
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is
>>>> running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity:
>>>> line 1: parser error : attributes construct error
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is
>>>> running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity:
>>>> line 1: parser error : Couldn't find end of Start Tag lrmd_notify line 1
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is
>>>> running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error: Entity:
>>>> line 1: parser error : Extra content at the end of the document
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> a-72fc-47e1-81b4-51b500c967f9" lrmd_rsc_output="tomcat6 (pid 3282) is
>>>> running...
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_xml_err: XML Error:
>>>> ^
>>>> Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: Parsing
>>>> failed (domain=1, level=3, code=5): Extra content at the end of the
>>>> document Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String
>>>> start:
>>>> <lrmd_notify lrmd_origin="send_cmd_complete_notify
>>>> Apr 25 09:46:22 xxxx crmd[1764]: warning: string2xml: String start+688:
>>>> 0000" CRM_meta_start_delay="15000"/></lrmd_notify>
>>>> Apr 25 09:46:22 xxxx crmd[1764]: error: crm_abort: string2xml: Forked
>>>> child 4182 to record non-fatal assert at xml.c:605 : String parsing error
>>
>> Also, it would be very useful if you could open up the core file for
>> 4182 and print the contents of the input passed to string2xml()
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
_______________________________________________
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
