[
https://issues.apache.org/jira/browse/DISPATCH-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281303#comment-16281303
]
Saurabh Badhwar commented on DISPATCH-895:
------------------------------------------
A more indepth information from debug analysis on per thread basis:
[root@c10-h19-r730xd-vm3 ~]# gdb qdrouterd
/var/spool/abrt/ccpp-2017-12-05-09\:16\:21-14072/coredump -ex 'info thread' -ex
'thread apply all bt' -batch
[New LWP 14075]
[New LWP 14072]
[New LWP 14073]
[New LWP 14076]
[New LWP 14074]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/qdrouterd -c
/etc/qpid-dispatch/qdrouterd.conf'.
Program terminated with signal 11, Segmentation fault.
#0 pni_add_tpwork (delivery=0x7f0b881447c0) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:727
727 pn_connection_t *connection = delivery->link->session->connection;
Id Target Id Frame
5 Thread 0x7f0b9ed34700 (LWP 14074) 0x00007f0bac2a0c0b in send () from
/lib64/libpthread.so.0
4 Thread 0x7f0b9dd32700 (LWP 14076) 0x00007f0bac2a042d in __lll_lock_wait
() from /lib64/libpthread.so.0
3 Thread 0x7f0b9f748700 (LWP 14073) 0x00007f0bac29d945 in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
2 Thread 0x7f0bacb6b1c0 (LWP 14072) 0x00007f0baaed9052 in mul4x_internal
() from /lib64/libcrypto.so.10
* 1 Thread 0x7f0b9e533700 (LWP 14075) pni_add_tpwork
(delivery=0x7f0b881447c0) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:727
Thread 5 (Thread 0x7f0b9ed34700 (LWP 14074)):
#0 0x00007f0bac2a0c0b in send () from /lib64/libpthread.so.0
#1 0x00007f0bac72a5e7 in connector_process (c=0x7f0b94098150) at
/usr/src/debug/qpid-dispatch-0.8.0/src/posix/driver.c:776
#2 0x00007f0bac742d09 in process_connector (cxtr=<optimized out>,
qd_server=0x141a370) at /usr/src/debug/qpid-dispatch-0.8.0/src/server.c:734
#3 thread_run (arg=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/server.c:1012
#4 0x00007f0bac299e25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f0bab5cd34d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f0b9dd32700 (LWP 14076)):
#0 0x00007f0bac2a042d in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f0bac2a2c6c in _L_cond_lock_789 () from /lib64/libpthread.so.0
#2 0x00007f0bac2a2b01 in __pthread_mutex_cond_lock () from
/lib64/libpthread.so.0
#3 0x00007f0bac29d9d5 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#4 0x00007f0bac72b319 in sys_cond_wait (cond=<optimized out>,
held_mutex=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/posix/threading.c:91
#5 0x00007f0bac74292c in thread_run (arg=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/server.c:894
#6 0x00007f0bac299e25 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f0bab5cd34d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f0b9f748700 (LWP 14073)):
#0 0x00007f0bac29d945 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007f0bac72b319 in sys_cond_wait (cond=<optimized out>,
held_mutex=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/posix/threading.c:91
#2 0x00007f0bac7392d5 in router_core_thread (arg=0x1448e20) at
/usr/src/debug/qpid-dispatch-0.8.0/src/router_core/router_core_thread.c:66
#3 0x00007f0bac299e25 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f0bab5cd34d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0bacb6b1c0 (LWP 14072)):
#0 0x00007f0baaed9052 in mul4x_internal () from /lib64/libcrypto.so.10
#1 0x00007f0baaed92c4 in bn_power5 () from /lib64/libcrypto.so.10
#2 0x000000000175b600 in ?? ()
#3 0x00007ffd21c408f0 in ?? ()
#4 0x000000000175b380 in ?? ()
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f0b9e533700 (LWP 14075)):
#0 pni_add_tpwork (delivery=0x7f0b881447c0) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:727
#1 0x00007f0bac4cdcae in pn_delivery_update (delivery=<optimized out>,
state=<optimized out>) at
/usr/src/debug/qpid-proton-0.16.0/proton-c/src/core/engine.c:2026
#2 0x00007f0bac73dcc8 in CORE_delivery_update (context=0x144b970,
dlv=0x7f0b90074f10, disp=39, settled=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/router_node.c:1068
#3 0x00007f0bac732c5d in qdr_connection_process (conn=0x7f0b88030bd0) at
/usr/src/debug/qpid-dispatch-0.8.0/src/router_core/connections.c:306
#4 0x00007f0bac71e1d8 in writable_handler (container=0xe658c0,
container=0xe658c0, conn=<optimized out>, qd_conn=0x7f0b90000b10) at
/usr/src/debug/qpid-dispatch-0.8.0/src/container.c:307
#5 handler (handler_context=0xe658c0, conn_context=0x1294f40, event=<optimized
out>, qd_conn=0x7f0b90000b10) at
/usr/src/debug/qpid-dispatch-0.8.0/src/container.c:614
#6 0x00007f0bac742ccb in process_connector (cxtr=<optimized out>,
qd_server=0x141a370) at /usr/src/debug/qpid-dispatch-0.8.0/src/server.c:792
#7 thread_run (arg=<optimized out>) at
/usr/src/debug/qpid-dispatch-0.8.0/src/server.c:1012
#8 0x00007f0bac299e25 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f0bab5cd34d in clone () from /lib64/libc.so.6
> qpid-dispatch crashes with a SEGFAULT in libqpid-proton
> -------------------------------------------------------
>
> Key: DISPATCH-895
> URL: https://issues.apache.org/jira/browse/DISPATCH-895
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Routing Engine
> Affects Versions: 0.8.0
> Environment: OS: Red Hat Enterprise Linux 7.4
> Arch: x86_64
> Installed Packages
> Name : qpid-cpp-client
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 4.4 M
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : Libraries for Qpid C++ client applications
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : Run-time libraries for AMQP client applications developed using
> Qpid
> : C++. Clients exchange messages with an AMQP message broker using
> : the AMQP protocol.
> Name : qpid-cpp-server
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 5.2 M
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : An AMQP message broker daemon
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : A message broker daemon that receives stores and routes
> messages using
> : the open AMQP messaging protocol.
> Name : qpid-cpp-server-linearstore
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 797 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : Red Hat persistence extension to the Qpid messaging sytem
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : Red Hat persistence extension to the Qpid AMQP broker:
> persistent message
> : storage using a libaio-based asynchronous journal.
> Name : qpid-dispatch-debuginfo
> Arch : x86_64
> Version : 0.8.0
> Release : 10.el7
> Size : 2.1 M
> Repo : installed
> Summary : Debug information for package qpid-dispatch
> URL : http://qpid.apache.org/
> License : ASL 2.0
> Description : This package provides debug information for package
> qpid-dispatch.
> : Debug information is useful when developing applications that
> use this
> : package or when debugging this package.
> Name : qpid-dispatch-router
> Arch : x86_64
> Version : 0.8.0
> Release : 10.el7
> Size : 1.0 M
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : The Qpid Dispatch Router executable
> URL : http://qpid.apache.org/
> License : ASL 2.0
> Description : The Qpid Dispatch Router executable.
> Name : qpid-proton-c
> Arch : x86_64
> Version : 0.16.0
> Release : 7.el7
> Size : 334 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : C libraries for Qpid Proton
> URL : http://qpid.apache.org/proton/
> License : ASL 2.0
> Description : C libraries for Qpid Proton.
> Name : qpid-proton-debuginfo
> Arch : x86_64
> Version : 0.16.0
> Release : 7.el7
> Size : 6.8 M
> Repo : installed
> Summary : Debug information for package qpid-proton
> URL : http://qpid.apache.org/proton/
> License : ASL 2.0
> Description : This package provides debug information for package qpid-proton.
> : Debug information is useful when developing applications that
> use this
> : package or when debugging this package.
> Name : qpid-qmf
> Arch : x86_64
> Version : 1.36.0
> Release : 9.el7
> Size : 476 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : The QPID Management Framework
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : The Qpid Management Framework is a general-purpose management
> bus built on Qpid
> : messaging. It takes advantage of the scalability, security, and
> rich
> : capabilities of Qpid to provide flexible and easy-to-use
> manageability to a
> : large set of applications.
> Name : qpid-tools
> Arch : noarch
> Version : 1.36.0
> Release : 9.el7
> Size : 384 k
> Repo : installed
> From repo : rhel-server-7-satellite-6-beta-rpms
> Summary : Management and diagostic tools for Apache Qpid
> URL : http://qpid.apache.org
> License : ASL 2.0
> Description : Management and diagnostic tools for Apache Qpid brokers and
> clients.
> Reporter: Saurabh Badhwar
>
> During the work process associated with Red Hat Satellite, we found that
> Qpid-dispatch router started to crash abruptly without any changes being made
> to it. On checking the logs, it seems like qpid-dispatch-router experienced a
> SEGFAULT and crashed.
> Once this crash starts to happen there is no way we are able to restart
> qpid-dispatch since it continues to fail.
> A trace from the /var/log/messages
> [root@c10-h19-r730xd-vm1 qdrouterd]# cat /var/log/messages | grep qdrouterd
> Nov 28 00:59:50 c10-h19-r730xd-vm1 kernel: qdrouterd[28352]: segfault at 180
> ip 00007fdd86d37291 sp 00007fdd7951c2b8 error 4 in
> libqpid-proton.so.10.0.0[7fdd86d19000+4b000]
> Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 03:44:04 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 03:45:58 c10-h19-r730xd-vm1 kernel: qdrouterd[26607]: segfault at 180
> ip 00007f9f4d230291 sp 00007f9f3fa152b8 error 4 in
> libqpid-proton.so.10.0.0[7f9f4d212000+4b000]
> Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 03:50:12 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:06:07 c10-h19-r730xd-vm1 ansible-file: Invoked with
> directory_mode=None force=False remote_src=None
> path=/etc/systemd/system/qdrouterd.service.d owner=root follow=False
> group=root unsafe_writes=None state=directory content=NOT_LOGGING_PARAMETER
> serole=None diff_peek=None setype=None selevel=None original_basename=None
> regexp=None validate=None src=None seuser=None recurse=False delimiter=None
> mode=0755 attributes=None backup=None
> Nov 30 04:06:10 c10-h19-r730xd-vm1 ansible-stat: Invoked with
> checksum_algorithm=sha1 get_checksum=True
> path=/etc/systemd/system/qdrouterd.service.d/limits.conf checksum_algo=sha1
> follow=False get_md5=False get_mime=True get_attributes=True
> Nov 30 04:06:13 c10-h19-r730xd-vm1 ansible-copy: Invoked with
> directory_mode=None force=True remote_src=None owner=root follow=False
> local_follow=None group=root unsafe_writes=None setype=None
> content=NOT_LOGGING_PARAMETER serole=None
> dest=/etc/systemd/system/qdrouterd.service.d/limits.conf selevel=None
> original_basename=limits-qdrouterd.conf regexp=None validate=None
> src=/root/.ansible/tmp/ansible-tmp-1512032767.73-18163995376563/source
> seuser=None delimiter=None mode=0644 attributes=None backup=False
> Nov 30 04:06:15 c10-h19-r730xd-vm1 ansible-command: Invoked with warn=True
> executable=None _uses_shell=False _raw_params=restorecon -vR
> /etc/systemd/system/qdrouterd.service.d removes=None creates=None chdir=None
> stdin=None
> Nov 30 04:06:42 c10-h19-r730xd-vm1 kernel: qdrouterd[29818]: segfault at 180
> ip 00007f0863215291 sp 00007ffd3666b678 error 4 in
> libqpid-proton.so.10.0.0[7f08631f7000+4b000]
> Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 04:06:58 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:08:02 c10-h19-r730xd-vm1 ansible-file: Invoked with
> directory_mode=None force=False remote_src=None
> path=/etc/systemd/system/qdrouterd.service.d owner=root follow=False
> group=root unsafe_writes=None state=directory content=NOT_LOGGING_PARAMETER
> serole=None diff_peek=None setype=None selevel=None original_basename=None
> regexp=None validate=None src=None seuser=None recurse=False delimiter=None
> mode=0755 attributes=None backup=None
> Nov 30 04:08:05 c10-h19-r730xd-vm1 ansible-stat: Invoked with
> checksum_algorithm=sha1 get_checksum=True
> path=/etc/systemd/system/qdrouterd.service.d/limits.conf checksum_algo=sha1
> follow=False get_md5=False get_mime=True get_attributes=True
> Nov 30 04:08:07 c10-h19-r730xd-vm1 ansible-file: Invoked with
> directory_mode=None force=False remote_src=None
> path=/etc/systemd/system/qdrouterd.service.d/limits.conf owner=root
> follow=False group=root unsafe_writes=None state=file
> content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None
> dest=/etc/systemd/system/qdrouterd.service.d/limits.conf selevel=None
> original_basename=limits-qdrouterd.conf regexp=None validate=None
> src=limits-qdrouterd.conf seuser=None recurse=False delimiter=None mode=0644
> attributes=None backup=None
> Nov 30 04:08:09 c10-h19-r730xd-vm1 ansible-command: Invoked with warn=True
> executable=None _uses_shell=False _raw_params=restorecon -vR
> /etc/systemd/system/qdrouterd.service.d removes=None creates=None chdir=None
> stdin=None
> Nov 30 04:08:20 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:08:47 c10-h19-r730xd-vm1 kernel: qdrouterd[31775]: segfault at 180
> ip 00007f56880e4291 sp 00007f56799852b8 error 4 in
> libqpid-proton.so.10.0.0[7f56880c6000+4b000]
> Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 04:09:06 c10-h19-r730xd-vm1 kernel: qdrouterd[32150]: segfault at 180
> ip 00007fb455eea291 sp 00007fb446f8a2b8 error 4 in
> libqpid-proton.so.10.0.0[7fb455ecc000+4b000]
> Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process
> exited, code=killed, status=11/SEGV
> Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered
> failed state.
> Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
> Nov 30 04:09:21 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> Nov 30 04:11:20 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status
> failed: qdrouterd ***
> From the messages, it seems like, the SEGFAULT happen in libqpid-proton.so
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
