rsyslog crashes after a few seconds on redhat enterprise linux 6.2. gdb info:
[root@spip tmp]# gdb /usr/sbin/rsyslogd /tmp/core-rs\:main\ Q\:Reg-11-0-0-11100-1339415918 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/rsyslogd...done. [New Thread 11105] [New Thread 11106] [New Thread 11107] [New Thread 11108] [New Thread 11109] [New Thread 11100] [New Thread 11101] [New Thread 11102] [New Thread 11103] [New Thread 11104] Missing separate debuginfo for /usr/lib64/libee.so.0 Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/26/3bc5ceafcb55a9fb9f5c4ae5a4e019c775aab3 Missing separate debuginfo for /usr/lib64/libestr.so.0 Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/15/681c37f2f6a1dd2a6190386635a7229e1914ab Missing separate debuginfo for /usr/lib64/rsyslog/lmnet.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/2d/360a22d60889862f301cd8d37fd9a1a1e0d107 Missing separate debuginfo for /usr/lib64/rsyslog/imuxsock.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/aa/45b277a2f11088b520394287405c9e51358534 Missing separate debuginfo for /usr/lib64/rsyslog/imklog.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/da/2f2093ba0218533b61e31cd8ef10430982a411 Missing separate debuginfo for /usr/lib64/rsyslog/imudp.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/f1/0363af97bc6c333bda7f9a1f4cf83412a97620 Missing separate debuginfo for /usr/lib64/rsyslog/imtcp.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/18/a5b382bdd6080f189c8f1d7c3980e2980888b6 Missing separate debuginfo for /usr/lib64/rsyslog/lmnetstrms.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/21/04a207202ae056e2c0528540b6207647a3dd41 Missing separate debuginfo for /usr/lib64/rsyslog/lmtcpsrv.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/8b/3884eb17f5eb053ef309a1af492042b2504e79 Missing separate debuginfo for /usr/lib64/rsyslog/ommongodb.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/3c/e13e40b6a06c38813268d1c0ded376c5a0bbe2 Missing separate debuginfo for /usr/lib64/libmongo-client.so.0 Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/15/78059d9c73e4553a2d6be2a39fa64c406443e5 Missing separate debuginfo for /usr/lib64/rsyslog/lmnsd_ptcp.so Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/fa/7df5e39d94b1cd5cf7b3cda7d5970942c2f36b Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/3b/5709b4dddefce3dd0b4bae733298285d52c602 Reading symbols from /lib64/libz.so.1.2.3...Reading symbols from /usr/lib/debug/lib64/libz.so.1.2.3.debug...done. done. Loaded symbols for /lib64/libz.so.1.2.3 Reading symbols from /lib64/libpthread-2.12.so...Reading symbols from /usr/lib/debug/lib64/libpthread-2.12.so.debug...done. [Thread debugging using libthread_db enabled] done. Loaded symbols for /lib64/libpthread-2.12.so Reading symbols from /lib64/libdl-2.12.so...Reading symbols from /usr/lib/debug/lib64/libdl-2.12.so.debug...done. done. Loaded symbols for /lib64/libdl-2.12.so Reading symbols from /lib64/librt-2.12.so...Reading symbols from /usr/lib/debug/lib64/librt-2.12.so.debug...done. done. Loaded symbols for /lib64/librt-2.12.so Reading symbols from /usr/lib64/libee.so.0...done. Loaded symbols for /usr/lib64/libee.so.0 Reading symbols from /usr/lib64/libestr.so.0...done. Loaded symbols for /usr/lib64/libestr.so.0 Reading symbols from /lib64/libm-2.12.so...Reading symbols from /usr/lib/debug/lib64/libm-2.12.so.debug...done. done. Loaded symbols for /lib64/libm-2.12.so Reading symbols from /lib64/libc-2.12.so...Reading symbols from /usr/lib/debug/lib64/libc-2.12.so.debug...done. done. Loaded symbols for /lib64/libc-2.12.so Reading symbols from /lib64/ld-2.12.so...Reading symbols from /usr/lib/debug/lib64/ld-2.12.so.debug...done. done. Loaded symbols for /lib64/ld-2.12.so Reading symbols from /usr/lib64/rsyslog/lmnet.so...done. Loaded symbols for /usr/lib64/rsyslog/lmnet.so Reading symbols from /usr/lib64/rsyslog/imuxsock.so...done. Loaded symbols for /usr/lib64/rsyslog/imuxsock.so Reading symbols from /usr/lib64/rsyslog/imklog.so...done. Loaded symbols for /usr/lib64/rsyslog/imklog.so Reading symbols from /usr/lib64/rsyslog/imudp.so...done. Loaded symbols for /usr/lib64/rsyslog/imudp.so Reading symbols from /usr/lib64/rsyslog/imtcp.so...done. Loaded symbols for /usr/lib64/rsyslog/imtcp.so Reading symbols from /usr/lib64/rsyslog/lmnetstrms.so...done. Loaded symbols for /usr/lib64/rsyslog/lmnetstrms.so Reading symbols from /usr/lib64/rsyslog/lmtcpsrv.so...done. Loaded symbols for /usr/lib64/rsyslog/lmtcpsrv.so Reading symbols from /usr/lib64/rsyslog/ommongodb.so...done. Loaded symbols for /usr/lib64/rsyslog/ommongodb.so Reading symbols from /usr/lib64/libmongo-client.so.0...done. Loaded symbols for /usr/lib64/libmongo-client.so.0 Reading symbols from /lib64/libglib-2.0.so.0.2200.5...Reading symbols from /usr/lib/debug/lib64/libglib-2.0.so.0.2200.5.debug...done. done. Loaded symbols for /lib64/libglib-2.0.so.0.2200.5 Reading symbols from /usr/lib64/rsyslog/lmnsd_ptcp.so...done. Loaded symbols for /usr/lib64/rsyslog/lmnsd_ptcp.so Reading symbols from /lib64/libnss_files-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_files-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Reading symbols from /lib64/libresolv-2.12.so...Reading symbols from /usr/lib/debug/lib64/libresolv-2.12.so.debug...done. done. Loaded symbols for /lib64/libresolv-2.12.so Core was generated by `/usr/sbin/rsyslogd -f /etc/rsyslog.conf.v6.mongodb'. Program terminated with signal 11, Segmentation fault. #0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at malloc.c:4879 4879 old_idx = fastbin_index(chunksize(old)); (gdb) info thread 10 Thread 0x7f42b1f36700 (LWP 11104) 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 9 Thread 0x7f42b2937700 (LWP 11103) 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 8 Thread 0x7f42b3338700 (LWP 11102) 0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82 7 Thread 0x7f42b3d39700 (LWP 11101) 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82 6 Thread 0x7f42b692c700 (LWP 11100) 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82 5 Thread 0x7f42b6894700 (LWP 11109) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 4 Thread 0x7f42b68c6700 (LWP 11108) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 3 Thread 0x7f42b68f8700 (LWP 11107) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 2 Thread 0x7f42b692a700 (LWP 11106) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 * 1 Thread 0x7f42b1535700 (LWP 11105) _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at malloc.c:4879 (gdb) thread 1 [Switching to thread 1 (Thread 0x7f42b1535700 (LWP 11105))]#0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at malloc.c:4879 4879 old_idx = fastbin_index(chunksize(old)); (gdb) bt #0 _int_free (av=0x7f42a4000020, p=0x7f42a4007d50, have_lock=1) at malloc.c:4879 #1 0x00007f42b5535900 in _int_realloc (av=0x7f42a4000020, oldp=0x7f42a4007d50, oldsize=<value optimized out>, nb=<value optimized out>) at malloc.c:5345 #2 0x00007f42b5535af5 in __libc_realloc (oldmem=0x7f42a4007d60, bytes=32) at malloc.c:3822 #3 0x00007f42b3f81621 in IA__g_realloc (mem=<value optimized out>, n_bytes=32) at gmem.c:171 #4 0x00007f42b3f542db in g_array_maybe_expand (array=0x7f42a4002e40, len=<value optimized out>) at garray.c:416 #5 0x00007f42b3f54712 in IA__g_array_append_vals (farray=<value optimized out>, data=0x7f42a4004550, len=8) at garray.c:207 #6 0x00007f42b3f54769 in IA__g_byte_array_append (array=0x7f42a4002e40, data=<value optimized out>, len=<value optimized out>) at garray.c:880 #7 0x00007f42b422bc19 in _bson_append_string_element (b=0x7f42a40078b0, type=<value optimized out>, name=0x7f42b443a1c4 "sys", val=<value optimized out>, length=<value optimized out>) at bson.c:138 #8 0x00007f42b422d24c in bson_build (type=BSON_TYPE_STRING, name=0x7f42b443a1c4 "sys") at bson.c:458 #9 0x00007f42b4439cf0 in writeMongoDB_msg (pMsg=<value optimized out>, pData=0x1dc1550) at ommongodb.c:261 #10 0x0000000000446dd6 in actionCallDoAction (pThis=0x1dc1910, pMsg=0x7f42a8000df0, actParams=0x1ddca50) at ../action.c:915 #11 0x000000000044737f in actionProcessMessage (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at ../action.c:962 #12 tryDoAction (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at ../action.c:1057 #13 submitBatch (pAction=0x1dc1910, pBatch=0x1dafb80, nElem=2) at ../action.c:1133 #14 0x0000000000447790 in processAction (pAction=0x1dc1910, pBatch=0x1dafb80, pbShutdownImmediate=<value optimized out>) at ../action.c:1224 #15 processBatchMain (pAction=0x1dc1910, pBatch=0x1dafb80, pbShutdownImmediate=<value optimized out>) at ../action.c:1258 #16 0x0000000000444fdf in doQueueEnqObjDirectBatch (pAction=0x1dc1910, pBatch=0x1dafb80) at ../action.c:1727 #17 0x0000000000445c2b in doSubmitToActionQNotAllMarkBatch (pAction=0x1dc1910, pBatch=0x1dafb80) at ../action.c:1636 #18 0x0000000000440bf3 in processBatchDoActions (pData=0x1dc1910, pParam=<value optimized out>) at rule.c:103 #19 0x000000000042aec7 in llExecFunc (pThis=0x1dc12a0, pFunc=0x440bc0 <processBatchDoActions>, pParam=0x1dafb80) at linkedlist.c:389 #20 0x00000000004411f3 in processBatch (pThis=0x1dc1240, pBatch=0x1dafb80) at rule.c:293 #21 0x000000000043ff22 in processBatchDoRules (pData=0x1dc1240, pParam=0x1dafb80) at ruleset.c:147 #22 0x000000000042aec7 in llExecFunc (pThis=0x1da3a68, pFunc=0x43fee0 <processBatchDoRules>, pParam=0x1dafb80) at linkedlist.c:389 #23 0x00000000004401fa in processBatch (pBatch=0x1dafb80) at ruleset.c:229 #24 0x000000000040d056 in msgConsumer (notNeeded=<value optimized out>, pBatch=0x1dafb80, pbShutdownImmediate=0x1dc8eb0) at syslogd.c:608 #25 0x000000000043f42b in ConsumerReg (pThis=0x1dc8e90, pWti=0x1dafb50) at queue.c:1791 #26 0x000000000043a415 in wtiWorker (pThis=0x1dafb50) at wti.c:313 #27 0x0000000000438efa in wtpWorker (arg=0x1dafb50) at wtp.c:387 #28 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b1535700) at pthread_create.c:301 #29 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 2 [Switching to thread 2 (Thread 0x7f42b692a700 (LWP 11106))]#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 162 62: movl (%rsp), %edi (gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842da0) at tcpsrv.c:629 #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b692a700) at pthread_create.c:301 #3 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 3 [Switching to thread 3 (Thread 0x7f42b68f8700 (LWP 11107))]#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 162 62: movl (%rsp), %edi (gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842e08) at tcpsrv.c:629 #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b68f8700) at pthread_create.c:301 #3 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 4 [Switching to thread 4 (Thread 0x7f42b68c6700 (LWP 11108))]#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 162 62: movl (%rsp), %edi (gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842e70) at tcpsrv.c:629 #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b68c6700) at pthread_create.c:301 #3 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 5 [Switching to thread 5 (Thread 0x7f42b6894700 (LWP 11109))]#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 162 62: movl (%rsp), %edi (gdb) bt #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162 #1 0x00007f42b46406bf in wrkr (myself=0x7f42b4842ed8) at tcpsrv.c:629 #2 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b6894700) at pthread_create.c:301 #3 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 6 [Switching to thread 6 (Thread 0x7f42b692c700 (LWP 11100))]#0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82 #1 0x000000000040e36d in mainloop (argc=<value optimized out>, argv=<value optimized out>) at syslogd.c:1314 #2 realMain (argc=<value optimized out>, argv=<value optimized out>) at syslogd.c:2061 #3 0x00007f42b54d9cdd in __libc_start_main (main=0x40e750 <main>, argc=3, ubp_av=0x7fff59743d08, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7fff59743cf8) at libc-start.c:226 #4 0x000000000040bf89 in _start () (gdb) thread 7 [Switching to thread 7 (Thread 0x7f42b3d39700 (LWP 11101))]#0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007f42b55998b3 in select () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f42b50648bb in runInput (pThrd=<value optimized out>) at imuxsock.c:1140 #2 0x000000000044857a in thrdStarter (arg=0x1ddd700) at ../threads.c:210 #3 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b3d39700) at pthread_create.c:301 #4 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 8 [Switching to thread 8 (Thread 0x7f42b3338700 (LWP 11102))]#0 0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007f42b62f64ed in read () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f42b4e513d8 in readklog () at bsd.c:224 #2 0x00007f42b4e51509 in klogLogKMsg (pModConf=<value optimized out>) at bsd.c:282 #3 0x00007f42b4e50b73 in runInput (pThrd=0x1ddd8f0) at imklog.c:271 #4 0x000000000044857a in thrdStarter (arg=0x1ddd8f0) at ../threads.c:210 #5 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b3338700) at pthread_create.c:301 #6 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 9 [Switching to thread 9 (Thread 0x7f42b2937700 (LWP 11103))]#0 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f42b4c4d0a8 in rcvMainLoop (pThrd=0x1dddae0) at imudp.c:593 #2 0x000000000044857a in thrdStarter (arg=0x1dddae0) at ../threads.c:210 #3 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b2937700) at pthread_create.c:301 #4 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) thread 10 [Switching to thread 10 (Thread 0x7f42b1f36700 (LWP 11104))]#0 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 82 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007f42b55a12c3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #1 0x00007f42b3d3da4f in Wait (pNsdpoll=0x7f42ac000e10, timeout=-1, numEntries=0x7f42b1f35dbc, workset=0x7f42b1f35170) at nsdpoll_ptcp.c:242 #2 0x00007f42b464038c in Run (pThis=0x1dae7d0) at tcpsrv.c:884 #3 0x000000000044857a in thrdStarter (arg=0x1dddcd0) at ../threads.c:210 #4 0x00007f42b62ef7f1 in start_thread (arg=0x7f42b1f36700) at pthread_create.c:301 #5 0x00007f42b55a0ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) If you need more, please let me know, but remember: i am not a C programmer, Stefan Arts Holland. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards
