Hello Alan, I've seen a similar problem while updating from 7 to 8 on Ubuntu 14.04 (which has 7.x by default). The new one didn't start, I saw the defunct process, killed everything and from then on restarts worked. But I didn't try to reproduce the problem, so I have no idea what the problem/solutions might be :(
-- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Jul 30, 2014 at 6:00 AM, Edmonds, Alan <[email protected]> wrote: > > I'm seeing a strange startup deadlock issue. The basic problem is when > rsyslog starts, it hangs during /etc/init.d script processing. > > This is using the centos/redhat adiscon repo for centos 5. I only > encountered this problem when upgrading from rsyslog v7 to rsyslog v8. > > 1) rsyslog 7 does not have this problem. > 2) using the v7_stable and v8_stable RPMs. > 3) it hangs on some of the servers but not all. It is repeatable though. > 4) When it happens, it happens on both CentOS 5.9 and 5.10. > 5) I'm using the stock rsyslog config from the RPMs. > 5) if I kill the child, it is a <zombie> > > It is some kind of race condition. Here's what it looks like: > > DR/UAT [root@drukrf56 ~]# ps -efL| grep rsyslog > root 2699 2203 2699 0 1 12:37 ? 00:00:00 /bin/bash > /etc/rc3.d/S26rsyslog start > root 2702 2699 2702 0 1 12:37 ? 00:00:00 /bin/bash -c > ulimit -S -c 0 >/dev/null 2>&1 ; /sbin/rsyslogd -i /var/run/rsyslogd.pid -d > root 2703 2702 2703 0 1 12:37 ? 00:00:00 /sbin/rsyslogd > -i /var/run/rsyslogd.pid -d > root 2704 2703 2704 0 4 12:37 ? 00:00:00 /sbin/rsyslogd > -i /var/run/rsyslogd.pid -d > root 2704 2703 2705 0 4 12:37 ? 00:00:00 /sbin/rsyslogd > -i /var/run/rsyslogd.pid -d > root 2704 2703 2706 0 4 12:37 ? 00:00:00 /sbin/rsyslogd > -i /var/run/rsyslogd.pid -d > root 2704 2703 2707 0 4 12:37 ? 00:00:00 /sbin/rsyslogd > -i /var/run/rsyslogd.pid -d > root 2912 2845 2912 0 1 13:47 pts/0 00:00:00 grep rsyslog > > DR/UAT [root@drukrf56 ~]# strace -p 2703 > Process 2703 attached - interrupt to quit > futex(0x2b27ab66a200, FUTEX_WAIT_PRIVATE, 1, NULL > > <unfinished ...> > Process 2703 detached > DR/UAT [root@drukrf56 ~]# strace -p 2704 > Process 2704 attached - interrupt to quit > select(1, NULL, NULL, NULL, {83914, 94000} > > <unfinished ...> > Process 2704 detached > > Messages are being saved to /var/log/messages as expected. It just seems > like the parent doesn't realize the child has completed startup. > > If I kill the child, the parent doesn't notice. > > DR/UAT [root@drukrf56 ~]# kill 2704 > DR/UAT [root@drukrf56 ~]# ps aux | grep rsys > root 2699 0.0 0.0 10924 1448 ? S 12:37 0:00 /bin/bash > /etc/rc3.d/S26rsyslog start > root 2702 0.0 0.0 10788 1172 ? S 12:37 0:00 /bin/bash > -c ulimit -S -c 0 >/dev/null 2>&1 ; /sbin/rsyslogd -i /var/run/rsyslogd.pid > -d > root 2703 0.0 0.0 42724 1916 ? S 12:37 0:00 > /sbin/rsyslogd -i /var/run/rsyslogd.pid -d > root 2704 0.0 0.0 0 0 ? Z 12:37 0:00 > [rsyslogd] <defunct> > root 2939 0.0 0.0 6056 624 pts/0 S+ 13:53 0:00 grep rsys > > > If I kill -9 the child and the parent, then start rsyslog > (/etc/init.d/rsyslog start), it starts normally. > > One other interesting thing is if I remove the rsyslog from the startup, > and put /etc/init.d/rsyslog start in /etc/rc.local (ie - at the end), it > works. > > Has anyone else seen this kind of problem? > > Alan Edmonds > > > > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
