this sounds like rsyslog is failing to send the logs out to the RELP server, and so is building up a large queue. restarting rsyslog would clear the queued up log messages and make it fast again.
David Lang On Tue, 5 Jan 2010, Kenneth Marshall wrote: > Date: Tue, 5 Jan 2010 13:53:49 -0600 > From: Kenneth Marshall <[email protected]> > Reply-To: rsyslog-users <[email protected]> > To: [email protected] > Cc: [email protected] > Subject: [rsyslog] rsyslog hang with imklog (/dev/log) on a RHEL5 machine > > I am running rsyslog version 4.2.0 on a Redhat 5 machine > and noticed slow logins to the box. The strace on the login > sshd shows the following: > > 9937 0.000045 socket(PF_FILE, SOCK_DGRAM, 0) = 4 > 9937 0.000025 fcntl(4, F_SETFD, FD_CLOEXEC) = 0 > 9937 0.000019 connect(4, {sa_family=AF_FILE, path="/dev/log"...}, 110) > = 0 > 9937 0.000040 sendto(4, "<86>Jan 5 13:36:12 sshd[9937]: "..., 90, > MSG_NOSIGNAL, NULL, 0) = ? ERESTARTSYS (To be restarted) > 9937 0.000042 --- SIGCHLD (Child exited) @ 0 (0) --- > 9937 0.000018 sendto(4, "<86>Jan 5 13:36:12 sshd[9937]: "..., 90, > MSG_NOSIGNAL, NULL, 0 <unfinished ...> > 5095 7.001495 <... select resumed> ) = ? ERESTARTNOHAND (To be > restarted) > 5095 0.000040 --- SIGCHLD (Child exited) @ 0 (0) --- > 5095 0.000025 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], > WNOHANG, NULL) = 9844 > 5095 0.000055 wait4(-1, 0x7fffbf6d198c, WNOHANG, NULL) = 0 > 5095 0.000021 rt_sigaction(SIGCHLD, NULL, {0x2ad5c3ab2740, [], > SA_RESTORER, 0x2ad5c65922d0}, 8) = 0 > 5095 0.000028 rt_sigreturn(0x11) = -1 EINTR (Interrupted system call) > 5095 0.000027 select(7, [3 5], NULL, NULL, NULL <unfinished ...> > 9937 8.001608 <... sendto resumed> ) = 90 > 9937 0.000028 close(4) = 0 > 9937 0.000039 read(6, "\0\0\5\36", 4) = 4 > 9937 0.000037 read(6, "\31\0\0\0\24'\363w{\376B\364Ye > !\365\232\216\220\352\343\"\262\334\0\0\0\20\0\0\0"..., 1310) = 1310 > 9937 0.000104 close(6) = 0 > 9937 0.000029 mmap(NULL, 1310720, PROT_READ|PROT_WRITE, > MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 0x2ad8627d9000 > 9937 0.000074 munmap(0x2ad85caed000, 65536) = 0 > 9937 0.000037 wait4(9938, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, > NULL) = 9938 > 9937 0.000032 alarm(0) = 102 > 9937 0.000023 rt_sigaction(SIGALRM, NULL, {0x2ad85c8637a0, [], > SA_RESTORER|SA_INTERRUPT, 0x2ad85f3432d0}, 8) = 0 > 9937 0.000029 rt_sigaction(SIGALRM, {SIG_DFL, [], > SA_RESTORER|SA_INTERRUPT, 0x2ad85f3432d0}, NULL, 8) = 0 > ... > > The problem seems to be caused by writing to /dev/log which should > be being managed by the rsyslog program. I see a similar problem > reported earlier on the forum: > > rsyslog hangs with imklog + omrelp (Same bug a imuxlog FC ?) > > This was for version 3.18.4 but the symptom sounded very similar. > I restarted the rsyslog process and the login times returned to normal. > Let me know if there is something further I can do to help you debug > this matter. > > Regards, > Ken > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
