Hello all,
I've run into an interesting problem and would welcome comments about
the possible solution.
It concerns rsyslog's TCP server and an uncommon scenario of running out
of file descriptors. The symptoms are that the daemon gets into a
busy-loop and hogs the CPU until another file descriptor becomes
available. It affects all versions since v6.1.5 till current master.
(I've actually only tested 7.4.2 and a patched v5 but judging from the
code changes.)
This is the setup:
$ rpm -q rsyslog
rsyslog-7.4.2-2.fc20.x86_64
$ cat >> /etc/rsyslog.conf << EOF
$ModLoad imtcp
$InputTCPServerRun 1234
EOF
$ service rsyslog stop
$ setenforce 0
$ ulimit -n 32
$ rsyslogd -n &
$ screen
$ export ctr=0; for i in {1..30}; do screen -X screen sh -c "while true;
do echo foo-$((ctr++)); sleep 30; done | nc localhost 1234"; sleep 0.5; done
It's a simple tcp listener. The 'ulimit' is there to speed up the
reproducer. screen is used to setup an arbitrary number of tcp clients.
Now the symptoms:
After the fd count reaches the limit, the next connection that goes over
the limit will make nsd_ptcp.c:294:accept() fail with errno EMFILE -
"The per-process limit of open file descriptors has been reached".
So far so good. The problem is that the code will just go back to poll
on the server socket (tcpsrv.c:889:nspoll.Wait()) which will return
immediately, call accept() and the whole cycle repeats.
Here's a strace snippet:
epoll_wait(7, {{EPOLLIN, {u32=3690991456, u64=140200013467488}}}, 128,
-1) = 1
accept(5, 0x7f82f33fcbb0, [128]) = -1 EMFILE (Too many open files)
epoll_wait(7, {{EPOLLIN, {u32=3690991456, u64=140200013467488}}}, 128,
-1) = 1
accept(5, 0x7f82f33fcbb0, [128]) = -1 EMFILE (Too many open files)
I'm thinking whether a backoff or some proactive way of clearing the
redundant connections would be the proper way to handle this corner case.
Tomas
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
