remember that there are many files involved
/etc/default/rsyslog
/etc/init/rsyslog.conf
/etc/init.d/rsyslog
you have to put the right thing in the right file
ulimit -n # is what you would want in the final script that starts rsyslog
ulimit nofile # # is what you would want in the upstart config file. If you try
to put the one type inside others, it won't work as expected.
Also, with the upstart config changes, you want to do service rsyslog stop;
service rsyslog start, not service rsyslog restart (and you may actually need to
reboot, I don't remember) to make sure the new version of the file is in effect.
David Lang
On Tue, 15 Dec 2015, Giles Thomas wrote:
Date: Tue, 15 Dec 2015 20:32:37 +0000
From: Giles Thomas <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: [rsyslog] Increasing the number of available file handles in Ubuntu
14.04
Hi all,
After much battling I've managed to increase the number of file handles
available to rsyslog in Ubuntu 14.04. It was significantly harder than you
might expect, even given that Ubuntu uses upstart, which does strange stuff
with ulimits. So I thought it would be worth sharing the trick that works
with the list. Perhaps someone could even explain to me *why* it works :-)
Or, even better, tell me a better way to do it. Or both...
Scenario: I wanted to increase the number of available file handles for
syslog, because I'm writing to a large number of files via a large number of
queues.
Short version: you need to put "ulimit -n 16384" (or whatever) inside
/etc/default/rsyslog. The normal way of setting the open file limit for an
upstart job (ie. "limit nofile XXX") does not work, at least for me. Putting
the ulimit call inside the upstart rsyslog.conf doesn't work either, which is
weird (for reasons explained below).
Long version: the default Ubuntu upstart script for starting rsyslog looks
like this:
# rsyslog - system logging daemon
#
# rsyslog is an enhanced multi-threaded replacement for the traditional
# syslog daemon, logging messages from applications
description "system logging daemon"
start on filesystem
stop on runlevel [06]
expect fork
respawn
pre-start script
/lib/init/apparmor-profile-load usr.sbin.rsyslogd
end script
script
. /etc/default/rsyslog
exec rsyslogd $RSYSLOGD_OPTIONS
end script
Now, upstart does its own ulimit handling, and changes to limits.conf don't
affect processes launched by it, so the normal way to increase the number of
file handles is to add a line like this just after the "expect
fork"/"respawn" bit:
limit nofile 16384 16384
Indeed, David Lang suggested doing exactly that a couple of months ago in a
different thread (archive
<http://lists.adiscon.net/pipermail/rsyslog/2015-October/041404.html>).
But, weirdly, it doesn't work, at least for me. Inspecting
/proc/<rsyslog-pid>/limits after adding that line and bouncing rsyslog showed
that the file handle limit was still 1024 soft, 4096 hard. And I found that
my system was not writing to a number of the files it was meant to; "ls
/proc/<rsyslog-pid>/fd/ | wc -l" confirmed that it was capping out at 1024
files open.
My first guess was that it's something to do with the fact that Ubuntu's
upstart script launches rsyslog via a "script" section. The normal way to
start a process in upstart is simply something like
exec <binary> <options>
...without a "script" block around it. The script block just allows you to
add normal shell script to the upstart config so that you can do something
more complicated. So, I thought, perhaps the "limit nofile" stuff only works
if you're using a simple exec, and for scripts you have to do stuff manually.
I added a
ulimit -n 16384
...inside the script tag. But that didn't work.
The next thing was investigate *why* the default upstart config uses the
script tag. Best guess is that it's there so that the contents of
/etc/default/rsyslog can be read, so that configuration options can be placed
in that file. Looking inside /etc/default/rsyslog showed that the default
file simply contained this:
# Options for rsyslogd
# -x disables DNS lookups for remote messages
# See rsyslogd(8) for more details
RSYSLOGD_OPTIONS=""
So, this was basically a no-op -- rsyslog was being run with no options by
default. Because I hadn't needed to customise it, that meant that I could
rewrite the upstart script so that the whole "script...end script" block was
replaced by
exec rsyslogd
If the problem was the script block not playing well with the "limit nofile"
option, then you'd think that would work. But it didn't.
By this point (5 days in, working on this intermittently) I was clutching at
straws, and as a last ditch attempt I tried reverting back to the original
upstart script (so, no "limit nofile" bit, and the original "script...end
script" stuff sourcing the file from /etc/default), and put a call to "ulimit
-n 16384" inside /etc/default/rsyslog, and restarted rsyslog.
It worked.
This seems really weird to me. I can sort-of-kind-of see why the normal
"limit nofile" upstart thing might not work, given that rsyslog forks and
daemonises itself, though it seems odd that it does work for other processes
(and there's a "expect fork" line in there to tell it to handle that kind of
behaviour).
But the fact that the ulimit has to be in /etc/default/rsyslog completely
baffles me. If it works when it's in a file that is sourced into the script
block, then why does it not work inside the script block directly? I'm
pretty sure that in normal shell commands, using "." is basically equivalent
to running the commands in the file inside your current shell, one-by-one...
Anyway, I hope the solution is useful to someone out there, and that the
(somewhat rambling) explanation is of interest to someone too. And if anyone
does know the reason why it works in /etc/default/rsyslog but not in the
script block, or knows a better solution to the problem, I'd love to hear!
All the best,
Giles
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
