Hello,
I'm experiencing an issue with intermittent rsyslog hangs requiring a
restart. I'm running 5.8.5. I have ommysql running, and am doing a fair
amount of parsing (regex) into MySQL. I have a peak load of approx. 43
msgs/sec. according to impstats (which is running every 120 seconds at
the moment). The hangs are happening anywhere from 30 minutes to 6
hours, and don't appear to have any correlation to either message rate
(as per impstats output) or other processes on the box (at least that I
can tell).
I'm attempting to run on-demand debugging, and have followed
http://www.rsyslog.com/doc/troubleshoot.html but even when running
without problems, the process doesn't appear to respond to SIGUSR1 or
SIGUSR2.
in my init script:
export RSYSLOG_DEBUGLOG="/var/log/rsyslog-debuglog"
export RSYSLOG_DEBUG="DebugOnDemand NoStdOut LogFuncFlow PrintFuncDB
PrintMutexAction"
called as: /sbin/rsyslogd -c 5 -d
[root@css-dhcp log]# rsyslogd -v
rsyslogd 5.8.5, compiled with:
FEATURE_REGEXP: Yes
FEATURE_LARGEFILE: No
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: No
Runtime Instrumentation (slow code): No
When I restart rsyslog, I see on console:
0699.281483000:2b18292632d0: Note: debug on demand turned on via
configuraton file, use USR1 signal to activate.
I have attached my /var/log/rsyslog-debuglog. As far as I can tell, it
will write to it on startup, always ending with the "Checking pidfile"
line, and then never write to it again no matter how many times I send
USR1 or USR2.
This is a *production* box, and I no longer have test hardware. I can
deal with some short losses of syslog (as that's happening already when
it hangs) but can't feasibly replace with a debug-enabled version.
At the moment, I have a cron job running every 2 minutes that restarts
rsyslog if my busiest log file is >= 2 minutes old.
Any advice (on the debug log, or even better, the hang) would be greatly
appreciated.
Thanks,
Jason Antman
Rutgers University
9888.877947000:2afff6dab2d0: rsyslogd 5.8.5 startup, compatibility mode 5,
module path '', cwd:/
9888.878179000:2afff6dab2d0: caller requested object 'net', not found (iRet
-3003)
9888.878204000:2afff6dab2d0: Requested to load module 'lmnet'
9888.878220000:2afff6dab2d0: loading module '/lib64/rsyslog/lmnet.so'
9888.878360000:2afff6dab2d0: module of type 2 being loaded.
9888.878376000:2afff6dab2d0: entry point 'isCompatibleWithFeature' not present
in module
9888.878392000:2afff6dab2d0: source file conf.c requested reference for module
'lmnet', reference count now 1
9888.878419000:2afff6dab2d0: rsyslog runtime initialized, version 5.8.5,
current users 1
9888.878492000:2afff6dab2d0: source file syslogd.c requested reference for
module 'lmnet', reference count now 2
9888.878516000:2afff6dab2d0: GenerateLocalHostName uses 'css-dhcp'
9888.878534000:2afff6dab2d0: omfile: using transactional output interface.
9888.878572000:2afff6dab2d0: module of type 1 being loaded.
9888.878591000:2afff6dab2d0: module of type 1 being loaded.
9888.878605000:2afff6dab2d0: entry point 'beginTransaction' not present in
module
9888.878619000:2afff6dab2d0: entry point 'endTransaction' not present in module
9888.878636000:2afff6dab2d0: source file omfwd.c requested reference for module
'lmnet', reference count now 3
9888.878667000:2afff6dab2d0: module of type 1 being loaded.
9888.878681000:2afff6dab2d0: entry point 'doHUP' not present in module
9888.878694000:2afff6dab2d0: entry point 'beginTransaction' not present in
module
9888.878708000:2afff6dab2d0: entry point 'endTransaction' not present in module
9888.878723000:2afff6dab2d0: module of type 1 being loaded.
9888.878737000:2afff6dab2d0: entry point 'doHUP' not present in module
9888.878750000:2afff6dab2d0: entry point 'beginTransaction' not present in
module
9888.878764000:2afff6dab2d0: entry point 'endTransaction' not present in module
9888.878780000:2afff6dab2d0: module of type 1 being loaded.
9888.878794000:2afff6dab2d0: entry point 'doHUP' not present in module
9888.878808000:2afff6dab2d0: entry point 'beginTransaction' not present in
module
9888.878822000:2afff6dab2d0: entry point 'endTransaction' not present in module
9888.878836000:2afff6dab2d0: module of type 1 being loaded.
9888.878851000:2afff6dab2d0: entry point 'doHUP' not present in module
9888.878864000:2afff6dab2d0: entry point 'beginTransaction' not present in
module
9888.878878000:2afff6dab2d0: entry point 'endTransaction' not present in module
9888.878897000:2afff6dab2d0: rfc5424 parser init called
9888.878911000:2afff6dab2d0: GetParserName addr 0x2afff5ba23f0
9888.878925000:2afff6dab2d0: module of type 3 being loaded.
9888.878941000:2afff6dab2d0: Parser 'rsyslog.rfc5424' added to list of
available parsers.
9888.878956000:2afff6dab2d0: rfc3164 parser init called
9888.878970000:2afff6dab2d0: module of type 3 being loaded.
9888.878985000:2afff6dab2d0: Parser 'rsyslog.rfc3164' added to list of
available parsers.
9888.879004000:2afff6dab2d0: Parser 'rsyslog.rfc5424' added to default parser
set.
9888.879018000:2afff6dab2d0: Parser 'rsyslog.rfc3164' added to default parser
set.
9888.879034000:2afff6dab2d0: rsyslog standard file format strgen init called,
compiled with version 5.8.5
9888.879048000:2afff6dab2d0: module of type 4 being loaded.
9888.879062000:2afff6dab2d0: entry point 'isCompatibleWithFeature' not present
in module
9888.879078000:2afff6dab2d0: Strgen 'RSYSLOG_FileFormat' added to list of
available strgens.
9888.879092000:2afff6dab2d0: traditional file format strgen init called,
compiled with version 5.8.5
9888.879105000:2afff6dab2d0: module of type 4 being loaded.
9888.879119000:2afff6dab2d0: entry point 'isCompatibleWithFeature' not present
in module
9888.879133000:2afff6dab2d0: Strgen 'RSYSLOG_TraditionalFileFormat' added to
list of available strgens.
9888.879148000:2afff6dab2d0: rsyslog standard (network) forward format strgen
init called, compiled with version 5.8.5
9888.879162000:2afff6dab2d0: module of type 4 being loaded.
9888.879175000:2afff6dab2d0: entry point 'isCompatibleWithFeature' not present
in module
9888.879189000:2afff6dab2d0: Strgen 'RSYSLOG_ForwardFormat' added to list of
available strgens.
9888.879221000:2afff6dab2d0: rsyslog traditional (network) forward format
strgen init called, compiled with version 5.8.5
9888.879234000:2afff6dab2d0: module of type 4 being loaded.
9888.879248000:2afff6dab2d0: entry point 'isCompatibleWithFeature' not present
in module
9888.879262000:2afff6dab2d0: Strgen 'RSYSLOG_TraditionalForwardFormat' added to
list of available strgens.
9888.879374000:2afff6dab2d0: Checking pidfile.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
