Hey all,
I've got a log server (running the latest and greatest rsyslog as of
yesterday) which i've been seeing randomly dying. I load balance and have
scripts to check if rsyslog isn't running, and if it is restart it, but i'm
having a really tough time tracking down what could be making rsyslog
crash. Sometimes it seems like the rsyslog daemon dies, and because of it
I can't login via SSH. Because of this i've been forced to keep a screen
open with the rsyslog boxes just in case i need to restart rsyslog (which
allows me to login again).
I'm taking logs in from a number of different sources (asa, snare, etc...
all of them with their own disk assisted output queues, all outputting to
Elasticsearch). If i'm monitoring the queues, i don't see any queues
backing up or anything which would lead me to believe rsyslog is balooning
memory and going to die.
I have a number of these logs in my catchall bucket:
Framing Error in received TCP message: delimiter is not SP but has ASCII
value 46. [v8.9.0]
I see a few of these within the error logs too:
"UnavailableShardsException[[cisco-20150420][4] [3] shardIt, [1] active :
Timeout waiting for [3m], request:
org.elasticsearch.action.bulk.BulkShardRequest@5d48afd4]"
Could either of these cause rsyslog to hard die? How would you recommend
finding these seemingly random failures?
Here are what most of my ES output queues look like:
<snip>
if $rawmsg contains "%ASA-" or $rawmsg contains "%PIX-" then{
action(type="mmnormalize" userawmsg="on"
rulebase="/etc/rsyslog.d/asa.rule")
action(type="omelasticsearch"
name="rsys_asa"
server="10.10.10.10"
serverport="9200"
template="ciscoasa"
asyncrepl="on"
searchType="asa"
searchIndex="ciscoasaindex"
timeout="3m"
dynSearchIndex="on"
bulkmode="on"
errorfile="asa_err.log"
queue.type="linkedlist"
queue.filename="cisco.rsysq"
queue.size="15000000"
queue.saveonshutdown="on"
queue.maxdiskspace="100g"
queue.dequeuebatchsize="5000"
action.resumeretrycount="30")stop}
</snip>
Anything glaring here? Could my retries be killing rsyslog? Any ideas on
how i should go about troubleshooting?
Cheers,
JB
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
