Try setting your memory queue sizes very small (so that the disk queue will have
to be used) and then run with debugging turned on so that you can see any errors
that happen when it tries to use the disk queue.
If it is able to successfully use the disk queue, then you managed to get a
corrupted .qi file somehow (probably due to a crash or kill -9 while rsyslog was
manipulating it)
David Lang
On Mon, 28 Mar 2016, Alec Swan wrote:
Ciprian, your issue looks very similar. Have you found a work-around?
David, permissions seem to be fine and SELinux is disabled (see below). Any
other thoughts?
[root@m0058601 rsyslog.d]# ls -la /var/lib/ | grep rsyslog
drwx------ 2 root root 4096 Nov 3 13:00 rsyslog
[root@m0058601 rsyslog.d]# ls -la /var/spool/ | grep rsyslog
drwxr-xr-x 2 root root 4096 Mar 27 22:34 rsyslog
[root@m0058601 rsyslog.d]# sestatus
SELinux status: disabled
Thanks,
Alec
On Mon, Mar 28, 2016 at 12:59 PM, Ciprian Hacman <
[email protected]> wrote:
Seems very similar to this discussion. Unfortunately, never got the chance
to understand what happened.
http://lists.adiscon.net/pipermail/rsyslog/2015-August/041020.html
Ciprian
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Mar 28, 2016 at 7:25 PM, David Lang <[email protected]> wrote:
On Sat, 26 Mar 2016, Alec Swan wrote:
Hi there,
I am using omelasticsearch module to send logs to elasticsearch server
and
started noticing the "fatal error on disk queue" error shown below. I
also
noticed a 560 byte .qi file created for the queue configured for
omelasticsearch action as shown below. Once I removed the .qi file the
error went away.
Is there anything wrong with the configuration? If not, how do I go
about
troubleshooting this issue?
CONFIGURATION
action(
type = "omelasticsearch"
template = "es-payload"
dynSearchIndex = "on"
searchIndex = "logstash-index"
searchType = "syslog"
server = "127.0.0.1"
serverport = "9200"
uid = "xxx"
pwd = "yyy"
errorFile = "/var/log/rsyslog/ES-error.log"
bulkmode = "on"
action.resumeretrycount="-1" # retry if ES is unreachable (-1
for
infinite retries)
action.resumeInterval="60"
queue.dequeuebatchsize="1000" # ES bulk size
queue.type="linkedlist"
queue.size="100000"
queue.workerthreads="5"
queue.timeoutworkerthreadshutdown="2000"
queue.spoolDirectory="/var/spool/rsyslog"
queue.filename="omelasticsearch-queue"
queue.maxfilesize="100m"
queue.maxdiskspace="1g"
queue.highwatermark="80000" # when to start spilling to disk
queue.lowwatermark="20000" # when to stop spilling to disk
queue.saveonshutdown="on"
)
ERROR
Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue
'action
4
queue[DA]', emergency switch to direct mode [v8.14.0 try
http://www.rsyslog.com/e/2040 ]
Mar 27 04:02:04 m0058180 rsyslogd: [origin software="rsyslogd"
swVersion="8.14.0" x-pid="2648" x-info="http://www.rsyslog.com";] start
Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue
'action
2
queue[DA]', emergency switch to direct mode [v8.14.0 try
http://www.rsyslog.com/e/2040 ]
usually this means that there is a permission problem (including SELinux
permissions) when trying to access the files in the work directory.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
