Do you happen to have a large amount of data flowing through the queue? Have you thought about using the checkpoint.interval directive to ensure the data you want queued is written to disk in a timely manner? As far as processing the corrupted queue files, there should be a recover_qi.pl script kicking around to fix the queue files and send the data upstream via your existing queue.
I've been toying with the idea of creating/using a separate instance of rsyslog to deal with queue backlogs so the queue can be emptied and get back to a healthy state without losing the backlog or dealing with the slowness issues that come with emptying massive disk backed queues in the event of upstream failure, specifically with larger feeds. Cheers, JB Original Message From:[email protected] Sent:March 28, 2016 4:15 PM To:[email protected] Reply-to:[email protected] Subject:Re: [rsyslog] Fatal error on disk queue Hi Alec, No, I don't have any answers here. Somehow the queue files get corrupted. On restart, only the data files are removed, but not the .qi so the disk queue cannot be properly initialised. Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Mar 28, 2016 at 10:51 PM, Alec Swan <[email protected]> wrote: > Ciprian, your issue looks very similar. Have you found a work-around? > > David, permissions seem to be fine and SELinux is disabled (see below). Any > other thoughts? > > [root@m0058601 rsyslog.d]# ls -la /var/lib/ | grep rsyslog > drwx------ 2 root root 4096 Nov 3 13:00 rsyslog > > [root@m0058601 rsyslog.d]# ls -la /var/spool/ | grep rsyslog > drwxr-xr-x 2 root root 4096 Mar 27 22:34 rsyslog > > [root@m0058601 rsyslog.d]# sestatus > SELinux status: disabled > > Thanks, > > Alec > > On Mon, Mar 28, 2016 at 12:59 PM, Ciprian Hacman < > [email protected]> wrote: > > > Seems very similar to this discussion. Unfortunately, never got the > chance > > to understand what happened. > > http://lists.adiscon.net/pipermail/rsyslog/2015-August/041020.html > > > > Ciprian > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Mon, Mar 28, 2016 at 7:25 PM, David Lang <[email protected]> wrote: > > > > > On Sat, 26 Mar 2016, Alec Swan wrote: > > > > > > Hi there, > > >> > > >> I am using omelasticsearch module to send logs to elasticsearch server > > and > > >> started noticing the "fatal error on disk queue" error shown below. I > > also > > >> noticed a 560 byte .qi file created for the queue configured for > > >> omelasticsearch action as shown below. Once I removed the .qi file the > > >> error went away. > > >> > > >> Is there anything wrong with the configuration? If not, how do I go > > about > > >> troubleshooting this issue? > > >> > > >> CONFIGURATION > > >> > > >> action( > > >> type = "omelasticsearch" > > >> template = "es-payload" > > >> dynSearchIndex = "on" > > >> searchIndex = "logstash-index" > > >> searchType = "syslog" > > >> server = "127.0.0.1" > > >> serverport = "9200" > > >> uid = "xxx" > > >> pwd = "yyy" > > >> errorFile = "/var/log/rsyslog/ES-error.log" > > >> bulkmode = "on" > > >> action.resumeretrycount="-1" # retry if ES is unreachable (-1 > > for > > >> infinite retries) > > >> action.resumeInterval="60" > > >> queue.dequeuebatchsize="1000" # ES bulk size > > >> queue.type="linkedlist" > > >> queue.size="100000" > > >> queue.workerthreads="5" > > >> queue.timeoutworkerthreadshutdown="2000" > > >> queue.spoolDirectory="/var/spool/rsyslog" > > >> queue.filename="omelasticsearch-queue" > > >> queue.maxfilesize="100m" > > >> queue.maxdiskspace="1g" > > >> queue.highwatermark="80000" # when to start spilling to disk > > >> queue.lowwatermark="20000" # when to stop spilling to disk > > >> queue.saveonshutdown="on" > > >> ) > > >> > > >> > > >> ERROR > > >> > > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue > > 'action > > >> 4 > > >> queue[DA]', emergency switch to direct mode [v8.14.0 try > > >> http://www.rsyslog.com/e/2040 ] > > >> Mar 27 04:02:04 m0058180 rsyslogd: [origin software="rsyslogd" > > >> swVersion="8.14.0" x-pid="2648" x-info="http://www.rsyslog.com";] > start > > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue > > 'action > > >> 2 > > >> queue[DA]', emergency switch to direct mode [v8.14.0 try > > >> http://www.rsyslog.com/e/2040 ] > > >> > > > > > > usually this means that there is a permission problem (including > SELinux > > > permissions) when trying to access the files in the work directory. > > > > > > David Lang > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
