From what I remember the qi file is some type of binary file. I'll have a gander tomorrow and reply back.
Since I had a pain finding the script I'll just put a github link up once I get some free time and link it back here in case anyone wants it. Cheers, JB Original Message From:[email protected] Sent:March 28, 2016 5:40 PM To:[email protected] Reply-to:[email protected] Subject:Re: [rsyslog] Fatal error on disk queue On Mon, 28 Mar 2016, Joe Blow wrote: > Do you happen to have a large amount of data flowing through the queue? Have > you thought about using the checkpoint.interval directive to ensure the data > you want queued is written to disk in a timely manner? As far as processing > the corrupted queue files, there should be a recover_qi.pl script kicking > around to fix the queue files and send the data upstream via your existing > queue. My understanding is that in this case, all the data files are gone, the only thing left is the .qi file. That would indicate the all the logs were delivered, but the question is why the .qi file is still around. I haven't looked inside a .qi file, is it text, or is there a script to parse it and dump the info from it? > I've been toying with the idea of creating/using a separate instance of > rsyslog to deal with queue backlogs so the queue can be emptied and get back > to a healthy state without losing the backlog or dealing with the slowness > issues that come with emptying massive disk backed queues in the event of > upstream failure, specifically with larger feeds. I just had fun where a log destination was offline for 2 months. Periodically I needed to stop rsyslog, copy all the queue files to a new directory, restart rsyslog, and then compress the files so that I didn't run out of space on the relay box. When replaying them, I copied /etc/rsyslog.conf to /etc/rsyslog.conf.relay and did a few tweaks (not listening to the network or /dev/log, writing stats to different files, etc), and use a separate copy of rsyslog with this config file to replay the logs. With all the discussions about log replay speed from queues, I was worried, but it turns out not to be the bottleneck by a large margin in my case. David Lang > Cheers, > > JB > > Original Message > From:[email protected] > Sent:March 28, 2016 4:15 PM > To:[email protected] > Reply-to:[email protected] > Subject:Re: [rsyslog] Fatal error on disk queue > > Hi Alec, > > No, I don't have any answers here. > Somehow the queue files get corrupted. On restart, only the data files are > removed, but not the .qi so the disk queue cannot be properly initialised. > > Ciprian > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Mon, Mar 28, 2016 at 10:51 PM, Alec Swan <[email protected]> wrote: > >> Ciprian, your issue looks very similar. Have you found a work-around? >> >> David, permissions seem to be fine and SELinux is disabled (see below). Any >> other thoughts? >> >> [root@m0058601 rsyslog.d]# ls -la /var/lib/ | grep rsyslog >> drwx------ 2 root root 4096 Nov 3 13:00 rsyslog >> >> [root@m0058601 rsyslog.d]# ls -la /var/spool/ | grep rsyslog >> drwxr-xr-x 2 root root 4096 Mar 27 22:34 rsyslog >> >> [root@m0058601 rsyslog.d]# sestatus >> SELinux status: disabled >> >> Thanks, >> >> Alec >> >> On Mon, Mar 28, 2016 at 12:59 PM, Ciprian Hacman < >> [email protected]> wrote: >> >> > Seems very similar to this discussion. Unfortunately, never got the >> chance >> > to understand what happened. >> > http://lists.adiscon.net/pipermail/rsyslog/2015-August/041020.html >> > >> > Ciprian >> > >> > -- >> > Performance Monitoring * Log Analytics * Search Analytics >> > Solr & Elasticsearch Support * http://sematext.com/ >> > >> > On Mon, Mar 28, 2016 at 7:25 PM, David Lang <[email protected]> wrote: >> > >> > > On Sat, 26 Mar 2016, Alec Swan wrote: >> > > >> > > Hi there, >> > >> >> > >> I am using omelasticsearch module to send logs to elasticsearch server >> > and >> > >> started noticing the "fatal error on disk queue" error shown below. I >> > also >> > >> noticed a 560 byte .qi file created for the queue configured for >> > >> omelasticsearch action as shown below. Once I removed the .qi file the >> > >> error went away. >> > >> >> > >> Is there anything wrong with the configuration? If not, how do I go >> > about >> > >> troubleshooting this issue? >> > >> >> > >> CONFIGURATION >> > >> >> > >> action( >> > >> type = "omelasticsearch" >> > >> template = "es-payload" >> > >> dynSearchIndex = "on" >> > >> searchIndex = "logstash-index" >> > >> searchType = "syslog" >> > >> server = "127.0.0.1" >> > >> serverport = "9200" >> > >> uid = "xxx" >> > >> pwd = "yyy" >> > >> errorFile = "/var/log/rsyslog/ES-error.log" >> > >> bulkmode = "on" >> > >> action.resumeretrycount="-1" # retry if ES is unreachable (-1 >> > for >> > >> infinite retries) >> > >> action.resumeInterval="60" >> > >> queue.dequeuebatchsize="1000" # ES bulk size >> > >> queue.type="linkedlist" >> > >> queue.size="100000" >> > >> queue.workerthreads="5" >> > >> queue.timeoutworkerthreadshutdown="2000" >> > >> queue.spoolDirectory="/var/spool/rsyslog" >> > >> queue.filename="omelasticsearch-queue" >> > >> queue.maxfilesize="100m" >> > >> queue.maxdiskspace="1g" >> > >> queue.highwatermark="80000" # when to start spilling to disk >> > >> queue.lowwatermark="20000" # when to stop spilling to disk >> > >> queue.saveonshutdown="on" >> > >> ) >> > >> >> > >> >> > >> ERROR >> > >> >> > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue >> > 'action >> > >> 4 >> > >> queue[DA]', emergency switch to direct mode [v8.14.0 try >> > >> http://www.rsyslog.com/e/2040 ] >> > >> Mar 27 04:02:04 m0058180 rsyslogd: [origin software="rsyslogd" >> > >> swVersion="8.14.0" x-pid="2648" x-info="http://www.rsyslog.com";] >> start >> > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue >> > 'action >> > >> 2 >> > >> queue[DA]', emergency switch to direct mode [v8.14.0 try >> > >> http://www.rsyslog.com/e/2040 ] >> > >> >> > > >> > > usually this means that there is a permission problem (including >> SELinux >> > > permissions) when trying to access the files in the work directory. >> > > >> > > David Lang >> > > >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > > DON'T LIKE THAT. >> > > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
