.qi files are text files. Ciprian posted his here: http://lists.adiscon.net/pipermail/rsyslog/2015-August/041020.html
Here is mine: <OPB:1:qqueue:1: +iQueueSize:2:2:13: +tVars.disk.sizeOnDisk:2:5:15841: >End . <Obj:1:strm:1: +iCurrFNum:2:1:1: +pszFName:1:21:omelasticsearch-queue: +iMaxFiles:2:8:10000000: +bDeleteOnClose:2:1:0: +sType:2:1:1: +tOperationsMode:2:1:2: +tOpenMode:2:3:384: +iCurrOffs:2:5:15841: +inode:2:1:0: +bPrevWasNL:2:1:0: >End . <Obj:1:strm:1: +iCurrFNum:2:1:1: +pszFName:1:21:omelasticsearch-queue: +iMaxFiles:2:8:10000000: +bDeleteOnClose:2:1:1: +sType:2:1:1: +tOperationsMode:2:1:1: +tOpenMode:2:3:384: +iCurrOffs:2:4:1137: +inode:2:6:136136: +bPrevWasNL:2:1:0: >End . Thanks, Alec On Mon, Mar 28, 2016 at 3:45 PM, Joe Blow <[email protected]> wrote: > From what I remember the qi file is some type of binary file. I'll have a > gander tomorrow and reply back. > > Since I had a pain finding the script I'll just put a github link up once > I get some free time and link it back here in case anyone wants it. > > Cheers, > > JB > > Original Message > From:[email protected] > Sent:March 28, 2016 5:40 PM > To:[email protected] > Reply-to:[email protected] > Subject:Re: [rsyslog] Fatal error on disk queue > > On Mon, 28 Mar 2016, Joe Blow wrote: > > > Do you happen to have a large amount of data flowing through the queue? > Have > > you thought about using the checkpoint.interval directive to ensure the > data > > you want queued is written to disk in a timely manner? As far as > processing > > the corrupted queue files, there should be a recover_qi.pl script > kicking > > around to fix the queue files and send the data upstream via your > existing > > queue. > > My understanding is that in this case, all the data files are gone, the > only > thing left is the .qi file. > > That would indicate the all the logs were delivered, but the question is > why the > .qi file is still around. > > I haven't looked inside a .qi file, is it text, or is there a script to > parse it > and dump the info from it? > > > I've been toying with the idea of creating/using a separate instance of > > rsyslog to deal with queue backlogs so the queue can be emptied and get > back > > to a healthy state without losing the backlog or dealing with the > slowness > > issues that come with emptying massive disk backed queues in the event of > > upstream failure, specifically with larger feeds. > > I just had fun where a log destination was offline for 2 months. > Periodically I > needed to stop rsyslog, copy all the queue files to a new directory, > restart > rsyslog, and then compress the files so that I didn't run out of space on > the > relay box. > > When replaying them, I copied /etc/rsyslog.conf to /etc/rsyslog.conf.relay > and > did a few tweaks (not listening to the network or /dev/log, writing stats > to > different files, etc), and use a separate copy of rsyslog with this config > file > to replay the logs. With all the discussions about log replay speed from > queues, > I was worried, but it turns out not to be the bottleneck by a large margin > in my > case. > > David Lang > > > Cheers, > > > > JB > > > > Original Message > > From:[email protected] > > Sent:March 28, 2016 4:15 PM > > To:[email protected] > > Reply-to:[email protected] > > Subject:Re: [rsyslog] Fatal error on disk queue > > > > Hi Alec, > > > > No, I don't have any answers here. > > Somehow the queue files get corrupted. On restart, only the data files > are > > removed, but not the .qi so the disk queue cannot be properly > initialised. > > > > Ciprian > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Mon, Mar 28, 2016 at 10:51 PM, Alec Swan <[email protected]> wrote: > > > >> Ciprian, your issue looks very similar. Have you found a work-around? > >> > >> David, permissions seem to be fine and SELinux is disabled (see below). > Any > >> other thoughts? > >> > >> [root@m0058601 rsyslog.d]# ls -la /var/lib/ | grep rsyslog > >> drwx------ 2 root root 4096 Nov 3 13:00 rsyslog > >> > >> [root@m0058601 rsyslog.d]# ls -la /var/spool/ | grep rsyslog > >> drwxr-xr-x 2 root root 4096 Mar 27 22:34 rsyslog > >> > >> [root@m0058601 rsyslog.d]# sestatus > >> SELinux status: disabled > >> > >> Thanks, > >> > >> Alec > >> > >> On Mon, Mar 28, 2016 at 12:59 PM, Ciprian Hacman < > >> [email protected]> wrote: > >> > >> > Seems very similar to this discussion. Unfortunately, never got the > >> chance > >> > to understand what happened. > >> > http://lists.adiscon.net/pipermail/rsyslog/2015-August/041020.html > >> > > >> > Ciprian > >> > > >> > -- > >> > Performance Monitoring * Log Analytics * Search Analytics > >> > Solr & Elasticsearch Support * http://sematext.com/ > >> > > >> > On Mon, Mar 28, 2016 at 7:25 PM, David Lang <[email protected]> wrote: > >> > > >> > > On Sat, 26 Mar 2016, Alec Swan wrote: > >> > > > >> > > Hi there, > >> > >> > >> > >> I am using omelasticsearch module to send logs to elasticsearch > server > >> > and > >> > >> started noticing the "fatal error on disk queue" error shown > below. I > >> > also > >> > >> noticed a 560 byte .qi file created for the queue configured for > >> > >> omelasticsearch action as shown below. Once I removed the .qi file > the > >> > >> error went away. > >> > >> > >> > >> Is there anything wrong with the configuration? If not, how do I go > >> > about > >> > >> troubleshooting this issue? > >> > >> > >> > >> CONFIGURATION > >> > >> > >> > >> action( > >> > >> type = "omelasticsearch" > >> > >> template = "es-payload" > >> > >> dynSearchIndex = "on" > >> > >> searchIndex = "logstash-index" > >> > >> searchType = "syslog" > >> > >> server = "127.0.0.1" > >> > >> serverport = "9200" > >> > >> uid = "xxx" > >> > >> pwd = "yyy" > >> > >> errorFile = "/var/log/rsyslog/ES-error.log" > >> > >> bulkmode = "on" > >> > >> action.resumeretrycount="-1" # retry if ES is unreachable > (-1 > >> > for > >> > >> infinite retries) > >> > >> action.resumeInterval="60" > >> > >> queue.dequeuebatchsize="1000" # ES bulk size > >> > >> queue.type="linkedlist" > >> > >> queue.size="100000" > >> > >> queue.workerthreads="5" > >> > >> queue.timeoutworkerthreadshutdown="2000" > >> > >> queue.spoolDirectory="/var/spool/rsyslog" > >> > >> queue.filename="omelasticsearch-queue" > >> > >> queue.maxfilesize="100m" > >> > >> queue.maxdiskspace="1g" > >> > >> queue.highwatermark="80000" # when to start spilling to > disk > >> > >> queue.lowwatermark="20000" # when to stop spilling to disk > >> > >> queue.saveonshutdown="on" > >> > >> ) > >> > >> > >> > >> > >> > >> ERROR > >> > >> > >> > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue > >> > 'action > >> > >> 4 > >> > >> queue[DA]', emergency switch to direct mode [v8.14.0 try > >> > >> http://www.rsyslog.com/e/2040 ] > >> > >> Mar 27 04:02:04 m0058180 rsyslogd: [origin software="rsyslogd" > >> > >> swVersion="8.14.0" x-pid="2648" x-info="http://www.rsyslog.com";] > >> start > >> > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue > >> > 'action > >> > >> 2 > >> > >> queue[DA]', emergency switch to direct mode [v8.14.0 try > >> > >> http://www.rsyslog.com/e/2040 ] > >> > >> > >> > > > >> > > usually this means that there is a permission problem (including > >> SELinux > >> > > permissions) when trying to access the files in the work directory. > >> > > > >> > > David Lang > >> > > > >> > > _______________________________________________ > >> > > rsyslog mailing list > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > http://www.rsyslog.com/professional-services/ > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> > > DON'T LIKE THAT. > >> > > > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >> > > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
