Hi Alec, No, I don't have any answers here. Somehow the queue files get corrupted. On restart, only the data files are removed, but not the .qi so the disk queue cannot be properly initialised.
Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Mar 28, 2016 at 10:51 PM, Alec Swan <alecs...@gmail.com> wrote: > Ciprian, your issue looks very similar. Have you found a work-around? > > David, permissions seem to be fine and SELinux is disabled (see below). Any > other thoughts? > > [root@m0058601 rsyslog.d]# ls -la /var/lib/ | grep rsyslog > drwx------ 2 root root 4096 Nov 3 13:00 rsyslog > > [root@m0058601 rsyslog.d]# ls -la /var/spool/ | grep rsyslog > drwxr-xr-x 2 root root 4096 Mar 27 22:34 rsyslog > > [root@m0058601 rsyslog.d]# sestatus > SELinux status: disabled > > Thanks, > > Alec > > On Mon, Mar 28, 2016 at 12:59 PM, Ciprian Hacman < > ciprian.hac...@sematext.com> wrote: > > > Seems very similar to this discussion. Unfortunately, never got the > chance > > to understand what happened. > > http://lists.adiscon.net/pipermail/rsyslog/2015-August/041020.html > > > > Ciprian > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Mon, Mar 28, 2016 at 7:25 PM, David Lang <da...@lang.hm> wrote: > > > > > On Sat, 26 Mar 2016, Alec Swan wrote: > > > > > > Hi there, > > >> > > >> I am using omelasticsearch module to send logs to elasticsearch server > > and > > >> started noticing the "fatal error on disk queue" error shown below. I > > also > > >> noticed a 560 byte .qi file created for the queue configured for > > >> omelasticsearch action as shown below. Once I removed the .qi file the > > >> error went away. > > >> > > >> Is there anything wrong with the configuration? If not, how do I go > > about > > >> troubleshooting this issue? > > >> > > >> CONFIGURATION > > >> > > >> action( > > >> type = "omelasticsearch" > > >> template = "es-payload" > > >> dynSearchIndex = "on" > > >> searchIndex = "logstash-index" > > >> searchType = "syslog" > > >> server = "127.0.0.1" > > >> serverport = "9200" > > >> uid = "xxx" > > >> pwd = "yyy" > > >> errorFile = "/var/log/rsyslog/ES-error.log" > > >> bulkmode = "on" > > >> action.resumeretrycount="-1" # retry if ES is unreachable (-1 > > for > > >> infinite retries) > > >> action.resumeInterval="60" > > >> queue.dequeuebatchsize="1000" # ES bulk size > > >> queue.type="linkedlist" > > >> queue.size="100000" > > >> queue.workerthreads="5" > > >> queue.timeoutworkerthreadshutdown="2000" > > >> queue.spoolDirectory="/var/spool/rsyslog" > > >> queue.filename="omelasticsearch-queue" > > >> queue.maxfilesize="100m" > > >> queue.maxdiskspace="1g" > > >> queue.highwatermark="80000" # when to start spilling to disk > > >> queue.lowwatermark="20000" # when to stop spilling to disk > > >> queue.saveonshutdown="on" > > >> ) > > >> > > >> > > >> ERROR > > >> > > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue > > 'action > > >> 4 > > >> queue[DA]', emergency switch to direct mode [v8.14.0 try > > >> http://www.rsyslog.com/e/2040 ] > > >> Mar 27 04:02:04 m0058180 rsyslogd: [origin software="rsyslogd" > > >> swVersion="8.14.0" x-pid="2648" x-info="http://www.rsyslog.com";] > start > > >> Mar 27 04:02:04 m0058180 rsyslogd-2040: fatal error on disk queue > > 'action > > >> 2 > > >> queue[DA]', emergency switch to direct mode [v8.14.0 try > > >> http://www.rsyslog.com/e/2040 ] > > >> > > > > > > usually this means that there is a permission problem (including > SELinux > > > permissions) when trying to access the files in the work directory. > > > > > > David Lang > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
