On Sat, Dec 12, 2015 at 10:44 PM, Alec Swan <[email protected]> wrote:
> David, I don't have name= attribute in the action statement. However, there > are log messages from omelasticsearch module after rsyslog restart (see my > previous email). > > Cassandra does not generate a log of logs and I noticed that the few that > get generated are not sent to elasticsearch until rsyslog is restarted. It > feels like logs are being buffered and then sent to ES on subsequent > startup. > > So, here is what I see happening: > 1. I start Rsyslog > Do you see logs from rsyslog show up? I would add a line like: *.* /var/log/rsyslog-debug.log;RSYSLOG_DebugFormat So you can see if rsyslog is getting anything, and make sure you add this line before any other actions but after all your input modules and mm modules. > 2. I start Cassandra and it generates several dozens of log lines > So Cassandra generates logs in its own log file and logs via syslog() APIs? Or is it started under systemd and is it using a service script of type "simple" where systemd gathers stderr and stdout and sends them to the journal logs? Perhaps you could use the "logger" command to send a trace message. > 3. No logs show up in ES (even if I restart Cassandra several times) > 4. I restart Rsyslog > 5. The logs generated in step 2 show up in ES > So is this a RHEL or CentOS box? Can you share your configuration? -peter > Alec > > > On Sat, Dec 12, 2015 at 2:56 PM, David Lang <[email protected]> wrote: > > > On Sat, 12 Dec 2015, Alec Swan wrote: > > > > Thanks, Ciprian. I ran rsyslogd -dn and I can see that imfile is reading > >> changes from cassandra.log, but it's not sending them to elasticsearch > >> until I restart. Thoughts? > >> > >> Rsyslog trace before restart doesn't have any omelasticsearch logs: > >> ... > >> 5623.332950664:imfile.c : DDDD: imfile: in_processEvent (wd=2) > event > >> Mask='0x00000002' > >> 5623.332954364:imfile.c : DDDD: imfile: wd 2 got file > >> 0x7ffe24002190, > >> dir -1 > >> 5623.332965540:imfile.c : strm 0x7ffe240057c0: file 7 read 0 bytes > >> 5623.332978458:imfile.c : stream checking for file change on > >> '/var/log/cassandra/cassandra.log', inode 264465/264465 > >> 5623.332982339:imfile.c : DDDDD: readLine returns[-2026]: '(null)' > >> [*ppCStr 0x7ffe24026850] > >> > >> Rsyslog after restart which causes logs to be sent to elasticsearch: > >> ... > >> 6033.733447868:action 2 queue:Reg/w0: omelasticsearch: result doAction: > >> -2121 (bulkmode 1) > >> 6033.733450384:action 2 queue:Reg/w0: omelasticsearch: endTransaction > init > >> 6033.733482066:action 2 queue:Reg/w0: omelasticsearch: endTransaction, > >> batch: '{"index":{"_index": "logstash-2015.12.12","_type":"cassandra"}} > >> { "@timestamp": "2015-12-12T21:33:53.484399+00:00", "host": "m0051948", > >> "severity": "notice", "facility": "local2", "syslogtag": "cassandra", > >> "filename": "cassandra.log", "message": "Enqueuing flush of > >> Memtable-local@518770933(84\/840 serialized\/live bytes, 4 ops)", > >> "log_time": "21:26:55,179", "log_level": "INFO" } > >> {"index":{"_index": "logstash-2015.12.12","_type":"cassandra"}} > >> > > > > there should be something in the logs about the elasticsearch action, do > > you have it named? (name= in the action statement) > > > > are you saying that when rsyslog starts, it doesn't send, but if you > > restart it, it then sends normally? or are you saying that nothing goes > > through until you do a restart, then onebatch of messages get through and > > nothing beyond that until you do a restart again? > > > > David Lang > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
