Happy Friday! One last question. I modified the template a tad bit and added the following.
$template RemoteHost,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogseverity-text%.log.gz" When looking at the http://www.rsyslog.com/doc/property_replacer.html documentation I see both "syslogseverity-text" and "syslogpriority-text" My question is in this case I'm basically separating the files based on the severity. In what instance would I use the alias for? I guess I'm not fully understanding what its purpose is. Thanks and have a great weekend! Josh -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Radu Gheorghe Sent: Friday, May 09, 2014 12:15 AM To: rsyslog-users Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server Hi, Just a quick comment here (as it seems that all is working now, and if it works we shouldn't fix it :p): Kibana is not reading logs files, it queries ES and shows results. If Kibana can't see stuff, it's most likely that these are not in ES, though you can query ES without kibana with something like (let's say you logged "test" somewhere): curl 'localhost:9200/_search?q=test&pretty' Because of this, Kibana and ES won't care about how your omfile action looks like. Just the omelasticsearch action (and how the template looks like) are relevant. If omelasticsearch works fine and you still have missing long, then the cause might be in between the input and the action. Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Fri, May 9, 2014 at 1:45 AM, David Lang <[email protected]> wrote: > This sounds like kibana is reading logs from /var/log/remote, but you > should be getting logs into elasticsearch by having ES listening on > localhost:9200 > > so it shouldn't care what's getting written to disk. > > David Lang > > > On Thu, 8 May 2014, Josh Bitto wrote: > > Thanks David, >> >> Had to change the ziplevel '5' to "5". >> >> Now it's compressing and writing to the file, but it isn't showing up >> in kibana when I run a test that creates logs. Is placement in my >> config the key? Or should I be adjusting the part for Elastic search >> accordingly? >> >> >> >> >> -----Original Message----- >> From: [email protected] [mailto:rsyslog-bounces@lists. >> adiscon.com] On Behalf Of David Lang >> Sent: Thursday, May 08, 2014 3:26 PM >> To: rsyslog-users >> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >> server >> >> On Thu, 8 May 2014, Josh Bitto wrote: >> >> Yes your correct, but my template should be writing the logs to a file. >>> So I guess what I should have said is that omelasticsearch sends the >>> logs to ES while rsyslog "writes" the logs to a file. Would that be >>> the better terminology? >>> >>> As far as a omfile module are you saying that... >>> >>> module(load="builtin:omfile" FileCreateMode="0644" >>> DirCreateMode="0700" FileOwner="syslog" ZipLevel="2") >>> *.* action(type="omfile" >>> File="/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log") >>> >>> Would be better than doing... >>> >>> $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$ >>> DAY%.log" >>> >>> *.* ?RemoteHost >>> >> >> $template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$ >> DAY%.log.gz" >> >> action(name="PerHostFile" type="omfile" dynafile="RemoteHost" >> DynaFileCacheSize="1000" ziplevel='5') >> >> with v7 you don't need *.* >> >> this creates a compressed file to begin with, and every day will get >> a new file >> >> if rsyslog crashes (as opposed to shutting down) it's best to mv the >> file before starting a new rsyslog. VeryRobustZip='on' solves this >> problem, at the cost of worse compression >> >> if the omfile falls behind, you can add ASyncWriting="on" to this so >> that a different thread does the compression than the rest of the >> work (only needed at high traffic levels) >> >> if you are writing for more than 1000 hosts/day, change the >> DynaFileCachSize variable to be larger than the number of hosts you >> write for (the default is 10) >> >> >> Does that also recreate the file each day? Meaning....Day1 logs are >>> written and compressed. Then day2 new file is created and compressed >>> and so on... >>> >> >> it doesn't write it and then compress it, it compresses it and then >> writes it, so it's always compressed. >> >> David Lang >> >> >>> >>> >>> -----Original Message----- >>> From: [email protected] [mailto:rsyslog-bounces@lists. >>> adiscon.com] On Behalf Of David Lang >>> Sent: Thursday, May 08, 2014 2:46 PM >>> To: rsyslog-users >>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana >>> server >>> >>> On Thu, 8 May 2014, Josh Bitto wrote: >>> >>> I wanted to thank everyone that gave their input into this >>> question. It >>>> helped out tremendously. >>>> >>>> So I'm still in the test phase of creating a central log server, >>>> but I think I've landed what my final setup is going to be. I'm >>>> going give a rundown of what that is and ask to see if I might have >>>> potential problems. >>>> >>>> Rsyslog->Elasticsearch->Kibana >>>> >>>> The rsyslog.conf is pretty much the default file that comes with >>>> CentOS repository with the exception of adding the following: >>>> >>>> ################################### >>>> >>>> template(name="logstash-index" >>>> type="list") { >>>> constant(value="logstash-") >>>> property(name="timereported" dateFormat="rfc3339" position.from="1" >>>> position.to="4") >>>> constant(value=".") >>>> property(name="timereported" dateFormat="rfc3339" position.from="6" >>>> position.to="7") >>>> constant(value=".") >>>> property(name="timereported" dateFormat="rfc3339" position.from="9" >>>> position.to="10") } >>>> >>>> # this is for formatting our syslog in JSON with @timestamp >>>> template(name="plain-syslog" >>>> type="list") { >>>> constant(value="{") >>>> constant(value="\"@timestamp\":\"") >>>> property(name="timereported" dateFormat="rfc3339") >>>> constant(value="\",\"host\":\"") property(name="hostname") >>>> constant(value="\",\"severity\":\"") >>>> property(name="syslogseverity-text") >>>> constant(value="\",\"facility\":\"") >>>> property(name="syslogfacility-text") >>>> constant(value="\",\"tag\":\"") property(name="syslogtag" >>>> format="json") >>>> constant(value="\",\"message\":\"") property(name="msg" >>>> format="json") >>>> constant(value="\"}") >>>> } >>>> >>>> # this is where we actually send the logs to Elasticsearch >>>> (localhost:9200 by default) action(type="omelasticsearch" >>>> template="plain-syslog" >>>> searchIndex="logstash-index" >>>> dynSearchIndex="on") >>>> >>>> $ModLoad imfile # Load the imfile input module >>>> >>>> # Watch /var/log/httpd/access_log >>>> $InputFileName /var/log/httpd/access_log $InputFileTag apache-access: >>>> $InputFileStateFile state-apache-access $InputRunFileMonitor >>>> >>>> # Watch /var/log/httpd/error_log >>>> $InputFileName /var/log/httpd/error_log $InputFileTag apache-error: >>>> $InputFileStateFile state-apache-error $InputRunFileMonitor >>>> >>>> # Watch /var/ossec/logs/alerts/alerts.log $InputFileName >>>> /var/ossec/logs/alerts/alerts.log $InputFileTag ossec-alert: >>>> $InputFileStateFile state-ossec-alert $InputRunFileMonitor >>>> >>>> >>>> $template >>>> RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$ >>>> DAY%.log" >>>> >>>> *.* ?RemoteHost >>>> >>>> >>>> if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' >>>> or $msg contains 'Received SNMP packet(s) from UDP' ) then ~ >>>> >>>> ############################################################# >>>> >>>> So basically I have rsyslog writing incoming logs to a file and >>>> then they are being sent for ES and kibana. Eventually I will have >>>> a logrotation done so that they can be rotated and compressed to save >>>> space. >>>> >>>> Thanks guys! >>>> >>> >>> a couple nits >>> >>> omelasticsearch isn't writing them to a file (your description of >>> write them to a file then send them is misleading) >>> >>> take a look at the options for omfile that can compress the output >>> in place. It means that the logs are buffered a little bit more, and >>> things can get a bit interesting if rsyslog crashes in the middle of >>> a write, but it avoids the need to play the rotation game. >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: >>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>> you DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
