Ok so that fixed my original issue, but created another one.
I took out the whole template part of the rsyslog.conf and place it the
template and elasticsearch config in /etc/rsyslog.d/elasticsearch.conf
---------------------------------------------------------------------------------
module(load="imuxsock") # for listening to /dev/log
module(load="omelasticsearch") # for outputting to Elasticsearch
# this is for index names to be like: logstash-YYYY.MM.DD
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
# this is for formatting our syslog in JSON with @timestamp
template(name="plain-syslog"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"tag\":\"") property(name="syslogtag" format="json")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\"}")
}
# this is where we actually send the logs to Elasticsearch (localhost:9200 by
default)
action(type="omelasticsearch"
template="plain-syslog"
searchIndex="logstash-index"
dynSearchIndex="on")
$template
RemoteHost,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogseverity-text%.log.gz"
if $fromhost-ip !='127.0.0.1' then {
action(name="PerHostFile" type="omfile" dynafile="RemoteHost"
DynaFileCacheSize="1000" ziplevel="5")
action(type="omelasticsearch" template="plain-syslog"
searchIndex="logstash-index" dynSearchIndex="on")
stop
} else {
action(type="omelasticsearch" template="plain-syslog"
searchIndex="logstash-index" dynSearchIndex="on") }
--------------------------------------------------------------------------------------------------------------------
So that stops other host logs from going to /var/log/messages and that also
goes to elasticsearch, but now it does not create the file that I want to
compress.
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Orangepeel Beef
Sent: Wednesday, May 14, 2014 3:05 PM
To: rsyslog-users
Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
That's really strange Josh, I would expect your config to do exactly what you
are trying to do...
I don't write output direct to elasticsearch, too many problems with that
imho, but I don't see any reason in the config it's not working. I
usually leave the existing rsyslog.conf file alone, and add additional files in
/etc/rsyslog.d/
try this:
if $fromhost-ip !='127.0.0.1' then {
action(name="PerHostFile" type="omfile" dynafile="RemoteHost"
DynaFileCacheSize="1000" ziplevel="5")
action(type="omelasticsearch" template="plain-syslog"
searchIndex="logstash-index" dynSearchIndex="on")
stop
} else {
action(type="omelasticsearch" template="plain-syslog"
searchIndex="logstash-index" dynSearchIndex="on") }
or create a new file in /etc/rsyslog.d/ and copy your template
definitions and those lines, then remove them from the rsyslog.conf file,
restart rsyslog
On Wed, May 14, 2014 at 2:37 PM, Josh Bitto <[email protected]> wrote:
> I want to send everything to elasticsearch. Yes nothing is going into
> elasticsearch local or from other hosts. I will try that and see how
> it goes.
>
>
>
> -----Original Message-----
> From: [email protected] [mailto:
> [email protected]] On Behalf Of Orangepeel Beef
> Sent: Wednesday, May 14, 2014 2:33 PM
> To: rsyslog-users
> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana server
>
> Do you want to send everything to elasticsearch or just non local logs?
>
>
> are local logs going into elasticsearch now or nothing is?
>
> You can probably just add your
>
> action(type="omelasticsearch"
> template="plain-syslog"
> searchIndex="logstash-index"
> dynSearchIndex="on")
>
> again inside the if block, before the stop.
>
>
>
>
> On Wed, May 14, 2014 at 2:23 PM, Josh Bitto <[email protected]>
> wrote:
>
> > I tried the method you suggested and it stops sending logs to
> > elasticsearch. It doesn't matter what order I put the operation in
> > either above or below the template to send logs to elasticsearch. It
> won't send.
> > It does process the logs though according to the template for the
> > local IP, but I have to have both. I've racked my brain after a Dr.
> > Pepper and some beef jerky and still can't think of a way to do it.
> > Any
> ideas?
> >
> >
> >
> >
> > -----Original Message-----
> > From: [email protected] [mailto:
> > [email protected]] On Behalf Of Orangepeel Beef
> > Sent: Wednesday, May 14, 2014 12:25 PM
> > To: rsyslog-users
> > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
> > server
> >
> > That will send all logs that are not from localhost to your per host
> > files, and it should happen AFTER its already been sent to
> > elasticsearch based on your config.
> >
> >
> > the stop keyword just means don't send that log anywhere else.
> > Wrapping it in the if statement keeps stop from effecting ALL the
> > logs. You may need to adjust your positioning, but that should work.
> >
> >
> > On Wed, May 14, 2014 at 12:21 PM, Orangepeel Beef
> > <[email protected]>wrote:
> >
> > > if $fromhost-ip !='127.0.0.1' then {
> > > action(name="PerHostFile" type="omfile" dynafile="RemoteHost"
> > > DynaFileCacheSize="1000" ziplevel="5")
> > >
> > > stop
> > > }
> > >
> > >
> > >
> > > On Wed, May 14, 2014 at 12:11 PM, Josh Bitto
> > ><[email protected]
> > >wrote:
> > >
> > >> Ok so I did what you suggested, but that broke some things. It
> > >> stopped my other template action to send the logs to elasticsearch.
> > >> (From there Kibana sees the logs)
> > >>
> > >> Here is a snippet from my config.
> > >> http://pastebin.com/2W4g6nUS
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: [email protected] [mailto:
> > >> [email protected]] On Behalf Of Orangepeel Beef
> > >> Sent: Wednesday, May 14, 2014 11:44 AM
> > >> To: rsyslog-users
> > >> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
> > >> server
> > >>
> > >> that's odd, try going to
> > >> https://github.com/embalmed/so-logstashy
> > then
> > >> browse into configsamples/rsyslog-30-remote.txt
> > >>
> > >> but i've tested the link and it works for me, so not sure whats
> > >> up there
> > >>
> > >>
> > >> On Wed, May 14, 2014 at 11:41 AM, Josh Bitto
> > >> <[email protected]>
> > >> wrote:
> > >>
> > >> > When I click on the link I get a 404 "this is not the page you
> > >> > are looking for" with a cute star wars themed character.
> > >> >
> > >> >
> > >> >
> > >> > -----Original Message-----
> > >> > From: [email protected] [mailto:
> > >> > [email protected]] On Behalf Of Orangepeel Beef
> > >> > Sent: Wednesday, May 14, 2014 11:40 AM
> > >> > To: rsyslog-users
> > >> > Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana
> > >> > server
> > >> >
> > >> > Doesn't really matter.
> > >> >
> > >> >
> > >> > https://github.com/embalmed/so-logstashy/blob/master/configsamp
> > >> > le
> > >> > s/
> > >> > rsy
> > >> > slog-30-remote.txtis
> > >> > the one my buddy uses.
> > >> >
> > >> >
> > >> > On Wed, May 14, 2014 at 10:55 AM, Josh Bitto
> > >> > <[email protected]>
> > >> > wrote:
> > >> >
> > >> > > Should this template be before ###RULES### config or does it
> matter?
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: [email protected] [mailto:
> > >> > > [email protected]] On Behalf Of Orangepeel
> > >> > > Beef
> > >> > > Sent: Wednesday, May 14, 2014 10:52 AM
> > >> > > To: rsyslog-users
> > >> > > Subject: Re: [rsyslog] Rsyslog w/
> > >> > > logstash-elasticsearch-kibana server
> > >> > >
> > >> > > Add the word 'stop' on the next line.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, May 14, 2014 at 10:21 AM, Josh Bitto
> > >> > > <[email protected]>
> > >> > > wrote:
> > >> > >
> > >> > > > Hey David,
> > >> > > >
> > >> > > > I had a question for you and anyone else that know's the
> > >> > > > answer
> > to.
> > >> > > > Currently I'm running the omfile you suggested on my
> > >> > > > development server and I'm noticing that the code is
> > >> > > > working, but also that my messages log file is also filling
> > >> > > > up with the
> > same log information.
> > >> > > > Is there a way to filter logs to only go to their
> > >> > > > destination and not log into the messages log file that is in
> > >> > > > linux?
> > >> > > >
> > >> > > > Here is the part of my config that I have done.
> > >> > > > -------------------------------------------
> > >> > > > $template
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> RemoteHost,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogseverity-text%.log.gz"
> > >> > > >
> > >> > > > action(name="PerHostFile" type="omfile" dynafile="RemoteHost"
> > >> > > > DynaFileCacheSize="1000" ziplevel="5"
> > >> > > > -------------------------------------------
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > -----Original Message-----
> > >> > > > From: [email protected] [mailto:
> > >> > > > [email protected]] On Behalf Of David Lang
> > >> > > > Sent: Friday, May 09, 2014 3:36 PM
> > >> > > > To: rsyslog-users
> > >> > > > Subject: Re: [rsyslog] Rsyslog w/
> > >> > > > logstash-elasticsearch-kibana server
> > >> > > >
> > >> > > > On Fri, 9 May 2014, Josh Bitto wrote:
> > >> > > >
> > >> > > > > In the link I posted there is a description of
> > >> > > > > syslogpriority-text -an alias for syslogseverity-text
> > >> > > > >
> > >> > > > > And that's my question as to what it is referencing.
> > >> > > >
> > >> > > > ahh, in that case you use whichever one makes sense to you,
> > >> > > > some people think of it as priority, some as severity,
> > >> > > > rsyslog supports both names with identical content.
> > >> > > >
> > >> > > > David Lang
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > -----Original Message-----
> > >> > > > > From: [email protected]
> > >> > > > > [mailto:[email protected]] On Behalf Of
> > >> > > > > David Lang
> > >> > > > > Sent: Friday, May 09, 2014 3:33 PM
> > >> > > > > To: rsyslog-users
> > >> > > > > Subject: Re: [rsyslog] Rsyslog w/
> > >> > > > > logstash-elasticsearch-kibana server
> > >> > > > >
> > >> > > > > On Fri, 9 May 2014, Josh Bitto wrote:
> > >> > > > >
> > >> > > > >> Happy Friday!
> > >> > > > >>
> > >> > > > >> One last question. I modified the template a tad bit and
> > >> > > > >> added the
> > >> > > > following.
> > >> > > > >>
> > >> > > > >> $template
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> RemoteHost,"/var/log/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/%syslogseverity-text%.log.gz"
> > >> > > > >>
> > >> > > > >> When looking at the
> > >> > > http://www.rsyslog.com/doc/property_replacer.htmldocumentatio
> > >> > > n I see both "syslogseverity-text" and "syslogpriority-text"
> > >> > > > >>
> > >> > > > >> My question is in this case I'm basically separating the
> > >> > > > >> files based on the severity. In what instance would I
> > >> > > > >> use the alias
> > >> for?
> > >> > > > >> I guess I'm not fully understanding what its purpose is.
> > >> > > > >
> > >> > > > > what are you referring to as the 'alias'?
> > >> > > > >
> > >> > > > > I'm not understanding your question.
> > >> > > > >
> > >> > > > > David Lang
> > >> > > > >
> > >> > > > > _______________________________________________
> > >> > > > > rsyslog mailing list
> > >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > > > > http://www.rsyslog.com/professional-services/
> > >> > > > > What's up with rsyslog? Follow
> > >> > > > > https://twitter.com/rgerhards NOTE
> > >> > > WELL:
> > >> > > > This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >> > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
> > >> > > > DO NOT POST if you DON'T LIKE THAT.
> > >> > > > > _______________________________________________
> > >> > > > > rsyslog mailing list
> > >> > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > > > > http://www.rsyslog.com/professional-services/
> > >> > > > > What's up with rsyslog? Follow
> > >> > > > > https://twitter.com/rgerhards NOTE
> > >> > > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> > >> > > > > by a myriad of
> > >> > > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > >> > > > POST if you DON'T LIKE THAT.
> > >> > > > >
> > >> > > > _______________________________________________
> > >> > > > rsyslog mailing list
> > >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > > > http://www.rsyslog.com/professional-services/
> > >> > > > What's up with rsyslog? Follow
> > >> > > > https://twitter.com/rgerhards NOTE
> > >> > WELL:
> > >> > > > This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >> > > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
> > >> > > > DO NOT POST if you DON'T LIKE THAT.
> > >> > > > _______________________________________________
> > >> > > > rsyslog mailing list
> > >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > > > http://www.rsyslog.com/professional-services/
> > >> > > > What's up with rsyslog? Follow
> > >> > > > https://twitter.com/rgerhards NOTE
> > >> > > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
> > >> > > > a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
> > >> > > > and DO NOT POST if you DON'T LIKE THAT.
> > >> > > >
> > >> > > _______________________________________________
> > >> > > rsyslog mailing list
> > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > > http://www.rsyslog.com/professional-services/
> > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> > > NOTE
> > >> WELL:
> > >> > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > >> > > POST if you DON'T LIKE THAT.
> > >> > > _______________________________________________
> > >> > > rsyslog mailing list
> > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > > http://www.rsyslog.com/professional-services/
> > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> > > NOTE
> > >> > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >> > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> > >> > > NOT POST if you DON'T LIKE THAT.
> > >> > >
> > >> > _______________________________________________
> > >> > rsyslog mailing list
> > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > http://www.rsyslog.com/professional-services/
> > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> > NOTE
> > WELL:
> > >> > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > >> > if you DON'T LIKE THAT.
> > >> > _______________________________________________
> > >> > rsyslog mailing list
> > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> > http://www.rsyslog.com/professional-services/
> > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> > NOTE
> > >> > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> > >> > NOT POST if you DON'T LIKE THAT.
> > >> >
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL:
> > >> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > >> you DON'T LIKE THAT.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > >> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > >> POST if you DON'T LIKE THAT.
> > >>
> > >
> > >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
