On Fri, 18 Jul 2014, Eugene Istomin wrote:
What do you think about such concept:
Syslog clients are in any UTC zone & have a sending template with
unixtimestamp:
template(name="cee" type="list") {
...
constant(value="@cee: {")
...
##ES TIMESTAMP
constant(value="\"@timestamp\":\"")
property(name="timegenerated" dateFormat="unixtimestamp")
property(name="timegenerated" dateFormat="subseconds"
position.to="3")
constant(value="\", ")
}
Server is in UTC+00 and use own $year, $month and $day as ES index
mapping:
## DEFAULT ES INDEX ##
template(name="es_index-default" type="list") {
property(name="$!msg_class" )
constant(value="-")
property(name="$!msg_view" )
constant(value="-")
property(name="$year")
constant(value=".")
property(name="$month")
constant(value=".")
property(name="$day")
}
I think that this sort of thing is a lot more reliable than trying to use
client-generate timestamps.
I've just seen too many cases where systems (especially infrastructure things
like routers, switches, UPSs, etc) end up with the wrong time and nobody cares
enough to chase it down for months or years.
using the local time on the server to split things up is much more reliable,
even if the server time is wrong, it still keeps things together. If you use
client times and some of them are wrong, your data can end up getting split all
over.
I like to have a three-tier system where all clients send their logs to a
'local' relay, which then relays the logs to one or more central systems for
analysis.
This lets me trust the relay boxes a bit and I have them encapsulate the logs
that they get from the clients in a JSON structure, cleaning things up as needed
and adding fields for the original IP address, timestamp, custom tags (what
network is this, is it a dev box...), etc and then relaying the clean logs to
the central server(s) which then have a lot less 'strange stuff' to deal with
and can handle the combined volume better.
the relay servers and central boxes can be monitored and managed far more
strictly than the clients, and so I can trust them more for things like
timestamps and sources of the data.
David Lang
---
Best regards,
Eugene Istomin
Also i read about another approach:
"...I think you are far better off running your servers (and the timestamps
on the logs) on UTC. This not only avoids the problems of "where is this
server located, so what time zone is it in", but it also doesn't have
daylight savings time changes (with all the related problems of jobs
running multiple times or not at all)"
Well, sounds good. But needs additional configuration in every
time-dependent application (DB/schedulers/etc).
---
Best regards,
Eugene Istomin
On Thursday, July 17, 2014 10:49:02 PM Eugene Istomin wrote:
> Can you describe a little bit two 8.3.4 changelog lines?
>
> + new parser config object — permits to define custom parser
definitions
> + new tzinfo config object — permits to define time zone offsets
> /---/
> */Best regards,/*
> /Eugene Istomin/
>
> > Hello Rainer,
> >
> > during implementing of the flexible ES templates we are stopped by
time
>
> zone
>
> > problem, let's me describe:
> >
> > 1) We have a message template like:
> >
> > ##CEE TEMPLATE
> > template(name="cee" type="list") {
> > ...
> > constant(value="@cee: {")
> > ...
> >
> > property(name="timegenerated" dateFormat="rfc3339"
>
> format="jsonf"
>
> > outname="@timestamp") constant(value=", ") ...
> >
> >
> > 2) Sending this message using RELP:
> >
> > ruleset(name="relp_cee") {
> >
> > action(type="omrelp" Template="cee" Target="core"
Port="20514")
> >
> > }
> >
> >
> > 3) Log server is receiving this message:
> >
> > input(type="imrelp" Port="20514")
> > action(type="mmjsonparse")
> >
> > if $parsesuccess == "OK" then {
> >
> > action(type="omelasticsearch" server="localhost"
>
> template="ES-All"
>
> > searchIndex="es_index-default" searchType="events"
>
> dynSearchIndex="on"
>
> > bulkmode="on" queue.dequeuebatchsize="5000"
queue.size="100000"
> > queue.workerthreads="5" ); }
> >
> >
> > and puts to ES using index
> >
> > ## ES Index template
> > template(name="es_index-default" type="list") {
> >
> > property(name="$!msg_class" ) constant(value="-")
> > property(name="$!msg_view" ) constant(value="-")
> > property(name="@timestamp" dateFormat="rfc3339"
>
> position.from="1"
>
> > position.to="4") constant(value=".") property(name="@timestamp"
> > dateFormat="rfc3339" position.from="6" position.to="7")
>
> constant(value=".")
>
> > property(name="@timestamp" dateFormat="rfc3339"
position.from="9"
> > position.to="10") }
> >
> >
> > Let's date will be 2014-07-01T01:30:00.000000+03:00
> >
> > if we extract template index using "es_index-default" - we'll get
> > "X-Y-2014-07-01". But in fact, the correct date relatively to ES
> > timestamp
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
