Thanks for all the input. I verified that the data is being inserted
into graylog2. It is outside of the normal time window when looking at
the dashboard. When doing a search for All messages, I am able to find
the logs.
I want to clarify the problem one last time. I have several nodes that
run Linux/Windows etc and these all seem to pass the timestamp in a way
that is understood using the default timezone of the machine. VMware
ESXi hosts send a timestamp without the offset 9 I learned this after my
first post). VMware doesn't have an easy way to set the timezone in the
5.1+ ESXi versions (vCenter might be different).
I think I figured it out. It appears that when the message comes from an
ESXi host there is NO offset/TZ data attached. Rsyslog appends the
default timezone offset (-06:00 in my case) to the message. The message
is then forwarded on using omfwd into the logstore host, which applies
the GRAYLOGRFC5424 template and then forwards into graylog2 with a
timestamp like this 2015-01-30T00:51:11-06:00 instead of
2015-01-30T00:51:11.331Z. Graylog2 can handle the later just fine and
adjust for the UI's TZ offset.
One thing to note, at-least for my setup, which uses a relay host to
ship to a log repository is the template has to be applied on the
log-shipper node since that is where first contact is made.
Best Regards,
Brandon
On 01/29/2015 02:52 AM, Rainer Gerhards wrote:
2015-01-29 9:13 GMT+01:00 Radu Gheorghe <[email protected]>:
On Thu, Jan 29, 2015 at 9:48 AM, Rainer Gerhards <[email protected]
wrote:
2015-01-29 8:46 GMT+01:00 Radu Gheorghe <[email protected]>:
[...]
2015-01-27T16:17:57Z
And you can output everything except that Z (or it may be +00:00, I
don't
remember) and append a hardcoded timezone (like -05:00 or something)
directly in the template.
Ugly? You bet! But maybe less ugly that using something else just for
timestamp changing. Though a cleaner method could be to add this
functionality to rsyslog.
I would love to if someone could point me at a method that doesn't mean
duplicating tenthousands of operating system code lines...
Aha, I think I understand what you're saying (and your previous Email). But
let me double-check first :)
Say rsyslog receives a timestamp with no timezone. And in the config I say
that my timezone is "Europe/Bucharest" or something else from
/usr/share/zoneinfo/. There's no easy way to tell that this is GMT+03:00
right now and add it to a rfc-3339 timestamp?
First of all, you don't know the sender is in the same timezone. So you may
be guessing wrong.
Let's assume it is in the same zone. I think we get the +3 offset, and we
already do this today. As long as everything is in one zone, all is fine.
Because that's what Linux (and POSIX) assume. It just gets messy if you
have multiple zones...
Because that sounds quite complicated. I'm such a noob, all I can say is I
found this on the Internet and I have no idea if it helps or not:
http://stackoverflow.com/questions/13804095/get-the-time-zone-gmt-offset-in-c
That's the simple "one zone only" case.
On the other hand, if I say "+03:00" in my config, wouldn't be easy to just
do a string replace somewhere and come up with a GMT+3 timestamp?
If just adding a "+03:00" is fine, we are all set. No problem here. Some of
the input modules permit you to specify this (experimentally, but so for
two years now, so I'd say it's actually mature).
Of course
this implies changing the config for every DST change.
this is where it begins to get complicated. There is no clean way to get
DST begin/end information other than for the current time zone. And that's
the real problem. All API calls depend on the TZ environment variable. The
only solution to conversion is to
1. set TZ to a different zone
2. do the time API calls
3. reset TZ to the correct zone
This, of course, needs to be done under mutex protection, which also means
that all other TZ-based calls (like message creation) need to be done under
that same mutex. It's horrible inefficient to the point that this is not
something you can actually do in production code.
But to make matters wores, the original poster wanted the timestamp to be
*converted* to UTC, not just the offset added. Now think about these
timestamp
2000-03-01T01:00:00+02:00 (leap year)
2015-03-29T02:30:00+02:00 (right in the middle of DST transistion)
What are these in UTC? To do that right, you need to full tzinfo, which is
right there sitting in your Linux system but without any access to it other
than the strange TZ env var mangling.
David once suggested to copy that code and utilize it. While I agree that
it is a decent solution, I don't want to do that. Alone keeping up on
security I consider a nightmare.
The real solution would be to submit a patch that makes a decent API
available, but that would probably mean I would need to set of quite a bit
of time to read and understand that code. As the TZ conversion question
doesn't come up very often, I don't consider the functionality important
enough to warrant a very lengthy implementation.
I hope that explains. Again, all suggestions are very welcome.
Rainer
Not sure if this helps...
Best regards,
Radu
P.S. I hate time zones. And DST. DST more than timezones.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
