On 12/15/2013 10:44 PM, Rainer Gerhards wrote:
On Mon, Dec 16, 2013 at 4:05 AM, Erik Steffl <[email protected]> wrote:
changed config so now the MARK messages are sent (and received). BTW
they use kern.info facility, not sure why it's not the same as what
Rainer found in source (syslog.info).
That's actually a bug - and even though this may break some older things,
one I really need to fix. We should never emit messages under kern....
ok, I guess we should filter on both kern.info and syslog.info to
make our config forward compatible.
any idea why would messages from /dev/log be treated differently then
MARK messages? See also my other email about MARK messages being resent
and grouped together... Think I might be missing something obvious about
immark or how the messages are processed or something...
thanks!
erik
Rainer
This solves the simple test case in which I only have sender, load
balancer (ELB) and receiver. I strace both sender and receiver, see the
MARK messages and the connection is fine, i.e. I send a log message using
e.g. logger, then wait 5 min, then send another one and it works. Without
MARK the second message never arrives.
However in a more complex scenario this does not help at all. Complex
scenario looks like this (ascii arrows are flows of syslog messages):
- 6 machines -> ELB-prod -> collector-prod
- 1 machine -> ELB-test -> collector-test
- every 5 minutes: collector-test -> ELB-prod -> collector-prod
- every 5 minutes: collector-prod -> ELB-prod -> collector-prod (yes,
program on collector-prod sends message to rsyslog on collector-prod over
ELB)
that 5 minute pause cause the connection to go stale somehow which
results in periods of silence. I configured both collector-prod and
collector-test to send MARK messages (to collector-prod since that's where
they send regular messages) but I still see the periods of silence (on both
collectors). Used strace to verify that MARK messages arrive (I guess it's
possible that I confused the MARK messages from collector-prod and
collector-test, will continue investigating on that front)
However adding cron entry to send few log messages every minute DOES
solve the problem (there is no silence anymore).
Any ideas why that would be? Is it possible that MARK messages are being
sent through different connections than other messages?
As far as I can tell the only difference between MARK messages and the
cron'd messages is that the MARK messages are generated by immark and use
kern.info facilty and the cron'd messages arrive via /dev/log and use
local0.info facility.
Any ideas why would the simpler scenario behave differently than the
complex scenario? Or why MARK messages do not solve the problem in complex
scenario while cron'd messages do?
thanks!
erik
On 12/12/2013 02:39 PM, David Lang wrote:
On Thu, 12 Dec 2013, Erik Steffl wrote:
I will try to test librelp (with keepalive) but I need some
workaround in the meantime (sort of right now).
Already tested that cron-ing logger once per minute keeps the
connection alive so that's my backup workaround.
immark would be better cause then I only need to install rsyslog
config (easier deployment) plus it would be more efficient, do you
think that what David suggested is the best option?
if I understood David's comment something like this is what I am
looking for:
if
prifilt("local0.*") or
prifilt("local1.*") or
prifilt("local2.*") or
prifilt("local3.*") or
prifilt("local4.*") or
prifilt("local5.*") or
prifilt("local6.*") or
prifilt("local7.*") or
( prifilt("syslog.info") and ... message is --MARK--)
pretty much, I would do $msg == '--MARK--' as the second test
David Lang
then {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
action(
type="omrelp"
target="someHost"
port="5140"
template="json"
# see http://www.rsyslog.com/doc/node32.html
# disk used if forwarding blocked
queue.filename="json"
queue.maxdiskspace="75161927680" # 70GB (valuable data)
action.writeAllMarkMessages="on"
)
} else {
...
}
reasonable? Can be improved?
thanks!
erik
On 12/12/2013 12:44 PM, Rainer Gerhards wrote:
On Thu, Dec 12, 2013 at 9:17 PM, David Lang <[email protected]> wrote:
On Thu, 12 Dec 2013, Erik Steffl wrote:
On 12/12/2013 08:29 AM, David Lang wrote:
what facility and severity do the immark messages show up as?
immark just generates messages, normal filtering rules determine
where
theya re sent, and the transport used (in this case RELP) has no
effect
on if they are sent or not, it's all in the filters.
thanks, that makes my question a lot more specific. How do I
configure
immark to use a specific facility?
I don't think you do. I think they are using the syslog or kernel
facility, but I'd have to setup a quick test to check. I'll try to
do it
tonight if I can, but since you are seeing the messages locally, log
with
RSYSLOG_DebugFormat for a couple of minutes and look at what they are
logged as.
its syslog.=info:
http://git.adiscon.com/?p=rsyslog.git;a=blob;f=plugins/
immark/immark.c;h=0e946c0b92d555174b38de42dd236ac4432b98e7;hb=HEAD#l196
All I found when searching is this:
$ModLoad immark.so
$MarkMessagePeriod 60
which is what I have in my config.
Given that I see the --MARK-- messages in /var/log/syslog and
/var/log/kern.log I guess they are going to kern facility. Given
the config
below I need to use e.g. local0 facility.
no, you need to change your filtering config to send these messages,
not
try to change the messages to match your current config.
you actually can't. I considered mark a legacy feature and have not
enhanced it since 8 yrs.
Keepalive is the better option. librelp is not yet build due to the
current
workload. The code actually right now is at github only, as I have some
problems with the Adiscon repo. Easy to clone from here
https://github.com/rgerhards/librelp
messages have the facility that they have, you don't change the
facility
any more than you re-write the message to say something different.
actually, in this case a config option would make sense. But again, I
thought this is just legacy...
Rainer
David Lang
Unfortunately can't find anything related to --MARK-- and
facilities (or
anything else other than the two settings above).
Any ideas/pointers? Or if not possible to configure immark can I
catch
the --MARK-- message and change its facility? Or catch the --MARK--
message
and have action that uses omrelp and same target (would that use
same TCP
connection)?
Thanks!
erik
David Lang
On Thu, 12 Dec 2013, Erik Steffl wrote:
Date: Thu, 12 Dec 2013 02:30:52 -0800
From: Erik Steffl <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] immark - how to use with action(...)
How would I use immark to send mark messages for defined
actions that
use omrelp?
I have tried something like this:
$ModLoad immark.so
$MarkMessagePeriod 60
if(..)
if
prifilt("local0.*") or
prifilt("local1.*") or
prifilt("local2.*") or
prifilt("local3.*") or
prifilt("local4.*") or
prifilt("local5.*") or
prifilt("local6.*") or
prifilt("local7.*")
then {
action(type="mmjsonparse")
if $parsesuccess == "OK" then {
action(
type="omrelp"
target="someHost"
port="5140"
template="json"
# see http://www.rsyslog.com/doc/node32.html
# disk used if forwarding blocked
queue.filename="json"
queue.maxdiskspace="75161927680" # 70GB (valuable data)
action.writeAllMarkMessages="on"
)
} else {
...
}
I see --MARK-- messages in /var/log/syslog and /var/log/kern.log but
they are not send by omrelp action (the action works fine, normal
messages are going through).
Verified where the --MARK-- messages are going using strace so
pretty
sure they are only going to those two local files, nothing goes over
RELP. Also checked on the receiving side of RELP, no incoming
messages
have --MARK-- in them. And the connection goes down which is also
very
strong indicator that there are no --MARK-- messages.
How do I configure it so that the --MARK-- messages are send over
RELP
protocol to someHost (over same TCP connection that the given action
uses, it's for purpose to keep alive the connection since RELP does
not support KeepAlive (yet, Rainer just added it to master,
thanks!))
This is on Ubuntu 13.10 using rsyslog 7.5.6, librelp 1.2.0 from
adiscon repo.
Thanks!
erik
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
