Re: [rsyslog] Another approach to action load balancing

David Lang Thu, 24 Oct 2013 13:48:37 -0700

try setting a variable=$!msgid % 3 and add that to your output template, thatmay indicate what's going wrong.


David Lang

On Thu, 24 Oct 2013, Robert McIntyre wrote:

Date: Thu, 24 Oct 2013 13:17:49 -0700
From: Robert McIntyre <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Another approach to action load balancing

Hrrm, thanks for verifiying.  Very strange, then.  I'm definitely not getting 
any distribution of messages, but have verified that the serial (or SN) numbers 
are incrementing.

Are you seeing the rest of the logic (writing to the different files) working, 
perchance?

Thanks!
Robert

Date: Thu, 24 Oct 2013 23:59:54 +0400
From: [email protected]
To: [email protected]
Subject: Re: [rsyslog] Another approach to action load balancing


Very strange, because it works for me with some 7.4 from git.

     set $!msg = "vpn_tunnel=\"N/A\" src_int=\"port6\" dst_int=\"port5\"
app=\"N/A\" app_cat=\"N/A\" user=\"N/A\" group=\"N/A\" serial=1249572
app-type=\"N/A\"";
     set $!msgid = re_extract($!msg, " serial=([0-9]*)", 0, 1, "0");

I'm using this template to look at results:

template(
     name="common1"
     type="string"
     string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%
_%$!%_\n"
)

And they are:

  _{ "msg": "vpn_tunnel=\"N\/A\" src_int=\"port6\" dst_int=\"port5\"
app=\"N\/A\" app_cat=\"N\/A\" user=\"N\/A\" group=\"N\/A\"
serial=1249572 app-type=\"N\/A\"", "msgid": "1249572" }_


--
Pavel Levshin


24.10.2013 23:44, Robert McIntyre:

Thanks, Andre!  Downloaded and testing.  Unfortunately, I'm just getting almost 
nowhere with my regexes. :|  I've pasted the appropriate part of the config, 
and some sections of events for reference below

Let me start with what *is* working.  I can use re_match(), and it finds the 
strings and routes appropriately (This is Option 1 in the config below).  But, 
for some reason, I can't get the re_extract() to pull a value out (this is the 
commented out Option 2 in the config below).  Based on what I can see from 
debug logs, it never finds a match.  I've tested the regexs and strings with 
the rsyslog regex tester, and it looks like they should be working.

Any suggestions?

# Option 1, this verifies that the regex's work, and do what is expected
if (re_match($msg,"serial=([0-9]*)"))
     then set $!msgid = "0";
else if (re_match ($msg, "SN=([0-9]*)"))
     then set $!msgid = "1";

# Option 2, this should extract either the serial number, or SN from the 
message, but doesn't
#set $!msgid = re_extract($msg, " serial=([0-9]*)", 0, 1, "0");
#if ($!msgid == "0")
#     then set $!msgid = re_extract($msg, " SN=([0-9]*)", 0, 1, "0");

  if ($!msgid % 3 == '0')
  then {
   action(name="Act_File1"
    type="omfile"
    file="/syslogdata/testing/1.txt")
   stop
   }
  if ($!msgid % 3 == '1')
  then {
   action(name="Act_File2"
    type="omfile"
    file="/syslogdata/testing/2.txt")
   stop
   }
  if ($!msgid % 3 == '2')
  then {
   action(name="Act_File3"
    type="omfile"
    file="/syslogdata/testing/3.txt")
   stop
   }
  action(name="Act_File4"
   type="omfile"
   file="/syslogdata/testing/4.txt")

Extracts from logs:
vpn_tunnel="N/A" src_int="port6" dst_int="port5" SN=1718017351 app="N/A" app_cat="N/A" 
user="N/A" group="N/A"

vpn_tunnel="N/A" src_int="port6" dst_int="port5" app="N/A" app_cat="N/A" user="N/A" 
group="N/A" serial=1249572 app-type="N/A"

Thanks!
Robert

From: [email protected]
Date: Thu, 24 Oct 2013 09:15:09 +0200
To: [email protected]
Subject: Re: [rsyslog] Another approach to action load balancing

I am sorry for the missing rpm packages for 7.4.5. For some reason, my build
environment failed to build packages for EHEL 6 x64.
Packages have been created now and should be available in the repository.

Best regards,
Andre Lorbach

-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Robert McIntyre
Sent: Thursday, October 24, 2013 12:43 AM
To: rsyslog-users
Subject: Re: [rsyslog] Another approach to action load balancing

Make sense!  Now just waiting for the RPMs get updated, and will give this
a
go!

Thanks!
Robert
________________________________
From: David Lang<mailto:[email protected]>
Sent: 10/23/2013 3:33 PM
To: rsyslog-users<mailto:[email protected]>
Subject: Re: [rsyslog] Another approach to action load balancing

local variables only exist in the 7.5 branch.

in 7.4 you only have $! variables.

David Lang

On Wed, 23 Oct 2013, Robert McIntyre wrote:

Date: Wed, 23 Oct 2013 11:48:19 -0700
From: Robert McIntyre <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Another approach to action load balancing

So, I've had decent luck with Pavel's suggestion

(field($timegenerated,':',3), and it rotates around nicely based on the
second.

I'm trying a slightly different approach, though, to try to get
sub-second

rotation.  My firewall logs have a log sequence number that I'd like to
use as
the input to my modulus, but I'm having trouble extracting it.  Using the
rsyslog regex builder/tester, I came up with this regex:

%msg:R,ERE,1,DFLT:SN=([0-9]*)--end%

But, when I try to use it in my config it doesn't work.  I've tried
setting a

local variable:

$.msgid = "%msg:R,ERE,1,DFLT:SN=([0-9]*)--end%"
or
# $.msgid = "msg:R,ERE,1,DFLT:SN=([0-9]*)--end"
or
set $.msgid = "msg:R,ERE,1,DFLT:SN=([0-9]*)--end"
or
set # $.msgid = "%msg:R,ERE,1,DFLT:SN=([0-9]*)--end%"

But get config errors regardless.

I've tried putting these variations directly in the if clause:

if ("msg:R,ERE,1,DFLT:SN=([0-9]*)--end" % 3 == '0')

(and all the iterations), but no luck with that.

So, under 7.4.4, what is the recommended way to extract a string/number

from a message, and then use that extracted value in an expression?

I've got some other lessons learned from this that I plan to write up
for the

group, but want to get this final bit sorted first.

Thanks!!!
Robert

Date: Wed, 23 Oct 2013 08:41:20 -0700
From: [email protected]
To: [email protected]
Subject: Re: [rsyslog] Another approach to action load balancing

there should be per the docs, but in practice there is not. At least
not as it is accessed via the scripting variables. I think if you use
it as a property in a template you get the higher precision.

David Lang

On Wed, 23 Oct 2013, Robert McIntyre wrote:

Thanks, Pavel!  This works as expected.  The docs say that
$timegenerated is "always in high resolution".  Is that max
resolution seconds?  I'm trying to figure out how to just see the
value of $timegenerated to see what format it is (I'm assuming
HH:MM:SS based on the field statement, but wonder if there's a .XX at

the end).

Thanks!!!
Robert

Date: Wed, 23 Oct 2013 18:00:04 +0400
From: [email protected]
To: [email protected]
Subject: Re: [rsyslog] Another approach to action load balancing

Here is what you looked for:

field($timegenerated,':',3);

It is a number, so you can balance per second based on it. And it
works with 7.4.4.

--
Pavel Levshin

23.10.2013 17:12, Robert McIntyre:

Thanks, that's too bad.  I was quite excited yesterday, thinking
about

the problem and reading the docs, but couldn't figure it out after
spending
some time with my test server.

I recall the other thread recently about the documentation, and how
to

make it clear what's applicable to which version, as well as what's
possible
across features (Rainer script crossed with property replacer in this
case).
This is an example of that issue.

I don't have much to contribute to this project other than
questions, the occasional answer for someone else, and thanks, so
I'll reiterate: thanks to everyone working on this project! :)

Thanks!
Robert
________________________________
From: Rainer Gerhards<mailto:[email protected]>
Sent: 10/23/2013 4:01 AM
To: rsyslog-users<mailto:[email protected]>
Subject: Re: [rsyslog] Another approach to action load balancing

On Wed, Oct 23, 2013 at 12:41 PM, Pavel Levshin

<[email protected]>wrote:

So, not all system properties are accessible from RainerScript, in
7.4.
There is none having resolution of seconds. Here they are:

$now (this is just a date, unfortunately) $year $month $day $hour
$minute $myhostname

And that's all. In 7.5, all is complicated right now.

I am working on that ;) I could promise to add some $$nowseconds
sysvar,

but looking at the current schedule I better do not do that...

Rainer

--
Pavel


23.10.2013 10:33, Pavel Levshin:

It seemes that you are unable to access $uptime property (as
$$uptime, I suppose). The same is true for 7.4 and 7.5.5.

It works for me, because there is a regression after latest fixes
for global variables. In HEAD, I can access $uptime (as $uptime),
but do not see any property without $ at start.

As for more precise counter, it is timegenerated. But it is also
unusable because you cannot access subseconds from RainerScript.

AFAIK.

______________________________**_________________
rsyslog mailing list
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.a
discon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/**professional-

services/<http://www.rsyslog

.com/professional-services/> What's up with rsyslog? Follow
https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing
list, posts are ARCHIVED by a myriad of sites beyond our control.
PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of

sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of

sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of

sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites

beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites

beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites

beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Another approach to action load balancing

Reply via email to