Probably the last email on this, I swear! :) I've had a chance to do a
bunch of digging around, and here's some thoughts/notes on why this was so
weird for me to get sorted out:
1. Match and sub-match for re_extract() start with 0, not 1, which is
noted in the docs, but didn't register initially
2. Re_extract doesn't like the typical IP regex (eg.
\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}), it didn't seem to like the escape "\"
characters, so I wound up having to use [0-9]{1,3} and so forth. It works,
but not sure why the escaped characters didn't work. (Unless I needed to
escape the escape characters because it's in a string, which makes my head
hurt; hrrm.)
3. Cnum(), or the "type-less" conversion, (probably, can't check the
sources) uses longs, which means that really long sequence/serial numbers
from firewalls cause it to overload and do weird things which make modulus
also weird.
4. Doing type-less comparisons leaves me wondering whether they're not
evaluating properly, or whether the problem lies upstream when things don't
work properly. The $.msgid == "2" vs. $.msgid == 2 thing left me uncertain
what was correct for a while.
So, I wound up using the last octet of the source ip address for the
firewall traffic as my load-balancing input, extracted with a re_extract(),
and Pavel's field($timegenerated,':',3) for when that fails (admin messages
and IPv6 addresses). Certainly not the most elegant solution, but looking
at the distribution over 30+ minutes it's pretty good, and the single regex
doesn't seem to put too much load on the system.
Thanks, everyone, for your patience and input!
Robert
From: Robert McIntyre [mailto:[email protected]]
Sent: Thursday, October 24, 2013 1:38 PM
To: rsyslog-users
Subject: RE: [rsyslog] Another approach to action load balancing
Aaaand, I feel stupid. The modulus comparison needs to be something like
this:
if ($!msgid % 3 == 2)
without quotes around the "2".
Thanks so much, everyone!!!
--Robert
> From: <mailto:[email protected]> [email protected]
> To: <mailto:[email protected]> [email protected]
> Date: Thu, 24 Oct 2013 13:17:49 -0700
> Subject: Re: [rsyslog] Another approach to action load balancing
>
> Hrrm, thanks for verifiying. Very strange, then. I'm definitely not
getting any distribution of messages, but have verified that the serial (or
SN) numbers are incrementing.
>
> Are you seeing the rest of the logic (writing to the different files)
working, perchance?
>
> Thanks!
> Robert
>
> > Date: Thu, 24 Oct 2013 23:59:54 +0400
> > From: <mailto:[email protected]> [email protected]
> > To: <mailto:[email protected]> [email protected]
> > Subject: Re: [rsyslog] Another approach to action load balancing
> >
> >
> > Very strange, because it works for me with some 7.4 from git.
> >
> > set $!msg = "vpn_tunnel=\"N/A\" src_int=\"port6\" dst_int=\"port5\"
> > app=\"N/A\" app_cat=\"N/A\" user=\"N/A\" group=\"N/A\" serial=1249572
> > app-type=\"N/A\"";
> > set $!msgid = re_extract($!msg, " serial=([0-9]*)", 0, 1, "0");
> >
> > I'm using this template to look at results:
> >
> > template(
> > name="common1"
> > type="string"
> > string="%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%
> > _%$!%_\n"
> > )
> >
> > And they are:
> >
> > _{ "msg": "vpn_tunnel=\"N\/A\" src_int=\"port6\" dst_int=\"port5\"
> > app=\"N\/A\" app_cat=\"N\/A\" user=\"N\/A\" group=\"N\/A\"
> > serial=1249572 app-type=\"N\/A\"", "msgid": "1249572" }_
> >
> >
> > --
> > Pavel Levshin
> >
> >
> > 24.10.2013 23:44, Robert McIntyre:
> > > Thanks, Andre! Downloaded and testing. Unfortunately, I'm just getting
almost nowhere with my regexes. :| I've pasted the appropriate part of the
config, and some sections of events for reference below
> > >
> > > Let me start with what *is* working. I can use re_match(), and it
finds the strings and routes appropriately (This is Option 1 in the config
below). But, for some reason, I can't get the re_extract() to pull a value
out (this is the commented out Option 2 in the config below). Based on what
I can see from debug logs, it never finds a match. I've tested the regexs
and strings with the rsyslog regex tester, and it looks like they should be
working.
> > >
> > > Any suggestions?
> > >
> > > # Option 1, this verifies that the regex's work, and do what is
expected
> > > if (re_match($msg,"serial=([0-9]*)"))
> > > then set $!msgid = "0";
> > > else if (re_match ($msg, "SN=([0-9]*)"))
> > > then set $!msgid = "1";
> > >
> > > # Option 2, this should extract either the serial number, or SN from
the message, but doesn't
> > > #set $!msgid = re_extract($msg, " serial=([0-9]*)", 0, 1, "0");
> > > #if ($!msgid == "0")
> > > # then set $!msgid = re_extract($msg, " SN=([0-9]*)", 0, 1, "0");
> > >
> > > if ($!msgid % 3 == '0')
> > > then {
> > > action(name="Act_File1"
> > > type="omfile"
> > > file="/syslogdata/testing/1.txt")
> > > stop
> > > }
> > > if ($!msgid % 3 == '1')
> > > then {
> > > action(name="Act_File2"
> > > type="omfile"
> > > file="/syslogdata/testing/2.txt")
> > > stop
> > > }
> > > if ($!msgid % 3 == '2')
> > > then {
> > > action(name="Act_File3"
> > > type="omfile"
> > > file="/syslogdata/testing/3.txt")
> > > stop
> > > }
> > > action(name="Act_File4"
> > > type="omfile"
> > > file="/syslogdata/testing/4.txt")
> > >
> > > Extracts from logs:
> > > vpn_tunnel="N/A" src_int="port6" dst_int="port5" SN=1718017351
app="N/A" app_cat="N/A" user="N/A" group="N/A"
> > >
> > > vpn_tunnel="N/A" src_int="port6" dst_int="port5" app="N/A"
app_cat="N/A" user="N/A" group="N/A" serial=1249572 app-type="N/A"
> > >
> > > Thanks!
> > > Robert
> > >
> > >> From: <mailto:[email protected]> [email protected]
> > >> Date: Thu, 24 Oct 2013 09:15:09 +0200
> > >> To: <mailto:[email protected]> [email protected]
> > >> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>
> > >> I am sorry for the missing rpm packages for 7.4.5. For some reason,
my build
> > >> environment failed to build packages for EHEL 6 x64.
> > >> Packages have been created now and should be available in the
repository.
> > >>
> > >> Best regards,
> > >> Andre Lorbach
> > >>
> > >>> -----Original Message-----
> > >>> From: <mailto:[email protected]>
[email protected] [mailto:rsyslog-
> > >>> <mailto:[email protected]> [email protected]] On
Behalf Of Robert McIntyre
> > >>> Sent: Thursday, October 24, 2013 12:43 AM
> > >>> To: rsyslog-users
> > >>> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>>
> > >>> Make sense! Now just waiting for the RPMs get updated, and will give
this
> > >>> a
> > >>> go!
> > >>>
> > >>> Thanks!
> > >>> Robert
> > >>> ________________________________
> > >>> From: David Lang< <mailto:[email protected]> mailto:[email protected]>
> > >>> Sent: 10/23/2013 3:33 PM
> > >>> To: rsyslog-users< <mailto:[email protected]>
mailto:[email protected]>
> > >>> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>>
> > >>> local variables only exist in the 7.5 branch.
> > >>>
> > >>> in 7.4 you only have $! variables.
> > >>>
> > >>> David Lang
> > >>>
> > >>> On Wed, 23 Oct 2013, Robert McIntyre wrote:
> > >>>
> > >>>> Date: Wed, 23 Oct 2013 11:48:19 -0700
> > >>>> From: Robert McIntyre < <mailto:[email protected]>
[email protected]>
> > >>>> Reply-To: rsyslog-users < <mailto:[email protected]>
[email protected]>
> > >>>> To: rsyslog-users < <mailto:[email protected]>
[email protected]>
> > >>>> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>>>
> > >>>> So, I've had decent luck with Pavel's suggestion
> > >>> (field($timegenerated,':',3), and it rotates around nicely based on
the
> > >>> second.
> > >>>> I'm trying a slightly different approach, though, to try to get
> > >>>> sub-second
> > >>> rotation. My firewall logs have a log sequence number that I'd like
to
> > >>> use as
> > >>> the input to my modulus, but I'm having trouble extracting it. Using
the
> > >>> rsyslog regex builder/tester, I came up with this regex:
> > >>>> %msg:R,ERE,1,DFLT:SN=([0-9]*)--end%
> > >>>>
> > >>>> But, when I try to use it in my config it doesn't work. I've tried
> > >>>> setting a
> > >>> local variable:
> > >>>> $.msgid = "%msg:R,ERE,1,DFLT:SN=([0-9]*)--end%"
> > >>>> or
> > >>>> # $.msgid = "msg:R,ERE,1,DFLT:SN=([0-9]*)--end"
> > >>>> or
> > >>>> set $.msgid = "msg:R,ERE,1,DFLT:SN=([0-9]*)--end"
> > >>>> or
> > >>>> set # $.msgid = "%msg:R,ERE,1,DFLT:SN=([0-9]*)--end%"
> > >>>>
> > >>>> But get config errors regardless.
> > >>>>
> > >>>> I've tried putting these variations directly in the if clause:
> > >>>>
> > >>>> if ("msg:R,ERE,1,DFLT:SN=([0-9]*)--end" % 3 == '0')
> > >>>>
> > >>>> (and all the iterations), but no luck with that.
> > >>>>
> > >>>> So, under 7.4.4, what is the recommended way to extract a
string/number
> > >>> from a message, and then use that extracted value in an expression?
> > >>>> I've got some other lessons learned from this that I plan to write
up
> > >>>> for the
> > >>> group, but want to get this final bit sorted first.
> > >>>> Thanks!!!
> > >>>> Robert
> > >>>>
> > >>>>
> > >>>> Date: Wed, 23 Oct 2013 08:41:20 -0700
> > >>>> From: <mailto:[email protected]> [email protected]
> > >>>> To: <mailto:[email protected]> [email protected]
> > >>>> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>>>
> > >>>> there should be per the docs, but in practice there is not. At
least
> > >>>> not as it is accessed via the scripting variables. I think if you
use
> > >>>> it as a property in a template you get the higher precision.
> > >>>>
> > >>>> David Lang
> > >>>>
> > >>>> On Wed, 23 Oct 2013, Robert McIntyre wrote:
> > >>>>
> > >>>>> Thanks, Pavel! This works as expected. The docs say that
> > >>>>> $timegenerated is "always in high resolution". Is that max
> > >>>>> resolution seconds? I'm trying to figure out how to just see the
> > >>>>> value of $timegenerated to see what format it is (I'm assuming
> > >>>>> HH:MM:SS based on the field statement, but wonder if there's a .XX
at
> > >>> the end).
> > >>>>> Thanks!!!
> > >>>>> Robert
> > >>>>>
> > >>>>>
> > >>>>>> Date: Wed, 23 Oct 2013 18:00:04 +0400
> > >>>>>> From: <mailto:[email protected]> [email protected]
> > >>>>>> To: <mailto:[email protected]> [email protected]
> > >>>>>> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Here is what you looked for:
> > >>>>>>
> > >>>>>> field($timegenerated,':',3);
> > >>>>>>
> > >>>>>> It is a number, so you can balance per second based on it. And it
> > >>>>>> works with 7.4.4.
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Pavel Levshin
> > >>>>>>
> > >>>>>>
> > >>>>>> 23.10.2013 17:12, Robert McIntyre:
> > >>>>>>> Thanks, that's too bad. I was quite excited yesterday, thinking
> > >>>>>>> about
> > >>> the problem and reading the docs, but couldn't figure it out after
> > >>> spending
> > >>> some time with my test server.
> > >>>>>>> I recall the other thread recently about the documentation, and
how
> > >>>>>>> to
> > >>> make it clear what's applicable to which version, as well as what's
> > >>> possible
> > >>> across features (Rainer script crossed with property replacer in
this
> > >>> case).
> > >>> This is an example of that issue.
> > >>>>>>> I don't have much to contribute to this project other than
> > >>>>>>> questions, the occasional answer for someone else, and thanks,
so
> > >>>>>>> I'll reiterate: thanks to everyone working on this project! :)
> > >>>>>>>
> > >>>>>>> Thanks!
> > >>>>>>> Robert
> > >>>>>>> ________________________________
> > >>>>>>> From: Rainer Gerhards< <mailto:[email protected]>
mailto:[email protected]>
> > >>>>>>> Sent: 10/23/2013 4:01 AM
> > >>>>>>> To: rsyslog-users< <mailto:[email protected]>
mailto:[email protected]>
> > >>>>>>> Subject: Re: [rsyslog] Another approach to action load balancing
> > >>>>>>>
> > >>>>>>> On Wed, Oct 23, 2013 at 12:41 PM, Pavel Levshin
> > >>> < <mailto:[email protected]> [email protected]>wrote:
> > >>>>>>>> So, not all system properties are accessible from RainerScript,
in
> > >>>>>>>> 7.4.
> > >>>>>>>> There is none having resolution of seconds. Here they are:
> > >>>>>>>>
> > >>>>>>>> $now (this is just a date, unfortunately) $year $month $day
$hour
> > >>>>>>>> $minute $myhostname
> > >>>>>>>>
> > >>>>>>>> And that's all. In 7.5, all is complicated right now.
> > >>>>>>>>
> > >>>>>>>> I am working on that ;) I could promise to add some
$$nowseconds
> > >>>>>>>> sysvar,
> > >>>>>>> but looking at the current schedule I better do not do that...
> > >>>>>>>
> > >>>>>>> Rainer
> > >>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Pavel
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> 23.10.2013 10:33, Pavel Levshin:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> It seemes that you are unable to access $uptime property (as
> > >>>>>>>>> $$uptime, I suppose). The same is true for 7.4 and 7.5.5.
> > >>>>>>>>>
> > >>>>>>>>> It works for me, because there is a regression after latest
fixes
> > >>>>>>>>> for global variables. In HEAD, I can access $uptime (as
$uptime),
> > >>>>>>>>> but do not see any property without $ at start.
> > >>>>>>>>>
> > >>>>>>>>> As for more precise counter, it is timegenerated. But it is
also
> > >>>>>>>>> unusable because you cannot access subseconds from
RainerScript.
> > >>> AFAIK.
> > >>>>>>>>>
> > >>>>>>>> ______________________________**_________________
> > >>>>>>>> rsyslog mailing list
> > >>>>>>>>
<http://lists.adiscon.net/**mailman/listinfo/rsyslog%3chttp:/lists.a>
http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.a
> > >>>>>>>> discon.net/mailman/listinfo/rsyslog>
> > >>>>>>>> <http://www.rsyslog.com/**professional->
http://www.rsyslog.com/**professional-
> > >>> services/< <http://www.rsyslog%0b> http://www.rsyslog
> > >>>>>>>> .com/professional-services/> What's up with rsyslog? Follow
> > >>>>>>>> <https://twitter.com/rgerhards> https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing
> > >>>>>>>> list, posts are ARCHIVED by a myriad of sites beyond our
control.
> > >>>>>>>> PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> > >>>>>>>>
> > >>>>>>> _______________________________________________
> > >>>>>>> rsyslog mailing list
> > >>>>>>> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>>>>> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >>>>>>> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards NOTE
> > >>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
> > >>>>>>> of
> > >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >>> DON'T LIKE THAT.
> > >>>>>>> _______________________________________________
> > >>>>>>> rsyslog mailing list
> > >>>>>>> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>>>>> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >>>>>>> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards NOTE
> > >>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
> > >>>>>>> of
> > >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >>> DON'T LIKE THAT.
> > >>>>>> _______________________________________________
> > >>>>>> rsyslog mailing list
> > >>>>>> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>>>> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >>>>>> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards NOTE
> > >>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of
> > >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > >>> DON'T LIKE THAT.
> > >>>>> _______________________________________________
> > >>>>> rsyslog mailing list
> > >>>>> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>>> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >>>>> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards NOTE
> > >>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of
> > >>>>> sites
> > >>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > >>> LIKE THAT.
> > >>>> _______________________________________________
> > >>>> rsyslog mailing list
> > >>>> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >>>> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards NOTE
> > >>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
> > >>>> sites
> > >>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > >>> LIKE THAT.
> > >>>> _______________________________________________
> > >>>> rsyslog mailing list
> > >>>> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>>> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >>>> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards NOTE
> > >>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
> > >>>> sites
> > >>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > >>> LIKE THAT.
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
> >
> > _______________________________________________
> > rsyslog mailing list
> > <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> > <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> <http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://lists.adiscon.net/mailman/listinfo/rsyslog
> <http://www.rsyslog.com/professional-services/>
http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow <https://twitter.com/rgerhards>
https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
