[Simple-evcorr-users] possible change in variable substitution process?

2010-11-12 Thread Risto Vaarandi
hi all, due to work and travelling, I have been unable to work on the new version for couple of weeks, but I am returning to the track. I'd like to ask an open question from list members that concerns the variable substitution process. Currently, the variable is not substituted if it either does

Re: [Simple-evcorr-users] Debug help on using context expressions and %alnum

2010-11-24 Thread Risto Vaarandi
On 11/24/2010 12:15 AM, Tim Peiffer wrote: > On 11/17/10 3:13 PM, Tim Peiffer wrote: >> >> I need some help in debugging a context expression. I create intercept >> zones in my recursive DNS configurations in a somewhat automated >> manner. Conversely, I would like to know when the zones expire,

Re: [Simple-evcorr-users] handling dated logfiles

2010-11-26 Thread Risto Vaarandi
Tim, that's a good question. In fact, SEC uses the stat(2) system call for checking the attributes of the log file, and the check is applied both for the open file descriptor and file name. Normally, those two checks return identical results, but there are some special cases. First, suppose the fil

Re: [Simple-evcorr-users] handling dated logfiles

2010-11-27 Thread Risto Vaarandi
same rule to create the link. kind regards, risto > > > Regards, > Tim Peiffer > > On 11/26/10 4:34 PM, Risto Vaarandi wrote: >> >> Tim, >> that's a good question. In fact, SEC uses the stat(2) system call for >> checking the attributes of the log file,

Re: [Simple-evcorr-users] handling dated logfiles

2010-11-28 Thread Risto Vaarandi
2010/11/28 Peter Wolfenden : > If you want to make absolutely sure to process all the log lines *and* you > are in a position to control how your backup application writes to its log > files, then it may be worth considering using "multilog" to send one copy of > the data to an automatically rotate

Re: [Simple-evcorr-users] query about sending data to SEC

2010-12-01 Thread Risto Vaarandi
On 12/01/2010 03:55 PM, M Haris Farooque wrote: > dear all, > > I have a very lame question to ask. > > how to send data to SEC from command line. The SEC is running as daemon. > I am using a FIFO (Pipe) from SEC to write some data instantly as log > data through pipe and its working fine but I l

Re: [Simple-evcorr-users] query about sending data to SEC

2010-12-02 Thread Risto Vaarandi
> Hello Risto, > > In the begininnig No, but later Yes, I did specify > -input=/usr/local/etc/SEC_Log_Pipe. And /tmp/sec.dump also not showing > my input. here is the snapshot of sec.dump; > > Program options: --conf=/usr/local/etc/testmodel/myrules/main.cfg > --conf=/usr/local/etc/testmodel/myrule

Re: [Simple-evcorr-users] query about sending data to SEC

2010-12-03 Thread Risto Vaarandi
On 12/03/2010 11:42 AM, M Haris Farooque wrote: > Am 02.12.2010 11:17, schrieb Risto Vaarandi: >>> Hello Risto, >>> >>> In the begininnig No, but later Yes, I did specify >>> -input=/usr/local/etc/SEC_Log_Pipe. And /tmp/sec.dump also not showing >>

Re: [Simple-evcorr-users] read/write operations through named pipe

2010-12-08 Thread Risto Vaarandi
Am I understanding correctly that you want to write some of the data from named pipe to an external log? Since SEC is already reading from the pipe, it's a bad idea to start another process that reads the same pipe, since reading removes data from pipe. If would recommend a separate rule that would

Re: [Simple-evcorr-users] read/write operations through named pipe

2010-12-08 Thread Risto Vaarandi
On 12/08/2010 06:45 PM, John P. Rouillard wrote: > > In message<4cffa586.70...@fleetboard.com>, > M Haris Farooque writes: >> I have another question related to Named Pipe. >> I have made a "/usr/local/etc/SEC_Log_Pipe " and hang it with SEC so=20 >> that SEC can read input text from it; >> >> *cmd

Re: [Simple-evcorr-users] mysql database queries - example select debug

2010-12-09 Thread Risto Vaarandi
2010/12/9 Tim Peiffer : > > I am trying to use mysql table lookups to extract connector and contact > information to provide look aside for handling various correlator.  Can > I assign and dereference perl arrays in the eval minicode?  How do I > pass parameters from the regexp as quoted strings to

Re: [Simple-evcorr-users] Feature request - pass SECRC and create a title via commandline.

2010-12-11 Thread Risto Vaarandi
Tim, although I can see some rationale for the secrc flag, the -title option looks somewhat weird to me. Actually, in the UNIX world it is common to create a hard link to a program if there is a need to run the same program under different names. There are also other opportunities to distinguish pr

Re: [Simple-evcorr-users] Feature request - pass SECRC and create a title via commandline.

2010-12-11 Thread Risto Vaarandi
shorter. BR, risto 2010/12/12 Risto Vaarandi : > Tim, > although I can see some rationale for the secrc flag, the -title > option looks somewhat weird to me. Actually, in the UNIX world it is > common to create a hard link to a program if there is a need to run > the same program

Re: [Simple-evcorr-users] Feature request - pass SECRC and create a title via commandline.

2010-12-13 Thread Risto Vaarandi
On 12/12/2010 05:34 AM, John P. Rouillard wrote: > > In message<4d042711.7090...@umn.edu>, > Tim Peiffer writes: >> First, the -title is not needed unless you can't get enough of the >> command line in the buffer to see what is going on. In one case I use >> Solaris 5.10, and I can't get the whol

[Simple-evcorr-users] SEC-2.6.alpha1 released

2010-12-31 Thread Risto Vaarandi
hi folks, today, I've released SEC-2.6.alpha1 which includes a number of new features like the support for named match variables, variable maps, several performance improvements and one bug fix, and finally the EventGroup rule which allows for correlation of custom number of _different_ events in

Re: [Simple-evcorr-users] SEC-2.6.alpha1 released

2011-01-04 Thread Risto Vaarandi
2011/1/4 John P. Rouillard : > > Hi Risto: > > In message , > Risto Vaarandi writes: >>Also, I have made substantial changes to the man page and although the >>overall structure is roughly the same, a lot of content is completely >>rewritten. The old man page was

[Simple-evcorr-users] SEC-2.6.alpha2 released

2011-01-06 Thread Risto Vaarandi
lpha2.tar.gz/download kind regards, risto 2010/12/31 Risto Vaarandi : > hi folks, > > today, I've released SEC-2.6.alpha1 which includes a number of new > features like the support for named match variables, variable maps, > several performance improvements and one bug fix, and

Re: [Simple-evcorr-users] Invalid regular expression

2011-01-13 Thread Risto Vaarandi
On 01/13/2011 11:37 AM, M Haris Farooque wrote: > hi all, > > I am getting this error *Rule in /usr/local/sec/rules/snpv.cfg at line > 17: * *Invalid regular expression > '^\[\d+\]\sNEW\s(*SERVICE:\s(?|*snpv1**)\;(service1)|(**snpv**1)\;(service2)|(**snpv**)\;(service3)|(**snpv3**)\;(service4)|(*

Re: [Simple-evcorr-users] Invalid regular expression

2011-01-14 Thread Risto Vaarandi
> I used (?| to enable branch reset operation in order to make following > pattern variables; > > |m{ > ^ \[ \d+ \] : \s > (?| (|snpv1|);(|service1|)# $1, $2 >| (|snpv|2);(|service|2)# $1, $2 >| (|snpv|3);(|service|3)# $1, $2 >| (|snpv|2);(|ser

Re: [Simple-evcorr-users] Pair question

2011-01-20 Thread Risto Vaarandi
As I understand, you would like to do some sort of balance checking if every foo has a corresponding bar? The event correlation operations that Pair and PairWithWindow rules trigger actually consume repeated instances of the first event silently (in your case foo). In the case of your rule, mail wi

Re: [Simple-evcorr-users] Pair question

2011-01-20 Thread Risto Vaarandi
On 01/20/2011 06:50 PM, Kim Scarborough wrote: > Risto Vaarandi wrote: >> As I understand, you would like to do some sort of balance checking if >> every foo has a corresponding bar? The event correlation operations >> that Pair and PairWithWindow rules trigger actu

Re: [Simple-evcorr-users] Looking for advice on a creating sec rule for a problem...

2011-01-21 Thread Risto Vaarandi
On 01/21/2011 01:25 AM, Mark D. Nagel wrote: > On 1/20/2011 12:40 PM, Morris, Patrick wrote: >> On 1/20/2011 11:26 AM, Morris, Christopher wrote: >>> >>> >>> type=SingleWithThreshold >>> >>> continue=takenext >>> >>> ptype=RegExp >>> >>> pattern=:\d\d \S+ .*Liberty app at (\S+) (.*) >>> >>> desc=

Re: [Simple-evcorr-users] a query regarding input file auto-rotation

2011-01-25 Thread Risto Vaarandi
On 01/25/2011 05:51 PM, M Haris Farooque wrote: > > Hi all, > > I just like to know, what will happen when a file (data.log) is rotated > automatically which is actually hooked with SEC_STARTUP event as shown > in following rule definition. > > /type=single > ptype=substr > pattern=SEC_STARTUP > co

Re: [Simple-evcorr-users] how do you make a report on a sliding window

2011-02-18 Thread Risto Vaarandi
On 02/18/2011 01:48 AM, da...@lang.hm wrote: > I want to do a 'alert if more than X events in Y min' type of thing, but > with the resulting alert containing the logs of the X events. > > I know that I can do something along the lines of > > if event > add log message to context (with an expiration

[Simple-evcorr-users] opinions on implementing pattern match caching

2011-02-25 Thread Risto Vaarandi
hi all, I'd like to return to a topic recently discussed in the mailing list. I am in the process of implementing a pattern match caching for SEC, and it is really hard to decide which way is the best one. Currently, I have already implemented a separate Parse rule for this purpose. For example,

Re: [Simple-evcorr-users] opinions on implementing pattern match caching

2011-02-27 Thread Risto Vaarandi
2011/2/27 John P. Rouillard : > > In message <4d679119.5060...@seb.ee>, > Risto Vaarandi writes: >>I'd like to return to a topic recently discussed in the mailing list. >>I am in the process of implementing a pattern match caching for SEC, and >>it is reall

[Simple-evcorr-users] SEC-2.6.0 available

2011-03-14 Thread Risto Vaarandi
hi all, I've released SEC-2.6.0 which provides some additions to the 2.6.alpha versions, most notably support for pattern match caching. Also, the input status polling functions have been updated, in order to lessen the number of system calls. The new version is available at: http://sourcefor

Re: [Simple-evcorr-users] Working With SEC, Part One now available

2011-03-18 Thread Risto Vaarandi
hi all, just to inform you -- Jim's tutorial has changed its location and is now hosted at SourceForge. I have also changed the pointers at SEC home page. I would like to use this opportunity and thank Jim for putting together this tutorial which has served the users for 8 long years :) kind r

[Simple-evcorr-users] 10th birthday of SEC

2011-03-23 Thread Risto Vaarandi
hi all, a small note that might be interesting for some. The first public version of SEC (1.0) was released 10 years ago, in March 23 2001. Few facts about the 1.0 version: - it had 3,059 lines and 75KB of code (in contrast, the latest 2.6.0 version has almost 10,000 lines and 280KB of code) -

Re: [Simple-evcorr-users] 10th birthday of SEC

2011-03-24 Thread Risto Vaarandi
> > My deepest admiration for your tool and general work on this very interesting > topic! > > C. > > On 23 Mar 2011, at 11:15, Eric Smith wrote: > >> On 03/23/2011 05:25 AM, Risto Vaarandi wrote: >>> I would like to thank all list members for exchanging ma

Re: [Simple-evcorr-users] Context expirations and counts

2011-03-25 Thread Risto Vaarandi
hi Miles, in fact, your question touches nicely couple of new features of the 2.6.0 version. With previous versions, there were several options for addressing this question. First, you could set up two rules in the style described in Q17 of the FAQ (http://simple-evcorr.sourceforge.net/FAQ.html#

Re: [Simple-evcorr-users] Extract a block of text between two markers

2011-03-25 Thread Risto Vaarandi
On 03/25/2011 02:27 PM, Supratik Goswami wrote: > Hi > > I want to extract everything with multiple lines between two markers. > > I want to display everything between: > > /WY_LOG_TYPE_ERROR <<**/ > > > /**>>/ > / > / > / > / > So if the text entered in the log file > > /WY_LOG_TYPE_ERROR <<**/ >

Re: [Simple-evcorr-users] Extract a block of text between two markers

2011-03-28 Thread Risto Vaarandi
e lines are logged in the input file, but this time the > previous 20 lines are still there in the buffer. So the regular > expression matches them also. > How can I clear the input buffer each time? > > > On Fri, Mar 25, 2011 at 8:32 PM, Risto Vaarandi <mailto:risto.vaara...@s

Re: [Simple-evcorr-users] Extract a block of text between two markers

2011-03-28 Thread Risto Vaarandi
(?!.*WY_LOG_TYPE_ERROR)(.*)\*\*>>$/ > /action=logonly/ > > The problem now is SEC is firing the event twice. So if I set > action=logonly, its getting logged twice. If I set action=(send mail to > me), its sending two mails. > > Any help will be highly appreciated. > >

Re: [Simple-evcorr-users] Extract a block of text between two markers

2011-03-28 Thread Risto Vaarandi
In fact, I like David's approach more, since it this prevents expensive multiline matching against *all* input. Or to put it differently, decomposing a problem into several simple questions is often more efficient than attacking the original issue. kind regards, risto 2011/3/28 : > On Mon, 28 Mar

Re: [Simple-evcorr-users] Extract a block of text between two markers

2011-03-28 Thread Risto Vaarandi
2011/3/29 : > On Mon, 28 Mar 2011, Risto Vaarandi wrote: > >> In fact, I like David's approach more, since it this prevents >> expensive multiline matching against *all* input. Or to put it >> differently, decomposing a problem into several simple questions is

Re: [Simple-evcorr-users] Count the number of event filtering by parameters.

2011-03-29 Thread Risto Vaarandi
Ludovic, there are several ways to address the problem, but it depends what exactly you would like to do. Do you want to keep track of different user names, and report current counters for all users once in X minutes, or do you rather want to send a report for each user after the user has been

Re: [Simple-evcorr-users] Count the number of event filtering by parameters.

2011-03-29 Thread Risto Vaarandi
'action' field of Single. In fact, I posted the previous example for illustrating the capabilities of the new EventGroup rule type :) kind regards, risto > > Le 29/03/2011 11:40, Risto Vaarandi a écrit : >> Ludovic, >> >> there are several ways to address the pro

Re: [Simple-evcorr-users] Using $variable with script

2011-04-05 Thread Risto Vaarandi
hi Rafael, there are three kind of variables that can be used in SEC rules: 1) action list variables which are visible in action lists only (e.g., %t or %s), 2) match variables which are set by patterns (e.g., $1 or $+{varname}), 3) Perl variables that are set and used in Perl code snippets defined

Re: [Simple-evcorr-users] Using $variable with script

2011-04-06 Thread Risto Vaarandi
g pattern and stored in $1. If the time elapsed is less than n seconds > I would like to execute action; otherwise, execute action2. > > Hope it makes sense. > > Regards, > Rafael > > On Apr 5, 2011, at 2:12 PM, Risto Vaarandi wrote: > >> hi Rafael, >> there

Re: [Simple-evcorr-users] Input field within rule definition

2011-04-07 Thread Risto Vaarandi
On 04/05/2011 11:54 PM, MILLS, ROCKY (ATTSI) wrote: > For discussion only -- not an immediate need to be addressed. > > ~ > Well, the 'input' field looks like a synonym to the file context to me... Maybe I haven't got all the details for the 'input' field, though. However, there is one danger re

Re: [Simple-evcorr-users] Question about clearing of the pair rule

2011-04-07 Thread Risto Vaarandi
hi Uwe, the problem you are seeing is caused by a side effect of Pair rule, but can easily be fixed by changing the 'pattern2' field just a bit. Let me explain why this happening. After you have submitted SEC the first 4 input lines, SEC has two event correlation operations running that have be

Re: [Simple-evcorr-users] Input field within rule definition

2011-04-07 Thread Risto Vaarandi
2011/4/7 MILLS, ROCKY (ATTSI) : > Risto, et al, > > Advantages of input field within rule sets: > ... > 4. eliminates extraneous perl code to extract "input" file/source name > Actually, this issue is best addressed with named match variables that were introduced into the 2.6 version. Previously,

Re: [Simple-evcorr-users] Referencing perl hashes with eval

2011-04-08 Thread Risto Vaarandi
Rafael, you need to escape the %-sign with another %, so %hash should be written as %%hash. The problem is that action list variables also begin with %, and they are substituted before the Perl code is evaluated. With pre 2.6 versions, variables without values were not substituted, but this created

Re: [Simple-evcorr-users] can't get SingleWithSuppress to work

2011-04-14 Thread Risto Vaarandi
hi Marc, you have set the 'desc' field of the rule to $0 which holds th entire matching line. However, 'desc' field defines the event correlation operation key. The input lines tend to contain timestamps which make lines different, and therefore for every line a separate operation is started with a

Re: [Simple-evcorr-users] Suggestions for handling multiple streams/events - disabling alerting based on source, etc.

2011-04-15 Thread Risto Vaarandi
hi John, I would recommend to use contexts, once you have seen a match from particular rules that should disable matches for several other rules. The contexts are visible across all rules and it is easy to check their presence or absence with Boolean expressions. For example, for disabling input co

Re: [Simple-evcorr-users] Suggestions for handling multiple streams/events - disabling alerting based on source, etc.

2011-04-16 Thread Risto Vaarandi
irst into a separate variable, and then evaluating m/.*\/(.*)\.SystemOut\.log$ against the input file name. hope this helps, risto > > > > > -Risto Vaarandi wrote: - > > To: John Grasett > From: Risto Vaarandi > Date: 04/15/2011 12:43PM > Cc: simple-ev

Re: [Simple-evcorr-users] Suggestions for handling multiple streams/events - disabling alerting based on source, etc.

2011-04-17 Thread Risto Vaarandi
2011/4/17 Risto Vaarandi : > 2011/4/15 John Grasett : >> Yes, that sounds perfect. I could also then do this to not repeat on the >> same event in the same log. >> >> type=single >> ptype=perlfunc >> pattern=sub { if ($_[0] =~ /.*(SRVE0255E).*/) { \ >

Re: [Simple-evcorr-users] Suggestions for handling multiple streams/events - disabling alerting based on source, etc.

2011-04-19 Thread Risto Vaarandi
se the value of the manipulkated log message if I try to > manipulate the value of the log source. > > I have everything else I wanted to rest working fine, short lived > context blocking of further alerts works just great...my email > script is being triggered perfectly

Re: [Simple-evcorr-users] SEC Rule Help

2011-04-29 Thread Risto Vaarandi
hi Edward, the task you have can be addressed with the help of context aliases. The following simplistic rule sets up a context and an alias for an observed event. The alias will suppress further events with the same FRQ number, but if an event with different number comes in, the context is del

Re: [Simple-evcorr-users] Calling GeoIP function

2011-05-04 Thread Risto Vaarandi
hi Ray, since %geoip is an action list variable, it can only be used in other action lists (which are executed after a successful pattern match). However, matching users not coming from particular countries should be done with a pattern (or pattern with a context expression, as done below). I woul

Re: [Simple-evcorr-users] How to generate OID and make SEC to communicate with zenoss.

2011-05-17 Thread Risto Vaarandi
On 05/16/2011 01:17 PM, Waseem Hawaldar wrote: > Hi , > > I have installed both SEC (ver 2.5.2) and zenoss in 2 different VMs. I > have tested SEC with simple expression matching and writing the results > to file. > > Now I am trying to make SEC to talk with zenoss so that I can monitor > the event

Re: [Simple-evcorr-users] How to generate OID and make SEC to communicate with zenoss.

2011-05-17 Thread Risto Vaarandi
l expression > and send the filtered contents to zenoss. > > Please suggest me some tips for this problem. Any help will be appreciated. > > > > Regards, > Waseem > > > > -Original Message- > From: Risto Vaarandi [mailto:risto.vaara...@seb.ee] > Sent: Tuesday,

Re: [Simple-evcorr-users] How to generate OID and make SEC to communicate with zenoss.

2011-05-17 Thread Risto Vaarandi
program is actually found. If you want to suppress this non-critical warning, however, please specify snmptrap with full path. kind regards, risto > > > Regards, > Waseem > > > -Original Message- > From: Risto Vaarandi [mailto:risto.vaara...@seb.ee] >

Re: [Simple-evcorr-users] How to generate OID and make SEC to communicate with zenoss.

2011-05-17 Thread Risto Vaarandi
; Well, it is hard to comment on that, since these error messages are coming from the snmptrap utility of the Net-SNMP software suite. Please either search Net-SNMP mailing lists for this particular problem solution, or try posting your question to these lists: http://net-snmp.sourceforge

Re: [Simple-evcorr-users] shellcmd with timeout

2011-05-20 Thread Risto Vaarandi
On 05/20/2011 02:56 PM, Matthieu Pérotin wrote: > Hi, > > we recently experienced an annoying problem with processes that, in some > circonstances, would get stuck and never return. The fault here is > clearly on the processes side, but one can never be sure that a process > will return nicely... T

Re: [Simple-evcorr-users] shellcmd with timeout

2011-05-23 Thread Risto Vaarandi
Pérotin : > Le vendredi 20 mai 2011 à 18:11 +0300, Risto Vaarandi a écrit : >> On 05/20/2011 02:56 PM, Matthieu Pérotin wrote: >> > Hi, >> > >> > we recently experienced an annoying problem with processes that, in some >> > circonstances, would get stuck and ne

Re: [Simple-evcorr-users] shellcmd with timeout

2011-05-24 Thread Risto Vaarandi
On 05/23/2011 04:51 PM, Matthieu Pérotin wrote: > Hi Risto, > > the solutions you list are fine with us. The only objections I may have > are: > - the shell only solution induce an additional fork, which is not > necessary with a patch. It may be problematic in heavy loaded systems; > - we are loos

Re: [Simple-evcorr-users] Few questions

2011-06-13 Thread Risto Vaarandi
hi Ludovic, here are quick answers to your questions. The %t variable is set according to the clock of the node where SEC is running. However, the timestamps of log messages are often set by the network node which emitted the messages. Therefore, the value of %t variable can differ from the time

Re: [Simple-evcorr-users] Few questions

2011-06-14 Thread Risto Vaarandi
2011/6/14 Ludovic Hutin : > Hi, > >     I need help (again)... > >     I got many entry like PATTERN1(time) PATTERN2(username) PATTERN3(ip) >     I want to log in a file unique ip of a user with a timeout of 10mn > >     12h20    USERA    10.10.10.10 => this entry have to be log. (with a > action=w

Re: [Simple-evcorr-users] Few questions

2011-06-17 Thread Risto Vaarandi
hi Ludovic, it is impossible to set case insensitive processing for context names. However, there are several workarounds. First, with a PerlFunc pattern it is possible to convert the user name to lower (or upper) case format, and return it for the match variable. Second, if you would like to corre

Re: [Simple-evcorr-users] Few questions

2011-06-17 Thread Risto Vaarandi
R,,,,,13001,SUCCESS,,T,,,,userlogin,XXX,XXX > > > There is no order in the event correlation. The two event are generated > on differents computer in a very short period, they can be redirected to > the centralized syslog server in

Re: [Simple-evcorr-users] Another question.. (sorry)

2011-06-21 Thread Risto Vaarandi
Ludovic, if you are willing to tolerate a slight inaccuracy, the following fairly simple rule could do: type=EventGroup init=create COUNTING_$2 end=delete COUNTING_$2 ptype=RegExp pattern=([\d.]+) (\w+) context=!$1_COUNTED_FOR_$2 count=alias COUNTING_$2 $1_COUNTED_FOR_$2 desc=3 logins from diffe

Re: [Simple-evcorr-users] Few questions

2011-06-21 Thread Risto Vaarandi
On 06/21/2011 12:01 PM, Ludovic Hutin wrote: > Hi all, > > The solution work, so it's enough for us. Add another tool for > "transforming" the log is not the best idea (i think) > It's impossible for us to configure the tool to write in syslog. I > think we have to best solution. > >

Re: [Simple-evcorr-users] Question about some special case

2011-06-22 Thread Risto Vaarandi
On 06/22/2011 03:34 PM, Ludovic Hutin wrote: > Hi, > > I have a question about the "write" action > I got something like > write /var/log/result/$13.login %t $8 ; \ > > If we got a $13 like "&& rm -fr /" or something like that, what > should append ? > Are we prot

Re: [Simple-evcorr-users] Another question.. (sorry)

2011-06-26 Thread Risto Vaarandi
t;         count=lcall %ret $2 $1 -> (  sub { use Data::Dumper; > $ucountsIP{$_[0]}->{$_[1]} = 1 ; print Dumper($ucountsIP{$_[0]}) ;  } ); >         // This one is not a perl function ;) >         end=lcall %ret $2 -> ( sub { foreach keys in $ucountsIP{$_[0]}; > ( write_to_file $keys

Re: [Simple-evcorr-users] How to limit the number of fork process running in parallel

2011-06-30 Thread Risto Vaarandi
hi Kevin, although there is no command line option for limiting the number of child processes, you can check their number from a context expression. The info about all children is stored to SEC's internal %children hash with PIDs acting as keys (you can access this hash by using the main:: pre

Re: [Simple-evcorr-users] shellcmd with timeout

2011-07-12 Thread Risto Vaarandi
On 05/23/2011 05:10 PM, Jeff Schroeder wrote: > 2011/5/23 Matthieu Pérotin: >> Le vendredi 20 mai 2011 à 18:11 +0300, Risto Vaarandi a écrit : >>> On 05/20/2011 02:56 PM, Matthieu Pérotin wrote: >>>> Hi, >>>> >>>> we recently experi

Re: [Simple-evcorr-users] Count & report event number

2011-08-07 Thread Risto Vaarandi
hi Jean Baptiste, you might try the following rule: type=EventGroup ptype=regexp pattern=" (\d)[0-9]{2} \d+ count=lcall %ret $1 -> ( sub { ++$req; if ($_[0] == 2) { ++$req2; } } ); desc=Count HTTP request response codes action=none init=lcall %ret -> ( sub { $req = 0; $req2 = 0; } ) end=lcall %re

Re: [Simple-evcorr-users] Count & report event number

2011-08-16 Thread Risto Vaarandi
tely, with regular files there are no such issues. kind regards, risto > > Best regards, > JB > > > On 07/08/2011 23:08, Risto Vaarandi wrote: >> hi Jean Baptiste, >> >> you might try the following rule: >> >> type=EventGroup >> ptype=regexp

Re: [Simple-evcorr-users] Count & report event number

2011-08-16 Thread Risto Vaarandi
On 08/13/2011 09:28 PM, John P. Rouillard wrote: > In message, > da...@lang.hm writes: > >> SEC works just fine reading from named pipe or stdin from a socket. I have >> this working with rsyslog with the only problem being that when I want to >> change the SEC rules, it involves a restart of rsysl

Re: [Simple-evcorr-users] Store/restore contexts in sec 2.6.0

2011-08-22 Thread Risto Vaarandi
Pedro, you need to use two percent-signs (%%) in front of main::context_list. This rule used to work with earlier SEC version only because the 2.4 version left unset action list variables intact. Note that since there is % in front of main::, and variable substitution is carried out inside Perl

Re: [Simple-evcorr-users] Need to kill -ABRT my sec process periodically

2011-09-10 Thread Risto Vaarandi
2011/9/9 Joe Prosser : > Hi Folks, > I have an extremely busy sec process running with 17675 active > contexts.  I'm finding that I need to kill -ABRT the process every > hour or so or else it hogs the CPU and lags in reading the input file. >   I know the number of active contexts has been growing

Re: [Simple-evcorr-users] Need to kill -ABRT my sec process periodically

2011-09-10 Thread Risto Vaarandi
Just out of curiosity -- if SIGUSR1 signal is sent to the sec process for creating the dump file (tmp/sec.dump), what does the "Performance statistics" section look like in the dump? Look at the "Run time", "User time", "System time", "Processed input lines" readings -- the figures tell you how man

Re: [Simple-evcorr-users] Need to kill -ABRT my sec process periodically

2011-09-10 Thread Risto Vaarandi
ump rules? hope this helps, risto > > On Sat, Sep 10, 2011 at 3:12 PM, Risto Vaarandi > wrote: >> Just out of curiosity -- if SIGUSR1 signal is sent to the sec process >> for creating the dump file (tmp/sec.dump), what does the "Performance >> statistics" secti

[Simple-evcorr-users] SEC-2.6.1 released

2011-09-16 Thread Risto Vaarandi
hi all, SEC-2.6.1 has been released which contains some improvements over the 2.6.0 version, most notably the support for $+{_inputsrc} match variable which holds input file name(s), and --keepopen / --nokeepopen flags which change input handling for soft restarts. The new versions is availabl

Re: [Simple-evcorr-users] assign on SEC_STARTUP

2011-09-30 Thread Risto Vaarandi
hi Thomas, these error messages are actually not caused by the rule below, but rather by other rules which employ the %n variable. When SEC loads its rules, all paths to external programs are checked and if the program is not found, a warning message is logged. In your case, you have of course

Re: [Simple-evcorr-users] Questions about thread

2011-10-13 Thread Risto Vaarandi
hi Ludovic, SEC is mono-threaded. Although some parts of the code could be run in parallel, there are many parts in the code which require specific order of execution. Unfortunately, this also applies to rule processing and pattern matching (which usually consume most of the CPU time). Quite often,

Re: [Simple-evcorr-users] Selective deletion from context

2011-10-13 Thread Risto Vaarandi
hi Ralf, although there is no separate action for this purpose, the context event store can be filtered in various ways through several actions. I would assign the event store to an action list variable, pass this variable to a Perl code for filtering, and assign the result back to the context even

Re: [Simple-evcorr-users] Selective deletion from context

2011-10-14 Thread Risto Vaarandi
it is impossible to permanently memorize all details you have read from such a document :) regards, risto > > Thank you, > > -Ralf- > > > > From "Risto Vaarandi" : > >>hi Ralf, >>although there is no separate action for this purpose, the context >>

Re: [Simple-evcorr-users] Selective deletion from context

2011-10-16 Thread Risto Vaarandi
2011/10/15 Ralf Schmitt : > Hi Risto, > > thank you for the encouraging pat on the back ;) > > I'm taking a closer look at the documentation and the tutorial from > Jim Brown (nice document) at the moment. > > But I'm still not able to solve my problem. Here is a rule set that: > > - saves "IP not

Re: [Simple-evcorr-users] How to force SEC not to process a file from start when it is being edited.

2011-10-19 Thread Risto Vaarandi
On 10/19/2011 01:55 PM, Supratik Goswami wrote: > When monitoring a file using SEC, it normally tails on that file and > any new changes can be matched against some pattern. > If someone edits that file using any editor SEC recognizes that the > file has been recreated and shows the below message >

Re: [Simple-evcorr-users] How to group a problem and its symptoms

2011-11-04 Thread Risto Vaarandi
hi, yes, the EventGroup rule is probably the best solution for this case, since it does matching for unordered event groups. However, in order to verify that P1 is the first event to match the rule and trigger the event correlation operation, you could create a context from the 'count' field for P1

Re: [Simple-evcorr-users] How to group a problem and its symptoms

2011-11-11 Thread Risto Vaarandi
. why use context alias? This is done, in order to delete all contexts at the end of the event correlation operation (ie, garbage collection). The deletion is done from the 'end' field. regards, risto > Thanks again for your help, meanwhile I will keep playing with sec. > Gaok

Re: [Simple-evcorr-users] Making Ncached pattern type more useful and per event contexts

2011-11-17 Thread Risto Vaarandi
hi John, I think that I already answered some of the questions off the list, but I'll try to comment unanswered questions below. 2011/11/7 John P. Rouillard : > Hi all: > > I have a rather complex set of rules that I used to do flood > suppression to prevent jabbering services/hosts from making it

Re: [Simple-evcorr-users] How to group a problem and its symptoms

2011-11-19 Thread Risto Vaarandi
w. If that's the case, the EventGroup might not be the best solution, since it expects *all* events to occur. If you want to act on lone instances of events, several Single rules that use the same context might be a much better idea. Can you actually provide us with a somewhat more detailed

Re: [Simple-evcorr-users] Multiple Occurrences with Count Reset

2011-12-05 Thread Risto Vaarandi
On 12/04/2011 11:31 PM, Alan Deasy wrote: > Hi > > Thanks to Risto for SEC. It has been a great success in one of the banks here. > > An app developer recently approached me to ask if it is possible for SEC to > monitor multiple occurrences of an event, but if one is different, then reset > the c

Re: [Simple-evcorr-users] How to group a problem and its symptoms

2011-12-05 Thread Risto Vaarandi
I've been a little bit busy, so I could'nt answer before. > > Below you can find my comments after yours. > > On Sat, 19 Nov 2011 11:52:17 -0800 Risto > Vaarandi wrote > > > 2011/11/17 mindman101: > > > Hi! > > > > >

Re: [Simple-evcorr-users] Data normalization

2011-12-13 Thread Risto Vaarandi
On 12/12/2011 04:01 PM, Alberto Cortón wrote: > Hi, > > I would like to know if any of you have used SEC for normalizing log data. My > first approach to this was to generate normalized events like this: > > action = event > 'TIME=$1:::CODE=$3:::SRC_IP=$4:::SRC_PORT=$5:::DST_IP=$6:::DST_PORT=$7::

Re: [Simple-evcorr-users] Data normalization

2011-12-13 Thread Risto Vaarandi
On 12/12/2011 04:01 PM, Alberto Cortón wrote: > Hi, > > I would like to know if any of you have used SEC for normalizing log data. My > first approach to this was to generate normalized events like this: > > action = event > 'TIME=$1:::CODE=$3:::SRC_IP=$4:::SRC_PORT=$5:::DST_IP=$6:::DST_PORT=$7::

[Simple-evcorr-users] rewriting input

2011-12-13 Thread Risto Vaarandi
hi all, some months ago, we had a discussion on rewriting input events: http://sourceforge.net/mailarchive/forum.php?thread_name=4E066179.3010304%40willingminds.com&forum_name=simple-evcorr-users Would a similar feature be of interest to the end users? :) I was thinking about attacking the proble

Re: [Simple-evcorr-users] Data normalization

2011-12-13 Thread Risto Vaarandi
2011/12/13 : > On Tue, 13 Dec 2011, Risto Vaarandi wrote: > >> ...to add another idea -- if you want to run a very fast normalization >> on logs with multi-line events, you could also take advantage of the >> LogPP (Log PreProcessor) utility at http://logpp.sourceforge

Re: [Simple-evcorr-users] How to group a problem and its symptoms

2011-12-14 Thread Risto Vaarandi
On 12/14/2011 04:19 PM, mindman101 wrote: > Hi Risto, > > Thanks for your answer, you've got the idea. > > However, I still have a final doubt. > > The association among root cause and its son events are both IP and device > name. So, the Single rule and the Eventgroup rule type work fine from you

Re: [Simple-evcorr-users] rewriting input

2011-12-19 Thread Risto Vaarandi
On 12/13/2011 08:26 PM, Mark D. Nagel wrote: > On 12/13/2011 4:20 AM, Risto Vaarandi wrote: >> hi all, >> some months ago, we had a discussion on rewriting input events: >> >> http://sourceforge.net/mailarchive/forum.php?thread_name=4E066179.3010304%40willingminds.c

Re: [Simple-evcorr-users] Sec with Rsyslog

2011-12-21 Thread Risto Vaarandi
On 12/21/2011 01:43 AM, Kaushal Shriyan wrote: > Hi > > I have gone through http://simple-evcorr.sourceforge.net/ and it is > quite interesting and there is also a learning process. At present I am > using rsyslog daemon as a centralized server and several rsyslog clients > connecting to it. Not su

Re: [Simple-evcorr-users] Sec with Rsyslog

2011-12-21 Thread Risto Vaarandi
On 12/21/2011 12:33 PM, Kaushal Shriyan wrote: > > > On Wed, Dec 21, 2011 at 2:02 PM, Risto Vaarandi <mailto:risto.vaara...@seb.ee>> wrote: > > On 12/21/2011 01:43 AM, Kaushal Shriyan wrote: > > Hi > > I have gone through http://simple-evcor

Re: [Simple-evcorr-users] Sec with Rsyslog

2011-12-21 Thread Risto Vaarandi
> The following NEW packages will be installed: >   sec > 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. > Need to get 0B/74.1kB of archives. > After this operation, 360kB of additional disk space will be used. > Selecting previously deselected package sec. > (Reading database ... 5

Re: [Simple-evcorr-users] SEC & unix sockets

2011-12-28 Thread Risto Vaarandi
On 12/28/2011 05:05 PM, sylver_b wrote: > Hello, > > I've just discovered SEC and it seems to be the perfect fit for what we > are trying to do . > > Basically , we are running a voip peering service but have to face fraud > on a daily basis. We tried to imagine all sort of ways to detect/stop > fr

Re: [Simple-evcorr-users] SEC & unix sockets

2011-12-28 Thread Risto Vaarandi
2011/12/28 John P. Rouillard : > > In message <4efb49c8.30...@seb.ee>, > Risto Vaarandi writes: >>On 12/28/2011 05:05 PM, sylver_b wrote: >>> Basically , we are running a voip peering service but have to face fraud >>> on a daily basis. We tried to

Re: [Simple-evcorr-users] rewriting input

2011-12-29 Thread Risto Vaarandi
On 12/13/2011 08:26 PM, Mark D. Nagel wrote: > On 12/13/2011 4:20 AM, Risto Vaarandi wrote: >> hi all, >> some months ago, we had a discussion on rewriting input events: >> >> http://sourceforge.net/mailarchive/forum.php?thread_name=4E066179.3010304%40willingminds.c

<    1   2   3   4   5   6   7   8   9   10   >