2012/5/21 Jyothi Madallapalli <mnjyo...@hotmail.com>: > Thanks Risto. I was unable to get this to work. First rule works where we > create the context if it does not exist and print the line and start the > counter, but when I add the second rule (rule that you provided) it compiles > fine but nothing matches or is written to the output file.
That's probably because in your rule there is $0 specified for output file. I have tested the ruleset below myself and it writes output strings correctly to standard output: type=single ptype=regexp continue=takenext pattern=<DicXmlAsyncMsgService> Sending Ack to (\S+) failed context=!REPEATING_DICXML_EVENTS_BUCKET desc=$0 action=create REPEATING_DICXML_EVENTS_BUCKET; write - $0; \ eval %o ($count = 0) type=single ptype=regexp pattern=<DicXmlAsyncMsgService> Sending Ack to (\S+) failed context=REPEATING_DICXML_EVENTS_BUCKET && =(++$count == 100) desc=$0 action=delete REPEATING_DICXML_EVENTS_BUCKET; \ write - $0 repeats 100 times, don't want to spam so capturing in one single line For debugging purposes, I replaced the threshold 100 with 5, and here is my output: <DicXmlAsyncMsgService> Sending Ack to 1 failed Evaluating code '$count = 0' and setting variable '%o' Variable '%o' set to '0' <DicXmlAsyncMsgService> Sending Ack to 2 failed <DicXmlAsyncMsgService> Sending Ack to 3 failed <DicXmlAsyncMsgService> Sending Ack to 4 failed <DicXmlAsyncMsgService> Sending Ack to 5 failed Deleting context 'REPEATING_DICXML_EVENTS_BUCKET' Context 'REPEATING_DICXML_EVENTS_BUCKET' deleted Writing event '<DicXmlAsyncMsgService> Sending Ack to 5 failed repeats 100 times, don't want to spam so capturing in one single line' to file - <DicXmlAsyncMsgService> Sending Ack to 5 failed repeats 100 times, don't want to spam so capturing in one single line <DicXmlAsyncMsgService> Sending Ack to 6 failed Creating context 'REPEATING_DICXML_EVENTS_BUCKET' Writing event '<DicXmlAsyncMsgService> Sending Ack to 6 failed' to file - <DicXmlAsyncMsgService> Sending Ack to 6 failed Evaluating code '$count = 0' and setting variable '%o' Variable '%o' set to '0' <DicXmlAsyncMsgService> Sending Ack to 7 failed <DicXmlAsyncMsgService> Sending Ack to 8 failed <DicXmlAsyncMsgService> Sending Ack to 9 failed <DicXmlAsyncMsgService> Sending Ack to 10 failed Deleting context 'REPEATING_DICXML_EVENTS_BUCKET' Context 'REPEATING_DICXML_EVENTS_BUCKET' deleted Writing event '<DicXmlAsyncMsgService> Sending Ack to 10 failed repeats 100 times, don't want to spam so capturing in one single line' to file - <DicXmlAsyncMsgService> Sending Ack to 10 failed repeats 100 times, don't want to spam so capturing in one single line > > Another question I have is, how can I capture N same consecutive lines as one > single line like "XYZ repeats 10 times consecutively" For this, you can employ RegExpN pattern type (e.g., for use RegExp10 for matching N consecutive lines). regards, risto > > Thanks, > Jyothi > > > > > On May 19, 2012, at 3:56 PM, "Risto Vaarandi" <risto.vaara...@gmail.com> > wrote: > >> hi Jyothi, >> for offline processing, the use of Perl variables for counting is >> indeed the best approach. >> In fact, you can optimize this ruleset a bit by capturing the last two >> rules into one rule (I haven't tested it, but the idea is to increment >> the counter in the context evaluation): >> >> type=single >> ptype=regexp >> pattern=<DicXmlAsyncMsgService> Sending Ack to (\S+) failed >> context=REPEATING_DICXML_EVENTS_BUCKET && =(++$count == 100) >> desc=$0 >> action=delete REPEATING_DICXML_EVENTS_BUCKET; write $0 repeats 100 >> times, don't want to spam so capturing in one single line >> >> To optimize things even further, you can use the ->() operator instead >> of =() in the context expression, and lcall action instead of eval -- >> now the perl code is compiled only once, instead of compiling it >> before each execution. >> >> Hope this helps, >> risto >> >> 2012/5/18 Jyothi Madallapalli <mnjyo...@hotmail.com>: >>> Thanks Risto. This helps. >>> Another question, again in the offline mode. I want to set up a threshold to >>> a simple pattern matching event without using the window factor. >>> The requirement is : Find a pattern, if it matches write the line to an >>> output file, do not write the next N number of lines. After N same lines >>> mare matched, start the count and write again. >>> Could eval help here with something like this? >>> >>> #DicXmlAsyncMsgService Ack failed events >>> type=single >>> ptype=regexp >>> continue=takenext >>> pattern=<DicXmlAsyncMsgService> Sending Ack to (\S+) failed >>> context=!REPEATING_DICXML_EVENTS_BUCKET >>> desc=$0 >>> action=create REPEATING_DICXML_EVENTS_BUCKET; write secOutput.log $0; eval >>> %o ($count = 0);\ >>> >>> type=single >>> ptype=regexp >>> continue=takenext >>> pattern=<DicXmlAsyncMsgService> Sending Ack to (\S+) failed >>> context=REPEATING_DICXML_EVENTS_BUCKET >>> desc=$0 >>> action=eval %o ($count = $count + 1); \ >>> >>> >>> type=single >>> ptype=regexp >>> pattern=<DicXmlAsyncMsgService> Sending Ack to (\S+) failed >>> context=REPEATING_DICXML_EVENTS_BUCKET && =($count == 100) >>> desc=$0 >>> action=delete REPEATING_DICXML_EVENTS_BUCKET; write $0 repeats 100 times, >>> don't want to spam so capturing in one single line >>> >>> Thanks, >>> Jyothi >>> >>> >>>> Date: Tue, 8 May 2012 22:37:45 +0300 >>>> Subject: Re: [Simple-evcorr-users] Using SEC in offline mode and writing >>>> rules for complex event co-relation >>>> From: risto.vaara...@gmail.com >>>> To: mnjyo...@hotmail.com >>>> CC: simple-evcorr-users@lists.sourceforge.net >>>> >>>> hi Jyothi, >>>> the main working mode of SEC is real-time event log processing -- >>>> events are correlated as they come in, and intervals between events >>>> are measured according to the real system clock. Although you can do >>>> some event matching for past logs, much of the event correlation >>>> features would not work, since SEC does not support artificial clock >>>> for replaying past scenarios. In principle, you can implement "if A is >>>> not followed by B, process file F" scenario for real-time events by >>>> using PairWithWindow rule for matching A and B, and invoking 'spawn' >>>> action for reading F if B has not been observed. However, the question >>>> is how do you define "is not followed" relationship for offline events >>>> -- would it mean B is not among the following N events, or is not >>>> present at all in the log? (If B would have to come immediately after >>>> A, then you could use RegExp2 pattern type for the task.) Also, would >>>> the events from F have to be inserted into a specific place in the >>>> event stream? (Doing that is not easy for offline logs.) >>>> In short, the answer depends on the scenario you have. >>>> kind regards, >>>> risto >>>> >>>> 2012/5/6 Jyothi Madallapalli <mnjyo...@hotmail.com>: >>>>> We are planning to use SEC in offline mode using a huge set of logs. >>>>> One of the issues we are trying to write rules for has to do with logic >>>>> described like below. >>>>> 1. Check if certain pattern exists “server <n> started”…. and see if it >>>>> is >>>>> followed by another line that says “server <n> started successfully..” >>>>> 2. If that line is not found, sec needs to now start looking thru the >>>>> corresponding server <n> log file (where should I define the mapping >>>>> between >>>>> the server <n> and it’s corresponding log file?) >>>>> 3. And further such logic… >>>>> >>>>> Would I be able to do such complex event co-relation using SEC in the >>>>> offline mode? Do I have to pass in all the log files as input to SEC or >>>>> would merging all the logs file into one single log file sorted by >>>>> timestamp >>>>> work? Is it the perlfucc rule type I should be using? Does anyone have >>>>> any >>>>> example rule sets I could take a look at? >>>>> >>>>> Thanks, >>>>> Jyothi >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------------ >>>>> Live Security Virtual Conference >>>>> Exclusive live event will cover all the ways today's security and >>>>> threat landscape has changed and how IT managers can respond. >>>>> Discussions >>>>> will include endpoint security, mobile security and the latest in >>>>> malware >>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >>>>> _______________________________________________ >>>>> Simple-evcorr-users mailing list >>>>> Simple-evcorr-users@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >>>>> ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
