Hi Risto,
Sorry for late answer, I was busy on other topics this week.
Thanks your answer, I've been able to increase analysis rate from 120k
lines per minutes to...700k on the same hardware !
I had to adapt you solution, since I really need to output results at
each round minute. Therefore, Calendar type is really helpfull for that :)
I'm now testing this SEC configuration in real conditions. I'll post
here my configuration when validated.
Now I have to find a way to get more metrics, like average and median
time without too much performance impact. That will be an interesting
challenge, especially for median, with 700k long data set :)
By the way, I got 2 other questions:
- is there any way to know if SEC is late or not ? I mean, knowing if
log's arrival rate is higher than analysis one ? I guess answer is no
but that would be awesome if I can get this information too.
- second is: in order to increase performances, and since logs are
collected with rsyslog and stored in a file, is there any solution to
use a socket or a pipe as entry for SEC ? That could avoid disks I/O
operations. Disks I/O are well known bottleneck on Amazon, which is were
I host my server.
Best regards,
JB
On 07/08/2011 23:08, Risto Vaarandi wrote:
> hi Jean Baptiste,
>
> you might try the following rule:
>
> type=EventGroup
> ptype=regexp
> pattern=" (\d)[0-9]{2} \d+
> count=lcall %ret $1 -> ( sub { ++$req; if ($_[0] == 2) { ++$req2; } } );
> desc=Count HTTP request response codes
> action=none
> init=lcall %ret -> ( sub { $req = 0; $req2 = 0; } )
> end=lcall %ret -> ( sub { return "$req requests, $req2 2xx requests"; } ); \
> write - %ret
> window=60
>
> This rule keeps two Perl variables ($req and $req2) in memory which
> are accessed with the 'lcall' action. I am proposing 'lcall' instead
> of 'eval', since the 'eval' action compiles code before each execution
> and is thus much slower. Since your event rate is 3000 per second,
> 'lcall' is much better suited for maintaining the counters.
> The above rule starts an event correlation operation at the first
> occurrence of the web server event. Unlike the example rule you found
> from the archives, this rule does not have 'multact' field set to yes,
> which means that the first event freezes the event correlation window
> (since 'thresh' field defaults to 1), and the operation terminates
> after 60 seconds. Before terminating, the 'end' action creates the
> report string and writes it to standard output. The next web server
> event will start the operation again, and the $req and $req2 counters
> will be reset to 0 with the 'init' action. Note that we are not using
> the 'action' field at all, but tackle the problem by putting auxiliary
> 'init', 'count' and 'end' actions together. In other words, although
> the main purpose of EventGroup rule is to detect groups of different
> events in a time window, it can also employed for handling various
> counting tasks for events of one type only.
> Hope this example is helpful.
> regards,
> risto
>
> 2011/8/5 Jean Baptiste Favre <simple-evc...@jbfavre.org>:
>> Hello,
>> I need to count event number and report it every minutes.
>> I'm looking for a more elegant and powerfull way (I have to analyze 3000
>> req/s) to do it than what I found already (see bellow).
>>
>> I saw http://sourceforge.net/mailarchive/message.php?msg_id=27274908 but
>> can't figure out if this can help me.
>>
>> My current config
>>
>> ## INITialize SEC
>> type=Single
>> ptype=RegExp
>> pattern=(SEC_STARTUP|SEC_RESTART)
>> context=SEC_INTERNAL_EVENT
>> desc=Init counters with 0
>> action=assign %REQ_NBR_COUNT 0; \
>> assign %REQ_2xx_COUNT 0;
>>
>> ## Extract HTTP code from request and through event
>> type=Single
>> ptype=RegExp
>> pattern=.*" (\d)[0-9]{2} \d+ .*
>> desc=Request received
>> action=event 0 REQ_$1
>>
>> ## Counts 2xx HTTP code number
>> type=Single
>> ptype=SubStr
>> pattern=REQ_2
>> continue=TakeNext
>> desc=HTTP 2xx code found
>> action=eval %REQ_2xx_COUNT ( ++$REQ_2xx_COUNT )
>>
>> [...]
>> skipped other counters as they use the same definition as above
>> [...]
>>
>> ## Counts requests number
>> type=Single
>> ptype=SubStr
>> pattern=REQ_
>> desc=Request counter
>> action=eval %REQ_NBR_COUNT ( ++$REQ_NBR_COUNT )
>>
>> ## Flush counters every minute
>> type=Calendar
>> time=* * * * *
>> desc=monitoring.logs.http
>> context=!SEC_INTERNAL_EVENT
>> action=write - - %s.req[tot] %REQ_NBR_COUNT; \
>> eval %REQ_NBR_COUNT ( $REQ_NBR_COUNT = 0 ); \
>> write - - %s.req[2xx] %REQ_2xx_COUNT; \
>> eval %REQ_2xx_COUNT ( $REQ_2xx_COUNT = 0 );
>>
>> Any comment welcomed,
>>
>> Regards,
>> JB
>>
>> ------------------------------------------------------------------------------
>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
>> The must-attend event for mobile developers. Connect with experts.
>> Get tools for creating Super Apps. See the latest technologies.
>> Sessions, hands-on labs, demos & much more. Register early & save!
>> http://p.sf.net/sfu/rim-blackberry-1
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
> ------------------------------------------------------------------------------
> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
> The must-attend event for mobile developers. Connect with experts.
> Get tools for creating Super Apps. See the latest technologies.
> Sessions, hands-on labs, demos & much more. Register early & save!
> http://p.sf.net/sfu/rim-blackberry-1
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
------------------------------------------------------------------------------
FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
Subversion made easy with a complete admin console. Easy
to use, easy to manage, easy to install, easy to extend.
Get a Free download of the new open ALM Subversion platform now.
http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
