On 08/13/2011 02:30 PM, Jean Baptiste Favre wrote:
> Hi Risto,
> Sorry for late answer, I was busy on other topics this week.
>
> Thanks your answer, I've been able to increase analysis rate from 120k
> lines per minutes to...700k on the same hardware !
>
> I had to adapt you solution, since I really need to output results at
> each round minute. Therefore, Calendar type is really helpfull for that :)
>
> I'm now testing this SEC configuration in real conditions. I'll post
> here my configuration when validated.
>
> Now I have to find a way to get more metrics, like average and median
> time without too much performance impact. That will be an interesting
> challenge, especially for median, with 700k long data set :)
>
> By the way, I got 2 other questions:
> - is there any way to know if SEC is late or not ? I mean, knowing if
> log's arrival rate is higher than analysis one ? I guess answer is no
> but that would be awesome if I can get this information too.
If you send SIGUSR1 signal to the SEC process, it will write its status
information to dump file. The status information includes the current
content of the input buffer -- by default, 10 last lines that have been
read from input. If those lines include timestamps, you can easily tell
if SEC is lagging or not. Since the dump file also includes the time of
the dump, you can also calculate how many seconds SEC has to catch up.
> - second is: in order to increase performances, and since logs are
> collected with rsyslog and stored in a file, is there any solution to
> use a socket or a pipe as entry for SEC ? That could avoid disks I/O
> operations. Disks I/O are well known bottleneck on Amazon, which is were
> I host my server.
Reading from sockets is not supported. However, SEC can receive input
from names pipes. Nevertheless, it is important to understand that
unlike files, pipes (be it a named pipe or pipe connected to SEC stdin)
are not always reliable for transferring data, especially if you have
large amounts of bytes to transfer. When writing a large volume of data
to pipe, pipe can easily become full, and as a result subsequent writes
into pipe can (partially) fail. This issue has also been discussed in
the mailing list before, and the search might reveal relevant threads.
Fortunately, with regular files there are no such issues.
kind regards,
risto
>
> Best regards,
> JB
>
>
> On 07/08/2011 23:08, Risto Vaarandi wrote:
>> hi Jean Baptiste,
>>
>> you might try the following rule:
>>
>> type=EventGroup
>> ptype=regexp
>> pattern=" (\d)[0-9]{2} \d+
>> count=lcall %ret $1 -> ( sub { ++$req; if ($_[0] == 2) { ++$req2; } } );
>> desc=Count HTTP request response codes
>> action=none
>> init=lcall %ret -> ( sub { $req = 0; $req2 = 0; } )
>> end=lcall %ret -> ( sub { return "$req requests, $req2 2xx requests"; } ); \
>> write - %ret
>> window=60
>>
>> This rule keeps two Perl variables ($req and $req2) in memory which
>> are accessed with the 'lcall' action. I am proposing 'lcall' instead
>> of 'eval', since the 'eval' action compiles code before each execution
>> and is thus much slower. Since your event rate is 3000 per second,
>> 'lcall' is much better suited for maintaining the counters.
>> The above rule starts an event correlation operation at the first
>> occurrence of the web server event. Unlike the example rule you found
>> from the archives, this rule does not have 'multact' field set to yes,
>> which means that the first event freezes the event correlation window
>> (since 'thresh' field defaults to 1), and the operation terminates
>> after 60 seconds. Before terminating, the 'end' action creates the
>> report string and writes it to standard output. The next web server
>> event will start the operation again, and the $req and $req2 counters
>> will be reset to 0 with the 'init' action. Note that we are not using
>> the 'action' field at all, but tackle the problem by putting auxiliary
>> 'init', 'count' and 'end' actions together. In other words, although
>> the main purpose of EventGroup rule is to detect groups of different
>> events in a time window, it can also employed for handling various
>> counting tasks for events of one type only.
>> Hope this example is helpful.
>> regards,
>> risto
>>
>> 2011/8/5 Jean Baptiste Favre<simple-evc...@jbfavre.org>:
>>> Hello,
>>> I need to count event number and report it every minutes.
>>> I'm looking for a more elegant and powerfull way (I have to analyze 3000
>>> req/s) to do it than what I found already (see bellow).
>>>
>>> I saw http://sourceforge.net/mailarchive/message.php?msg_id=27274908 but
>>> can't figure out if this can help me.
>>>
>>> My current config
>>>
>>> ## INITialize SEC
>>> type=Single
>>> ptype=RegExp
>>> pattern=(SEC_STARTUP|SEC_RESTART)
>>> context=SEC_INTERNAL_EVENT
>>> desc=Init counters with 0
>>> action=assign %REQ_NBR_COUNT 0; \
>>> assign %REQ_2xx_COUNT 0;
>>>
>>> ## Extract HTTP code from request and through event
>>> type=Single
>>> ptype=RegExp
>>> pattern=.*" (\d)[0-9]{2} \d+ .*
>>> desc=Request received
>>> action=event 0 REQ_$1
>>>
>>> ## Counts 2xx HTTP code number
>>> type=Single
>>> ptype=SubStr
>>> pattern=REQ_2
>>> continue=TakeNext
>>> desc=HTTP 2xx code found
>>> action=eval %REQ_2xx_COUNT ( ++$REQ_2xx_COUNT )
>>>
>>> [...]
>>> skipped other counters as they use the same definition as above
>>> [...]
>>>
>>> ## Counts requests number
>>> type=Single
>>> ptype=SubStr
>>> pattern=REQ_
>>> desc=Request counter
>>> action=eval %REQ_NBR_COUNT ( ++$REQ_NBR_COUNT )
>>>
>>> ## Flush counters every minute
>>> type=Calendar
>>> time=* * * * *
>>> desc=monitoring.logs.http
>>> context=!SEC_INTERNAL_EVENT
>>> action=write - - %s.req[tot] %REQ_NBR_COUNT; \
>>> eval %REQ_NBR_COUNT ( $REQ_NBR_COUNT = 0 ); \
>>> write - - %s.req[2xx] %REQ_2xx_COUNT; \
>>> eval %REQ_2xx_COUNT ( $REQ_2xx_COUNT = 0 );
>>>
>>> Any comment welcomed,
>>>
>>> Regards,
>>> JB
>>>
>>> ------------------------------------------------------------------------------
>>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
>>> The must-attend event for mobile developers. Connect with experts.
>>> Get tools for creating Super Apps. See the latest technologies.
>>> Sessions, hands-on labs, demos& much more. Register early& save!
>>> http://p.sf.net/sfu/rim-blackberry-1
>>> _______________________________________________
>>> Simple-evcorr-users mailing list
>>> Simple-evcorr-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>
>>
>> ------------------------------------------------------------------------------
>> BlackBerry® DevCon Americas, Oct. 18-20, San Francisco, CA
>> The must-attend event for mobile developers. Connect with experts.
>> Get tools for creating Super Apps. See the latest technologies.
>> Sessions, hands-on labs, demos& much more. Register early& save!
>> http://p.sf.net/sfu/rim-blackberry-1
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>
>
> ------------------------------------------------------------------------------
> FREE DOWNLOAD - uberSVN with Social Coding for Subversion.
> Subversion made easy with a complete admin console. Easy
> to use, easy to manage, easy to install, easy to extend.
> Get a Free download of the new open ALM Subversion platform now.
> http://p.sf.net/sfu/wandisco-dev2dev
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
