hi Jaren,
note that the 'add' function does not numerically add a value to the
context event store, but it rather appends textual line(s) to context event
store. Since the context event store holds textual data only, you can't use
it in the way you would use the stack in Forth language.
If you wish to implement byte summing, it is sufficient to implement couple
of perl code snippets into your ruleset. Here is one example which you
might use for addressing the problem:
type=single
ptype=regexp
pattern=user (?<user>[\w.-]+) bytes (?<bytes_uploaded>\d+)
varmap=data_upload
continue=takenext
desc=parse data upload events
action=none
type=Single
ptype=Cached
pattern=data_upload
continue=TakeNext
context=!BYTES_$+{user}
desc=create a context for new user $+{user}
action=create BYTES_$+{user} 300; add BYTES_$+{user} 0
type= Single
ptype=Cached
pattern=data_upload
continue=TakeNext
context=BYTES_$+{user}
desc=add $+{bytes_uploaded} to the upload counter of user $+{user}
action=pop BYTES_$+{user} %prev; \
lcall %sum %prev $+{bytes_uploaded} -> ( sub { $_[0] + $_[1] } ); \
add BYTES_$+{user} %sum;
type=Single
ptype=Cached
pattern=data_upload
desc=check if user $+{user} has uploaded more than 100000 bytes
continue=TakeNext
context=BYTES_$+{user}
action=pop BYTES_$+{user} %sum; lcall %o %sum -> ( sub { $_[0] > 100000 }
); \
if %o (write - User $+{user} has uploaded too many bytes (%sum)); \
add BYTES_$+{user} %sum
Hope this helps,
risto
2015-10-07 12:27 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
> Hi,
>
> Sorry, i couldn´t answer before but i thought something like that:
>
> Here i can detect and create a context for each user.
>
> type = Single
>
> ptype = Cached
>
> desc = $0
>
> continue = TakeNext
>
> context = !BYTES_$+{user}
>
> pattern = pattern
>
> action = create BYTES_$+{user} 300;
>
> We check that the context exists and we add bytes in each context:
>
>
> type = Single
>
> ptype = Cached
>
> desc = $0
>
> continue = TakeNext
>
> context = BYTES_$+{user}
>
> pattern = pattern_que_necesites
>
> action = *add* BYTES_$+{user} $bytes_uploaded
>
>
> Finally i pop the result:
>
>
> type = Single
>
> ptype = Cached
>
> desc = $+{user}
>
> continue = TakeNext
>
> context = BYTES_$+{user}
>
> pattern = pattern
>
> action = *pop* BYTES_$+{user} %bytes_uploaded_$+{user} ; \
>
>
> I don´t know how read the bytes added to the context. is it possible?
>
>
> Regards.
>
>
>
> 2015-09-30 14:34 GMT+02:00 Risto Vaarandi <risto.vaara...@gmail.com>:
>
>> hi Jaren,
>> for using a separate window for each individual user, you can't use
>> SingleWithThreshold operations, since their internal counters reflect event
>> count, and the occurrence of a matching events always adds 1 to the
>> counter. Since you want to implement event counters where values greater
>> than 1 can be added, the rules need some additional Perl code. For
>> instance, the following ruleset addresses the counting scheme for users
>> Korsakof, Bob and Alice, and triggers an alarm if one of those users
>> uploads more than 100MB within 900 seconds (15 minutes):
>>
>> # Clean entries from the %bytes hash table which are older than 15 minutes
>>
>> type=Calendar
>> time=* * * * *
>> desc=clean old entries from the bytes hash table
>> action=lcall %o -> ( sub { my($user); my($time) = time(); \
>> foreach $user (keys %bytes) { \
>> if ($bytes{$user}->[-1]->[0] < $time - 900) \
>> { delete $bytes{$user}; } } } )
>>
>> # Suppress the user if he/she is not Korsakof, Bob or Alice
>>
>> type=Suppress
>> ptype=RegExp
>> pattern=^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ (?!(?:Korsakof|Bob|Alice)
>> )\S+ \d+
>>
>> # Use the %bytes hash table for memorizing each upload event for the user,
>> # and record the time and bytecount for each upload. When the upload event
>> # occurs, drop all previous upload events which are older than 15 minutes,
>> # and sum the bytecounts of remaining uploads. If the sum is greater than
>> # 100MB, produce an alarm, and suppress further alarms for 2 hours.
>>
>> type=Single
>> ptype=RegExp
>> pattern=^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ ([\w.@-]+) (\d+)
>> context=$1 $2 -> ( sub { my($time) = time(); my($sum) = 0; \
>> push @{$bytes{$_[0]}}, [ $time, $_[1] ]; \
>> while ($bytes{$_[0]}->[0]->[0] < $time - 900) \
>> { shift @{$bytes{$_[0]}}; } \
>> map { $sum += $_->[1] } @{$bytes{$_[0]}}; \
>> return ($sum > 104857600); } ) \
>> && !SUPPRESS_ALARMS_FOR_$1
>> desc=User $1 has transmitted too many bytes
>> action=write - %s; create SUPPRESS_ALARMS_FOR_$1 7200
>>
>>
>> The most complex part of the above ruleset is the context expression of
>> the last rule which employs the %bytes hash table for memorizing all
>> uploads for the last 15 minutes. As in my previous ruleset example, the
>> hash table keys are user names, but this time each user name points to a
>> list of upload events. Each upload event is in turn memorized as a tuple
>> (time_of_upload, bytecount). In other words, $bytes{"Bob"}->[0]->[0] refers
>> to the time of the earliest upload event for user Bob, while
>> $bytes{"Alice"}->[0]->[1] refers to the bytecount of the earliest upload
>> event for user Alice. The Perl function in the context expression uses the
>> time and bytecount fields for dropping events older than 15 minutes first,
>> and then summing up the bytes of remaining events and comparing the sum to
>> 104857600 (100MB). This ensures that you are always doing thresholding for
>> the last 15 minutes.
>>
>> If you wish to filter relevant user names in a different way (e.g., you
>> want to exclude few user names and apply thresholding for the rest), you
>> need to modify the regular expression of the second rule accordingly.
>>
>> Hope this helps,
>> risto
>>
>> 2015-09-30 14:20 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
>>
>>> Hi,
>>>
>>> Thanks for your help. I was thinking and i can not block user during 2H
>>> without alerts or in any time.
>>> I was thinking and how can i add a temporaly window?.Can i change the
>>> alert to SingleWithThreshold and just add a window variable without
>>> thresh?Could i limit the number of users?
>>>
>>> The idea its like that: User Jaren has uploaded 100MB in 15 min.
>>>
>>>
>>>
>>> Regards.
>>>
>>> 2015-09-30 13:19 GMT+02:00 Jaren Peich <burkol...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> Thanks for your help. I was thinking and i can not block user during 2H
>>>> without alerts or in any time.
>>>> I was thinking and how can i add a temporaly window?.Can i change the
>>>> alert to SingleWithThreshold and just add a window variable without
>>>> thresh?Could i limit the number of users?
>>>>
>>>> The idea its like that: User Jaren has uploaded 100MB in 15 min.
>>>>
>>>>
>>>>
>>>> Regards.
>>>>
>>>> 2015-09-29 16:19 GMT+02:00 Risto Vaarandi <risto.vaara...@gmail.com>:
>>>>
>>>>> hi Jaren,
>>>>> you could try the following simple ruleset and see if this fits your
>>>>> needs:
>>>>>
>>>>> type=Calendar
>>>>> time=0 * * * *
>>>>> desc=drop byte counters
>>>>> action=lcall %o -> ( sub { %bytes = () } )
>>>>>
>>>>> type=Single
>>>>> ptype=RegExp
>>>>> pattern=^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ ([\w.@-]+) (\d+)
>>>>> context=$1 $2 -> ( sub { ($bytes{$_[0]} += $_[1]) > 10000 } ) && \
>>>>> !SUPPRESS_ALARMS_FOR_$1
>>>>> desc=User $1 has transmitted too many bytes
>>>>> action=write - %s; create SUPPRESS_ALARMS_FOR_$1 7200
>>>>>
>>>>> This ruleset employs the Perl hash table %bytes for keeping track of
>>>>> user data transfers. The keys to %bytes hash table are user names, and the
>>>>> value which corresponds to each user name is the number of bytes
>>>>> transmitted.
>>>>>
>>>>> Since you didn't mention in which particular time window you would
>>>>> like to accomplish upload tracking, I have chosen a fixed window of 1 hour
>>>>> which is applied to all users. The purpose of the first rule is to drop
>>>>> byte counters for all users once every hour. If you wish to accomplish the
>>>>> monitoring in a larger window, it is easy to adjust the 'time' field of
>>>>> the
>>>>> Calendar rule accordingly (the field takes a value in crontab syntax).
>>>>>
>>>>> The second rule matches individual upload events with the regular
>>>>> expression
>>>>>
>>>>> ^\d+\/\d+\/\d+ \d{2}:\d{2}:\s*POST \S+ ([\w.@-]+) (\d+)
>>>>>
>>>>> which sets $1 to user name and $2 to transmitted bytes. I have assumed
>>>>> that legal characters for user names are letters, digits, underscores,
>>>>> dots, dashes and @-signs, and therefore I've used ([\w.@-]+) for
>>>>> matching the username. If you want your username to be just any sequence
>>>>> of
>>>>> non-whitespace characters, you can use (\S+) instead. In order to track
>>>>> the
>>>>> number of bytes user has uploaded, the rule uses the following context
>>>>> expression
>>>>>
>>>>> $1 $2 -> ( sub { ($bytes{$_[0]} += $_[1]) > 10000 } ) && \
>>>>> !SUPPRESS_ALARMS_FOR_$1
>>>>>
>>>>> which consists of two operands joined by logical AND (&&).
>>>>>
>>>>> The first operand '$1 $2 -> ( sub { ($bytes{$_[0]} += $_[1]) > 10000 }
>>>>> )' is a call to a precompiled perl function which simply adds bytes for
>>>>> the
>>>>> current upload to the total of the given user. Note that the variables $1
>>>>> (user name) and $2 (bytes) serve as input parameters for the function.
>>>>> Also, the function checks if the resulting user total exceeds the
>>>>> threshold
>>>>> (I have used 10000 bytes for threshold which probably needs replacement by
>>>>> a more reasonable value). The second operand of the context expression
>>>>> !SUPPRESS_ALARMS_FOR_$1 is true if the context SUPPRESS_ALARMS_FOR_$1 does
>>>>> not exist (which is the case if we haven't generated alarm for the given
>>>>> user within the last 2 hours -- see the following explanation). If both
>>>>> operands are true (in other words, the upload threshold is violated and we
>>>>> haven't produced an alert about the user in last 2h), the rule will write
>>>>> a
>>>>> warning string to standard output with the 'write' action. Also, after
>>>>> generating the alarm, the rule creates the SUPPRESS_ALARMS_FOR_$1 context
>>>>> with a lifetime of 7200 seconds. Since the context will stay around for 2
>>>>> hours, it suppresses further alarms for the same user for this time frame.
>>>>>
>>>>> Hope this helps,
>>>>> risto
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2015-09-29 11:21 GMT+03:00 Jaren Peich <burkol...@gmail.com>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Sorry for my English and thanks for the perl library. Thanks for
>>>>>> accepting me in mailing list. :)
>>>>>>
>>>>>> I have to program a code with SEC to detect a quantity of bytes
>>>>>> uploaded to the web from one user with a limit. I dont know how to sum
>>>>>> the
>>>>>> quantity of bytes that comes from each proxy log line till reach the
>>>>>> limit
>>>>>> and generate an alert.
>>>>>>
>>>>>> Log Line
>>>>>>
>>>>>> Time: Method "Url" User Bytes
>>>>>> 29/09/2015 10:14:POST "www.google.com" Korsakof 1250
>>>>>>
>>>>>> Thank you. Regards.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> Simple-evcorr-users mailing list
>>>>>> Simple-evcorr-users@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
------------------------------------------------------------------------------
Full-scale, agent-less Infrastructure Monitoring from a single dashboard
Integrate with 40+ ManageEngine ITSM Solutions for complete visibility
Physical-Virtual-Cloud Infrastructure monitoring from one console
Real user monitoring with APM Insights and performance trend reports
Learn More http://pubads.g.doubleclick.net/gampad/clk?id=247754911&iu=/4140
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
