Dear Risto,
Thanks for your reply,
i have a few question. im doing some testing by receive firewall deny logs,
web logs, and modsecurity logs.
Here my config,
root@ubuntu:/home/rsyslog/sec-2.7.2# perl sec -conf=sec.conf
-input=/home/rsyslog/sec-2.7.2/192.168.0.120/firewall.log-input=/home/rsyslog/sec-2.7.2/
192.168.0.112/web.log
SEC (Simple Event Correlator) 2.7.2
Reading configuration from sec.conf
4 rules loaded from sec.conf
Opening input file /home/rsyslog/sec-2.7.2/192.168.0.112/web.log
Opening input file /home/rsyslog/sec-2.7.2/192.168.0.120/firewall.log
Here my rules,
type=SingleWithSuppress
ptype=RegExp
pattern=(block).+(UDP|ICMP)
desc=$1 proto $2
action= write secoutput.txt Firewall Alert ( $2 ) Blocked suppressed for 20
seconds at %t
window=20
type=SingleWithSuppress
ptype=RegExp
pattern=\[client ([\d.]+)\].+\[msg "(.*?)"\].+\[unique_id "(.+?)"\]
continue=TakeNext
desc=client $1 unique_id $3
action= write secoutput.txt Alert - From $1 Type of Attack - $2 with unique
id $3 at %t
window=60
type=EventGroup2
ptype=RegExp
pattern=\[client ([\d.]+)\].+\[msg "(.*?SQL.*?)"\].+\[unique_id "(.+?)"\]
ptype2=RegExp
pattern2=\[client ([\d.]+)\].+\[msg "(.*?XSS.*?)"\].+\[unique_id "(.+?)"\]
desc=client $1 unique_id $3
action= write secoutput.txt Alert - From $1 Type of Attack - High
Possibility of XSS attacks with unique id $3 at %t
window=60
type=SingleWithSuppress
ptype=RegExp
pattern=apache-access: ([\d.]+).+(".+HTTP/1.1" (\d..))
desc=$1 $2
action=write secoutput.txt HTTP request $2 from $1 - code $3 at %t
window=30
And my problem is when i have a UDP flood attack, that will generate a huge
number of firewall deny traffic logs, the XSS rules did not match. Only
detect the first rules. and never detect for SQL or XSS during the flood
attacks. When i double check with the local web access logs, and
modsecurity logs, it actually generate logs and matched the pattern. Only
on SEC it not detect. Any suggestion? Thanks
On Thu, Dec 12, 2013 at 12:08 AM, Risto Vaarandi <risto.vaara...@seb.ee>wrote:
> On 12/10/2013 05:40 PM, termvrl term wrote:
> > Hi Risto,
> > Thanks for your helps,
> >
> > I have tried with your suggestion with a bit changes to the sec rule.
> >
> > type=SingleWithSuppress
> > ptype=RegExp
> > pattern=\[client ([\d.]+)\].+\[msg "(.*?)"\].+\[unique_id "(.+?)"\]
> > desc=client $1 unique_id $2
> > action= write secoutput.txt Alert - From $1 Type of Attack - $2 with
> > unique id $3
> > window=60
> >
> >
> > As i managed to get the output (as attached file, secoutput.txt), is it
> > possible if we aggregate the similar type of attacks alert with the same
> > host and unique id into a single alert? For example, This is for SQL
> > injection logs:-
> >
> > Alert - From 192.168.0.119 Type of Attack - SQL Injection Attack with
> > unique id Uqcub38AAQEAAAUsQ5kAAAAG
> > Alert - From 192.168.0.119 Type of Attack - SQL Character Anomaly
> > Detection Alert - Repetative Non-Word Characters with unique id
> > Uqcub38AAQEAAAUsQ5kAAAAG
> > Alert - From 192.168.0.119 Type of Attack - Restricted SQL Character
> > Anomaly Detection Alert - Total # of special characters exceeded with
> > unique id Uqcub38AAQEAAAUsQ5kAAAAG
> > Alert - From 192.168.0.119 Type of Attack - Detects basic SQL
> > authentication bypass attempts 1/3 with unique id
> Uqcub38AAQEAAAUsQ5kAAAAG
> > Alert - From 192.168.0.119 Type of Attack - Detects MSSQL code execution
> > and information gathering attempts with unique id
> Uqcub38AAQEAAAUsQ5kAAAAG
> > Alert - From 192.168.0.119 Type of Attack - Detects basic SQL
> > authentication bypass attempts 2/3 with unique id
> Uqcub38AAQEAAAUsQ5kAAAAG
> > Alert - From 192.168.0.119 Type of Attack - Detects classic SQL
> > injection probings 1/2 with unique id Uqcub38AAQEAAAUsQ5kAAAAG
> >
> > Aggregate into a single alert,
> >
> > Alert - From 192.168.0.119 Type of Attack - Possible SQL Injection
> > Attacks with unique id Uqcub38AAQEAAAUsQ5kAAAAG
>
> It is easy to achieve with the already existing SEC rule if you change
> the 'desc' field accordingly. Currently, the 'desc' field reads as:
>
> desc=client $1 unique_id $2
>
> but the $2 match variable does not correspond to unique_id, but rather
> to the message text. As a consequence, each alarm from the same client
> with a different text gets written to output file. In order to avoid
> logging alarms for the same client and the same unique_id, write the
> desc field as follows:
>
> desc=client $1 unique_id $3
>
> >
> > And Secondly,, For the XSS attacks,,it also match some rule from the SQL
> > , so i want to aggregate it to only a single alert.
> >
> > Here the sample logs for XSS attacks.
> >
> > Alert - From 192.168.0.111 Type of Attack - SQL Injection Attack with
> > unique id Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - SQL Character Anomaly
> > Detection Alert - Repetative Non-Word Characters with unique id
> > Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - Restricted SQL Character
> > Anomaly Detection Alert - Total # of special characters exceeded with
> > unique id Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - Detects classic SQL
> > injection probings 2/2 with unique id Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - Cross-site Scripting (XSS)
> > Attack with unique id Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - Possible XSS Attack Detected
> > - HTML Tag Handler with unique id Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - XSS Attack Detected with
> > unique id Uqcvg38AAQEAAANHCikAAAAE
> > Alert - From 192.168.0.111 Type of Attack - IE XSS Filters - Attack
> > Detected with unique id Uqcvg38AAQEAAANHCikAAAAE
> >
> > Can we make a rule, if SQL+XSS logs with the same unique id, then make a
> > new alert, like this,
> >
> > Alert - From 192.168.0.111 Type of Attack - High Possibility of XSS
> > attacks with unique id Uqcvg38AAQEAAANHCikAAAAE
>
> since the XSS alarms and SQL alarms can happen in any order (you could
> have SQL alarm first and then XSS, but also vice versa), you need to
> correlate these two alarms with the EventGroup2 rule. This rule type
> matches two different event types which can happen in arbitrary order.
> If you would like to use this rule together with the previous rule, your
> final ruleset might look like this:
>
> type=SingleWithSuppress
> ptype=RegExp
> pattern=\[client ([\d.]+)\].+\[msg "(.*?)"\].+\[unique_id "(.+?)"\]
> continue=TakeNext
> desc=client $1 unique_id $3
> action= write secoutput.txt Alert - From $1 Type of Attack - $2 with
> unique id $3
> window=60
>
> type=EventGroup2
> ptype=RegExp
> pattern=\[client ([\d.]+)\].+\[msg "(.*?SQL.*?)"\].+\[unique_id "(.+?)"\]
> ptype2=RegExp
> pattern2=\[client ([\d.]+)\].+\[msg "(.*?XSS.*?)"\].+\[unique_id "(.+?)"\]
> desc=client $1 unique_id $3
> action= write secoutput.txt Alert - From $1 Type of Attack - High
> Possibility of XSS attacks with unique id $3
> window=60
>
> Note that in the first rule the 'continue' parameter has been set to
> TakeNext, so that matching events are passed to the following
> Eventgroup2 rule.
>
> Also, you might want to change the 'window' parameter in those two rules
> to a smaller value than 60 seconds -- if the events of interest
> generally happen during 2-3 seconds, you could set 'window' to 5 or 10,
> for instance.
>
> hope this helps,
> risto
>
>
> >
> > Thanks again,
> >
> > Term
> >
> >
> >
> >
> >
> >
> > On Wed, Dec 4, 2013 at 1:20 AM, Risto Vaarandi <risto.vaara...@gmail.com
> > <mailto:risto.vaara...@gmail.com>> wrote:
> >
> >
> > hi,
> > yes, SEC is using the Perl dialect of the regular expression
> language.
> >
> > Also, from your letter it seems that you would like to lessen the
> > number of alerts, and somehow aggregate several alerts into one
> > alert which could be reported to the SIEM.
> > SEC allows to implement many different aggregation schemes, but in
> > order to implement them, there must be a clear understanding what
> > exactly to aggregate and what is the common identifier (or common
> > identifiers) for the aggregated events.
> >
> > From your log example it seems that there are several fields in
> > messages which could take this role:
> > 1) client (or hostname) -- seems to contain the IP address of the
> > remote node
> > 2) unique_id -- in your example, all the events have this field set
> > to the same value. Is this field somehow identifying the client
> > request? if so, you could take advantage of this.
> >
> > Once you have identified field(s) of interest which can be used for
> > aggregation, there must be clear understanding what exactly you want
> > to for achieving aggregation. For example, it is fairly easy to
> > write any of the following:
> > 1) a rule that reacts to the first event where field(s) have some
> > values, and suppresses the following events with same values during,
> > say, 1 minute
> > 2) a counting rule that reacts if N or more events have been seen
> > during some time window
> > 3) a rule that aggregates repeated events during some time window,
> > and sends them into a SIEM in a single message
> > etc. etc.
> >
> > In order to provide one example, you could have the following
> > simplistic rule:
> >
> > type=SingleWithSuppress
> > ptype=RegExp
> > pattern=\[client ([\d.]+)\].+\[unique_id "(.+?)"\]
> > desc=client $1 unique_id $2
> > action=pipe 'detected alert(s) for client $1 id $2' /bin/mail root
> > window=60
> >
> > This rule will match the client IP and unique_id from a message,
> > send an e-mail warning to the root, and then suppress during 60
> > seconds the following events with the same IP address *and*
> > unique_id. The event suppression by distinct (clientIP, unique_id)
> > value pairs is achieved by extracting IP address and unique_id from
> > each event, and setting the suppression scope properly with the
> > 'desc' field.
> >
> > The above rule is just one example out of many possible rules -- but
> > in order to write them down, it must be known what fields are
> > relevant in the events, and how to use them for aggregation. After
> > you have clarified this question, it is much more easy to provide
> > suggestions and advise.
> >
> > hope this helps,
> > risto
> >
> >
> > 2013/12/2 termvrl term <term...@gmail.com <mailto:term...@gmail.com
> >>
> >
> > Hi All, Thanks for your reply.
> >
> > ModSecurity is an open source web application firewall based on
> > rules, so it generate logs each time match with its rules.
> >
> > For one type of attacks it will matched with multiple rules up
> > to 4-5 pattern matched.
> >
> > Example, SQL attacks, will match at least 4 rules from
> > "sql_injestion.rules". So, i will receive 4 line of logs in my
> > SIEM for the same event.
> >
> > Unfortunately, for a some type of attacks it will match multiple
> > type of attacks rules, For example, when i try to do XSS attack,
> > the rules matched is a lot, it combine rules from SQL and XSS.
> > Futher checking, i found out that, the special character use in
> > XSS attacks matched with some SQL rules.
> >
> > So, i want to use SEC to correlate it just to generate a single
> > Alert send to SIEM.
> >
> > Hi Tim, i have try your conf and its work like i want.
> > But i have one request, can we log it only one time?
> >
> > here is my output,
> > root@ubuntu:/home/term/sec-2.7.2# perl sec -conf=tim.conf
> > -input=/var/log/apache2/error.log
> > SEC (Simple Event Correlator) 2.7.2
> > Reading configuration from tim.conf
> > 2 rules loaded from tim.conf
> > Opening input file /var/log/apache2/error.log
> > Stdin connected to terminal, SIGINT can't be used for changing
> > the logging level
> > Writing event 'SQL rule matched' to file -
> > SQL rule matched
> > Creating context 'SQL_INJECTION_ATTACK'
> > Writing event 'SQL rule matched' to file -
> > SQL rule matched
> > Creating context 'SQL_INJECTION_ATTACK'
> > Writing event 'SQL rule matched' to file -
> > SQL rule matched
> > Creating context 'SQL_INJECTION_ATTACK'
> > Writing event 'SQL rule matched' to file -
> > SQL rule matched
> > Creating context 'SQL_INJECTION_ATTACK'
> > Writing event 'XSS matched' to file secoutput.txt
> > Writing event 'XSS matched' to file secoutput.txt
> > Writing event 'XSS matched' to file secoutput.txt
> > Writing event 'XSS matched' to file secoutput.txt
> > Deleting stale context 'SQL_INJECTION_ATTACK'
> > Stale context 'SQL_INJECTION_ATTACK' deleted
> >
> >
> > Hi Risto,
> > you correct with the regex, the regex its not detect it.
> > And as for the regex use in SEC, may i know what type of regex
> > style it use? is it Perl-style regex?
> >
> > Thanks. Sorry if this is too long.
> >
> > Term
> >
> >
> > On Mon, Dec 2, 2013 at 3:28 AM, Risto Vaarandi
> > <risto.vaara...@gmail.com <mailto:risto.vaara...@gmail.com>>
> wrote:
> >
> > 2013/12/1 termvrl term <term...@gmail.com
> > <mailto:term...@gmail.com>>
> >
> > Hi all,
> >
> > i have working on correlate the alert from modsecurity.
> > when i simulate XSS attacks, modsec will generate alert
> > and it will match with SQL rule and XSS rule. So, i want
> > to use SEC to correlate if detect both signature then,
> > use write action to log a new message. Here is conf file.
> >
> > # Rule to match XSS attack.
> > # SQL + XSS
> >
> > type=Pair
> > ptype=RegExp
> > pattern=sql_injection_attacks
> > desc=$0
> > action=write - SQL rule matched
> > ptype2=RegExp
> > pattern2=xss_attacks\s*CRITICAL
> > desc2=$0
> > action2=write - XSS matched
> > window=5
> >
> > The problem is it detect only the first pattern, and the
> > second pattern never matched. Is it because the
> > modsecurity generate it with same timestamp?
> >
> > Attach is the sample log that i want to correlate.
> >
> > Thanks
> > Term
> >
> >
> > hi,
> > the Pair rule works as follows -- if an event is observed
> > which matches the pattern given with the 'pattern' field, a
> > waiting operation is started which gets its ID from the
> > 'desc' field, rule file name, and rule offset in the file
> > (the use of the rule file name and offset ensures that
> > operation IDs do not clash for different rules).
> > The waiting operation started by the Pair rule runs for the
> > 'window' seconds, and expects to see an event which is
> > matched by 'pattern2'. Your sample rule assumes the
> following:
> > 1) first an event which matches a regular expression
> > sql_injection_attacks must come in,
> > 2) the incoming event starts an operation which has the ID
> > containing the entire matching line (held by $0 variable),
> > 3) the operation runs for 5 seconds and waits an event
> > matching the regular expression xss_attacks\s*CRITICAL (this
> > regular expression means "the string xss_attacks, followed
> > by 0 or more whitespace characters, followed by the string
> > CRITICAL").
> >
> > Generally, it's a bad idea to set desc to $0, since entire
> > matching line contains a timestamp, which starts a new
> > waiting operation each time a matching line with a newer
> > timestamp is observed. Are you actually trying to do
> > processing on the client IP address basis? If so, your
> > regular expressions should match the IP address from
> > incoming events and assign it to a match variable, and this
> > variable should be used in the 'desc' field. In that way,
> > you run separate event correlation operations for distinct
> > IP addresses (also, have a look into the relevant section in
> > the official documentation:
> > http://simple-evcorr.sourceforge.net/man.html#lbAW)
> >
> > Also, the Pair rule assumes that events are ordered. If this
> > is not the case, I would recommend to consider the
> > EventGroup2 rule. Finally, your second regular expression
> > xss_attacks\s*CRITICAL assumes that only whitespace can
> > appear in between "xss_attacks" and "CRITICAL", but your log
> > messages seem to be different. This is probably the reason
> > why the second pattern of your Pair rule fails to match
> > incoming events.
> >
> > hope this helps,
> > risto
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Rapidly troubleshoot problems before they affect your
> > business. Most IT
> > organizations don't have a clear picture of how
> > application performance
> > affects their revenue. With AppDynamics, you get 100%
> > visibility into your
> > Java,.NET, & PHP application. Start your 15-day FREE
> > TRIAL of AppDynamics Pro!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=84349351&iu=/4140/ostg.clktrk
> > _______________________________________________
> > Simple-evcorr-users mailing list
> > Simple-evcorr-users@lists.sourceforge.net
> > <mailto:Simple-evcorr-users@lists.sourceforge.net>
> >
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
> >
> >
> >
> >
> >
> >
> >
> >
> ------------------------------------------------------------------------------
> > Rapidly troubleshoot problems before they affect your business. Most IT
> > organizations don't have a clear picture of how application performance
> > affects their revenue. With AppDynamics, you get 100% visibility into
> your
> > Java,.NET, & PHP application. Start your 15-day FREE TRIAL of
> AppDynamics Pro!
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> >
> >
> >
> > _______________________________________________
> > Simple-evcorr-users mailing list
> > Simple-evcorr-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
> >
>
>
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics
> Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> _______________________________________________
> Simple-evcorr-users mailing list
> Simple-evcorr-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>
------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
