Hello risto
I ran the tests with real logs. Suggested method works exactly as expected.
This resolves many of my other queries. Thank you for prompt response.
Regards,
Santhosh
On Fri, Aug 30, 2019, 20:59 Risto Vaarandi <risto.vaara...@gmail.com> wrote:
> hi Santhosh,
>
> since your task involves not only matching IP addresses against a
> blacklist but also includes reporting IoC information for a bad IP address,
> I would recommend loading IoC data from file into a Perl hash which allows
> for quick lookups. The example ruleset below uses a global hash %ioc which
> is a global variable and can thus be accessed from all rules:
>
> type=Single
> ptype=RegExp
> pattern=^(SEC_STARTUP|SEC_RESTART|SEC_SOFTRESTART)$
> context=SEC_INTERNAL_EVENT
> desc=load IoC information on SEC startup and HUP or ABRT signals
> action=lcall %count -> ( sub { %ioc = (); \
> if (!open(FILE, "/home/risto/IOC_data_proposal.txt")) { return -1;
> } \
> while (<FILE>) { if (/^\s*(([\d.]+):\d+\s*-.*)/) { \
> $ioc{$2} = $1; } \
> } close(FILE); return scalar keys %ioc; } ); \
> logonly Loaded %count entries from IoC data file
>
> type=Single
> ptype=PerlFunc
> pattern=sub { if ($_[0] !~ /ASA-\S+: Teardown \S+ connection \d+ for
> outside: ([\d.]+)/) { return 0; } \
> if (!exists($ioc{$1})) { return 0; } return ($1, $ioc{$1}); }
> desc=Connection to IP address $1 with IoC information $2
> action=pipe 'Log matched IoC:%{.nl}IoC: $2%{.nl}Log: $0' /bin/mail
> some...@example.com
>
> The first rule loads IoC information from file into %ioc hash table
> whenever SEC is started or HUP or ABRT signal is received by SEC. IP
> addresses serve as keys of the hash table, while each value is an entire
> line from the IoC file. For example, if the file contains the following two
> lines
>
> 187.163.222.244:465 - emotet
> 187.189.195.208:8443 - emotet
>
> the %ioc hash table will contain the following mappings (keys and values
> are separated by ->):
>
> 187.163.222.244 -> 187.163.222.244:465 - emotet
> 187.189.195.208 -> 187.189.195.208:8443 - emotet
>
> Currently, the first rule assumes that IoC file is in the same format as
> you described in your e-mail, and the rule uses regular expression
> ^\s*(([\d.]+):\d+\s*-.*) for parsing the file and extracting relevant
> information. Should the format of the file change, this regular expression
> needs to be adjusted accordingly. Also, the rule finds the number of
> entries loaded from IoC file, stores it in %count action list variable and
> logs a debug message with this value into SEC log file. If the rule was
> unable to open the file, the value -1 is logged which is useful for
> troubleshooting purposes.
>
> The second rule uses PerlFunc pattern for matching incoming ASA firewall
> events and first verifies that incoming event matches the regular expression
> ASA-\S+: Teardown \S+ connection \d+ for outside: ([\d.]+)
> If there is a match, IP address of remote host is extracted and assigned
> to $1 variable, and %ioc hash table is looked up for IoC information for
> that IP address. If lookup is successful, PerlFunc pattern returns a list
> with two elements (IP address, IoC info) which are mapped to match
> variables $1 and $2 by SEC ($0 variable will hold the entire matching event
> log line). The match variables are then used by 'pipe' action for sending
> an e-mail to relevant mailbox.
>
> Hope this helps,
> risto
>
> Kontakt Santhosh Kumar (<santhoshkmrre...@gmail.com>) kirjutas kuupäeval
> R, 30. august 2019 kell 06:37:
>
>> Hello
>>
>> Thanks for the brief..!!
>>
>> In my case, udgram creates a event loop however tcpsock functionality
>> achived the goal.
>>
>> Regarding second scenario, the goal is to match the IP address and notify.
>>
>> "ASA-6-302016: Teardown UDP connection 806353 for outside: 187.189.195.208
>> <http://187.189.195.208:8443/>/24057 to
>> identity: 172.18.124.136/161 duration 0:02:01 bytes 313"
>>
>> List of IOC's stored in a file(IOC_data_proposal.txt) as below. the
>> rule suppose to match the IP address from the file(IOC_data_proposal.txt)
>> though
>> the file contains other informations like port number and name of the
>> malware, etc. which is not part of the log.
>>
>> The reason for not removing extra data(port, ioc name) from the file(
>> IOC_data_proposal.txt) is that its required for a notification.
>>
>> ************************
>>
>> IOC_data_proposal.txt
>>
>> 187.163.222.244:465 - emotet
>>
>> 187.189.195.208:8443 - emotet
>>
>> 188.166.253.46:8080 - emotet
>>
>> 189.209.217.49:80 <http://189.209.217.49/> - heartbleed
>>
>> ************************
>>
>>
>> Expected mail Output:
>>
>>
>> Log Matched IOC,
>>
>> IOC: 187.189.195.208:8443 - emotet
>>
>> Log: "ASA-6-302016: Teardown UDP connection 806353 for outside:
>> 187.189.195.208 <http://187.189.195.208:8443/>/24057 to
>>
>> identity: 172.18.124.136/161 duration 0:02:01 bytes 313"
>>
>>
>> regards,
>> Santhosh
>>
>> On Wed, Aug 28, 2019 at 4:18 AM Risto Vaarandi <risto.vaara...@gmail.com>
>> wrote:
>>
>>> hi Santhosh,
>>>
>>> Kontakt Santhosh Kumar (<santhoshkmrre...@gmail.com>) kirjutas
>>> kuupäeval T, 27. august 2019 kell 04:55:
>>>
>>>> Hello Risto
>>>>
>>>>
>>>> I’ve been running tests on SEC for a while and stuck with below points.
>>>> I’m not familiar with Perl though I tried to find a solution from sec mail
>>>> bucket but no luck, please suggest if this can be achieved with high
>>>> performance,
>>>>
>>>>
>>>>
>>>> 1. I could see a log drops when I tested with the event rate of
>>>> 15000 logs/sec. A simple SEC rule to receive and forward all the logs
>>>> to a
>>>> destination. The output shows relatively less number of logs. This also
>>>> increases the cpu usage from 0.3% to 45%
>>>>
>>>> ************************
>>>>
>>>> Type=single
>>>>
>>>> Ptype=regexp
>>>>
>>>> Pattern=([.\d]+)
>>>>
>>>> Desc=$1
>>>>
>>>> Action=pipe $0 nc syslog101 514
>>>>
>>>> ************************
>>>>
>>>
>>> The above rule is very inefficient, since 'pipe' action pipes the value
>>> of $0 to "nc syslog101 514" command and then closes the pipe, forcing the
>>> nc command to terminate. In other words, each time the above rule matches
>>> an event, new command is forked, and if you have 15000 events per second,
>>> the rule attempts to fork 15000 processes each second. This imposes
>>> considerable load on the system and if you send in events at a high rate,
>>> the rule might easily exhaust your system resources.
>>>
>>> Instead of forking nc on each matching event, I would recommend to
>>> utilize 'tcpsock' action which transmits events over a single TCP socket
>>> (since you haven't used -u flag with nc tool, I assume that your syslog
>>> server listens on port 514/tcp). For example, consider the following rule
>>> (the rule terminates each transmitted line with the newline):
>>>
>>> type=single
>>> ptype=regexp
>>> pattern=([.\d]+)
>>> desc=test event $1
>>> action=tcpsock syslog101:514 $0%.nl
>>>
>>> If your syslog server speaks BSD syslog protocol (
>>> https://tools.ietf.org/html/rfc3164), and incoming events are not in
>>> that format, you could use sec builtin action list variables for formatting
>>> the event and providing fields that syslog server expects (such as
>>> timestamp). For example, the following rule transmits each event line over
>>> TCP in BSD syslog format with priority 14 (facility of "user" and severity
>>> of "info"), with hostname "myhost", with program name "myprog", and using
>>> newline as a separator between messages:
>>>
>>> type=single
>>> ptype=regexp
>>> pattern=([.\d]+)
>>> desc=test event $1
>>> action=tcpsock syslog101:514 <14>%.monstr %.mdaystr %.hmsstr myhost
>>> myprog: $0%.nl
>>>
>>> Finally, as David suggested, you can also pass messages to local syslog
>>> server via /dev/log socket, and let the local syslog server handle the
>>> messages (note that unlike for 'tcpsock' in previous example, there is no
>>> need for hostname and terminating newline for 'udgram' action):
>>>
>>> type=single
>>> ptype=regexp
>>> pattern=([.\d]+)
>>> desc=test event $1
>>> action=udgram /dev/log <14>%.monstr %.mdaystr %.hmsstr myprog: $0
>>>
>>> If your local syslog server is rsyslog, you could have the following
>>> rsyslog rule for forwarding messages:
>>>
>>> if $programname == "myprog" then @@syslog101:514
>>>
>>> As you can see, there are several ways for achieving your goal, and
>>> hopefully above examples are helpful for selecting the most convenient
>>> solution.
>>>
>>>
>>>>
>>>> 1. On a different scenario, I was interested to match the logs with
>>>> list of IOC’s. Here i was trying to mail the detected log along with IOC
>>>> name. I could achieve it to certain level as mentioned in example but no
>>>> luck with this cases, "Split IP's from the IOC file and use it on
>>>> the “pattern” to match IP from logs"
>>>>
>>>> ************************
>>>>
>>>> IOC_data_proposal.txt
>>>>
>>>> 187.163.222.244:465 - emotet
>>>>
>>>> 187.189.195.208:8443 - emotet
>>>>
>>>> 188.166.253.46:8080 - emotet
>>>>
>>>> 189.209.217.49:80 - heartbleed
>>>>
>>>> ************************
>>>>
>>>> Please check and share some insights.
>>>>
>>>
>>> I am not sure I fully understood what exactly you want to achieve here.
>>> Can you provide some examples of input events and what output you would
>>> like to generate on each match?
>>>
>>> kind regards,
>>> risto
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>> Eg: I currently tested below case and its working fine as this is a
>>>> straight forward IOC matches.
>>>>
>>>> ************************
>>>>
>>>> #Current Rule for matching IOC:
>>>>
>>>> type=Single
>>>>
>>>> ptype=RegExp
>>>>
>>>> pattern=(?:SEC_STARTUP|SEC_RESTART|SEC_SOFTRESTART)
>>>>
>>>> desc=load IOC data
>>>>
>>>> action=logonly; delete IP; create IP; \
>>>>
>>>> lcall %iocevents -> (sub{scalar `cat /usr/local/bin/sec-rules/
>>>> ioc_data.txt`}); \
>>>>
>>>> cevent IOC_IP 0 %iocevents;
>>>>
>>>>
>>>>
>>>> type=Single
>>>>
>>>> ptype=RegExp
>>>>
>>>> pattern=.
>>>>
>>>> context=IOC_IP
>>>>
>>>> desc=create an entry
>>>>
>>>> action=logonly; alias IOC IOC_$0
>>>>
>>>>
>>>>
>>>> type=Single
>>>>
>>>> ptype=regexp
>>>>
>>>> context=IOC_$2
>>>>
>>>> pattern= syslog.*hostname=([\w\-\d]+).*IP=([\d\.]+)
>>>>
>>>> desc=Matched host & ip: $2 && $3
>>>>
>>>> action=pipe '$0' mail -s ‘%s’ ‘test123.gmail.com’
>>>>
>>>>
>>>>
>>>> IOC_data.txt
>>>>
>>>> 187.163.222.244
>>>>
>>>> 187.189.195.208
>>>>
>>>> 188.166.253.46
>>>>
>>>> 189.209.217.49
>>>>
>>>> 187.163.222.244
>>>>
>>>> 187.189.195.208
>>>>
>>>> 188.166.253.46
>>>>
>>>> 189.209.217.49
>>>>
>>>> ************************
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> san
>>>>
>>>
>>
>> --
>> Regards,
>> SanthoshKumar S
>>
>
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
