[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Thanks again Jesus!

I will definitely share what I come up with and thanks for all your 
suggestions and bearing with me through this (long) thread :)

Fredrik 

On Thursday, August 18, 2016 at 12:17:20 PM UTC+2, Jesus Linares wrote:
>
> Hi Fredik,
>
> Long time no see!. It is a hot summer here and, as always, playing with 
> OSSEC ;).
>
> I don't have time to create all the decoders, but here a template to help 
> you:
> 
>   ^redirect \p|^prevent \p|^allow \p|^block \p|^mail System 
> Alert|\S+ alert Protection Name:
>   firewall
> 
>
>
> 
> 
>   checkpoint
>   ^block \p
>   src: (\S+); dst: (\S+); proto: (\S+);
> 
>   srcip,dstip,protocol
> 
>
>
> 
>
>
> 
> 
>   checkpoint
>   alert Protection Name: 
>   (\.+); Severity: (\d+);
>   url,status
> 
>
>
> 
>
>
> 
>
> If you have questions with a specific decoder, just post it here.
>
> To make useful decoders for the community, you should:
>
>- Add the version of your checkpoint at the begging of the file.
>- Use a standard log format: are you using the default log format of 
>checkpoint?. If not, you should use it or at least, use the syslog 
>standard. In this way, the decoders will work for other users.
>- Provide log samples for each decoder: I usually paste the log as 
>comments in the decoders.
>
> If you don't mind, when you have the decoders working, you could send them 
> to our ruleset repositoy  and to 
> ossec-hids .
>
> Also, you could use plugin decoders 
> 
>  
> instead of standard decoders. Plugin decoders ara coded in c and it is 
> useful with complex logs (like firewalls).
>
> I hope it helps.
> Regards.
>
> On Wednesday, August 17, 2016 at 2:38:47 PM UTC+2, Fredrik wrote:
>>
>> Hi Jesus!
>>
>>
>> Hope you have had a nice summer so far :) I'm revisiting this decoder 
>> with, what I hoped would be, a fresh (rested) pair of eyes ;) 
>> Unfortunately, I realize I still have trouble sorting this one out in an 
>> efficient manner. I was hoping I could ask you for a few additional 
>> pointers especially regarding how to best extract the relevant information 
>> from all five types of messages (outlined below). My current challenge is 
>> that the 'catch all' parent decoder and allow-block-child, result in events 
>> of types redirect/alert/system alert not be parsed completely i.e. 
>> information like action/src/dst/message not being extracted. I have 
>> attacked the problem from the few angles I can think of, but have come up 
>> short :(
>>
>> Note: All messages except "System Alert" now share a commonality in the 
>> hostname which is the active node in the cluster, possible values being 
>> st4600fw01n1 or st4600fw01n2 . Previously (in thread) you showed me how the 
>> actions could be used in parent decoder ^redirect \p|^prevent 
>> \p|^allow \p|^block \p|^mail System Alert|\S+ alert Protection 
>> Name:.
>>
>> - Should I use the hostname as the matching criteria for my parent 
>> decoder and group these four events together whilst creating a separate 
>> decoder for System Alert? 
>> - How would you then suggest I go about extracting the information 
>> action/srcip/dstip/url/extra data for the for types of messages in this 
>> 'group' using child decoders?
>>
>> BLOCK
>> Jun  2 13:24:13 st4600fw01n1 block > 54.164.78.72; proto: tcp; bytes: 1845; sent_bytes: 749; received_bytes: 
>> 1096; app_id: 1347922162; browse_time: **; Suppressed logs: 7; 
>> Referrer_self_uid: **; Referrer_Parent_uid: **; product: URL 
>> Filtering; service: http; s_port: 51096; product_family: Network;
>>
>> Jun  2 13:24:54 st4600fw01n1 block > 54.164.78.72; proto: tcp; bytes: 11122; sent_bytes: 4494; received_bytes: 
>> 6628; app_id: 1347922162; browse_time: **; Referrer_self_uid: **; 
>> product: URL Filtering; service: http; s_port: 51096; product_family: 
>> Network;
>>
>> Jun  2 13:31:57 st4600fw01n1 block > 152.115.75.210; proto: tcp; appi_name: **; app_desc: **; app_id: 
>> 1875144601; app_category: **; matched_category: **; app_properties: 
>> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
>> web_client_type: Other: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; 
>> Touch; rv:11.0) like Gecko; web_server_type: Other: nginx; resource: 
>> http://adx.adform.net/adx/?rp=3=1=_adform_cb_1464866991419_10159400672628005;
>>  
>> proxy_src_ip: 192.168.71.3; product: URL Filtering; service: http; s_port: 
>> 51311; product_family: Network;
>>
>>
>> ALLOW
>> Jun  2 13:50:15 st4600fw01n1 allow > 107.170.204.55; proto: tcp; appi_name: **; app_desc: **; app_id: 
>> 60520086; app_category: **; matched_category: **; app_properties: 
>> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
>> app_sig_id: 60520086:4; proxy_src_ip: 192.168.99.11; product: Application 
>> Control; service: https; 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Jesus Linares]

Hi Fredik,

Long time no see!. It is a hot summer here and, as always, playing with 
OSSEC ;).

I don't have time to create all the decoders, but here a template to help 
you:

  ^redirect \p|^prevent \p|^allow \p|^block \p|^mail System 
Alert|\S+ alert Protection Name:
  firewall





  checkpoint
  ^block \p
  src: (\S+); dst: (\S+); proto: (\S+);

  srcip,dstip,protocol








  checkpoint
  alert Protection Name: 
  (\.+); Severity: (\d+);
  url,status








If you have questions with a specific decoder, just post it here.

To make useful decoders for the community, you should:

   - Add the version of your checkpoint at the begging of the file.
   - Use a standard log format: are you using the default log format of 
   checkpoint?. If not, you should use it or at least, use the syslog 
   standard. In this way, the decoders will work for other users.
   - Provide log samples for each decoder: I usually paste the log as 
   comments in the decoders.

If you don't mind, when you have the decoders working, you could send them 
to our ruleset repositoy  and to 
ossec-hids .

Also, you could use plugin decoders 

 
instead of standard decoders. Plugin decoders ara coded in c and it is 
useful with complex logs (like firewalls).

I hope it helps.
Regards.

On Wednesday, August 17, 2016 at 2:38:47 PM UTC+2, Fredrik wrote:
>
> Hi Jesus!
>
>
> Hope you have had a nice summer so far :) I'm revisiting this decoder 
> with, what I hoped would be, a fresh (rested) pair of eyes ;) 
> Unfortunately, I realize I still have trouble sorting this one out in an 
> efficient manner. I was hoping I could ask you for a few additional 
> pointers especially regarding how to best extract the relevant information 
> from all five types of messages (outlined below). My current challenge is 
> that the 'catch all' parent decoder and allow-block-child, result in events 
> of types redirect/alert/system alert not be parsed completely i.e. 
> information like action/src/dst/message not being extracted. I have 
> attacked the problem from the few angles I can think of, but have come up 
> short :(
>
> Note: All messages except "System Alert" now share a commonality in the 
> hostname which is the active node in the cluster, possible values being 
> st4600fw01n1 or st4600fw01n2 . Previously (in thread) you showed me how the 
> actions could be used in parent decoder ^redirect \p|^prevent 
> \p|^allow \p|^block \p|^mail System Alert|\S+ alert Protection 
> Name:.
>
> - Should I use the hostname as the matching criteria for my parent decoder 
> and group these four events together whilst creating a separate decoder for 
> System Alert? 
> - How would you then suggest I go about extracting the information 
> action/srcip/dstip/url/extra data for the for types of messages in this 
> 'group' using child decoders?
>
> BLOCK
> Jun  2 13:24:13 st4600fw01n1 block  54.164.78.72; proto: tcp; bytes: 1845; sent_bytes: 749; received_bytes: 
> 1096; app_id: 1347922162; browse_time: **; Suppressed logs: 7; 
> Referrer_self_uid: **; Referrer_Parent_uid: **; product: URL 
> Filtering; service: http; s_port: 51096; product_family: Network;
>
> Jun  2 13:24:54 st4600fw01n1 block  54.164.78.72; proto: tcp; bytes: 11122; sent_bytes: 4494; received_bytes: 
> 6628; app_id: 1347922162; browse_time: **; Referrer_self_uid: **; 
> product: URL Filtering; service: http; s_port: 51096; product_family: 
> Network;
>
> Jun  2 13:31:57 st4600fw01n1 block  152.115.75.210; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 1875144601; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> web_client_type: Other: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; 
> Touch; rv:11.0) like Gecko; web_server_type: Other: nginx; resource: 
> http://adx.adform.net/adx/?rp=3=1=_adform_cb_1464866991419_10159400672628005;
>  
> proxy_src_ip: 192.168.71.3; product: URL Filtering; service: http; s_port: 
> 51311; product_family: Network;
>
>
> ALLOW
> Jun  2 13:50:15 st4600fw01n1 allow  107.170.204.55; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 60520086; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> app_sig_id: 60520086:4; proxy_src_ip: 192.168.99.11; product: Application 
> Control; service: https; s_port: 54159; product_family: Network;
>
> Jun  2 13:59:05 st4600fw01n1 allow  89.208.212.2; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 10063753; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> web_client_type: Chrome; web_server_type: Microsoft-IIS; app_sig_id: 
> 10063753:5; resource: 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hi Jesus!


Hope you have had a nice summer so far :) I'm revisiting this decoder with, 
what I hoped would be, a fresh (rested) pair of eyes ;) Unfortunately, I 
realize I still have trouble sorting this one out in an efficient manner. I 
was hoping I could ask you for a few additional pointers especially 
regarding how to best extract the relevant information from all five types 
of messages (outlined below). My current challenge is that the 'catch all' 
parent decoder and allow-block-child, result in events of types 
redirect/alert/system alert not be parsed completely i.e. information like 
action/src/dst/message not being extracted. I have attacked the problem 
from the few angles I can think of, but have come up short :(

Note: All messages except "System Alert" now share a commonality in the 
hostname which is the active node in the cluster, possible values being 
st4600fw01n1 or st4600fw01n2 . Previously (in thread) you showed me how the 
actions could be used in parent decoder ^redirect \p|^prevent 
\p|^allow \p|^block \p|^mail System Alert|\S+ alert Protection 
Name:.

- Should I use the hostname as the matching criteria for my parent decoder 
and group these four events together whilst creating a separate decoder for 
System Alert? 
- How would you then suggest I go about extracting the information 
action/srcip/dstip/url/extra data for the for types of messages in this 
'group' using child decoders?

BLOCK
Jun  2 13:24:13 st4600fw01n1 block http://adx.adform.net/adx/?rp=3=1=_adform_cb_1464866991419_10159400672628005;
 
proxy_src_ip: 192.168.71.3; product: URL Filtering; service: http; s_port: 
51311; product_family: Network;


ALLOW
Jun  2 13:50:15 st4600fw01n1 allow http://www.aliveproxy.com/; proxy_src_ip: 
192.168.99.11; product: Application Control; service: http; s_port: 54473; 
product_family: Network;


ALERT
May 31 08:05:18 > st4600fw01n1 alert Protection Name: TCP Urgent Data 
Enforcement; Severity: 0; Confidence Level: 0; protection_id: 
tcp_block_urg_bit_enable; SmartDefense Profile: Recommended_Protection; 
Performance Impact: 0; Protection Type: settings; Attack Info: TCP segment 
with urgent pointer (no data). Urgent data indication was stripped. Please 
refer to sk36869.; attack: Streaming Engine: TCP Urgent Data Enforcement; 
Total logs: 8; Suppressed logs: 7; proto: tcp; dst: 52.22.193.162; src: 
192.168.10.204; product: SmartDefense; service: http; s_port: 50869; 
FollowUp: Not Followed; product_family: Network;

May 31 17:51:04 > st4600fw01n1 alert Protection Name: TCP Urgent Data 
Enforcement; Severity: 0; Confidence Level: 0; protection_id: 
tcp_block_urg_bit_enable; SmartDefense Profile: Recommended_Protection; 
Performance Impact: 0; Protection Type: settings; rule: 21; rule_uid: 
{3F67BCCB-8087-4974-95FA-F4A4FF466D49}; rule_name: #name; Attack Info: TCP 
segment with urgent pointer (no data). Urgent data indication was stripped. 
Please refer to sk36869.; attack: Streaming Engine: TCP Urgent Data 
Enforcement; Total logs: 24; Suppressed logs: 23; proto: tcp; dst: 
54.239.168.11; src: 192.168.10.204; product: SmartDefense; service: http; 
s_port: 60324; FollowUp: Not Followed; product_family: Network;

Aug 17 12:37:14 > st4600fw01n1 alert Protection Name: TCP Urgent Data 
Enforcement; Severity: 0; Confidence Level: 0; protection_id: 
tcp_block_urg_bit_enable; SmartDefense Profile: Recommended_Protection; 
Performance Impact: 0; Protection Type: settings; rule: 23; rule_uid: 
{3F67BCCB-8087-4974-95FA-F4A4FF466D49}; rule_name: #name; Attack Info: TCP 
segment with urgent pointer (no data). Urgent data indication was stripped. 
Please refer to sk36869.; attack: Streaming Engine: TCP Urgent Data 
Enforcement; Total logs: 11; Suppressed logs: 10; proto: tcp; dst: 
80.251.201.102; src: 172.18.46.230; product: SmartDefense; service: https; 
s_port: 57991; FollowUp: Not Followed; product_family: Network;

Aug 17 04:33:21 > st4600fw01n1 alert Protection Name: Packet Sanity; 
Severity: 2; Confidence Level: 5; protection_id: PacketSanity; SmartDefense 
Profile: Recommended_Protection; Performance Impact: 1; Industry Reference: 
CAN-2002-1071; Protection Type: anomaly; Attack Info: Invalid TCP packet - 
source / destination port 0; attack: Malformed Packet; Total logs: 3; 
Suppressed logs: 2; proto: tcp; dst: 80.169.184.240; src: 185.65.132.121; 
product: SmartDefense; FollowUp: Not Followed; product_family: Network;


REDIRECT
Jun  2 13:54:09 st4600fw01n1 redirect http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
src: 192.168.99.11; dst: 172.226.217.148; proto: tcp; session_id: 
{0x57501e61,0x1001b,0xc50d2e0a,0xc000}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 

Re: [ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Thanks Antonio! Noted! 

Best regards,
Fredrik

On Saturday, April 16, 2016 at 1:14:53 AM UTC+2, Antonio Querubin wrote:
>
> On Fri, 15 Apr 2016, Fredrik wrote: 
>
> > Thanks for getting back to me. Again :) :) I'm trying out your 
> enhancement 
> > to the first decoder and trying to combine it with child-decoders from 
> our 
> > previous posts. I currently have this (which obviously doesn't work), 
> but 
> > how do I best create the parent-child tree to handle the slight 
> variations 
> > in messages? One per action (e.g. block, prevent) and where needed use 
> two 
> > child-decoders with same name to capture all field of interest - as you 
> > have instructed previously (i.e. one for fields action,srcip,dstip , the 
> > second for url, extra_data fields? 
>
> >  
> >  Checkpoint-test 
> >  (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> > (\d+.\d+.\d+.\d+) 
> >  action,srcip,dstip 
> >  
>
> If you plan on detecting IPv6 activity on your network you may want to 
> change \d+.\d+.\d+.\d+ to \S+ 
>
>
> Antonio Querubin 
> e-mail:  to...@lavanauts.org  
> xmpp:  antonio...@gmail.com  
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ossec-list] Re: Decoding long messages - multiple regex statements

[Antonio Querubin]

On Fri, 15 Apr 2016, Fredrik wrote:


Thanks for getting back to me. Again :) :) I'm trying out your enhancement
to the first decoder and trying to combine it with child-decoders from our
previous posts. I currently have this (which obviously doesn't work), but
how do I best create the parent-child tree to handle the slight variations
in messages? One per action (e.g. block, prevent) and where needed use two
child-decoders with same name to capture all field of interest - as you
have instructed previously (i.e. one for fields action,srcip,dstip , the
second for url, extra_data fields?




 Checkpoint-test
 (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst:
(\d+.\d+.\d+.\d+)
 action,srcip,dstip



If you plan on detecting IPv6 activity on your network you may want to 
change \d+.\d+.\d+.\d+ to \S+



Antonio Querubin
e-mail:  t...@lavanauts.org
xmpp:  antonioqueru...@gmail.com

[ossec-list] Re: Decoding long messages - multiple regex statements

[Jesus Linares]

Hi Fredrik,

It is good progress. You can capture all events with:


  ^redirect \p|^prevent \p|^allow \p|^block \p|^mail System 
Alert|\S+ alert Protection Name:
  firewall



I know... It is not very elegant, but it controls all your events. Also, 
you can add a tag in the beginning of the log (by the firewall settings or 
with *rsyslog*) and the decoder will be vey easy:

Logs:
*FredikFirewall *Apr 15 12:23:16 > st4600fw01n1 alert Protection Name: TCP 
Urgent Data Enforcement; Severity: 0; Confidence Level: 0; protection_id: 
tcp_block_urg_bit_enable; SmartDefense Profile: Recommended_Protection; 
Performance Impact: 0; Protection Type: settings; rule: 20; rule_uid: {
3F67BCCB-8087-4974-95FA-F4A4FF466D49}; rule_name: #sample; Attack Info: TCP 
segment with urgent pointer (no data). Urgent data indication was stripped. 
Please refer to sk36869.; attack: Streaming Engine: TCP Urgent Data 
Enforcement; Total logs: 3; Suppressed logs: 2; proto: tcp; dst: 
104.16.65.50; src: 192.168.10.204; product: SmartDefense; service: https; 
s_port: 56814; FollowUp: Not Followed; product_family: Network;



Decoder:

  ^FredikFirewall 
  firewall



Regards,
Jesus Linares.



On Friday, April 15, 2016 at 3:47:17 PM UTC+2, Fredrik wrote:
>
> Hello Jesus!
>
>
> Story continues. Just wanted to let you know that I have been able, with 
> help, to unify ALL the messages for easier handling in OSSEC. Thing is now 
> that the hostname is extracted automagically (by OSSEC) from the message 
> and I guess can't be used for my prematch, or? Ossec-logtest will treat the 
> hostname as part of the header and start the 'Log:' section with e.g. 
>
> block 
> How would you tackle this? Right a prematch with all operative words 
> (actions) that is used with the messages I'm interested in (e.g. 
> ^allow|^block|^prevent|^redirect)? In my scenario this shouldn't conflict 
> with other type of messages. I'm guessing that you Ossec-pros will have 
> options and better alternative though ;) I would also like to match the 
> decoder regardless of which node in the firewall cluster is the source of 
> the event? I The two possibilities are st4600fw01n1 and st4600fw01n2 .
>
> Here are more message samples:
>
> pr 15 14:41:53 st4600fw01n1 allow  216.131.91.92; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 60461422; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> web_client_type: Chrome; web_server_type: Apache; app_sig_id: 60461422:1; 
> resource: 
> http://strongvpn.com/difference_between_proxy_and_vpn.html?utm_source=adwords_medium=sem=Cj0KEQjwosK4BRCYhsngx4_SybcBEiQAowaCJTFp6qNVmL7E-BhfeTkQouJTwpHN5v1wslK79jD62k4aAqBB8P8HAQ;
>  
> proxy_src_ip: 192.168.5.133; product: Application Control; service: http; 
> s_port: 59319; product_family: Network;
>
> Apr 15 14:21:37 st4600fw01n1 redirect  resource: 
> http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
> src: 192.168.5.133; dst: 184.31.90.152; proto: tcp; session_id: 
> {0x5710dcd1,0x10002,0xc50d2e0a,0xc000}; Protection name: Check Point - 
> Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
> Level: 5; severity: 2; malware_action: Communication with C site; 
> rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
> reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
> protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 
> 192.168.5.133; product: Anti Malware; service: http; s_port: 57878;
>
> Apr 15 05:35:51 st4600fw01n1 prevent  192.168.99.4; proto: tcp; session_id: 
> {0x57106197,0x10003,0xc50d2e0a,0xc001}; Protection name: 
> Trojan.Win32.HackerDefender.C; malware_family: HackerDefender; Source OS: 
> Solaris; Confidence Level: 5; severity: 4; malware_action: Malicious 
> network activity; rule_uid: {25157EEE-C09C-4FE0-A872-E0A1486526B8}; 
> rule_name: #extweb; Protection Type: protection; malware_rule_id: 
> {00CE-00A4-0046-9658-621EA5468654}; protection_id: 43FBC; log_id: 
> 2; scope: 192.168.99.4; product: Anti Malware; service: http; s_port: 49228;
>
> Apr 15 14:13:17 st4600fw01n1 block  37.157.2.24; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 1875144601; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> web_client_type: Chrome; web_server_type: Other: nginx; resource: 
> http://adx.adform.net/adx/?rp=3=1_dXJsPQ_dXJsPQ_dXJsPQ=_adform_cb_1460722287088_3438587873323349;
>  
> proxy_src_ip: 192.168.7.196; product: URL Filtering; service: http; s_port: 
> 51190; product_family: Network;
>
> Apr 15 11:16:05 st4600fw01n1 block  64.207.139.185; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 3723664659; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hello Jesus!


Story continues. Just wanted to let you know that I have been able, with 
help, to unify ALL the messages for easier handling in OSSEC. Thing is now 
that the hostname is extracted automagically (by OSSEC) from the message 
and I guess can't be used for my prematch, or? Ossec-logtest will treat the 
hostname as part of the header and start the 'Log:' section with e.g. 

block http://strongvpn.com/difference_between_proxy_and_vpn.html?utm_source=adwords_medium=sem=Cj0KEQjwosK4BRCYhsngx4_SybcBEiQAowaCJTFp6qNVmL7E-BhfeTkQouJTwpHN5v1wslK79jD62k4aAqBB8P8HAQ;
 
proxy_src_ip: 192.168.5.133; product: Application Control; service: http; 
s_port: 59319; product_family: Network;

Apr 15 14:21:37 st4600fw01n1 redirect http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
src: 192.168.5.133; dst: 184.31.90.152; proto: tcp; session_id: 
{0x5710dcd1,0x10002,0xc50d2e0a,0xc000}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 
192.168.5.133; product: Anti Malware; service: http; s_port: 57878;

Apr 15 05:35:51 st4600fw01n1 prevent http://adx.adform.net/adx/?rp=3=1_dXJsPQ_dXJsPQ_dXJsPQ=_adform_cb_1460722287088_3438587873323349;
 
proxy_src_ip: 192.168.7.196; product: URL Filtering; service: http; s_port: 
51190; product_family: Network;

Apr 15 11:16:05 st4600fw01n1 block http://cdn.wibiya.com/Toolbars/dir_0650/Toolbar_650079/Loader_650079.js; 
proxy_src_ip: 192.168.8.67; product: URL Filtering; service: http; s_port: 
61907; product_family: Network;

The two outliers now are the messages below. Not quite sure how to handle 
them, but two additional decoders seem required, At least I'm down to two 
outliers and not a whole bunch of exceptions as previously :) :) What would 
be your take on how to treat these two?

Mar  7 13:07:53 sto-fwm03 mail System Alert message: A Firewall Policy has 
been successfully installed on st4600fw01n1; Object: st4600fw01n1; Event: 
Change; Parameter: policy_time; Condition: changes Mon Mar  7 13:03:42 
2016; Current value: Mon Mar  7 13:08:48 2016; product: Test Monitor; 
product_family: Network;

Apr 15 12:23:16 > st4600fw01n1 alert Protection Name: TCP Urgent Data 
Enforcement; Severity: 0; Confidence Level: 0; protection_id: 
tcp_block_urg_bit_enable; SmartDefense Profile: Recommended_Protection; 
Performance Impact: 0; Protection Type: settings; rule: 20; rule_uid: 
{3F67BCCB-8087-4974-95FA-F4A4FF466D49}; rule_name: #sample; Attack Info: 
TCP segment with urgent pointer (no data). Urgent data indication was 
stripped. Please refer to sk36869.; attack: Streaming Engine: TCP Urgent 
Data Enforcement; Total logs: 3; Suppressed logs: 2; proto: tcp; dst: 
104.16.65.50; src: 192.168.10.204; product: SmartDefense; service: https; 
s_port: 56814; FollowUp: Not Followed; product_family: Network;

Best regards,
Fredrik 


On Friday, April 1, 2016 at 1:18:17 PM UTC+2, Jesus Linares wrote:
>
> Hi Fredrik,
>
> here an example of decoding allow/block events (with the option 
> *after_regex*):
>
>
> 
> 
>   ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
>   firewall
> 
>
>
> 
> 
>   Checkpoint-test
>   ^block|^allow
>   (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> (\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
> 
>   Checkpoint-test
>   resource: (\S+); proxy_src_ip: \S+; 
> product: (\.+); 
>   url, extra_data
> 
>
>
> I recommend you configure all your checkpoint devices with the same log 
> format. If you can't you could use *several parents*:
>
> 
> 
>   ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
>   firewall
> 
>
>
> 
> 
>   ^redirect \p|^prevent \p
>   firewall
> 
>
>
> 
> 
>   Checkpoint-test
>   ^block|^allow
>   (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> (\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
> 
>   Checkpoint-test
>   resource: (\S+); proxy_src_ip: \S+; 
> product: (\.+); 
>   url, extra_data
> 
>
>
> P.S. My name is Jesus, not Jose ;).
>
> Regards,
> Jesus Linares.
>
>
>
> On Wednesday, March 30, 2016 at 10:28:09 AM UTC+2, Fredrik wrote:
>
> Hi Jose,
>
>
> I got some help to sort out the different timestamps (format) and all log 
> types now use "Jan 27 09:41:01".  You asked about the firewall, this 
> particular one is a Checkpoint currently running version R77.20.
>
> The remaining question, that might be of interest to others on the path to 
> OSSEC mastery ;) ;) is how to handle messages with different "format" 
> coming from the same host. I have collected a bunch of messages that I 
> would like to be able to decode, but I'm not sure about the most efficient 
> way to build the parent/child decoder tree for this. 
>
> With the help received previously in this thread, I currently have the 
> following in my local_decoder and 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hello Jesus!


Story continues. Just wanted to let you know that I have been able, with 
help, to unify ALL the messages for easier handling in OSSEC. It seems I 
will be able to only use the hostname for the decoder (see new messages 
samples below). However, how do I handle the fact that I would like to 
match the decoder regardless of which node in the firewall cluster is the 
source of the event? I can't use '|' within , right? 
The two possibilities are st4600fw01n1 and st4600fw01n2 .

Apr 15 14:41:53 st4600fw01n1 allow http://strongvpn.com/difference_between_proxy_and_vpn.html?utm_source=adwords_medium=sem=Cj0KEQjwosK4BRCYhsngx4_SybcBEiQAowaCJTFp6qNVmL7E-BhfeTkQouJTwpHN5v1wslK79jD62k4aAqBB8P8HAQ;
 
proxy_src_ip: 192.168.5.133; product: Application Control; service: http; 
s_port: 59319; product_family: Network;

Apr 15 14:21:37 st4600fw01n1 redirect http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
src: 192.168.5.133; dst: 184.31.90.152; proto: tcp; session_id: 
{0x5710dcd1,0x10002,0xc50d2e0a,0xc000}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 
192.168.5.133; product: Anti Malware; service: http; s_port: 57878;

Apr 15 05:35:51 st4600fw01n1 prevent http://adx.adform.net/adx/?rp=3=1_dXJsPQ_dXJsPQ_dXJsPQ=_adform_cb_1460722287088_3438587873323349;
 
proxy_src_ip: 192.168.7.196; product: URL Filtering; service: http; s_port: 
51190; product_family: Network;

Apr 15 11:16:05 st4600fw01n1 block http://cdn.wibiya.com/Toolbars/dir_0650/Toolbar_650079/Loader_650079.js; 
proxy_src_ip: 192.168.8.67; product: URL Filtering; service: http; s_port: 
61907; product_family: Network;



The two outliers now are the messages below. For the first one, I guess, 
the best way will be to create a separate decoder and match for the other 
hostname (message originates from the firewall management), right? The 
second one, I have to figure out why SmartDefense throws in a leading > 
before the hostname. Until then, perhaps only go with a prematch including 
the '>', right?

Mar  7 13:07:53 sto-fwm03 mail System Alert message: A Firewall Policy has 
been successfully installed on st4600fw01n1; Object: st4600fw01n1; Event: 
Change; Parameter: policy_time; Condition: changes Mon Mar  7 13:03:42 2016; 
Current value: Mon Mar  7 13:08:48 2016; product: Test Monitor; 
product_family: Network;

Apr 15 12:23:16 > st4600fw01n1 alert Protection Name: TCP Urgent Data 
Enforcement; Severity: 0; Confidence Level: 0; protection_id: 
tcp_block_urg_bit_enable; SmartDefense Profile: Recommended_Protection; 
Performance Impact: 0; Protection Type: settings; rule: 20; rule_uid: 
{3F67BCCB-8087-4974-95FA-F4A4FF466D49}; rule_name: #sample; Attack Info: 
TCP segment with urgent pointer (no data). Urgent data indication was 
stripped. Please refer to sk36869.; attack: Streaming Engine: TCP Urgent 
Data Enforcement; Total logs: 3; Suppressed logs: 2; proto: tcp; dst: 
104.16.65.50; src: 192.168.10.204; product: SmartDefense; service: https; 
s_port: 56814; FollowUp: Not Followed; product_family: Network;

Best regards,
Fredrik 

On Friday, April 1, 2016 at 1:18:17 PM UTC+2, Jesus Linares wrote:
>
> Hi Fredrik,
>
> here an example of decoding allow/block events (with the option 
> *after_regex*):
>
>
> 
> 
>   ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
>   firewall
> 
>
>
> 
> 
>   Checkpoint-test
>   ^block|^allow
>   (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> (\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
> 
>   Checkpoint-test
>   resource: (\S+); proxy_src_ip: \S+; 
> product: (\.+); 
>   url, extra_data
> 
>
>
> I recommend you configure all your checkpoint devices with the same log 
> format. If you can't you could use *several parents*:
>
> 
> 
>   ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
>   firewall
> 
>
>
> 
> 
>   ^redirect \p|^prevent \p
>   firewall
> 
>
>
> 
> 
>   Checkpoint-test
>   ^block|^allow
>   (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> (\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
> 
>   Checkpoint-test
>   resource: (\S+); proxy_src_ip: \S+; 
> product: (\.+); 
>   url, extra_data
> 
>
>
> P.S. My name is Jesus, not Jose ;).
>
> Regards,
> Jesus Linares.
>
>
>
> On Wednesday, March 30, 2016 at 10:28:09 AM UTC+2, Fredrik wrote:
>
> Hi Jose,
>
>
> I got some help to sort out the different timestamps (format) and all log 
> types now use "Jan 27 09:41:01".  You asked about the firewall, this 
> particular one is a Checkpoint currently running version R77.20.
>
> The remaining question, that might be of interest to others on the path to 
> OSSEC mastery ;) ;) is how to handle messages with different "format" 
> coming from the same host. I have collected a bunch of 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Very sorry about the mistake with your name - hope I haven't done it 
before!? 

Will try out your much much appreciated suggestions for decoders over the 
weekend! Very excited! :)

Thanks,
Fredrik 

On Friday, April 1, 2016 at 1:18:17 PM UTC+2, Jesus Linares wrote:
>
> Hi Fredrik,
>
> here an example of decoding allow/block events (with the option 
> *after_regex*):
>
>
> 
> 
>   ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
>   firewall
> 
>
>
> 
> 
>   Checkpoint-test
>   ^block|^allow
>   (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> (\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
> 
>   Checkpoint-test
>   resource: (\S+); proxy_src_ip: \S+; 
> product: (\.+); 
>   url, extra_data
> 
>
>
> I recommend you configure all your checkpoint devices with the same log 
> format. If you can't you could use *several parents*:
>
> 
> 
>   ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
>   firewall
> 
>
>
> 
> 
>   ^redirect \p|^prevent \p
>   firewall
> 
>
>
> 
> 
>   Checkpoint-test
>   ^block|^allow
>   (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
> (\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
> 
>   Checkpoint-test
>   resource: (\S+); proxy_src_ip: \S+; 
> product: (\.+); 
>   url, extra_data
> 
>
>
> P.S. My name is Jesus, not Jose ;).
>
> Regards,
> Jesus Linares.
>
>
>
> On Wednesday, March 30, 2016 at 10:28:09 AM UTC+2, Fredrik wrote:
>
> Hi Jose,
>
>
> I got some help to sort out the different timestamps (format) and all log 
> types now use "Jan 27 09:41:01".  You asked about the firewall, this 
> particular one is a Checkpoint currently running version R77.20.
>
> The remaining question, that might be of interest to others on the path to 
> OSSEC mastery ;) ;) is how to handle messages with different "format" 
> coming from the same host. I have collected a bunch of messages that I 
> would like to be able to decode, but I'm not sure about the most efficient 
> way to build the parent/child decoder tree for this. 
>
> With the help received previously in this thread, I currently have the 
> following in my local_decoder and I'm experimenting with different addition 
> - none of which is working so far ;) 
>
> 
>   ^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d
>   firewall
> 
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
> 
>   Checkpoint
>   \.*resource: (\.*);\.*product: (\.*);
>   url,extra_data
> 
>
>
> Below is a collection of syslog messages recieved from the firewall where 
> the first section is currently decoded using the local_decoder above:
>
>
> Mar 29 10:09:40 127.0.0.1 Mar 29 9:57:49 st4600fw01n1 block  src: 192.168.7.206; dst: 54.72.9.51; proto: tcp; bytes: 4962; sent_bytes: 
> 530; received_bytes: 4432; app_id: 3404393449; browse_time: **; 
> Suppressed logs: 1; Referrer_self_uid: **; product: URL Filtering; 
> service: http; s_port: 54693; product_family: Network;
>
> Mar 29 20:57:00, st4600fw01n1, allow  89.208.212.2; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 10063753; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> web_client_type: Microsoft IE; web_server_type: Microsoft-IIS; app_sig_id: 
> 10063753:5; resource: http://aliveproxy.com/; proxy_src_ip: 
> 192.168.5.133; product: Application Control; service: http; s_port: 63867; 
> product_family: Network;
>
> Mar 30 09:04:14 127.0.0.1 Mar 30 8:52:22 st4600fw01n1 allow  src: 192.168.5.133; dst: 23.67.132.180; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 10003219; app_category: **; matched_category: 
> **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; app_sig_id: 10003219:4; proxy_src_ip: 192.168.5.133; 
> product: Application Control; service: https; s_port: 64166; 
> product_family: Network;
>
> Mar 30 09:03:56 127.0.0.1 Mar 30 8:52:05 st4600fw01n1 allow  src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; bytes: 2396; sent_bytes: 
> 1180; received_bytes: 1216; app_id: 10063753; browse_time: **; 
> Suppressed logs: 19; Referrer_self_uid: **; product: Application 
> Control; service: http; s_port: 64136; product_family: Network;
>
> Mar 30 09:03:35 127.0.0.1 Mar 30 8:51:43 st4600fw01n1 allow  src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 10063753; app_category: **; matched_category: 
> **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: 
> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.5.133; product: 
> Application Control; service: http; s_port: 64136; product_family: Network;
>
> Mar 29 09:04:21 127.0.0.1 Mar 29 8:52:29 st4600fw01n1 block  src: 192.168.5.136; dst: 37.157.4.15; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 1875144601; app_category: **; 
> matched_category: **; 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Jesus Linares]

Hi Fredrik,

here an example of decoding allow/block events (with the option 
*after_regex*):




  ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
  firewall





  Checkpoint-test
  ^block|^allow
  (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
(\d+.\d+.\d+.\d+)
  action,srcip,dstip





  Checkpoint-test
  resource: (\S+); proxy_src_ip: \S+; product: 
(\.+); 
  url, extra_data



I recommend you configure all your checkpoint devices with the same log 
format. If you can't you could use *several parents*:



  ^\w\w\w \d+ \d+:\d+:\d+ \S+ 
  firewall





  ^redirect \p|^prevent \p
  firewall





  Checkpoint-test
  ^block|^allow
  (\w+) \S+ \S+ src: (\d+.\d+.\d+.\d+); dst: 
(\d+.\d+.\d+.\d+)
  action,srcip,dstip





  Checkpoint-test
  resource: (\S+); proxy_src_ip: \S+; product: 
(\.+); 
  url, extra_data



P.S. My name is Jesus, not Jose ;).

Regards,
Jesus Linares.



On Wednesday, March 30, 2016 at 10:28:09 AM UTC+2, Fredrik wrote:
>
> Hi Jose,
>
>
> I got some help to sort out the different timestamps (format) and all log 
> types now use "Jan 27 09:41:01".  You asked about the firewall, this 
> particular one is a Checkpoint currently running version R77.20.
>
> The remaining question, that might be of interest to others on the path to 
> OSSEC mastery ;) ;) is how to handle messages with different "format" 
> coming from the same host. I have collected a bunch of messages that I 
> would like to be able to decode, but I'm not sure about the most efficient 
> way to build the parent/child decoder tree for this. 
>
> With the help received previously in this thread, I currently have the 
> following in my local_decoder and I'm experimenting with different addition 
> - none of which is working so far ;) 
>
> 
>   ^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d
>   firewall
> 
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
> 
>   Checkpoint
>   \.*resource: (\.*);\.*product: (\.*);
>   url,extra_data
> 
>
>
> Below is a collection of syslog messages recieved from the firewall where 
> the first section is currently decoded using the local_decoder above:
>
>
> Mar 29 10:09:40 127.0.0.1 Mar 29 9:57:49 st4600fw01n1 block  src: 192.168.7.206; dst: 54.72.9.51; proto: tcp; bytes: 4962; sent_bytes: 
> 530; received_bytes: 4432; app_id: 3404393449; browse_time: **; 
> Suppressed logs: 1; Referrer_self_uid: **; product: URL Filtering; 
> service: http; s_port: 54693; product_family: Network;
>
> Mar 29 20:57:00, st4600fw01n1, allow  89.208.212.2; proto: tcp; appi_name: **; app_desc: **; app_id: 
> 10063753; app_category: **; matched_category: **; app_properties: 
> **; app_risk: **; app_rule_id: **; app_rule_name: **; 
> web_client_type: Microsoft IE; web_server_type: Microsoft-IIS; app_sig_id: 
> 10063753:5; resource: http://aliveproxy.com/; proxy_src_ip: 
> 192.168.5.133; product: Application Control; service: http; s_port: 63867; 
> product_family: Network;
>
> Mar 30 09:04:14 127.0.0.1 Mar 30 8:52:22 st4600fw01n1 allow  src: 192.168.5.133; dst: 23.67.132.180; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 10003219; app_category: **; matched_category: 
> **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; app_sig_id: 10003219:4; proxy_src_ip: 192.168.5.133; 
> product: Application Control; service: https; s_port: 64166; 
> product_family: Network;
>
> Mar 30 09:03:56 127.0.0.1 Mar 30 8:52:05 st4600fw01n1 allow  src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; bytes: 2396; sent_bytes: 
> 1180; received_bytes: 1216; app_id: 10063753; browse_time: **; 
> Suppressed logs: 19; Referrer_self_uid: **; product: Application 
> Control; service: http; s_port: 64136; product_family: Network;
>
> Mar 30 09:03:35 127.0.0.1 Mar 30 8:51:43 st4600fw01n1 allow  src: 192.168.5.133; dst: 89.208.212.2; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 10063753; app_category: **; matched_category: 
> **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: 
> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.5.133; product: 
> Application Control; service: http; s_port: 64136; product_family: Network;
>
> Mar 29 09:04:21 127.0.0.1 Mar 29 8:52:29 st4600fw01n1 block  src: 192.168.5.136; dst: 37.157.4.15; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 1875144601; app_category: **; 
> matched_category: **; app_properties: **; app_risk: **; 
> app_rule_id: **; app_rule_name: **; web_client_type: Other: 
> Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like 
> Gecko; web_server_type: Other: nginx; resource: 
> http://adx.adform.net/adx/?rp=3=1=_adform_cb_1459234242747_7491414591872548;
>  
> proxy_src_ip: 192.168.5.136; product: URL Filtering; service: http; s_port: 
> 54051; 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hi Jose,


I got some help to sort out the different timestamps (format) and all log 
types now use "Jan 27 09:41:01".  You asked about the firewall, this 
particular one is a Checkpoint currently running version R77.20.

The remaining question, that might be of interest to others on the path to 
OSSEC mastery ;) ;) is how to handle messages with different "format" 
coming from the same host. I have collected a bunch of messages that I 
would like to be able to decode, but I'm not sure about the most efficient 
way to build the parent/child decoder tree for this. 

With the help received previously in this thread, I currently have the 
following in my local_decoder and I'm experimenting with different addition 
- none of which is working so far ;) 


  ^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d
  firewall



  Checkpoint
  (\w+) \p\w+ \w+ 
src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
  action,srcip,dstip



  Checkpoint
  \.*resource: (\.*);\.*product: (\.*);
  url,extra_data



Below is a collection of syslog messages recieved from the firewall where 
the first section is currently decoded using the local_decoder above:


Mar 29 10:09:40 127.0.0.1 Mar 29 9:57:49 st4600fw01n1 block http://aliveproxy.com/; proxy_src_ip: 192.168.5.133; 
product: Application Control; service: http; s_port: 63867; product_family: 
Network;

Mar 30 09:04:14 127.0.0.1 Mar 30 8:52:22 st4600fw01n1 allow http://www.aliveproxy.com/; proxy_src_ip: 192.168.5.133; product: 
Application Control; service: http; s_port: 64136; product_family: Network;

Mar 29 09:04:21 127.0.0.1 Mar 29 8:52:29 st4600fw01n1 block http://adx.adform.net/adx/?rp=3=1=_adform_cb_1459234242747_7491414591872548;
 
proxy_src_ip: 192.168.5.136; product: URL Filtering; service: http; s_port: 
54051; product_family: Network;

Mar 29 09:06:12 127.0.0.1 Mar 29 8:54:21 st4600fw01n1 block http://adx.adform.net/adx/?rp=3=1=_adform_cb_1459234355177_008705563130792681;
 
proxy_src_ip: 192.168.6.157; product: URL Filtering; service: http; s_port: 
51746; product_family: Network;

Mar 29 08:37:54 127.0.0.1 Mar 29 8:26:03 st4600fw01n1 block http://adx.adform.net/adx/?rp=3=1_dXJsPQ_dXJsPQ=_adform_cb_1459232656248_7681010355476278;
 
proxy_src_ip: 192.168.6.157; product: URL Filtering; service: http; s_port: 
51104; product_family: Network;

Mar 29 08:35:17 127.0.0.1 Mar 29 8:23:24 st4600fw01n1 block http://adx.adform.net/adx/?rp=3=1=_adform_cb_1459232497393_6046677836175733;
 
proxy_src_ip: 192.168.6.157; product: URL Filtering; service: http; s_port: 
50904; product_family: Network;




Mar 30 10:04:39 127.0.0.1 redirect http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
src: 192.168.5.133; dst: 172.226.217.148; proto: tcp; session_id: 
{0x56fb8896,0x10009,0xc50d2e0a,0xc001}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 
192.168.5.133; product: Anti Malware; service: http; s_port: 49244;

Mar 29 20:13:14 127.0.0.1 prevent http://www.bing.com/fd/ls/GLinkPing.aspx?IG=9A0044152B65437D93F87086B9E730D9&=SERP,5118.1=http://sc1.checkpoint.com/z...;
 
src: 192.168.5.133; dst: 204.79.197.200; proto: tcp; session_id: 
{0x56fac5ba,0x10004,0xc50d2e0a,0xc000}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.5.133; scope: 
192.168.5.133; product: Anti Malware; service: http; s_port: 63119;

Mar 30 9:04:59, st4600fw01n1, allow http://www.bypassthat.com/; proxy_src_ip: 192.168.5.133; product: 
Application Control; service: http; s_port: 64499; product_family: Network;



Mar 30 08:55:41 127.0.0.1 Mar 30 8:49:25 < sto-fwm03 mail System Alert 
message: A Firewall Policy has been successfully installed on st4600fw01n2; 
Object: st4600fw01n2; Event: Change; Parameter: policy_time; Condition: 
changes Tue Mar 22 11:07:17 2016; Current value: Wed Mar 30 08:39:57 2016; 
product: System Monitor; product_family: Network;

Mar 30 08:56:02 127.0.0.1 Mar 30 8:49:47 < sto-fwm03 mail System Alert 
message: A Firewall Policy has been successfully installed on st4600fw01n1; 
Object: st4600fw01n1; Event: Change; Parameter: policy_time; Condition: 
changes Tue Mar 22 11:09:21 2016; Current value: Wed Mar 30 08:43:12 2016; 
product: System Monitor; product_family: Network;

 

On Tuesday, March 29, 2016 at 12:53:19 PM UTC+2, Jesus Linares wrote:
>
> Hi,
>
> first, I would use the same format for both messages. Two options:
>

[ossec-list] Re: Decoding long messages - multiple regex statements

[Jesus Linares]

Hi,

first, I would use the same format for both messages. Two options:

   - Change log format in each device. 
  - Choose one:
 - 1Mar2016 15:17:09 redirect st4600fw01n1
 - Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1
  - This part could be your parent decoder (using regular expressions)
   - Change the log received with rsyslog, for example, add a string:
  - *MyFirewall *1Mar2016 15:17:09 redirect st4600fw01n1
  - So, the parent decoder will be ^**MyFirewall *
   
The prematch of each sub-decoder (child decoder) could be the type of log, 
maybe "web_client_type" or "mail".

What firewall are you using? Version?.

Paste here more logs.

Regards,
Jesus Linares

On Thursday, March 24, 2016 at 9:47:28 PM UTC+1, Fredrik wrote:
>
> Hi Jesus,
>
>
> Got sidetracked with other projects, and finally getting back to my 
> questions about handling different messages from the same device 
> (firewall). Also, Jesus your suggestion about placing a prematch in the 
> suggested decoder in this thread - what would be a good prematch here? 
>
> Should I add an OR to the parent decoder to do the first match and then 
> use different subdecoders to extract the useful information from the other 
> type of message? How do you deal with these type of scenarios?
>
> Just so I got that part right. Giving two sections the same 
> Checkpoint-alert in essence means that it is one 
> decoder, but defined in two sections? 
>
>
> Please find the two message-types below for reference.
>
> MESSAGE1:
> 1Mar2016 15:17:09 redirect st4600fw01n1  Chrome; resource: http://
> sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; src: 
> 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: 
> {0x56d5a2de,0x4,0xc50d2e0a,0xc001}; Protection name: Check Point - 
> Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
> Level: 5; severity: 2; malware_action: Communication with C site; 
> rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
> reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
> protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: 
> 10.46.5.133; product: Anti Malware; service: http; s_port: 61834;
>
> MESSAGE2:
> Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 allow  : 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; app_desc
> : **; app_id: 10063753; app_category: **; matched_category: 
> **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: Microsoft
> -IIS; app_sig_id: 10063753:5; resource: http://www.aliveproxy.com/; 
> proxy_src_ip: 192.168.1.15 product: Application Control; service: http; 
> s_port: 58579; product_family: Network;
>
> On Monday, March 7, 2016 at 12:11:21 PM UTC+1, Jesus Linares wrote:
>
>> Hi Fredrik,
>>
>> The expression "\.+" matches for anything. Usually, it is not a good idea 
>> because is slow and maybe you capture something that you don't want. So, 
>> *when 
>> it is possible*, it is better to use something specific.
>>
>> When you have different decoders (different name) with the same parent, 
>> you should use a prematch. If you don't use prematch, it is fired the first 
>> rule. In the previous example:
>>
>> Log:
>> Mar  3 12:15:24 LinMV TestDecoder[1963]: TypeB field1: hi; value2: bye; 
>> value3: seeyou
>>
>> Without prematch:
>> **Phase 2: Completed decoding.
>>decoder: 'TestDecoder'
>>extra_data: 'seeyou'
>>
>> With prematch:
>> **Phase 2: Completed decoding.
>>decoder: 'TestDecoder'
>>id: 'bye;'
>>
>>
>> Without prematch, the decoder is TestDecoder-1, but it should be 
>> TestDecoder2 (because it has the string "field1". In my view, it is a good 
>> practice use prematch, but sometimes it is no necessary.
>>
>> Regarding your last question, could you use the same log format in your 
>> firewall and in the blade?. Paste here two logs of each one (firewall and 
>> blade) and your decoders, and we will take a look ;)
>>
>> Regards.
>> Jesus Linares
>>
>> On Friday, March 4, 2016 at 9:08:34 PM UTC+1, Fredrik wrote:
>>>
>>> Hi All,
>>>
>>>
>>> In this context and with your great response. What would you PROs 
>>> suggest I do when decoding another type of message from the same firewall - 
>>> but a different blade (i.e. module). Turns out that the messages look 
>>> somewhat different. This is a sample from the other module and it won't 
>>> match with the current decoder. Should I add an OR to the parent decoder to 
>>> do the first match and then use different subdecoders to extract the useful 
>>> information from the other type of message? How do you deal with these type 
>>> of scenarios?
>>>
>>> MESSAGE:
>>> 1Mar2016 15:17:09 redirect st4600fw01n1 >> Chrome; resource: http://
>>> sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
>>> src: 192.168.1.15; dst: 23.8.4.103; proto: tcp; 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hi Jesus,


Got sidetracked with other projects, and finally getting back to my 
questions about handling different messages from the same device 
(firewall). Also, Jesus your suggestion about placing a prematch in the 
suggested decoder in this thread - what would be a good prematch here? 

Should I add an OR to the parent decoder to do the first match and then use 
different subdecoders to extract the useful information from the other type 
of message? How do you deal with these type of scenarios?

Just so I got that part right. Giving two sections the same 
Checkpoint-alert in essence means that it is one 
decoder, but defined in two sections? 


Please find the two message-types below for reference.

MESSAGE1:
1Mar2016 15:17:09 redirect st4600fw01n1 http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
src: 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: 
{0x56d5a2de,0x4,0xc50d2e0a,0xc001}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: 
10.46.5.133; product: Anti Malware; service: http; s_port: 61834;

MESSAGE2:
Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 allow http://www.aliveproxy.com/; proxy_src_ip: 
192.168.1.15 product: Application Control; service: http; s_port: 58579; 
product_family: Network;

On Monday, March 7, 2016 at 12:11:21 PM UTC+1, Jesus Linares wrote:

> Hi Fredrik,
>
> The expression "\.+" matches for anything. Usually, it is not a good idea 
> because is slow and maybe you capture something that you don't want. So, 
> *when 
> it is possible*, it is better to use something specific.
>
> When you have different decoders (different name) with the same parent, 
> you should use a prematch. If you don't use prematch, it is fired the first 
> rule. In the previous example:
>
> Log:
> Mar  3 12:15:24 LinMV TestDecoder[1963]: TypeB field1: hi; value2: bye; 
> value3: seeyou
>
> Without prematch:
> **Phase 2: Completed decoding.
>decoder: 'TestDecoder'
>extra_data: 'seeyou'
>
> With prematch:
> **Phase 2: Completed decoding.
>decoder: 'TestDecoder'
>id: 'bye;'
>
>
> Without prematch, the decoder is TestDecoder-1, but it should be 
> TestDecoder2 (because it has the string "field1". In my view, it is a good 
> practice use prematch, but sometimes it is no necessary.
>
> Regarding your last question, could you use the same log format in your 
> firewall and in the blade?. Paste here two logs of each one (firewall and 
> blade) and your decoders, and we will take a look ;)
>
> Regards.
> Jesus Linares
>
> On Friday, March 4, 2016 at 9:08:34 PM UTC+1, Fredrik wrote:
>>
>> Hi All,
>>
>>
>> In this context and with your great response. What would you PROs suggest 
>> I do when decoding another type of message from the same firewall - but a 
>> different blade (i.e. module). Turns out that the messages look somewhat 
>> different. This is a sample from the other module and it won't match with 
>> the current decoder. Should I add an OR to the parent decoder to do the 
>> first match and then use different subdecoders to extract the useful 
>> information from the other type of message? How do you deal with these type 
>> of scenarios?
>>
>> MESSAGE:
>> 1Mar2016 15:17:09 redirect st4600fw01n1 > Chrome; resource: http://
>> sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
>> src: 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: 
>> {0x56d5a2de,0x4,0xc50d2e0a,0xc001}; Protection name: Check Point - 
>> Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
>> Level: 5; severity: 2; malware_action: Communication with C site; 
>> rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
>> reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
>> protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: 
>> 10.46.5.133; product: Anti Malware; service: http; s_port: 61834;
>>
>> Best regards,
>> Fredrik 
>>
>>
>> On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>>>
>>> Hi All,
>>>
>>>
>>> Came across this where I think I would be helped by extracting fields 
>>> both in forward (from beginning) and in reverse (from end) order of 
>>> messages!? Is this possible, if so, is it stupid given that there are other 
>>> (better) ways to accomplish the same thing :/ ? 
>>>
>>> In addition to the fields my current decoder extracts, I was hoping to 
>>> extract the resource (http://www.aliveproxy.com/) and also the product 
>>> (Application 
>>> Control;). My idea was to add a secondary statement, before the  
>>> statement, something in the lines of:
>>> $/p\w+\s[...] and work my way backward 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Jesus Linares]

Hi Fredrik,

The expression "\.+" matches for anything. Usually, it is not a good idea 
because is slow and maybe you capture something that you don't want. So, *when 
it is possible*, it is better to use something specific.

When you have different decoders (different name) with the same parent, you 
should use a prematch. If you don't use prematch, it is fired the first 
rule. In the previous example:

Log:
Mar  3 12:15:24 LinMV TestDecoder[1963]: TypeB field1: hi; value2: bye; 
value3: seeyou

Without prematch:
**Phase 2: Completed decoding.
   decoder: 'TestDecoder'
   extra_data: 'seeyou'

With prematch:
**Phase 2: Completed decoding.
   decoder: 'TestDecoder'
   id: 'bye;'


Without prematch, the decoder is TestDecoder-1, but it should be 
TestDecoder2 (because it has the string "field1". In my view, it is a good 
practice use prematch, but sometimes it is no necessary.

Regarding your last question, could you use the same log format in your 
firewall and in the blade?. Paste here two logs of each one (firewall and 
blade) and your decoders, and we will take a look ;)

Regards.
Jesus Linares

On Friday, March 4, 2016 at 9:08:34 PM UTC+1, Fredrik wrote:
>
> Hi All,
>
>
> In this context and with your great response. What would you PROs suggest 
> I do when decoding another type of message from the same firewall - but a 
> different blade (i.e. module). Turns out that the messages look somewhat 
> different. This is a sample from the other module and it won't match with 
> the current decoder. Should I add an OR to the parent decoder to do the 
> first match and then use different subdecoders to extract the useful 
> information from the other type of message? How do you deal with these type 
> of scenarios?
>
> MESSAGE:
> 1Mar2016 15:17:09 redirect st4600fw01n1  Chrome; resource: http://
> sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; src: 
> 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: 
> {0x56d5a2de,0x4,0xc50d2e0a,0xc001}; Protection name: Check Point - 
> Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
> Level: 5; severity: 2; malware_action: Communication with C site; 
> rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
> reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
> protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: 
> 10.46.5.133; product: Anti Malware; service: http; s_port: 61834;
>
> Best regards,
> Fredrik 
>
>
> On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>>
>> Hi All,
>>
>>
>> Came across this where I think I would be helped by extracting fields 
>> both in forward (from beginning) and in reverse (from end) order of 
>> messages!? Is this possible, if so, is it stupid given that there are other 
>> (better) ways to accomplish the same thing :/ ? 
>>
>> In addition to the fields my current decoder extracts, I was hoping to 
>> extract the resource (http://www.aliveproxy.com/) and also the product 
>> (Application 
>> Control;). My idea was to add a secondary statement, before the  
>> statement, something in the lines of:
>> $/p\w+\s[...] and work my way backward so that I can extract 
>> Application Control and resource . How would you suggest I do this?! 
>>
>> Thanks again for all the great help - hope my threads (and questions) can 
>> be useful for other newstarters outhere trying to get there feet off the 
>> ground ;) 
>>
>> Best regards,
>> Fredrik 
>>
>> LOG-MESSAGE
>>
>> *Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 *allow > src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; 
>> app_desc: **; app_id: 10063753; app_category: **; 
>> matched_category: **; app_properties: **; app_risk: **; 
>> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
>> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: http:
>> //www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: Application 
>> Control; service: http; s_port: 58579; product_family: Network;
>>
>> MY CURRENT DECODER
>>
>> 
>>   ^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d
>>   firewall
>> 
>>
>> 
>>   Checkpoint
>>   (\w+) \p\w+ \w+ 
>> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>>   action,srcip,dstip
>> 
>>
>> LOGTEST OUTPUT
>>
>>
>> **Phase 1: Completed pre-decoding.
>>full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 
>> allow > appi_name: **; app_desc: **; app_id: 10063753; app_category: 
>> **; matched_category: **; app_properties: **; app_risk: **; 
>> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
>> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: 
>> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15; product: 
>> Application Control; service: http; s_port: 58579; product_family: Network;'
>>hostname: '127.0.0.1'
>>program_name: '(null)'
>>log: 'Jan 27 9:32:28 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hi All,


In this context and with your great response. What would you PROs suggest I 
do when decoding another type of message from the same firewall - but a 
different blade (i.e. module). Turns out that the messages look somewhat 
different. This is a sample from the other module and it won't match with 
the current decoder. Should I add an OR to the parent decoder to do the 
first match and then use different subdecoders to extract the useful 
information from the other type of message? How do you deal with these type 
of scenarios?

MESSAGE:
1Mar2016 15:17:09 redirect st4600fw01n1 http://sc1.checkpoint.com/za/images/threatwiki/pages/TestAntiBotBlade.html; 
src: 192.168.1.15; dst: 23.8.4.103; proto: tcp; session_id: 
{0x56d5a2de,0x4,0xc50d2e0a,0xc001}; Protection name: Check Point - 
Testing Bot; malware_family: Check Point; Source OS: Windows; Confidence 
Level: 5; severity: 2; malware_action: Communication with C site; 
rule_uid: {9AF67731-0D35-4117-AF2B-9A47F9396D26}; Protection Type: URL 
reputation; malware_rule_id: {00CE-00A4-0046-9658-621EA5468654}; 
protection_id: 00233CFEE; log_id: 2; proxy_src_ip: 192.168.1.15; scope: 
10.46.5.133; product: Anti Malware; service: http; s_port: 61834;

Best regards,
Fredrik 


On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>
> Hi All,
>
>
> Came across this where I think I would be helped by extracting fields both 
> in forward (from beginning) and in reverse (from end) order of messages!? 
> Is this possible, if so, is it stupid given that there are other (better) 
> ways to accomplish the same thing :/ ? 
>
> In addition to the fields my current decoder extracts, I was hoping to 
> extract the resource (http://www.aliveproxy.com/) and also the product 
> (Application 
> Control;). My idea was to add a secondary statement, before the  
> statement, something in the lines of:
> $/p\w+\s[...] and work my way backward so that I can extract 
> Application Control and resource . How would you suggest I do this?! 
>
> Thanks again for all the great help - hope my threads (and questions) can 
> be useful for other newstarters outhere trying to get there feet off the 
> ground ;) 
>
> Best regards,
> Fredrik 
>
> LOG-MESSAGE
>
> *Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 *allow  src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 10063753; app_category: **; matched_category
> : **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: Microsoft
> -IIS; app_sig_id: 10063753:5; resource: http://www.aliveproxy.com/; 
> proxy_src_ip: 192.168.1.15 product: Application Control; service: http; 
> s_port: 58579; product_family: Network;
>
> MY CURRENT DECODER
>
> 
>   ^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d
>   firewall
> 
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
> LOGTEST OUTPUT
>
>
> **Phase 1: Completed pre-decoding.
>full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 
> allow  appi_name: **; app_desc: **; app_id: 10063753; app_category: 
> **; matched_category: **; app_properties: **; app_risk: **; 
> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15; product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>hostname: '127.0.0.1'
>program_name: '(null)'
>log: 'Jan 27 9:32:28 st4600fw01n1 allow  192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; app_desc: 
> **; app_id: 10063753; app_category: **; matched_category: **; 
> app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: 
> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15; product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>
> **Phase 2: Completed decoding.
>decoder: 'Checkpoint'
>action: 'allow'
>srcip: '192.168.1.15'
>dstip: '89.208.212.2'
>
> **Phase 3: Completed filtering (rules).
>Rule id: '4100'
>Level: '0'
>Description: 'Firewall rules grouped.'
>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Hi (again) Jesus!

Big thanks for your time and effort! Greatly appreciated! 

I tried your example below, to better understand why each decoder should 
have a prematch. I haven't quite figured out yet, but I see (obviously) 
that different results are produced. What would be a good prematch in my 
example? Just so I got that part right. Giving two sections the same 
Checkpoint-alert in essence means that it is one 
decoder, but defined in two sections? 

Your recommendation not to use \. where alternatives are viable is this due 
to performance, or implications coming with being too wide (as opposed to 
narrow) in matching a string? 

Best regards,
Fredrik 

On Thursday, March 3, 2016 at 12:51:19 PM UTC+1, Jesus Linares wrote:
>
> Hi,
>
> I would add a *prematch *tag:
>
> 
>   Checkpoint
>   **
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
> 
>   Checkpoint
>   \.*resource: (\.*);\.*product: (\.*);
> 
>   url,extra_data
> 
>
>
> Each decoder must have a *prematch* tag. Try this example without *prematch 
> *and see what happens.
>
> 
> 
> TestDecoder
> 
>
>
> 
> TestDecoder
> TypeA
> value1: hi; value2: (\S+)
> id
> 
>
>
> 
> TestDecoder
> value3: (\S+)
> extra_data
> 
>
>
> 
> TestDecoder
> TypeB
> field1: hi; value2: (\S+)
> id
> 
>
>
> Also, when it is possible, try to don't use the character "\.". Maybe you 
> can do it whit \S+.
>
> Regards,
> Jesus Linares.
>
> On Thursday, March 3, 2016 at 10:05:16 AM UTC+1, Pedro S wrote:
>>
>> Hi Fredrik,
>>
>> I don't think OSSEC allow regex to work backwards, from end to beginning, 
>> I know that can be specify on other languages with some flags, but I am not 
>> sure if we can do that here. 
>>
>> Regarding to your decoder, we have two options, include the extraction of 
>> "resource" and product" fields on the same decoder:
>>
>> FULL DECODER
>>
>> 
>>   Checkpoint
>>   (\w+) \p\w+ \w+ 
>> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)\.*resource: 
>> (\.*);\.*product: (\.*);
>>   action,srcip,dstip,url,extra_data
>> 
>>
>> Or in a better way, separate the extraction in two different decoders, so 
>> we can be sure that in case of "resource" and "product" fields do not 
>> exists, our decoder still will parse and work.
>>
>> SPLIT DECODERS:
>>
>> 
>>   Checkpoint
>>   (\w+) \p\w+ \w+ 
>> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>>   action,srcip,dstip
>> 
>>
>>
>> 
>>   Checkpoint
>>   \.*resource: (\.*);\.*product: (\.*);
>> 
>>   url,extra_data
>> 
>>
>>
>>
>> LOGTEST OUTPUT
>> **Phase 1: Completed pre-decoding.
>>full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 
>> st4600fw01n1 allow > tcp; appi_name: **; app_desc: **; app_id: 10063753; app_category: 
>> **; matched_category: **; app_properties: **; app_risk: **; 
>> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
>> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: 
>> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: 
>> Application Control; service: http; s_port: 58579; product_family: Network;'
>>hostname: '127.0.0.1'
>>program_name: '(null)'
>>log: 'Jan 27 9:32:28 st4600fw01n1 allow > 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; app_desc: 
>> **; app_id: 10063753; app_category: **; matched_category: **; 
>> app_properties: **; app_risk: **; app_rule_id: **; 
>> app_rule_name: **; web_client_type: Chrome; web_server_type: 
>> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
>> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: 
>> Application Control; service: http; s_port: 58579; product_family: Network;'
>>
>>
>> **Phase 2: Completed decoding.
>>decoder: 'Checkpoint'
>>action: 'allow'
>>srcip: '192.168.1.15'
>>dstip: '89.208.212.2'
>> *   url*
>> *: 'http://www.aliveproxy.com/ '  
>>  extra_data: 'Application Control'*
>>
>>
>> **Phase 3: Completed filtering (rules).
>>Rule id: '4100'
>>Level: '0'
>>Description: 'Firewall rules grouped.'
>>
>>
>> In both decoders, I am using wildcards *.* *and expecting always "
>> *resource*" before "*product*", either way won't work. 
>>
>> You asked about using another "regex" line in the same decoder, it will 
>> work too, like this:
>>
>> 
>>   Checkpoint
>>   (\w+) \p\w+ \w+ 
>> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>>   *\.*resource: (\.*);\.*product: (\.*);*
>>   action,srcip,dstip, url, extra_data
>> 
>>
>>
>>
>> Best regards,
>>
>> Pedro S.
>>
>>
>>
>> On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>>>
>>> Hi All,
>>>
>>>
>>> Came across this where I think I would be helped by extracting fields 
>>> both in forward (from beginning) and in reverse (from end) order of 
>>> messages!? Is this possible, if so, is it stupid given that there are 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Fredrik]

Fantastic! Thank you so much! Makes sense what you wrote about my suggested 
(mis)use of regex to try and get it start reading backwards. I should have 
realised that wildcard (*) was the way to go, without having to match the 
full string with the expressions available. I guess Jesus' recommendation 
not to use \. where alternatives are viable is due to performance? (Feel 
free to chime in Jesus :)) . Anyway, thanks alot. I learned a great deal 
from your suggestions, that should come in handy as write more decoders :)

/f

On Thursday, March 3, 2016 at 10:05:16 AM UTC+1, Pedro S wrote:
>
> Hi Fredrik,
>
> I don't think OSSEC allow regex to work backwards, from end to beginning, 
> I know that can be specify on other languages with some flags, but I am not 
> sure if we can do that here. 
>
> Regarding to your decoder, we have two options, include the extraction of 
> "resource" and product" fields on the same decoder:
>
> FULL DECODER
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)\.*resource: 
> (\.*);\.*product: (\.*);
>   action,srcip,dstip,url,extra_data
> 
>
> Or in a better way, separate the extraction in two different decoders, so 
> we can be sure that in case of "resource" and "product" fields do not 
> exists, our decoder still will parse and work.
>
> SPLIT DECODERS:
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
>   Checkpoint
>   \.*resource: (\.*);\.*product: (\.*);
> 
>   url,extra_data
> 
>
>
>
> LOGTEST OUTPUT
> **Phase 1: Completed pre-decoding.
>full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 
> allow  appi_name: **; app_desc: **; app_id: 10063753; app_category: 
> **; matched_category: **; app_properties: **; app_risk: **; 
> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>hostname: '127.0.0.1'
>program_name: '(null)'
>log: 'Jan 27 9:32:28 st4600fw01n1 allow  192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; app_desc: 
> **; app_id: 10063753; app_category: **; matched_category: **; 
> app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: 
> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>
>
> **Phase 2: Completed decoding.
>decoder: 'Checkpoint'
>action: 'allow'
>srcip: '192.168.1.15'
>dstip: '89.208.212.2'
> *   url*
> *: 'http://www.aliveproxy.com/ '  
>  extra_data: 'Application Control'*
>
>
> **Phase 3: Completed filtering (rules).
>Rule id: '4100'
>Level: '0'
>Description: 'Firewall rules grouped.'
>
>
> In both decoders, I am using wildcards *.* *and expecting always "
> *resource*" before "*product*", either way won't work. 
>
> You asked about using another "regex" line in the same decoder, it will 
> work too, like this:
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   *\.*resource: (\.*);\.*product: (\.*);*
>   action,srcip,dstip, url, extra_data
> 
>
>
>
> Best regards,
>
> Pedro S.
>
>
>
> On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>>
>> Hi All,
>>
>>
>> Came across this where I think I would be helped by extracting fields 
>> both in forward (from beginning) and in reverse (from end) order of 
>> messages!? Is this possible, if so, is it stupid given that there are other 
>> (better) ways to accomplish the same thing :/ ? 
>>
>> In addition to the fields my current decoder extracts, I was hoping to 
>> extract the resource (http://www.aliveproxy.com/) and also the product 
>> (Application 
>> Control;). My idea was to add a secondary statement, before the  
>> statement, something in the lines of:
>> $/p\w+\s[...] and work my way backward so that I can extract 
>> Application Control and resource . How would you suggest I do this?! 
>>
>> Thanks again for all the great help - hope my threads (and questions) can 
>> be useful for other newstarters outhere trying to get there feet off the 
>> ground ;) 
>>
>> Best regards,
>> Fredrik 
>>
>> LOG-MESSAGE
>>
>> *Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 *allow > src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; 
>> app_desc: **; app_id: 10063753; app_category: **; 
>> matched_category: **; app_properties: **; app_risk: **; 
>> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
>> web_server_type: Microsoft-IIS; 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Jesus Linares]

Hi,

I would add a *prematch *tag:


  Checkpoint
  **
  (\w+) \p\w+ \w+ 
src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
  action,srcip,dstip



  Checkpoint
  \.*resource: (\.*);\.*product: (\.*);
  url,extra_data



Each decoder must have a *prematch* tag. Try this example without *prematch 
*and see what happens.



TestDecoder




TestDecoder
TypeA
value1: hi; value2: (\S+)
id




TestDecoder
value3: (\S+)
extra_data




TestDecoder
TypeB
field1: hi; value2: (\S+)
id



Also, when it is possible, try to don't use the character "\.". Maybe you 
can do it whit \S+.

Regards,
Jesus Linares.

On Thursday, March 3, 2016 at 10:05:16 AM UTC+1, Pedro S wrote:
>
> Hi Fredrik,
>
> I don't think OSSEC allow regex to work backwards, from end to beginning, 
> I know that can be specify on other languages with some flags, but I am not 
> sure if we can do that here. 
>
> Regarding to your decoder, we have two options, include the extraction of 
> "resource" and product" fields on the same decoder:
>
> FULL DECODER
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)\.*resource: 
> (\.*);\.*product: (\.*);
>   action,srcip,dstip,url,extra_data
> 
>
> Or in a better way, separate the extraction in two different decoders, so 
> we can be sure that in case of "resource" and "product" fields do not 
> exists, our decoder still will parse and work.
>
> SPLIT DECODERS:
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
>
> 
>   Checkpoint
>   \.*resource: (\.*);\.*product: (\.*);
> 
>   url,extra_data
> 
>
>
>
> LOGTEST OUTPUT
> **Phase 1: Completed pre-decoding.
>full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 
> allow  appi_name: **; app_desc: **; app_id: 10063753; app_category: 
> **; matched_category: **; app_properties: **; app_risk: **; 
> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>hostname: '127.0.0.1'
>program_name: '(null)'
>log: 'Jan 27 9:32:28 st4600fw01n1 allow  192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; app_desc: 
> **; app_id: 10063753; app_category: **; matched_category: **; 
> app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: 
> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>
>
> **Phase 2: Completed decoding.
>decoder: 'Checkpoint'
>action: 'allow'
>srcip: '192.168.1.15'
>dstip: '89.208.212.2'
> *   url*
> *: 'http://www.aliveproxy.com/ '  
>  extra_data: 'Application Control'*
>
>
> **Phase 3: Completed filtering (rules).
>Rule id: '4100'
>Level: '0'
>Description: 'Firewall rules grouped.'
>
>
> In both decoders, I am using wildcards *.* *and expecting always "
> *resource*" before "*product*", either way won't work. 
>
> You asked about using another "regex" line in the same decoder, it will 
> work too, like this:
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   *\.*resource: (\.*);\.*product: (\.*);*
>   action,srcip,dstip, url, extra_data
> 
>
>
>
> Best regards,
>
> Pedro S.
>
>
>
> On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>>
>> Hi All,
>>
>>
>> Came across this where I think I would be helped by extracting fields 
>> both in forward (from beginning) and in reverse (from end) order of 
>> messages!? Is this possible, if so, is it stupid given that there are other 
>> (better) ways to accomplish the same thing :/ ? 
>>
>> In addition to the fields my current decoder extracts, I was hoping to 
>> extract the resource (http://www.aliveproxy.com/) and also the product 
>> (Application 
>> Control;). My idea was to add a secondary statement, before the  
>> statement, something in the lines of:
>> $/p\w+\s[...] and work my way backward so that I can extract 
>> Application Control and resource . How would you suggest I do this?! 
>>
>> Thanks again for all the great help - hope my threads (and questions) can 
>> be useful for other newstarters outhere trying to get there feet off the 
>> ground ;) 
>>
>> Best regards,
>> Fredrik 
>>
>> LOG-MESSAGE
>>
>> *Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 *allow > src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; 
>> app_desc: **; app_id: 10063753; app_category: **; 
>> matched_category: **; app_properties: **; 

[ossec-list] Re: Decoding long messages - multiple regex statements

[Pedro S]

Hi Fredrik,

I don't think OSSEC allow regex to work backwards, from end to beginning, I 
know that can be specify on other languages with some flags, but I am not 
sure if we can do that here. 

Regarding to your decoder, we have two options, include the extraction of 
"resource" and product" fields on the same decoder:

FULL DECODER


  Checkpoint
  (\w+) \p\w+ \w+ 
src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)\.*resource: 
(\.*);\.*product: (\.*);
  action,srcip,dstip,url,extra_data


Or in a better way, separate the extraction in two different decoders, so 
we can be sure that in case of "resource" and "product" fields do not 
exists, our decoder still will parse and work.

SPLIT DECODERS:


  Checkpoint
  (\w+) \p\w+ \w+ 
src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
  action,srcip,dstip




  Checkpoint
  \.*resource: (\.*);\.*product: (\.*);
  url,extra_data




LOGTEST OUTPUT
**Phase 1: Completed pre-decoding.
   full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 
allow http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: Application 
Control; service: http; s_port: 58579; product_family: Network;'
   hostname: '127.0.0.1'
   program_name: '(null)'
   log: 'Jan 27 9:32:28 st4600fw01n1 allow http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15 product: Application 
Control; service: http; s_port: 58579; product_family: Network;'


**Phase 2: Completed decoding.
   decoder: 'Checkpoint'
   action: 'allow'
   srcip: '192.168.1.15'
   dstip: '89.208.212.2'
*   url*
*: 'http://www.aliveproxy.com/'   extra_data: 'Application Control'*


**Phase 3: Completed filtering (rules).
   Rule id: '4100'
   Level: '0'
   Description: 'Firewall rules grouped.'


In both decoders, I am using wildcards *.* *and expecting always "*resource*" 
before "*product*", either way won't work. 

You asked about using another "regex" line in the same decoder, it will 
work too, like this:


  Checkpoint
  (\w+) \p\w+ \w+ 
src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
  *\.*resource: (\.*);\.*product: (\.*);*
  action,srcip,dstip, url, extra_data




Best regards,

Pedro S.



On Wednesday, March 2, 2016 at 11:03:08 AM UTC+1, Fredrik wrote:
>
> Hi All,
>
>
> Came across this where I think I would be helped by extracting fields both 
> in forward (from beginning) and in reverse (from end) order of messages!? 
> Is this possible, if so, is it stupid given that there are other (better) 
> ways to accomplish the same thing :/ ? 
>
> In addition to the fields my current decoder extracts, I was hoping to 
> extract the resource (http://www.aliveproxy.com/) and also the product 
> (Application 
> Control;). My idea was to add a secondary statement, before the  
> statement, something in the lines of:
> $/p\w+\s[...] and work my way backward so that I can extract 
> Application Control and resource . How would you suggest I do this?! 
>
> Thanks again for all the great help - hope my threads (and questions) can 
> be useful for other newstarters outhere trying to get there feet off the 
> ground ;) 
>
> Best regards,
> Fredrik 
>
> LOG-MESSAGE
>
> *Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 *allow  src: 192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; 
> app_desc: **; app_id: 10063753; app_category: **; matched_category
> : **; app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: Microsoft
> -IIS; app_sig_id: 10063753:5; resource: http://www.aliveproxy.com/; 
> proxy_src_ip: 192.168.1.15 product: Application Control; service: http; 
> s_port: 58579; product_family: Network;
>
> MY CURRENT DECODER
>
> 
>   ^\w+ \d+ \d+:\d+:\d+ st4600fw01n\d
>   firewall
> 
>
> 
>   Checkpoint
>   (\w+) \p\w+ \w+ 
> src:\s(\d+.\d+.\d+.\d+);\sdst:\s(\d+.\d+.\d+.\d+)
>   action,srcip,dstip
> 
>
> LOGTEST OUTPUT
>
>
> **Phase 1: Completed pre-decoding.
>full event: 'Jan 27 09:41:01 127.0.0.1 Jan 27 9:32:28 st4600fw01n1 
> allow  appi_name: **; app_desc: **; app_id: 10063753; app_category: 
> **; matched_category: **; app_properties: **; app_risk: **; 
> app_rule_id: **; app_rule_name: **; web_client_type: Chrome; 
> web_server_type: Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/; proxy_src_ip: 192.168.1.15; product: 
> Application Control; service: http; s_port: 58579; product_family: Network;'
>hostname: '127.0.0.1'
>program_name: '(null)'
>log: 'Jan 27 9:32:28 st4600fw01n1 allow  192.168.1.15; dst: 89.208.212.2; proto: tcp; appi_name: **; app_desc: 
> **; app_id: 10063753; app_category: **; matched_category: **; 
> app_properties: **; app_risk: **; app_rule_id: **; 
> app_rule_name: **; web_client_type: Chrome; web_server_type: 
> Microsoft-IIS; app_sig_id: 10063753:5; resource: 
> http://www.aliveproxy.com/;