Dan, that was extremely informative and helpful. And cleared up so much of the confusion I was having with writing decoders! Thanks a bunch! I'm still working through it but the initial efforts/changes have been productive! I will post more once I have things setup more concretely.
Thanks again! On Sun, Sep 19, 2010 at 1:39 PM, dan (ddp) <[email protected]> wrote: > This email went through a few different iterations, so if something > seems wonky let me know. It just means I didn't proofread the final > very well and I don't mind fixing it. > > On Sun, Sep 19, 2010 at 2:43 PM, Jeremy Lee <[email protected]> wrote: > > Hey guys, > > > > I have another one I can't figure out. > > > > A couple sample logs: > > > > Sep 18 08:01:03 linksux ser2sock[2515]: > > [000003011000--------],004,[f70000071004030028020000000000],"FAULT > > 04 "#012 > > > > Sep 19 11:37:11 linksux ser2sock[2515]: > > [100000011000--------],008,[f70000071008001c28020000000000]," DISARMED > > CHIME Ready to Arm "#012 > > > > > > My decoder rule - I checked this on a regex builder site, but it may have > > been different than what OSSEC uses: > > > > <decoder name="ad2usb"> > > <!-- <prematch>\S\d+\S+</prematch> --> > > > <prematch>^\w+\s\d*\d*\s\d+\S\d+\S\d+\s\w+\s\w+\d\w+\S\d+\S+\s</prematch> > > <regex > > > offset="after_prematch">\S\d+\S+\d+\S+\w\d+\w\d+\S+\s*\w+\s*\S*\S*\S*\w+\S*\S*\S*\s*\w+\s*\w+\s*\w+\s*\S+</regex> > > <order>id</order> > > </decoder> > > > > You have the <order> tag, but nothing is in parenthesis in the regex > line. The things in parenthesis are what will populate the entries in > <order>. > You've also complicated this A LOT. Start simple. VERY basic decoder > for alerts with ser2sock: > <decoder name="forj"> > <program_name>^ser2sock</program_name> > </decoder> > > I'll try to break this down into pieces with the following log you > provided: > Sep 18 08:01:03 linksux ser2sock[2515]: > [000003011000--------],004,[f70000071004030028020000000000],"FAULT 04 > "#012 > > 'Sep 18 08:01:03 ' This is the timestamp. This is a fairly standard > format so ossec recognizes it nicely. Ignore it. > 'linksux ' This is the hostname of the server. This is fairly > standard. Ignore it unless you need it in a rule. > 'ser2sock' This is the application name. This will populate > <program_name> in a decoder. This is the easiest way to make a basic > decoder for an application. > '[2515]: ' This is the pid of the program name. Ignore it. > '[000003011000--------],004,[f70000071004030028020000000000],"FAULT 04 > "#012' This is your log message. > > The decoder I pasted above will recognize all log messages from > ser2sock. Example (using the second log message you posted and > ossec-logtest): > > **Phase 1: Completed pre-decoding. > full event: 'Sep 19 11:37:11 linksux ser2sock[2515]: > [100000011000--------],008,[f70000071008001c28020000000000]," DISARMED > CHIME Ready to Arm "#012' > hostname: 'linksux' > program_name: 'ser2sock' > log: '[100000011000--------],008,[f70000071008001c28020000000000]," > DISARMED CHIME Ready to Arm "#012' > > **Phase 2: Completed decoding. > decoder: 'forj' > ------------------------------------------------------ > > Now that we have the basics out of the way, we want to pull out an > 'id' from the message. Which part of the log is the "id?" I'll assume > it is the > 000003011000 and 100000011000 in the messages for this example. If I > guessed wrong you can hopefully use the information I'm providing to > fix it. If you try and don't succeed, reply and I'll help. > > Assuming all ser2sock messages are VERY similar we'll just add to the > decoder above. If they aren't a child decoder can be created instead > so multiple message formats can be supported. > > Decoder from above: > <decoder name="forj"> > <program_name>^ser2sock</program_name> > </decoder> > > Adding the 'id': > <decoder name="forj"> > <program_name>^ser2sock</program_name> > <regex>^\p(\d+)\p\.+#\d+$</regex> > <order>id</order> > </decoder> > > The id field will be populated with the string of numbers near the > beginning the log message thanks to the parenthesis. Using this new > decoder we get the following information via ossec-logtest: > Sep 18 08:01:03 linksux ser2sock[2515]: > [000003011000--------],004,[f70000071004030028020000000000],"FAULT 04 > "#012 > > > **Phase 1: Completed pre-decoding. > full event: 'Sep 18 08:01:03 linksux ser2sock[2515]: > [000003011000--------],004,[f70000071004030028020000000000],"FAULT 04 > "#012' > hostname: 'linksux' > program_name: 'ser2sock' > log: > '[000003011000--------],004,[f70000071004030028020000000000],"FAULT > 04 "#012' > > **Phase 2: Completed decoding. > decoder: 'forj' > id: '000003011000' > > Sep 19 11:37:11 linksux ser2sock[2515]: > [100000011000--------],008,[f70000071008001c28020000000000]," DISARMED > CHIME Ready to Arm "#012 > > > **Phase 1: Completed pre-decoding. > full event: 'Sep 19 11:37:11 linksux ser2sock[2515]: > [100000011000--------],008,[f70000071008001c28020000000000]," DISARMED > CHIME Ready to Arm "#012' > hostname: 'linksux' > program_name: 'ser2sock' > log: '[100000011000--------],008,[f70000071008001c28020000000000]," > DISARMED CHIME Ready to Arm "#012' > > **Phase 2: Completed decoding. > decoder: 'forj' > id: '100000011000' > > ---------------------------------------------------------------- > > We now have an id. > > Breakdown of the regex: > <regex>^\p(\d+)\p\.+#\d+$</regex> > '^' This is the beginning of the log message. > '\p' This is a stand in for the '[' character. It actually matches a > number of characters, not just this one. > '(' This is the beginning of a section we would like to pull out of > the log message and store in a variable (id). > '\d+' This represents 1 or more numeric characters. This is the data > that will populate the id field. If this information could be > alpha-numeric, this will have to change. > ')' This is the end of the section we want to populate a variable. > Nothing after this will be put into the first variable. > '\p' There is a dash at the end of the string of numbers, and the \p > represents it. I don't know if some of your messages may include > longer numbers without the dashes. In those cases the \p will > represent the ending ']'. > '\.+' This means 1 or more characters. Any characters. > '"#\d+' These are the characters immediately following what we wanted > in the id variable ("#), and the numbers at the end. Using '\d+" means > that there will be 1 or more numbers (012 in this case). > '$' This is the end of the message. > > Now it's time to add rules to these decoders. (see below) > > > > > And lastly, the rules, I setup: > > > > <group name="local,syslog,errors,openvpnals,windows,ad2usb"> > > > > <rule id="100040" level="0"> // I also tried setting the level to 5 > > <decoded_as>ad2usb</decoded_as> > > <description>AD2USB Alarm Status.</description> > > </rule> > > > > <rule id="100041" level="12"> > > <if_sid>100040</if_sid> > > <match>100000011000</match> > > <description>Alarm armed</description> > > </rule> > > > > </group> > > > > Any log message that matches the "forj" decoder will trigger the > following rule (becuase of the <decoded_as> option): > <rule id="490019" level="0" noalert="1"> > <decoded_as>forj</decoded_as> > <description>Collected ser2sock alerts.</description> > </rule> > > However, the 'noalert="1"' setting will make this not show up as an > alert. We will use it to collect all log messages for ser2sock in one > place. So a followup rule will be necessary to complete this. > > So here is a second rule to alert on id '100000011000'. > <rule id="490020" level="12"> > <if_sid>490019</if_sid> > <id>100000011000</id> > <description>blah blah blah</description> > </rule> > > We have 2 fields that look for information in the log message in this > rule. First the log message has to match rule id 490019 (the first > rule above), as will any log message triggered by ser2sock. Second, we > look for the id field being populated by '100000011000' as the second > log message does. > > Now we feed the log messages through ossec-logtest again and get the > following: > Sep 18 08:01:03 linksux ser2sock[2515]: > [000003011000--------],004,[f70000071004030028020000000000],"FAULT 04 > "#012 > > > **Phase 1: Completed pre-decoding. > full event: 'Sep 18 08:01:03 linksux ser2sock[2515]: > [000003011000--------],004,[f70000071004030028020000000000],"FAULT 04 > "#012' > hostname: 'linksux' > program_name: 'ser2sock' > log: > '[000003011000--------],004,[f70000071004030028020000000000],"FAULT > 04 "#012' > > **Phase 2: Completed decoding. > decoder: 'forj' > id: '000003011000' > > Sep 19 11:37:11 linksux ser2sock[2515]: > [100000011000--------],008,[f70000071008001c28020000000000]," DISARMED > CHIME Ready to Arm "#012 > > > **Phase 1: Completed pre-decoding. > full event: 'Sep 19 11:37:11 linksux ser2sock[2515]: > [100000011000--------],008,[f70000071008001c28020000000000]," DISARMED > CHIME Ready to Arm "#012' > hostname: 'linksux' > program_name: 'ser2sock' > log: '[100000011000--------],008,[f70000071008001c28020000000000]," > DISARMED CHIME Ready to Arm "#012' > > **Phase 2: Completed decoding. > decoder: 'forj' > id: '100000011000' > > **Phase 3: Completed filtering (rules). > Rule id: '490020' > Level: '12' > Description: 'blah blah blah' > **Alert to be generated. > -------------------------------------------- > > Notice how the first message only makes it to Phase 2? The id field > for that message is populated with '000003011000' and there is no rule > associated with that content. A third rule would need to be setup for > that message (leaving this as an exercise for the reader, PLEASE post > if you get it/don't get it). > The second message makes it to Phase 3 since there is a rule > associated with that id. It is triggered and ossec-logtest tells you > that an alert would be sent out. > > > > > > I have debug turned on for analyzer, logcollector, and agent (on the > server > > and agent) and I see the logcolelctor reading hte file on the agent but > the > > server doesn't display any log messages (I'm guessing it would?). > > Interestingly, I echoed the same log message into /var/log/syslog and > > /var/log/messages on the server, which are supposed to be monitored by > OSSEC > > as well, but I didn't see anything show up from the debug output like it > > wason the OSSEC agent. > > > > > > > > > > > > Using ossec-logtest makes this easier/cleaner. Using ossec-logtest you > don't have to populate your real log files with junk. You could also > use logger to do this if you can't use ossec-logtest for some reason. > Example: > echo '[100000011000--------],008,[f70000071008001c28020000000000]," > DISARMED CHIME Ready to Arm "#012' | logger > > I usually write decoders and rules in pieces. I put in a basic decoder > (like the first one I posted above), and run ossec-logtest. If it > works I can then modify it do more (like adding the regex/order > lines). I often have to do the regex lines themselves in a similar > way, starting off very small and building it up as I get the syntax > correct. Run ossec-logtest between steps to make sure everything is > still working. > > > If anyone has comments or questions about the information above please > (please please please) let me know (on list if it's topical to this > thread, offlist or IRC if it isn't). I'm thinking about doing some > kind of post along these lines for the Week of OSSEC, and would love > feedback so I can make it better/clearer for everyone. >
