Hi Risto,
Thank You once again for your kind comment.
Actually, my script my.pl is used for taking out the syslog events from X
timestamp [intial time] to Y timestamp [New time after adding some
seconds]. This intial timestamp is determined by searching for particular
pattern [ALARM RAISE] of event in syslog.Finally printing all the syslog
events from X to Y timestamps.
I wanted to pipe this script output to SEC again, I have a second rule in
my .conf file which can match event in this script output. I'm implementing
all beacause my inputs are non realtime events.
Please try to make some time avaliable for yourself to check the below
details of my project.
I am working on a project for log correlation engine where we want to have
tool which can check if the issue reported is already known or not and
hence I thought of using SEC for this. The below is the abstract info about
the project and my requirement.
*Scenario:*
Basically our idea is to have a database of signatures ( conf files etc )
which consists of issues found in our product ( means set of events in
various log files which point to particular issue ) and we just want to run
this tool on new issue/error reported on our product and see if this issue
was already existing.
*Example: *
We have two logfiles, logfile1 and logfile2 and we want to know is a
pattern is seen in these logs which was reported in earlier known cases
*Logfile1 snippet:*
Aug 21 21:16:00.168055 info CFPU-1 rsyslogd: file /srv/Log/log/syslog was
successfully rotated
Aug 21 21:16:00.376903 info CFPU-0 ResourceMeas: COMMUTIL INFO [Ag]
Measurement job created, data for all objects is reported by default
Aug 21 21:16:01.319159 info CFPU-1 crond[30812]: (root) CMD (nice
/opt/nokiasiemens/bin/timestamp.sh)
*Logfile2 snippet:*
Aug 21 21:16:14.35245 info tid=0x93801200 <== set_state_and_signal -
tx_state TS_ACTIVE -> TS_PROCESSING
Aug 21 21:17:14.35540 info tid=0x77001200 <== set_state_and_signal -
tx_state TS_PROCESSING -> TS_SEARCHING_IN_PROCESSING
*Now my project is to check the Logfile1 for following event*:
rsyslogd: file /srv/Log/log/syslog was successfully rotated
*And this should be followed by another in logfile2:*
35245 info tid=0x93801200 <== set_state_and_signal - tx_state TS_ACTIVE ->
TS_PROCESSING
And both of these event should have occurred in *a span of 1 minute* for
correlation to succeed.
Limitation is that we will not be using these files at runtime means these
files ar not having same time as of system ( basically timestamps in
logfiles are not same as system time )
*So kindly help me with below*:
1) 1) Is SEC the right tool for this usecase ?
2) 2) Are there any examples to looks for these ?
3) 3) Do we have detailed knowledge base for all
features/keyword/functionality of SEC with examples ?
Thanks in advance and look forward for your advice.
Thanks & Regards,
Karthik
On Thu, Jul 2, 2015 at 4:07 PM, Risto Vaarandi <risto.vaara...@gmail.com>
wrote:
> ...to complement my previous mail, I think it is simpler to set up
> dedicated perl scripts for preparing input for sec, rather than trying to
> incorporate data preprocessing into sec rules.
>
> Here are two perl scripts which follow the example Mark has already
> provided. Firstly, generate_timestamp.pl looks like follows (note that it
> currently works for timestamps like "2015 Jun 20 12:00:00" and "Jun 21
> 12:13:41", and for other formats you need to adjust the timestamp parsing
> regular expression):
>
> #!/usr/bin/perl -w
> #
> # generate_timestamp.pl
> # Prepend 'seconds since epoch' to each input line
>
> use Time::Local;
>
> %months = ( 'Jan' => 0, 'Feb' => 1, 'Mar' => 2, 'Apr' => 3,
> 'May' => 4, 'Jun' => 5, 'Jul' => 6, 'Aug' => 7,
> 'Sep' => 8, 'Oct' => 9, 'Nov' => 10, 'Dec' => 11 );
>
> @time = localtime(time());
>
> $year = $time[5] + 1900;
> $month = $time[4];
>
> while (<STDIN>) {
> if
> (/^(?:(\d{4})\s+)?([A-Z][a-z]{2})\s+(\d{1,2})\s+(\d{2}):(\d{2}):(\d{2})/) {
> $y = defined($1)?$1:$year;
> $m = exists($months{$2})?$months{$2}:$month;
> $time = timelocal($6, $5, $4, $3, $m, $y);
> print "$time ", $_;
> }
> }
>
>
> And then the replay_events.pl script:
>
> #!/usr/bin/perl -w
> #
> # replay_events.pl
> # Replay sorted events generated by generate_timestamp.pl
>
> select STDOUT;
> $| = 1;
>
> while (<STDIN>) {
>
> if ($_ !~ /^(\d+) (.*)/) { next; }
>
> if (!defined($previous)) {
> print $2, "\n";
> $previous = $1;
> next;
> }
>
> $d = $1 - $previous;
> if ($d < 0) { next; }
> sleep($d);
> print $2, "\n";
> $previous = $1;
> }
>
>
> After having those scripts implemented, replaying past events becomes a
> matter of simple UNIX pipeline. For example, the following pipeline joins
> /var/log/messages and /var/log/secure into a single event stream and
> replays this to sec:
>
> cat /var/log/messages /var/log/secure | ./generate_timestamp.pl | sort |
> ./replay_events.pl | /usr/bin/sec --conf=test.conf --input=- --notail
>
> Also, I would suggest to include any database queries into a dedicated
> preprocessing script which precedes sec in the UNIX pipeline.
>
> To summarize, I strongly believe that the clear separation of data
> preprocessing from sec-based event correlation is the best solution for
> you, in order to keep your configuration manageable and efficient.
>
> hope this helps,
> risto
>
> 2015-07-01 13:15 GMT+03:00 Rajesh M <rajesh68.embed...@gmail.com>:
>
>> Hi Risto,
>>
>> I am implementing a perl script which basically accepts the input file
>> and time in sec, after search for particular pattern and take out the time
>> stamp and adds input time to that, generates "New Time stamp". Later the
>> output is syslog events from input file between Original TS and New TS.
>>
>> my .conf file:
>> ------------------
>> type=Single
>> ptype=RegExp
>> pattern="ALARM RAISE 70307" [Very first rasie event in alarm.log]
>> desc=$0
>> action=spawn /var/test/my.pl
>>
>> type=Single
>> ptype=RegExp
>> pattern=.*logf started
>> desc=Matched event
>> action=write /home/test.out
>>
>>
>> Basically I am trying to trigger the scrpit based on my 1st rule match
>> and wants to fed the script output to SEC for 2nd rule match.
>>
>> Whenever I was executing this .conf, the script was calling and it
>> wouldn't goto 2nd rule and also the script was exectuing how many number of
>> times the 1st rule matches.Please suggest what is the correct way of doing
>> this operation in SEC.
>>
>> Thanks & Regards,
>> Karthik
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>> https://www.gigenetcloud.com/
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>>
>
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
