Hi Risto,
As you pointed out, we couldn't able to implement all the functionality
with SEC while giving non real-time inputs.
We had integrated the SEC with our running cluster so that SEC will run as
a background process all the time and takes the real-time system logs as
inputs for processing.
Right now as part of testing we are running SEC with only one .conf file
with few rules in it and passing this real-time syslogs as inputs.
But, in future we have more .conf files like sshd.conf, restart.conf,
link_down.conf, etc....etc.
We are in way to customize SEC that, it will take all the .conf's files at
a time and start processing the inputs coming from the cluster.
Are there any ways in implementing the above functionality in SEC?, if not
please suggest us the alternatives to achieve this.
Thanks in advance.
Regards,
Karthik
On Thu, Jul 2, 2015 at 4:07 PM, Risto Vaarandi <risto.vaara...@gmail.com>
wrote:
> ...to complement my previous mail, I think it is simpler to set up
> dedicated perl scripts for preparing input for sec, rather than trying to
> incorporate data preprocessing into sec rules.
>
> Here are two perl scripts which follow the example Mark has already
> provided. Firstly, generate_timestamp.pl looks like follows (note that it
> currently works for timestamps like "2015 Jun 20 12:00:00" and "Jun 21
> 12:13:41", and for other formats you need to adjust the timestamp parsing
> regular expression):
>
> #!/usr/bin/perl -w
> #
> # generate_timestamp.pl
> # Prepend 'seconds since epoch' to each input line
>
> use Time::Local;
>
> %months = ( 'Jan' => 0, 'Feb' => 1, 'Mar' => 2, 'Apr' => 3,
> 'May' => 4, 'Jun' => 5, 'Jul' => 6, 'Aug' => 7,
> 'Sep' => 8, 'Oct' => 9, 'Nov' => 10, 'Dec' => 11 );
>
> @time = localtime(time());
>
> $year = $time[5] + 1900;
> $month = $time[4];
>
> while (<STDIN>) {
> if
> (/^(?:(\d{4})\s+)?([A-Z][a-z]{2})\s+(\d{1,2})\s+(\d{2}):(\d{2}):(\d{2})/) {
> $y = defined($1)?$1:$year;
> $m = exists($months{$2})?$months{$2}:$month;
> $time = timelocal($6, $5, $4, $3, $m, $y);
> print "$time ", $_;
> }
> }
>
>
> And then the replay_events.pl script:
>
> #!/usr/bin/perl -w
> #
> # replay_events.pl
> # Replay sorted events generated by generate_timestamp.pl
>
> select STDOUT;
> $| = 1;
>
> while (<STDIN>) {
>
> if ($_ !~ /^(\d+) (.*)/) { next; }
>
> if (!defined($previous)) {
> print $2, "\n";
> $previous = $1;
> next;
> }
>
> $d = $1 - $previous;
> if ($d < 0) { next; }
> sleep($d);
> print $2, "\n";
> $previous = $1;
> }
>
>
> After having those scripts implemented, replaying past events becomes a
> matter of simple UNIX pipeline. For example, the following pipeline joins
> /var/log/messages and /var/log/secure into a single event stream and
> replays this to sec:
>
> cat /var/log/messages /var/log/secure | ./generate_timestamp.pl | sort |
> ./replay_events.pl | /usr/bin/sec --conf=test.conf --input=- --notail
>
> Also, I would suggest to include any database queries into a dedicated
> preprocessing script which precedes sec in the UNIX pipeline.
>
> To summarize, I strongly believe that the clear separation of data
> preprocessing from sec-based event correlation is the best solution for
> you, in order to keep your configuration manageable and efficient.
>
> hope this helps,
> risto
>
> 2015-07-01 13:15 GMT+03:00 Rajesh M <rajesh68.embed...@gmail.com>:
>
>> Hi Risto,
>>
>> I am implementing a perl script which basically accepts the input file
>> and time in sec, after search for particular pattern and take out the time
>> stamp and adds input time to that, generates "New Time stamp". Later the
>> output is syslog events from input file between Original TS and New TS.
>>
>> my .conf file:
>> ------------------
>> type=Single
>> ptype=RegExp
>> pattern="ALARM RAISE 70307" [Very first rasie event in alarm.log]
>> desc=$0
>> action=spawn /var/test/my.pl
>>
>> type=Single
>> ptype=RegExp
>> pattern=.*logf started
>> desc=Matched event
>> action=write /home/test.out
>>
>>
>> Basically I am trying to trigger the scrpit based on my 1st rule match
>> and wants to fed the script output to SEC for 2nd rule match.
>>
>> Whenever I was executing this .conf, the script was calling and it
>> wouldn't goto 2nd rule and also the script was exectuing how many number of
>> times the 1st rule matches.Please suggest what is the correct way of doing
>> this operation in SEC.
>>
>> Thanks & Regards,
>> Karthik
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>> https://www.gigenetcloud.com/
>> _______________________________________________
>> Simple-evcorr-users mailing list
>> Simple-evcorr-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>>
>>
>
------------------------------------------------------------------------------
Don't Limit Your Business. Reach for the Cloud.
GigeNET's Cloud Solutions provide you with the tools and support that
you need to offload your IT needs and focus on growing your business.
Configured For All Businesses. Start Your Cloud Today.
https://www.gigenetcloud.com/
_______________________________________________
Simple-evcorr-users mailing list
Simple-evcorr-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
