hi Gonzalo Rodrigo, do I understand correctly that in some cases a script is not started when you are monitoring 65 input files with a single SEC process? Within the SEC code there are no restrictions to the number of input files. In order to investigate this issue a bit further, I have a couple of questions. First, how are the input files specified -- are they given implicitly as a pattern (for example, -input=*.log) or are they specified explicitly by name (for example, -input=log1.log -input=log2.log ... -input=log65.log)? Second, have you started SEC with -reopen_timeout option? In order to clarify these questions, if you have started SEC with using wildcards for the -input option, the file pattern is *only* evaluated at SEC startup (and configuration reloads). As a consequence, if a certain input log file does not exist when SEC starts (or does not exist when SEC is restarted automatically, e.g. by cron), then it will not be opened in the future when it is created. Therefore, it is always better to specify input files explicitly without using wildcards. Also, I strongly recommend to use -reopen_timeout option for SEC, since this will tell SEC to check periodically for newly created input files and open them for processing. I don't know if these advises are of any value, but some users have run into similar issues with input files before. Additional recommendation -- if you are monitoring a large number of input files and many of these files don't change frequently (for instance, they are only modified once in minute), you can take advantage of the -check_timeout option which could significantly reduce the CPU time consumed by SEC. Another question -- are there any error messages in the SEC log file about the scripts that have not been executed? Have you sent the SIGUSR1 signal to the SEC process, in order to check if SEC has all log files open? kind regards, risto
On 05/03/2010 01:19 PM, Gonzalo Rodrigo Sancho wrote: > Hi all, > > First of all i want to introduce myself as this is my first mail on this > list. My name is Gonzalo Rodrigo and i'm a security engineer on s21sec, a > security company based on Spain, and a sec user since december 2008. > > My current issue is related with one sec that checks about 65 files, this > sec simply sends an email and launch a connection with a remote jira. One > month ago it was controlling about 40 files and it was working properly, > but when we grow to 65 files i noticed some failures, for wxample, one > file exists but the email was not send and the conn with jira was not > stablished. > > I had checked everything that was involved on the process, the script that > sends the mail, the connection with the remote server, the mail server > itself, and the sec without success, i didn't find anything that could be > the reason of this problem. > > And finally, last thursday, at 12:00 pm, when i was in bed, i had a > "divine revelation", ¿Could be that the sec has too much files to check > and, for some reason that i cannot understand, it simply discard some of > them?" Next day i simply divided on two instances of the same sec, one > with 32 files an the order one with 33, one minute later on file was > created and the mail and conn were sent. And during this weekend > everything was working great, so looks like it was the root of all my > problems. > > The strange thing is that the machine was not heavily loaded after the > change, we have an Intel Xeon at 3 Ghz with 12GB of ram and a cabinet > connected via fiberchannel to the machine, the total cpu load was about > 45% and ram usage was 10 GB (i have other processes running on that > machine), and regarding the sec itself, it was at 12% more or less, and > 60% of one single core. > > So, do you think that my problem is solved?, do sec have any limits > regarding file handling or something related?, is any flag available to > check a big amount of files? Do you think is a better solution available? > Thans in advance and regards from Zaragoza (Spain) > > PD: Here you have the code of the sec, due to security issues i changed > the email address and the patterns, mainly i changed every word for a > single letter, but how the sec works weren't changed. > > It checks those files, if one of them is created it send an email and open > a conn with the first line, and also creates a temporaly context of one > and 12 hours. If we received the same event again, it was stored in one > file and after that hour is passed, the file is sent as an attachement, it > also do the same with 12 hours. > > type=Single > ptype=RegExp > desc=1h_storing_logs_$2 > continue=takenext > pattern=(.+?) bita.+?: > a=(.+)\|\|b=(.+)\|\|c=(.+)\|\|d=(.+)\|\|e=(.+)\|\|f=(.+)\|\|g=(.+)\|\|h=(.+)\|\|i=(.+)\|\|j=(.+)\|\|k=(.+)\|\|(l=/|l=)(.+)\|\|m=(.+)\|\|n=(.+) > context= context_1h_$2&& ! context_12h_$2 > action= shellcmd (/usr/local/xxxx/sec/bin/sec2jira.pl --a '$3' --b '$4' > --c '$5' --d '$9' --e '$10' --f '$11: $12' --g '$6' --h '$8' --i '$7' --j > '$2' --k '$14' --l '$15' --m '$16'); write > /usr/local/xxxxx/sec/tmp/1h-$2.txt A las $1, $11: $12 > > type=Single > ptype=RegExp > desc=1h_storing_logs_$2 > continue=takenext > pattern=(.+?) bita.+?: > a=(.+)\|\|b=(.+)\|\|c=(.+)\|\|d=(.+)\|\|e=(.+)\|\|f=(.+)\|\|g=(.+)\|\|h=(.+)\|\|i=(.+)\|\|j=(.+)\|\|k=(.+)\|\|(l=/|l=)(.+)\|\|m=(.+)\|\|n=(.+) > context= ! context_1h_$2&& context_12h_$2 > action= shellcmd (/usr/local/xxxx/sec/bin/sec2jira.pl --a '$3' --b '$4' > --c '$5' --d '$9' --e '$10' --f '$11: $12' --g '$6' --h '$8' --i '$7' --j > '$2' --k '$14' --l '$15' --m '$16'); write > /usr/local/xxxxx/sec/tmp/1h-$2.txt A las $1, $11: $12 > > type=Single > ptype=RegExp > desc=mail_notification_$2 > pattern=(.+?) bita.+?: > a=(.+)\|\|b=(.+)\|\|c=(.+)\|\|d=(.+)\|\|e=(.+)\|\|f=(.+)\|\|g=(.+)\|\|h=(.+)\|\|i=(.+)\|\|j=(.+)\|\|k=(.+)\|\|(l=/|l=)(.+)\|\|m=(.+)\|\|n=(.+) > context= ! context_1h_$2&& ! context_12h_$2 > action = shellcmd (/usr/bin/perl /usr/local/xxxxx/sec/bin/SendHTMLMail.pl > 'x...@xxxxx.es' 'alertas-test.x...@xxxx.es,xxxx....@xxxx.es' "bitALERT: $7 > - $8 $10" "$11" "$12" "blablabla %t"); shellcmd > (/usr/local/xxxx/sec/bin/sec2jira.pl --a '$3' --b '$4' --c '$5' --d '$9' > --e '$10' --f '$11: $12' --g '$6' --h '$8' --i '$7' --j '$2' --k '$14' --l > '$15' --m '$16'); create context_1h_$2 3600 (spawn > (/usr/local/xxxxx/sec/bin/alertaCorreo.sh 'x...@xxxxx.es' > 'alertas-test.x...@xxxx.es,xxxx....@xxxx.es' "One hour notification: $7 - > $8 $10" "$11" "$12" "blablabla %t" '/usr/local/xxxxx/sec/tmp/1h-$2.txt' > 'echoing_12h_$2||$7||$8||$10')) > > type=Single > ptype=RegExp > desc=creation_12h_$1 > pattern= echoing_12h_(.+?)\|\|(.+?)\|\|(.+?)\|\|(.+) > context= ! context_1h_$1&& ! context_12h_$1 > action= create context_12h_$1 43200 shellcmd > (/usr/local/xxxxx/sec/bin/alertaCorreo.sh 'xx...@yyyyy.es' > 'alertas-test.x...@yyyyy.es,xxxxx.zz...@wwwww.es' '12 hours notification' > 'alert is still active' 'New alerts detected:' 'Time: %t' > '/usr/local/xxxxx/sec/tmp/12h-$1.txt') > > > > ------------------------------------------------------------------------------ > _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users > > ------------------------------------------------------------------------------ _______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
