hi Richard, let me provide few comments here. Since you mentioned setting up an associative array (Perl hash) that contains input file names, contexts are actually implemented as a Perl hash as well in SEC code. Checking for the presence of the context involves a search for some key in a hash, and it is very fast in Perl. Therefore, if you are concerned about search speed, using your own hash would not offer significant performance improvements. However, using a custom hash has one important advantage -- internal hash of SEC contexts has to be scanned periodically (by default once a second), in order to detect expired contexts and execute their action lists. If you have stored input file names in a custom hash, you don't have that housekeeping overhead.
For estimating the impact of that overhead, you could run couple of tests with a larger number of contexts in your environment. I have just done couple of quick experiments on my 4 year old laptop (intel core i5) which involve creating N contexts and letting SEC run without any other input for about 15-20 minutes, measuring consumed CPU time at the end of the experiment. With 1,000 contexts, CPU was utilized by 0.4%, with 10,000 contexts the utilization was 2.2%, and with 100,000 contexts the utilization reached 16.3%. Given those figures, I would say that if you have only few thousand contexts, housekeeping should not consume too much of your CPU time. But if you want to employ contexts for different purposes like implementing a large blacklist of IP addresses, custom Perl hash is probably a better way to go (and it would be easier to search it from PerlFunc patterns). hope this helps, risto Kontakt Richard Ostrochovský (<richard.ostrochov...@gmail.com>) kirjutas kuupäeval N, 2. aprill 2020 kell 01:18: > Hi Risto, > > thank you for the solution. There is also concern about potential > performance impact in case of e.g. thousands of files being added by > addinput with creating extra context for each of them. > > Another way could be e.g. maintaining (associative) array in action list, > with keys of paths to addinput files, reduced by dropinput, similarly as > you suggest with context creation. > > And maybe the most efficient, having built-in way, how to access this list > of open files, without custom implementation of it. > > How do you see these potential options in comparison by performance? > > Richard > > št 20. 2. 2020 o 21:23 Risto Vaarandi <risto.vaara...@gmail.com> > napísal(a): > >> hi Richard, >> >> I think this scenario is best addressed by creating a relevant SEC >> context when 'addinput' action is called. In fact, handling such scenarios >> is one of the purposes of contexts, and here is an example rule which >> illustrates this idea: >> >> type=single >> ptype=regexp >> pattern=start monitoring (\S+) >> context=!FILE_$1_MONITORED >> desc=add $1 to list of inputs >> action=addinput $1; create FILE_$1_MONITORED >> >> Whenever "start monitoring <filename>" event appears, the rule will match >> only if context FILE_<filename>_MONITORED does not exist. If rule matches, >> it executes 'addinput' action for the given file and creates the context, >> in order to manifest the fact that 'addinput' has already been executed for >> the given file. Also, as you can see from the above rule, the presence of >> the context for a file will prevent the execution of 'addinput' again for >> this file. In order to keep contexts in sync with files that are monitored, >> the context for a file should be deleted when 'dropinput' action is >> executed for it. >> >> Note that when HUP signal is received, SEC will stop monitoring input >> files set up with 'addinput' action. However, on receiving HUP signal SEC >> will also drop all its contexts, so there is no need to take any extra >> steps in that case. >> >> Hope this helps, >> risto >> >> Kontakt Richard Ostrochovský (<richard.ostrochov...@gmail.com>) kirjutas >> kuupäeval N, 20. veebruar 2020 kell 21:43: >> >>> Hello Risto and friends, >>> >>> having mechanism for dynamic opening (addinput) and closing (dropinput) >>> files, I would like to be able to check, if the file is already opened, >>> before trying to open it again, to avoid it. That way I would like to >>> eliminate this error message from SEC log (present also in debug level 3): >>> >>> Dynamic input file '/path/to/file.log' already exists in the list of >>> inputs, can't add >>> >>> This information is present in sec.dump, but maybe there exists more >>> instant and straightforward way how to achieve it (without parsing >>> intermediary files). >>> >>> Thank you. >>> >>> Richard >>> _______________________________________________ >>> Simple-evcorr-users mailing list >>> Simple-evcorr-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >>> >> _______________________________________________ > Simple-evcorr-users mailing list > Simple-evcorr-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users >
_______________________________________________ Simple-evcorr-users mailing list Simple-evcorr-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
