[
https://bro-tracker.atlassian.net/browse/BIT-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=19907#comment-19907
]
Johanna Amann commented on BIT-1182:
------------------------------------
I am actually not sure if this is an input framework issue. The input framework
does not spawn a new thread for each change in an input file. Instead, all the
changes are serialized into events by a single input reader, which is
responsible for the file. For 5,000 changed lines, this should be rather fast -
it probably processes all changes in less then a second.
If I understood everything correctly, things work as long as you do not use the
exec framework. The problem here is actually that the exec framework spawns a
thread for each execution that you want to perform (...because one input reader
is spawned per execution...). As you get all change events near-simultaneously,
all of them are spawned near-simultaneously - and I can see that leading to all
kinds of problems.
I am not quite sure what the best way to handle that is though. Throttling the
number of events that the input framework sends should be possible -- however,
I am not sure if it is desirable since it should usually work without too much
troubles. You would run into the same problem if you try to spawn a lot of exec
framework tasks because of some other event (e.g. a lot of network packets that
trigger it at the same time).
> Input-framework thread spwan
> ----------------------------
>
> Key: BIT-1182
> URL: https://bro-tracker.atlassian.net/browse/BIT-1182
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
> Affects Versions: 2.2
> Reporter: Aashish Sharma
> Labels: input-framework
>
> Using the mode REREAD, I noticed that input-framework spawns a thread for
> every add/change/delete for the elements in the feed file.
> this is a VERY desired feature and powerful capability and works quite well
> in general settings.
> Since, all the changes in a file spawns a thread to process for: EVENT_NEW,
> EVENT_CHANGED, EVENT_REMOVED, If there are lets say 5000 Changes in the file,
> there would be 5000 threads spawned at the same time. this is still alright
> and system can handle load and processing is done in a few seconds.
> However, if I include a when statement along with exec framework usage to
> execute an action in Input::EVENT_NEW, Input::EVENT_CHANGED or
> Input::EVENT_REMOVED - all threads spawned together freezes bro from
> processing any packets at all.
> It would be nice if we can serialize this thread creation and spawn only a
> few at a time. This way we can spread out the increased load over next N mins
> instead of freezing bro to a standstill.
> (As always, please let me know if you want code to be able to re-produce this
> issue).
--
This message was sent by Atlassian JIRA
(v6.4-OD-15-055#64014)
_______________________________________________
bro-dev mailing list
[email protected]
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
