Jon Siwek created BIT-1388:
------------------------------
Summary: Broker's integration in Bro's main/run loop
Key: BIT-1388
URL: https://bro-tracker.atlassian.net/browse/BIT-1388
Project: Bro Issue Tracker
Issue Type: Problem
Components: Bro, Broker
Reporter: Jon Siwek
Fix For: 2.5
* There's a cost to Broker queues being idle. Whenever Broker gets a chance to
process messages, it looks for updates to all
connections/message-queues/data-stores. That involves sending synchronous
messages between actors, and for empty queues, it just gets back an empty deque
object it needs to destroy.
* Broker queues integrate into Bro's run loop by exposing a file descriptor
that's ready when the queue is non-empty. Users have the capability of adding
arbitrary numbers of queues at run-time (e.g. they can freely add subscriptions
to any amount of logs, events, etc.). Relying on select() may become a
bottleneck if someone has hundreds of Broker queues, or could possibly break on
some systems if FD_SETSIZE is limited to 1024.
Ideas on how to fix:
* Improve Bro's main run loop and dedicate an IOSource to each Broker queue
(instead of sharing a single IOSource like they do now). There might be
several things that could be tweaked in the main run loop, but at a minimum,
epoll()/kqueue() could alternatively replace select(). Could also think about
using something like libev
(http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod) to abstract what
particular polling backend is used. Might even be able to use libev's timers
to fix how Bro's timers are currently coupled w/ there being an active IOSource
consistently driving time forward.
* Move the draining of Broker queues completely off to their own threads. This
maybe is adding a bit too much complexity (Broker/CAF uses threads for queues,
then Bro would use more threads just to talk to those other threads...). Since
CAF becomes a requirement, it may be simpler to start replacing/allowing some
areas of Bro's threading to be done w/ CAF actors. And then if Broker exposed
an optional API to talk directly w/ CAF actors, the integration w/ Bro may
actually become more straightforward.
And those ideas don't have to be mutually exclusive.
--
This message was sent by Atlassian JIRA
(v6.5-OD-01-120#65000)
_______________________________________________
bro-dev mailing list
[email protected]
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
