Re: [Bro-Dev] Broker::publish API

[Robin Sommer]

On Mon, Jul 30, 2018 at 09:01 -0700, I wrote:

> Is there a summary somewhere of what events & topics the cluster nodes
> are currently exchanging?

So I went through the exercise of collecting this information: what
connections do we have between nodes, who's subscribing to what, and
who's publishing what; see the attached PDF. This is based on all the
standard scripts, with some special cases ignored (like the control
framework).

I'm not fully sure yet what to conclude from this, but a few quick
observations:

- The main topics are bro/cluster/ and
  bro/cluster/node/. For these we wouldn't have a problem
  with loops if we enabled automatic, topic-driven forwading as
  far as I can see.

- bro/cluster/broadcast seems to be the main case with a looping
  problem, because everybody subscribes to it. It's hardly used
  though. (bro/config/change is used similarly though).

- Relaying is hardly used.

- There are a couple of script-specific topics where I'm wondering
  if these could switch to using bro/cluster/ instead
  (bro/intel/*, bro/irc/dcc_transfer_update). In other words: when
  clusterizing scripts, prefer not to introduce new topics.

- There's a lot of checks in publishing code of the type "if I am
  (not) of node type X".

- Pools are used for two different things: 1. the known-* scripts
  pick a proxy to process and log the information; whereas 2. the
  Intel scripts pick a proxy just as a relay to broadcast stuff
  out, reducing load. That 1st application is a good, but the 2nd
  feels like should be handled differently.

Need to mull over this more, thoughts welcome.

Overall I have to say I found it pretty hard to follow this all
because we don't have much consistency right now in how scripts
structure their communication. That's not surprising, given that we're
just starting to use all this, but it suggests that we have room for
improvement in our abstractions. :)

Robin

-- 
Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com


Broker Communication.pdf
Description: Adobe PDF document
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker::publish API

[Robin Sommer]

On Fri, Aug 03, 2018 at 15:57 -0500, Jonathan Siwek wrote:

> Another use is hidden within Cluster::relay_rr():

Yeah, though at least from an API perspective this is different: The
caller gives relay_rr() only one topic to send to (indicator_topic).
It's then using a different topic internally to get it over to the
proxy first, but that feels more like an implementation detail. So in
that sense I would argue that this is not a use-case for the Broker
API letting users change the topic on relay. (I'm not saying that that
capability can't be useful, I'm just still looking for actual use
cases.)

I have another question about this specific case: we use relay_rr()
only for sending Intel::insert_indicator. Intel::remove_indicator gets
published normally through auto_publish(). Why the difference?

Robin

-- 
Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev