Re: [Bro-Dev] Broker::publish API
On Mon, Jul 30, 2018 at 13:30 -0500, Jonathan Siwek wrote: > Seems clunky and could get dicey Agreed. :) It'd just be a heuristic to catch some obvious errors. I don't think there's more we can do, we can't really catch loops statically by looking at the code. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker::publish API
On Mon, Jul 30, 2018 at 12:16 PM Robin Sommer wrote: > > I'd be more comfortable if one could automate answering the question: > > "if I add a subscription to a given node in the network, will I create > > a cycle?". > > Hmm ... What about a test mode where we'd spin up a dummy cluster > (similar to what the bests do), have each node send a message to all > subscribed topics, and watch for TTL violations? Seems clunky and could get dicey -- subscriptions that transient/dynamic may not be well-tested and you'd probably want to guarantee that sending such a dummy message actually does not result in any side-effects at the Bro-layer. If nodes start raising random events at unusual/unintended times I start to doubt the stability of things. - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker::publish API
On Mon, Jul 30, 2018 at 11:15 -0500, Jonathan Siwek wrote: > I don't see why not, but it takes planning and prudence on everyone's > part (including users) to not break that rule. Yeah, question is we can pre-configure the cluster so that user's don't need to worry about it most of the time. > I'd be more comfortable if one could automate answering the question: > "if I add a subscription to a given node in the network, will I create > a cycle?". Hmm ... What about a test mode where we'd spin up a dummy cluster (similar to what the bests do), have each node send a message to all subscribed topics, and watch for TTL violations? Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker::publish API
On Mon, Jul 30, 2018 at 11:02 AM Robin Sommer wrote: > True, although it's not cycles in the connection topology that matter, > it's cycles in topic subscriptions. Right, good point. > I need to think about this a bit > more (and I need to remind myself how our topics currently look like) I think we just have the "broadcast_topic" to which all nodes subscribe, but not sure if there's more. > but could we set up topics so that even in a cluster, messages don't > go into a cycle? I don't see why not, but it takes planning and prudence on everyone's part (including users) to not break that rule. I'd be more comfortable if one could automate answering the question: "if I add a subscription to a given node in the network, will I create a cycle?". - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker::publish API
On Fri, Jul 27, 2018 at 7:30 PM Azoff, Justin S wrote: > > > > On Jul 27, 2018, at 6:10 PM, Jon Siwek wrote: > > > > On Fri, Jul 27, 2018 at 3:55 PM Azoff, Justin S wrote: > > > >> I do agree that there's room for a lot of simplification, for example a > >> worker broadcasting a message efficiently to all > >> other workers needs to do something like this from the docs: > >> > >>Cluster::relay_rr(Cluster::proxy_pool, "example_key", > >> Cluster::worker_topic, worker_to_workers, > >> Cluster::node + " (via a proxy)"); > >> > >> But a lot of that could have defaults: > >> > >> Most use cases would want to relay through the default proxy pool > >> Since round robin is in use, they key shouldn't matter. > > > > At the moment, one could write their own wrapper function around that > > if they find it too verbose and always want to use certain defaults? > > Yeah.. The wrapper would be trivial.. Should bro include it so that the API > scripts use is simpler? Maybe. We can see how it fits in the mix of what Robin suggested: # Supports variadic args in place of Broker::Event. Broker::publish(topic: string, args: Broker::Event, relay_topic: string ="", process_on_relayer: bool =F) # Supports variadic args in place of Broker::Event. Cluster::publish(pool: Cluster::pool, key: any, strategy: enum, args: Broker::Event, relay_topic: string ="", process_on_relayer: bool =F) # Supports variadic args in place of Broker::Event. Use proxy pool and RR method w/ arbitrary, internal key by default. Cluster::publish_via_proxy(relay_topic: string, args: Broker::Event) That last one being the wrapper you're asking for. Also, I compressed the ideas of having a separate "relay: bool" / "relay_hops: int" and "relay_topic: string" args -- a non-empty relay topic implicitly means you want to enable relaying on the receiving node. Thinking more about original idea of giving the number of relay hops: it may be better to leave that until Broker multihop is more robust and allow it's automatic forwarding mechanisms to take care of those scenarios whereas a "relay" is a simple mechanism at the Bro application level (has it's own unique message format) that serves to aid load-balancing use-cases (rather than routing use-cases). - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker::publish API
On Fri, Jul 27, 2018 at 14:47 -0500, Jonathan Siwek wrote: > Broker does not yet have automatic multihop where subscriptions are > globally flooded automatically. Yep, that's what I meant: dynamic multihop where each node tracks what its peers are subscribing to, and forwards messages independent of its own subscriptions. > Possibly a downside is now you need to store original hop limit in > addition to current TTL in each message if you want to detect the "is > 1st hop" condition for the "relay_topic" option below. Yeah, that's right. Actually I think ideally the 1st hop wouldn't have any special role anyways if we didn't need that "relay_topic". > It's maybe both a concern and a reality -- Bro clusters currently > contain cycles (e.g. worker -> manager -> proxy -> worker) True, although it's not cycles in the connection topology that matter, it's cycles in topic subscriptions. I need to think about this a bit more (and I need to remind myself how our topics currently look like) but could we set up topics so that even in a cluster, messages don't go into a cycle? Is there a summary somewhere of what events & topics the cluster nodes are currently exchanging? > > - Add a second function publish_pool() that has all the same > > options, but receives a pool type instead of a topic (just an > > enum: RR, HRW). > > What's the goal of the enums instead of just publish_hrw() and publish_rr() ? Similar to what Justin wrote, it would more directly express the intent, with less emphasis on the mechanism; we could set a default to whatever we recommend people normally use; and it'd be more extensible. > At the moment, one could write their own wrapper function around that > if they find it too verbose and always want to use certain defaults? They could, but my general point is that it'd be nice to have a simple API that covers the most common uses cases directly and intuitively. Then let people change defaults if they have to and know what they are doing. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
