Re: [Bro-Dev] Broker::publish API
Yeah, and let me add one thing: What if as a starting point for modeling things, we assumed that we have global topic-based routing available. Meaning if node A publishes to topic X, the message will show up at all nodes that are subscribed to topic X anywhere, no matter what the topology --- Broker will somehow take care of that. I believe that's where we want to get eventually, through whatever mechanism; it's not trivial, but also not rocket science. Then we (A) design the API from that perspective and adapt our standard scripts accoordingly, and (B) see how we can get an approximation of that assumption for today's Broker and our simple clusters, by having the cluster framework hardcode what need. > (1) enable the "explicit/manual" forwarding by default? Coming from that assumption above, I'd say yes here, doing it like you suggest: differentiate between forwarding and locally raising an event by topic. Maybe instead of adding it to Broker::subscribe() as a boolean, we add a separate "Broker::forward(topic_prefix)" function, and use that to essentially hardcode forwarding on each node just like we want/need for the cluster. Behind the scenes Broker could still just store the information as a boolean, but API-wise it means we can later (once we have real routing) just rip out the forward() calls and let Magic take its role. :) As you say, we don't get load-balancing that way (today), but we still have pools for distributing analyses (like the known-* scripts do). And if distributing message load (like the Intel scripts do) is necessary, I think pools can solve that as well: we could use a RR proxy pool and funnel it through script-land there: send to one proxy and have an event handler there that triggers a new event to publish it back out to the workers. For proxies, that kind of additional load should be fine (if load-balancing is even necessary at all; just going through a single forwarding node might just as well be fine. > (2) re-implement any existing subscription cycles? Now, here I'm starting to change my mind a bit. Maybe in the end, in large topologies, it would be futile to insist on not having cycles after all. The assumption above doesn't care about it, putting Broker in charge of figuring it out. So with that, if we can set up forwarding through (1) in a way that cycles in subscriptions don't matter, it may be fine to just leave them in. But I guess in the end it doesn't matter, removing them can only make things better/easier. > Also maybe begs the question for later regarding the "real" routing > mechanism: I suppose that would need to be smart enough to do > automatic load-balancing in the case of there being more than one > route to a subscriber. Yeah, I'm becoming more and more convinced that in the end we won't get around adding a "real" routing layer that takes of such things under the hood. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Broker::publish API
On Wed, Aug 8, 2018 at 2:50 PM Robin Sommer wrote: > > * enable message forwarding by default (meaning re-implement the one > > or two subscription patterns that might create a cycle) > > Haven't quite made up my mind on this one. In principlel yes, but > right now a host needs to be subscribed to a topic to forward it if I > remember than right. That may limit how we use topics, not sure (e.g., > if a worker wanted to talk to other workers, with "real" > forwarding/routing they'd just publish to the worker topic and that > message would get routed there, but not be processed at the > intermediary hops as well. With our current forwarding, the hops would > need to subscribe to the worker topic as well and hence the event got > raised there, too.) Yeah, that's how I also understand the current mechanisms would work. Maybe can split it into two separate questions: (1) enable the "explicit/manual" forwarding by default? (2) re-implement any existing subscription cycles? Answer to (2) may pragmatically be "yes" because they'd be known to cause problems if ever (1) did become enabled (and also could be problematic for a more sophisticated/automatic/implicit routing system should that become available in the future... at least I think it's a problem, but then again maybe connection-cycles would also still be a problem at that point, not quite sure). Answer to (1) may be "no" because we don't have a use for it at the moment -- having the forwarding-nodes also raise events is not ideal, but if we solved that would it be useful? Maybe an idea would be extend the subscribe() API in Bro: function Broker::subscribe(topic_prefix: string, forward_only: bool =F); I recall that we have access to both the message/event as well as the topic string on the receiver side, so could be possible to detect whether or not to raise the event depending on whether the topic only has a matching subscription prefix that is marked as forward_only. With that you could do something like: # On Manager Broker::subscribe(worker_to_worker_topic, T); # On Worker Broker::subscribe(worker_to_worker_topic); Broker::publish(worker_to_worker_topic, my_event); There, my_event would be distributed from one worker to all workers via the manager, but not sure that's as usable/dynamic as the current "relay" mechanism because you also get a load-balancing scheme to go along with it. Here, you'd only ever want to pick a single manager or proxy to do the forwarding (subscribing like this on all proxies causes all proxies to forward to all workers resulting in undesired event duplication.) So I guess that's still to say I'm not sure what the use of the current forwarding mechanism would be if it were enabled. Also maybe begs the question for later regarding the "real" routing mechanism: I suppose that would need to be smart enough to do automatic load-balancing in the case of there being more than one route to a subscriber. - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
