On Thu, Aug 9, 2018 at 1:29 PM Robin Sommer <ro...@corelight.com> wrote:
> > (1) enable the "explicit/manual" forwarding by default? > > Coming from that assumption above, I'd say yes here, doing it like you > suggest: differentiate between forwarding and locally raising an event > by topic. Maybe instead of adding it to Broker::subscribe() as a > boolean, we add a separate "Broker::forward(topic_prefix)" function, > and use that to essentially hardcode forwarding on each node just like > we want/need for the cluster. Behind the scenes Broker could still > just store the information as a boolean, but API-wise it means we can > later (once we have real routing) just rip out the forward() calls and > let Magic take its role. :) Not sure there'd be anywhere we'd currently use Broker::forward() ? Or is it a matter of "if a user needed it for something, then it's available" ? The only intra-cluster communication that's more than 1 hop at the moment is worker-worker, but setting up a Broker::forward() route wouldn't be my first thought as it's not currently a scalable approach. I'd instead take the cautious approach of relaying via a RR-proxy so one can add proxies to handle more load as needed. However, I can see Broker::forward() could make it a bit easier for a user wanting to manually set up a forwarding route between clusters or other external applications. Is that a clear use-case we need to cater to now? If so, then it would indeed be just saying "hey, Broker::forward() is now a no-op since Broker has real routing mechanisms and you can remove them". > As you say, we don't get load-balancing that way (today), but we still > have pools for distributing analyses (like the known-* scripts do). > And if distributing message load (like the Intel scripts do) is > necessary, I think pools can solve that as well: we could use a RR > proxy pool and funnel it through script-land there: send to one proxy > and have an event handler there that triggers a new event to publish > it back out to the workers. For proxies, that kind of additional load > should be fine (if load-balancing is even necessary at all; just going > through a single forwarding node might just as well be fine. Seems more prudent not to guess whether a single, hardcoded forwarding node is good enough when writing the default cluster-enabled scripts. RR via proxy is not just load-balancing either, but fault-tolerance as well. But here you're talking more about removing the relay() functions and doing the RR-via-proxy "manually", right? That seems ok to me -- once "real" routing is available, you then have the option to simplify your script and get a minor optimization by not having to manually handle+forward the event on proxies. > > (2) re-implement any existing subscription cycles? > > Now, here I'm starting to change my mind a bit. Maybe in the end, in > large topologies, it would be futile to insist on not having cycles > after all. The assumption above doesn't care about it, putting Broker > in charge of figuring it out. So with that, if we can set up > forwarding through (1) in a way that cycles in subscriptions don't > matter, it may be fine to just leave them in. But I guess in the end > it doesn't matter, removing them can only make things better/easier. Again, I think we wouldn't have any Broker::forward() usages in the default cluster setup, but simply enabling the forwarding of messages at the Broker-layer would currently cause some messages to route in a cycle. Enabling the current message forwarding means we need to re-implement existing subscription cycles. If we instead waited for the "real" routing, then it doesn't matter if we leave them in. - Jon _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
