Yeah, and let me add one thing: What if as a starting point for modeling things, we assumed that we have global topic-based routing available. Meaning if node A publishes to topic X, the message will show up at all nodes that are subscribed to topic X anywhere, no matter what the topology --- Broker will somehow take care of that. I believe that's where we want to get eventually, through whatever mechanism; it's not trivial, but also not rocket science.
Then we (A) design the API from that perspective and adapt our standard scripts accoordingly, and (B) see how we can get an approximation of that assumption for today's Broker and our simple clusters, by having the cluster framework hardcode what need. > (1) enable the "explicit/manual" forwarding by default? Coming from that assumption above, I'd say yes here, doing it like you suggest: differentiate between forwarding and locally raising an event by topic. Maybe instead of adding it to Broker::subscribe() as a boolean, we add a separate "Broker::forward(topic_prefix)" function, and use that to essentially hardcode forwarding on each node just like we want/need for the cluster. Behind the scenes Broker could still just store the information as a boolean, but API-wise it means we can later (once we have real routing) just rip out the forward() calls and let Magic take its role. :) As you say, we don't get load-balancing that way (today), but we still have pools for distributing analyses (like the known-* scripts do). And if distributing message load (like the Intel scripts do) is necessary, I think pools can solve that as well: we could use a RR proxy pool and funnel it through script-land there: send to one proxy and have an event handler there that triggers a new event to publish it back out to the workers. For proxies, that kind of additional load should be fine (if load-balancing is even necessary at all; just going through a single forwarding node might just as well be fine. > (2) re-implement any existing subscription cycles? Now, here I'm starting to change my mind a bit. Maybe in the end, in large topologies, it would be futile to insist on not having cycles after all. The assumption above doesn't care about it, putting Broker in charge of figuring it out. So with that, if we can set up forwarding through (1) in a way that cycles in subscriptions don't matter, it may be fine to just leave them in. But I guess in the end it doesn't matter, removing them can only make things better/easier. > Also maybe begs the question for later regarding the "real" routing > mechanism: I suppose that would need to be smart enough to do > automatic load-balancing in the case of there being more than one > route to a subscriber. Yeah, I'm becoming more and more convinced that in the end we won't get around adding a "real" routing layer that takes of such things under the hood. Robin -- Robin Sommer * Corelight, Inc. * ro...@corelight.com * www.corelight.com _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev