> On Nov 2, 2017, at 2:37 PM, Aashish Sharma <asha...@lbl.gov> wrote:
>
>
>
> Now, while Justins' multiple data nodes idea has specticular merits, I am not
> much fan of it. Reason being having multiple data-notes results in same sets
> of problems
It does not have the same problems.. It may have different problems that I
haven't thought of yet, but it doesn't have the same problems.
> syncronization,
What synchronization problems?
> latencies
Adding multiple data nodes will reduce the load on each node and lower overall
latencies.
> mess of data2worker, worker2data events etc etc
you're projecting the current mess of worker2manager_events and
manager2worker_events onto what I am trying to replace them with.
Having
worker2manager_events
and
@if ( Cluster::is_enabled() && Cluster::local_node_type() !=
Cluster::MANAGER )
all over the place exists because bro doesn't have higher level methods for
distributing data and events across the cluster. I am not proposing replacing
that with
worker2datanode_events
and
@if ( Cluster::is_enabled() && Cluster::local_node_type() !=
Cluster::DATANODE )
I'm proposing getting rid of that sort of thing entirely. No '@if cluster'. no
'redef worker2manager_events'. All gone.
> I'd love to keep things rather simple. Cooked data goes to one (or more)
> datanodes (datastores). Just replicate for relibaility rather then pick and
> choose what goes where.
Then clusters will just change from having an overloaded manager process that
is falling under the load to 2 data nodes that are both failing. This is just
renaming the current bottlenecks and is not a solution.
I implemented a multi data node cluster back in March on top of
topic/mfischer/broker-integration . Porting my scan.bro from the
manager2worker_events stuff to sending events directly to one of N datanodes
was:
Remove:
redef Cluster::worker2manager_events ...
@if (Cluster ...
event Scan::scan_attempt(scanner, attempt);
Add:
local args = Broker::event_args(Scan::scan_attempt, scanner, attempt);
Cluster::send_event_hashed(scanner, args);
Other than having that wrapped in a single function, it doesn't get any easier
than that.
—
Justin Azoff
_______________________________________________
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
