Re: [Bro-Dev] Flare removal

[Seth Hall]

> On May 20, 2016, at 6:02 PM, Azoff, Justin S  wrote:
> 
> I think Robins branch doesn't fix the problem because I don't think the 
> flares were really the issue.. I think bro started having issues because 
> between 2.3 and 2.4 traffic volumes increased, cluster sizes increased, and 
> we added a ton of new analyzers and log files which put even more strain on 
> the communication system.

That's an interesting idea and at least sounds as plausible as anything else 
I've heard.

Have you tried making those small changes you outlined by themselves on the 
NCSA dev cluster yet?  I'd be curious to hear if that fixes the problem.  Or do 
you even see this issue on that cluster?

  .Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] CBAN design proposal

[Slagell, Adam J]

> On May 21, 2016, at 5:44 PM, Robin Sommer  wrote:
> 
> As I read through the design doc, I started questioning our plan of
> curating CBAN content. I know that's what we've been intending to do,
> but is that really the best approach? I don't know of script
> repositories for other languages that enforce quality control on
> submissions beyond checking technical conventions like certain meta
> data being there.

I think there is a broad spectrum from no interaction to vetting and pulling 
into the main repository. It is about finding the right balance. 

I agree with minimal restrictions that block submissions. There needs to be 
some basic quality control and standardization there. For example, do you have 
all the right pieces. 

I do think there is another level of non blocking checks and quality control we 
can provide. For example, we can do checks for exec calls and give warnings to 
users. I think Puppet Forge has a nice model here. We won't block a submission, 
but these checks encourage better development and help new users trust 
submissions. That said, I think these must be automated. They can't block on a 
human reviewing them. 

Finally, I think we need a way to let the whole community endorse scripts or 
script authors. 
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] CBAN design proposal

[Aashish Sharma]

> In other words, my proposal is to put authors into control of their
> code, and make them fully responsible for it too --- not us. We'd just
> connect authors with users, with as little friction as possible.
> 

I support this completely. 

> If we want some kind of quality measure, we could introduce stars for
> modules that user assign if they like something; or even some facility
> to leave comments or other feedback that's visible to everybody. We

I think community vetting and feedback (and experience stories) is the right 
way to go here. 

Bro team vetting something will be very hard. My personal experience says there 
are times when scripts bring surprises weeks and months down the line - so 
testing isn't merely run a few days and give an OK.

Aashish 


On Sat, May 21, 2016 at 03:44:01PM -0700, Robin Sommer wrote:
> 
> 
> On Fri, May 20, 2016 at 20:16 +, Jon wrote:
> 
> > Here’s an updated design doc for CBAN, a plugin manager for Bro, with
> > some new ideas on how it could work:
> 
> Cool, thanks. I'm going to send some feedback but first I wanted to
> bring something up that might be controversial. :-)
> 
> As I read through the design doc, I started questioning our plan of
> curating CBAN content. I know that's what we've been intending to do,
> but is that really the best approach? I don't know of script
> repositories for other languages that enforce quality control on
> submissions beyond checking technical conventions like certain meta
> data being there.
> 
> I'm getting concerned that we're creating an unncessary bottleneck by
> imposing the Bro Team into the critical path for submissions and
> updates. Why not let people control their stuff themselves? They'd
> register things with CBAN to make it available to everybody, but we'd
> stay out of the loop for anything further. We may still want to keep a
> local copy of the code to protect against stuff going offline, but
> that could be automated. Or we could even move that burden to the
> authors as well and just keep a reference where to find the code,
> without a local copy; if it disappears, so be it.
> 
> In other words, my proposal is to put authors into control of their
> code, and make them fully responsible for it too --- not us. We'd just
> connect authors with users, with as little friction as possible.
> 
> If we want some kind of quality measure, we could introduce stars for
> modules that user assign if they like something; or even some facility
> to leave comments or other feedback that's visible to everybody. We
> could also verify if a plugins builds and loads correctly, or if tests
> pass. But we wouldn't block it if something fails, just mark it (say,
> with a banner saying "tests pass", "tests fail", "no tests").
> 
> Thoughts?
> 
> Robin
> 
> -- 
> Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] CBAN design proposal

[Robin Sommer]

On Fri, May 20, 2016 at 20:16 +, Jon wrote:

> Here’s an updated design doc for CBAN, a plugin manager for Bro, with
> some new ideas on how it could work:

Cool, thanks. I'm going to send some feedback but first I wanted to
bring something up that might be controversial. :-)

As I read through the design doc, I started questioning our plan of
curating CBAN content. I know that's what we've been intending to do,
but is that really the best approach? I don't know of script
repositories for other languages that enforce quality control on
submissions beyond checking technical conventions like certain meta
data being there.

I'm getting concerned that we're creating an unncessary bottleneck by
imposing the Bro Team into the critical path for submissions and
updates. Why not let people control their stuff themselves? They'd
register things with CBAN to make it available to everybody, but we'd
stay out of the loop for anything further. We may still want to keep a
local copy of the code to protect against stuff going offline, but
that could be automated. Or we could even move that burden to the
authors as well and just keep a reference where to find the code,
without a local copy; if it disappears, so be it.

In other words, my proposal is to put authors into control of their
code, and make them fully responsible for it too --- not us. We'd just
connect authors with users, with as little friction as possible.

If we want some kind of quality measure, we could introduce stars for
modules that user assign if they like something; or even some facility
to leave comments or other feedback that's visible to everybody. We
could also verify if a plugins builds and loads correctly, or if tests
pass. But we wouldn't block it if something fails, just mark it (say,
with a banner saying "tests pass", "tests fail", "no tests").

Thoughts?

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Broker status update

[Matthias Vallentin]

This is just a brief summary of where the Broker overhaul is currently
at. The new Broker API will come two types of endpoints: blocking and
nonblocking. The former provides a synchronous API to receive messages,
whereas the latter operates fully asynchronous. Performance-wise, the
asynchronous version has a major advantage because it does not introduce
buffering. The callback provided upon endpoint construction will be
immediately executed by the Broker runtime, for each message as it
arrives. For a blocking endpoint, it's the users responsibility to
extract messages, otherwise they queue up.

Endpoints can only be constructed through a broker::context object:

// Encapsulates global states, such as announced types, thread pool,
// CAF scheduler, etc.
context ctx;

// Creates a synchronous endpoint.
auto be = ctx.spawn();
auto msg = be.receive();  // Block and wait for next message.
be.receive(
  [&](const topic&, const message&) {
// As above, but use a lambda to handle the topic-message pair.
  }
);  // Block and wait for next message.

// Creates an asynchronous endpoint.
auto ne = ctx.spawn(
  [=](const topic&, const message&) {
// invoked for every subscribed message
  }
);

Other than that, both brokers have a publish/subscribe interface, e.g.:

be.subscribe("foo");
be.subscribe("bar");
auto t = topic{"bar"};
auto msg = make_message(42, "string");
be.publish(t, msg);

I'll post more updates as the development progresses, but all of the
above examples are now working. 

The TODO list currently includes:

- Peerings: managing subscriptions across multiple endpoints. Here's
  an example:
  
// Make two endpoints.
auto x = ctx.spawn<...>(...);
auto y = ctx.spawn<...>(...);

  I'm torn between two interfaces. This is the first:

peer(x, y); // Create a peering between the two endpoints.
peer(y, x); // Idempotent. Peerings are symmetric.

// Peer with a remote endpoint. Requires that the remote side
// called e.listen(addr, port) beforehand.
auto r = ctx.spawn("1.2.3.4", 42000);
peer(y, r);

// Undo a peering.
unpeer(x, y);
unpeer(r, y);

  The second is:

x.peer(y); // Create a peering between the two endpoints.
y.peer(x); // Idempotent. Peerings are symmetric.
x.peer("1.2.3.4", 42000); // Peer with a remote endpoint.

x.unpeer(y);
x.unpeer("1.2.3.4", 42000); // Unpeer from a remote endpoint,
   must match previous peering data.

  Personally, I'm favoring the first version, as the functional API
  makes the symmetry of the peering relationship clearer. 

- Data store: integrate the new "main API" with the existing data
  stores. My plan is to use as much as possible from the existing
  data store API.

- Bindings: For Python, I'm considering switching to pybind11 [1],
  which provides a much more convenient API than SWIG and supports
  modern C++11.

Please chime in if you have questions/comment or see opportunities for
improvement.

Matthias

[1] http://pybind11.readthedocs.io/en/latest/classes.html
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev