Re: [Bro-Dev] Broker has landed in master, please test

[Azoff, Justin S]

> On Jun 1, 2018, at 11:37 AM, Azoff, Justin S  wrote:
> 
> I could never figure out what was causing the problem, and it's possible that 
>  not doing anything anymore is why it's better now.  I'm mostly 
> using  for syncing input files across all the workers and one of 
> them does have 300k entries in it.  That file is fairly constant though, only 
> a few k changes every 5 minutes and nothing that should use 20G of ram.

FWIW, I figured out what was causing this problem.  While the file wasn't 
changing that much, I was using something like

curl -o file.new $URL && mv file.new file.csv

to download the file, and apparently unless you pass -f to curl, it doesn't 
actually exit with a non-zero status code on server errors.

This was causing a server error page to be written to the csv file every now 
and then.  When this happened:

* the input reader would throw a warning that the file couldn't be parsed, and 
clear out the set
* bro would then clear the set, triggering a removal of 300k items across all 
nodes (56 in the case of the test cluster)
* 5 minutes later the next download would work
* bro would then fill back in the set, and trigger 300k items to be synced to 
all 56 nodes again.

so within 5 minutes, 300,000*56*2 updates would be kicked off, which is 
33million updates.  This seemed to max out the proxies for 30 minutes.
The raw size of the data is only ~4M, or 261M total, which makes it a little 
crazy that memory usage would blow up by dozens of gigabytes of ram.

 not having an effect in master made this problem go away, and 
adding a -f to curl on our pre-broker clusters fixed those too.

All the more reason to port the method of distributing the data off of 
  I think I will just run the curl command on the worker nodes 
too,
effectively replacing  with curl.


— 
Justin Azoff


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker has landed in master, please test

[Robin Sommer]

On Wed, May 23, 2018 at 20:16 +, Adam wrote:

> I think those question belong on the main list which is for using Bro
> and its language. This list is really more about development of Bro
> itself.

Just to give context here, the reason I sent the original mail about
Broker here, including the request for feedback, was to limit the
initial round of testing to folks quite familiar with Bro and its
internals. That gives us a chance to spot any obvious issues quickly
before annoying everybody else. :-) But discussing it at either place
is fine of course, whatever works best for folks. If things seem to
work, we should definitely also announce the merge more broadly.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker has landed in master, please test

[Jon Siwek]

On 5/23/18 3:12 PM, Michael Dopheide wrote:

> For here though, can you elaborate on the going down to one proxy?  My 
> understanding still isn't strong, but that seems to be opposed to the 
> idea of using Cluster::publish_hrw to spread memory across proxies.

The idea is to try starting with a single proxy and then scale your 
deployment based on what you actually need, and there may not be that 
great of a need at the moment as the default scripts that ship with Bro 
do not widely use the HRW/pool/partitioning APIs yet.

By default, it's currently just the Software framework that will use 
Cluster::publish_hrw.  I also plan to soon change the Intel framework to 
make use of Cluster::relay_rr.

There's also an option in the various Known::* scripts for users to 
opt-in to an alternate implementation that uses HRW + tables instead of 
the default approach of data stores.

Different sites could also have different requirements/usage of those 
default scripts and it's all too new to give better suggestions other 
than "try one proxy, add more as needed".

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker has landed in master, please test

[Slagell, Adam J]

On May 23, 2018, at 3:12 PM, Michael Dopheide 
> wrote:

Do you want dumb programming questions asked here or on the main Bro list?  
While most people might not need it yet, discussion there might help get more 
people interested or help avoid issues with custom policy conversion.

I think those question belong on the main list which is for using Bro and its 
language. This list is really more about development of Bro itself.

--

Adam J. Slagell
Director, Cybersecurity & Networking Division
Chief Information Security Officer
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written 
communication to or from University employees regarding University business is 
a public record and may be subject to public disclosure."








___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Broker has landed in master, please test

[Michael Dopheide]

Do you want dumb programming questions asked here or on the main Bro list?
While most people might not need it yet, discussion there might help get
more people interested or help avoid issues with custom policy conversion.

For here though, can you elaborate on the going down to one proxy?  My
understanding still isn't strong, but that seems to be opposed to the idea
of using Cluster::publish_hrw to spread memory across proxies.

Thanks,
-Dop







On Tue, May 22, 2018 at 9:26 AM, Robin Sommer  wrote:

> We merged the new Broker version into Bro master yesterday. As this a
> major change to one of Bro's core components, I wanted to send a quick
> heads-up here, along with a couple of notes.
>
> With this merge we are completely replacing usage of Bro's classic
> communication system with Broker. All the standard scripts have been
> ported over, and BroControl has been adapted as well. The old
> communication system remains available for the time being, but is now
> deprecated and scheduled to be removed in Bro 2.7 (not 2.6). Broccoli
> is now turned off by default.
>
> With such a large change, I'm sure there'll be some more kinks to iron
> out still; that's where we need everybody's help. If you have an
> environment where you can test drive new Bro versions, please give
> this a try. We're interested in any feedback you have, both specific
> issues you encounter (best to file tickets) and general experiences
> with the new version, including in particular any observations about
> performance (best to send to this list).
>
> >From a user's perspective, not much should even be changing, most of
> the new stuff is under the hood. The exception are custom scripts
> doing communication themselves, they need to be ported over to Broker.
> Documentation for that is here:
> https://www.bro.org/sphinx-git/frameworks/broker.html, including a
> porting guide for existing scripts. Let us know if there's anything
> missing there that would be helpful. The Broker library itself comes
> with a new user manual as well, we'll get that online shortly.
>
> One specific note on upgrading existing Bro clusters: the meaning of
> "proxy" has changed. They still exist, but play a quite different role
> now. If you're currently using more than one proxy, we recommend going
> back to one; that'll most likely be fine with the standard scripts
> (and if not, please let us know!)
>
> Many thanks to Jon Siwek for the recent integration work tying up all
> the loose ends and getting Broker mergeable. Also thanks to those who
> have tested it already from the actor-system branch.
>
> Robin
>
> --
> Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
> ___
> bro-dev mailing list
> bro-dev@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
>
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Broker has landed in master, please test

[Robin Sommer]

We merged the new Broker version into Bro master yesterday. As this a
major change to one of Bro's core components, I wanted to send a quick
heads-up here, along with a couple of notes.

With this merge we are completely replacing usage of Bro's classic
communication system with Broker. All the standard scripts have been
ported over, and BroControl has been adapted as well. The old
communication system remains available for the time being, but is now
deprecated and scheduled to be removed in Bro 2.7 (not 2.6). Broccoli
is now turned off by default.

With such a large change, I'm sure there'll be some more kinks to iron
out still; that's where we need everybody's help. If you have an
environment where you can test drive new Bro versions, please give
this a try. We're interested in any feedback you have, both specific
issues you encounter (best to file tickets) and general experiences
with the new version, including in particular any observations about
performance (best to send to this list).

>From a user's perspective, not much should even be changing, most of
the new stuff is under the hood. The exception are custom scripts
doing communication themselves, they need to be ported over to Broker.
Documentation for that is here:
https://www.bro.org/sphinx-git/frameworks/broker.html, including a
porting guide for existing scripts. Let us know if there's anything
missing there that would be helpful. The Broker library itself comes
with a new user manual as well, we'll get that online shortly.

One specific note on upgrading existing Bro clusters: the meaning of
"proxy" has changed. They still exist, but play a quite different role
now. If you're currently using more than one proxy, we recommend going
back to one; that'll most likely be fine with the standard scripts
(and if not, please let us know!)

Many thanks to Jon Siwek for the recent integration work tying up all
the loose ends and getting Broker mergeable. Also thanks to those who
have tested it already from the actor-system branch.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev