Re: [Bro-Dev] Performance Enhancements

[Jim Mellander]

Interesting info.  The > order of magnitude difference in time between
BaseList::remove & BaseList::removenth suggests the possibility that the
for loop in BaseList::remove is falling off the end in many cases (i.e.
attempting to remove an item that doesn't exist).  Maybe thats whats broken.



On Fri, Oct 6, 2017 at 3:49 PM, Azoff, Justin S  wrote:

>
> > On Oct 6, 2017, at 5:59 PM, Jim Mellander  wrote:
> >
> > I particularly like the idea of an allocation pool that per-packet
> information can be stored, and reused by the next packet.
> >
> > There also are probably some optimizations of frequent operations now
> that we're in a 64-bit world that could prove useful - the one's complement
> checksum calculation in net_util.cc is one that comes to mind, especially
> since it works effectively a byte at a time (and works with even byte
> counts only).  Seeing as this is done per-packet on all tcp payload,
> optimizing this seems reasonable.  Here's a discussion of do the checksum
> calc in 64-bit arithmetic: https://locklessinc.com/articles/tcp_checksum/
> - this website also has an x64 allocator that is claimed to be faster than
> tcmalloc, see: https://locklessinc.com/benchmarks_allocator.shtml  (note:
> I haven't tried anything from this source, but find it interesting).
> >
> > I'm guessing there are a number of such "small" optimizations that could
> provide significant performance gains.
> >
> > Take care,
> >
> > Jim
>
> I've been messing around with 'perf top', the one's complement function
> often shows up fairly high up.. that, PriorityQueue::BubbleDown, and
> BaseList::remove
>
> Something (on our configuration?) is doing a lot of
> PQ_TimerMgr::~PQ_TimerMgr... I don't think I've come across that class
> before in bro.. I think a script may be triggering something that is
> hurting performance.  I can't think of what it would be though.
>
> Running perf top on a random worker right now with -F 1 shows:
>
> Samples: 485K of event 'cycles', Event count (approx.): 26046568975
> Overhead  Shared Object Symbol
>   34.64%  bro   [.] BaseList::remove
>3.32%  libtcmalloc.so.4.2.6  [.] operator delete
>3.25%  bro   [.] PriorityQueue::BubbleDown
>2.31%  bro   [.] BaseList::remove_nth
>2.05%  libtcmalloc.so.4.2.6  [.] operator new
>1.90%  bro   [.] Attributes::FindAttr
>1.41%  bro   [.] Dictionary::NextEntry
>1.27%  libc-2.17.so  [.] __memcpy_ssse3_back
>0.97%  bro   [.] StmtList::Exec
>0.87%  bro   [.] Dictionary::Lookup
>0.85%  bro   [.] NameExpr::Eval
>0.84%  bro   [.] BroFunc::Call
>0.80%  libtcmalloc.so.4.2.6  [.] tc_free
>0.77%  libtcmalloc.so.4.2.6  [.] operator delete[]
>0.70%  bro   [.] ones_complement_checksum
>0.60%  libtcmalloc.so.4.2.6  [.] tcmalloc::ThreadCache::
> ReleaseToCentralCache
>0.60%  bro   [.] RecordVal::RecordVal
>0.53%  bro   [.] UnaryExpr::Eval
>0.51%  bro   [.] ExprStmt::Exec
>0.51%  bro   [.] iosource::Manager::FindSoonest
>0.50%  libtcmalloc.so.4.2.6  [.] operator new[]
>
>
> Which sums up to 59.2%
>
> BaseList::remove/BaseList::remove_nth seems particularly easy to
> optimize. Can't that loop be replaced by a memmove?
> I think something may be broken if it's being called that much though.
>
>
>
> —
> Justin Azoff
>
>
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] design summary: porting Bro scripts to use Broker

[Robin Sommer]

Nice!

On Fri, Oct 06, 2017 at 16:53 +, you wrote:

>   # contains topic prefixes
>   const Cluster::manager_subscriptions: set[string] 
> 
>   # contains (topic string, event name) pairs
>   const Cluster::manager_publications: set[string, string] 

I'm wondering if we can simplify this with Broker. With the old comm
system we needed the event names because that's what was subscribed
to. Now that we have topics, does the cluster framework still need to
know about the events at all? I'm thinking we could just go with a
topic convention and then the various scripts would publish there
directly.

In the most simple version of this, the cluster framework would just
hard-code a subscription to "bro/cluster/". And then scripts like the
Intel framework would just publish all their events to "bro/cluster/"
directly through Broker.

To allow for distinguishing by node type we can define separate topic
hierarchies: "bro/cluster/{manager,worker,logger}/". Each node
subscribes to the hierarchy corresponding to its type, and each script
publishes according to where it wants to send events to (again
directly using the Broker API).

I think we could fit in Justin's hashing here too: We add per node
topics as well ("bro/cluster/node/worker-1/",
"bro/cluster/node/worker-2/", etc.) and then the cluster framework can
provide a function that maps a hash key to a topic that corresponds to
currently active node:

local topic = Cluster:topic_for_key("abcdef"); # Returns, e.g., 
"bro/cluster/node/worker-1"
Broker::publish(topic, event);

And that scheme may suggest that instead of hard-coding topics on the
sender side, the Cluster framework could generally provide a set of
functions to retrieve the right topic:

# In SumStats framework:
local topic = Cluster::topic_for_manager() # Returns "bro/cluster/manager".
Broker::public(topic, event);

Bottom-line: If we can find a way to steer information by setting up
topics appropriately, we might not need much additional configuration
at all.

>   The old “communication” framework scripts can just go away as most
>   of its functions have direct corollaries in the new “broker”
>   framework.

Yep, agree.

> The one thing that is missing is the “Communication::nodes” table

Agree that it doesn't look useful from an API perspective. The Broker
framework may eventually need an equivalent table internally if we
want to offer robustness mechanisms like Justin's hashing.

> Broker Framework API
> 

I'm wondering if these store operations should become part of the
Cluster framework instead. If we added them to the Broker framework,
we'd have two separate store APIs there: one low-level version mapping
directly to the C++ Broker API, and one higher-level that configures
things like location of the DB files. That could be confusing.

>   Software::tracked_store = Broker::InitStore(Software::tracked_store_name);

I like this. One additional idea: while I see that it's generally the
user who wants to configure which backend to use, the script author
may know already if it's data that should be persistent across
execution; I'm guessing that's usually implied by the script's
semantics. We could give InitStore() an additional boolean
"persistent" to indicate that. If that's true, it'd use the
"default_backend" (or now maybe "default_db_backend"); if false, it'd
always use the MEMORY backend.

> # User needs to be able to choose data store backends and which cluster node 
> the
> # the master store lives on.  They can either do this manually, or BroControl
> # will autogenerate the following in cluster-layout.bro:

I don't really like the idea of autogenerating this, as it's pretty
complex information. Usually, the Broker::default_* values should be
fine, right? For the few cases where one wants to tweak that on a
per-store bassis, using a manual redef on the table sounds fine to me.

Hmm, actually, what would you think about using functions instead of
tables? We could model this similar to how the logging framework does
filters: there's a default filter installed, but you can retrieve and
update it. Here there'd be a default StoreInfo, which one can update.

> redef Broker::default_master_node = "manager";
> redef Broker::default_backend = Broker::MEMORY;
> redef Broker::default_store_dir = "/home/jon/stores";

Can the default_store_dir be set to some standard location through
BroControl? Would be neat if this all just worked in the standard case
without any custom configuration at all.

> BroControl Example Usage
> 

I'll skip commenting on this and wait for your response to the above
first, as I'm wondering if we need this BroControl functionality at
all.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Performance Enhancements

[Azoff, Justin S]

> On Oct 6, 2017, at 5:59 PM, Jim Mellander  wrote:
> 
> I particularly like the idea of an allocation pool that per-packet 
> information can be stored, and reused by the next packet.
> 
> There also are probably some optimizations of frequent operations now that 
> we're in a 64-bit world that could prove useful - the one's complement 
> checksum calculation in net_util.cc is one that comes to mind, especially 
> since it works effectively a byte at a time (and works with even byte counts 
> only).  Seeing as this is done per-packet on all tcp payload, optimizing this 
> seems reasonable.  Here's a discussion of do the checksum calc in 64-bit 
> arithmetic: https://locklessinc.com/articles/tcp_checksum/ - this website 
> also has an x64 allocator that is claimed to be faster than tcmalloc, see: 
> https://locklessinc.com/benchmarks_allocator.shtml  (note: I haven't tried 
> anything from this source, but find it interesting).
> 
> I'm guessing there are a number of such "small" optimizations that could 
> provide significant performance gains.
> 
> Take care,
> 
> Jim

I've been messing around with 'perf top', the one's complement function often 
shows up fairly high up.. that, PriorityQueue::BubbleDown, and BaseList::remove

Something (on our configuration?) is doing a lot of 
PQ_TimerMgr::~PQ_TimerMgr... I don't think I've come across that class before 
in bro.. I think a script may be triggering something that is hurting 
performance.  I can't think of what it would be though.

Running perf top on a random worker right now with -F 1 shows:

Samples: 485K of event 'cycles', Event count (approx.): 26046568975
Overhead  Shared Object Symbol
  34.64%  bro   [.] BaseList::remove
   3.32%  libtcmalloc.so.4.2.6  [.] operator delete
   3.25%  bro   [.] PriorityQueue::BubbleDown
   2.31%  bro   [.] BaseList::remove_nth
   2.05%  libtcmalloc.so.4.2.6  [.] operator new
   1.90%  bro   [.] Attributes::FindAttr
   1.41%  bro   [.] Dictionary::NextEntry
   1.27%  libc-2.17.so  [.] __memcpy_ssse3_back
   0.97%  bro   [.] StmtList::Exec
   0.87%  bro   [.] Dictionary::Lookup
   0.85%  bro   [.] NameExpr::Eval
   0.84%  bro   [.] BroFunc::Call
   0.80%  libtcmalloc.so.4.2.6  [.] tc_free
   0.77%  libtcmalloc.so.4.2.6  [.] operator delete[]
   0.70%  bro   [.] ones_complement_checksum
   0.60%  libtcmalloc.so.4.2.6  [.] 
tcmalloc::ThreadCache::ReleaseToCentralCache
   0.60%  bro   [.] RecordVal::RecordVal
   0.53%  bro   [.] UnaryExpr::Eval
   0.51%  bro   [.] ExprStmt::Exec
   0.51%  bro   [.] iosource::Manager::FindSoonest
   0.50%  libtcmalloc.so.4.2.6  [.] operator new[]


Which sums up to 59.2%

BaseList::remove/BaseList::remove_nth seems particularly easy to optimize. 
Can't that loop be replaced by a memmove?
I think something may be broken if it's being called that much though.



— 
Justin Azoff


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Performance Enhancements

[Jim Mellander]

I particularly like the idea of an allocation pool that per-packet
information can be stored, and reused by the next packet.

There also are probably some optimizations of frequent operations now that
we're in a 64-bit world that could prove useful - the one's complement
checksum calculation in net_util.cc is one that comes to mind, especially
since it works effectively a byte at a time (and works with even byte
counts only).  Seeing as this is done per-packet on all tcp payload,
optimizing this seems reasonable.  Here's a discussion of do the checksum
calc in 64-bit arithmetic: https://locklessinc.com/articles/tcp_checksum/ -
this website also has an x64 allocator that is claimed to be faster than
tcmalloc, see: https://locklessinc.com/benchmarks_allocator.shtml  (note: I
haven't tried anything from this source, but find it interesting).

I'm guessing there are a number of such "small" optimizations that could
provide significant performance gains.

Take care,

Jim




On Fri, Oct 6, 2017 at 7:26 AM, Azoff, Justin S  wrote:

>
> > On Oct 6, 2017, at 12:10 AM, Clark, Gilbert  wrote:
> >
> > I'll note that one of the challenges with profiling is that there are
> the bro scripts, and then there is the bro engine.  The scripting layer has
> a completely different set of optimizations that might make sense than the
> engine does: turning off / turning on / tweaking different scripts can have
> a huge impact on Bro's relative performance depending on the frequency with
> which those script fragments are executed.  Thus, one way to look at
> speeding things up might be to take a look at the scripts that are run most
> often and seeing about ways to accelerate core pieces of them ... possibly
> by moving pieces of those scripts to builtins (as C methods).
> >
>
> Re: scripts, I have some code I put together to do arbitrary benchmarks of
> templated bro scripts.  I need to clean it up and publish it, but I found
> some interesting things.  Function calls are relatively slow.. so things
> like
>
> ip in Site::local_nets
>
> Is faster than calling
>
> Site::is_local_addr(ip);
>
> inlining short functions could speed things up a bit.
>
> I also found that things like
>
> port == 22/tcp || port == 3389/tcp
>
> Is faster than checking if port in {22/tcp,3389/tcp}.. up to about 10
> ports.. Having the hash class fallback to a linear search when the hash
> only contains few items could speed things up there.  Things like
> 'likely_server_ports' have 1 or 2 ports in most cases.
>
>
> > If I had to guess at one engine-related thing that would've sped things
> up when I was profiling this stuff back in the day, it'd probably be
> rebuilding the memory allocation strategy / management.  From what I
> remember, Bro does do some malloc / free in the data path, which hurts
> quite a bit when one is trying to make things go fast.  It also means that
> the selection of a memory allocator and NUMA / per-node memory management
> is going to be important.  That's probably not going to qualify as
> something *small*, though ...
>
> Ah!  This reminds me of something I was thinking about a few weeks ago.
> I'm not sure to what extent bro uses memory allocation pools/interning for
> common immutable data structures.  Like for port objects or small strings.
> There's no reason bro should be mallocing/freeing memory to create port
> objects when they are only 65536 times 2 (or 3?) port objects... but bro
> does things like
>
> tcp_hdr->Assign(0, new PortVal(ntohs(tp->th_sport),
> TRANSPORT_TCP));
> tcp_hdr->Assign(1, new PortVal(ntohs(tp->th_dport),
> TRANSPORT_TCP));
>
> For every packet.  As well as allocating a ton of TYPE_COUNT vals for
> things like packet sizes and header lengths.. which will almost always be
> between 0 and 64k.
>
> For things that can't be interned, like ipv6 address, having an allocation
> pool could speed things up... Instead of freeing things like IPAddr objects
> they could just be returned to a pool, and then when a new IPAddr object is
> needed, an already initialized object could be grabbed from the pool and
> 'refreshed' with the new value.
>
> https://golang.org/pkg/sync/#Pool
>
> Talks about that sort of thing.
>
> > On a related note, a fun experiment is always to try running bro with a
> different allocator and seeing what happens ...
>
> I recently noticed our boxes were using jemalloc instead of tcmalloc..
> Switching that caused malloc to drop a few places down in 'perf top' output.
>
>
> —
> Justin Azoff
>
>
>
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] design summary: porting Bro scripts to use Broker

[Azoff, Justin S]

> On Oct 6, 2017, at 12:53 PM, Siwek, Jon  wrote:
> 
> I want to check if there’s any feedback on the approach I’m planning to take 
> when porting over Bro’s scripts to use Broker.  There’s two major areas to 
> consider: (1) how users specify network topology e.g. either for traditional 
> cluster configuration or manually connecting Bro instances and (2) replacing 
>  with Broker’s distributed data storage features.
> 

...

> Then subscriptions and auto-publications still get automatically set up by 
> the cluster framework in bro_init().
> 
> Other Manual/Custom Topologies
> --
> 
> I don’t see anything to do here as the Broker API already has enough to set 
> up peerings and subscriptions in arbitrary ways.  The old “communication” 
> framework scripts can just go away as most of its functions have direct 
> corollaries in the new “broker” framework.
> 
> The one thing that is missing is the “Communication::nodes” table which acts 
> as both a state-tracking structure and an API that users may use to have the 
> comm. framework automatically set up connections between the nodes in the 
> table.  I find this redundant — there’s two APIs to accomplish the same 
> thing, with the table being an additional layer of indirection to the actual 
> connect/listen functions a user can just as easily use themselves.  I also 
> think it’s not useful for state-tracking as a user operating at the level of 
> this use-case is can easily track nodes themselves or has some other notion 
> of the state structures they need to track that is more intuitive for the 
> particular problem they're solving.  Unless there’s arguments or I find it’s 
> actually needed, I don’t plan to port this to Broker.
> 

I had some feedback related to this sort of thing earlier in the year:

http://mailman.icsi.berkeley.edu/pipermail/bro-dev/2017-February/012386.html
http://mailman.icsi.berkeley.edu/pipermail/bro-dev/2017-March/012411.html

I got send_event_hashed to work via a bit of a hack 
(https://github.com/JustinAzoff/broker_distributed_events/blob/master/distributed_broker.bro),
but it needs support from inside broker or at least the bro/broker integration 
to work properly in the case of node failure.

My ultimate vision is a cluster with 2+ physical datanode/manager/logger boxes 
where one box can fail and the cluster will continue to function perfectly.
The only thing this requires is a send_event_hashed function that does 
consistent ring hashing and is aware of node failure.

For things that don't need necessarily need consistent partitioning - like 
maybe logs if you were using Kafka, a way to designate that a topic should be 
distributed round-robin between subscribers would be useful too.



— 
Justin Azoff

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] design summary: porting Bro scripts to use Broker

[Siwek, Jon]

I want to check if there’s any feedback on the approach I’m planning to take 
when porting over Bro’s scripts to use Broker.  There’s two major areas to 
consider: (1) how users specify network topology e.g. either for traditional 
cluster configuration or manually connecting Bro instances and (2) replacing 
 with Broker’s distributed data storage features.


Broker-Based Topology
=

It’s again useful to decompose topology specification it into two main 
use-cases:

Creating Clusters, e.g w/ BroControl


This use-case should look familiar once ported to use Broker: the existing 
“cluster” framework will be used for specifying the topology of the cluster and 
for automatically setting up the connections between nodes.  The one thing that 
will differ is the event subscription mechanism, which needs to change since 
Broker itself handles that differently, but I think the general idea can remain 
similar.

The current mechanism for handling event subscription/publication:

const Cluster::manager2worker_events = /Drop::.*/ 
# similar patterns follow for all node combinations...

And a script author that is making their script usable on clusters writes:

redef Cluster::manager2worker_events += 
/^Intel::(cluster_new_item|purge_item)$/;

The new mechanism:

# contains topic prefixes
const Cluster::manager_subscriptions: set[string] 

# contains (topic string, event name) pairs
const Cluster::manager_publications: set[string, string] 

# similar sets follow for all node types…

And a script author writes:

# topic naming convention relates to file hierarchy/organization of 
scripts
redef Cluster::manager_subscriptions += {
"bro/event/framework/control",
"bro/event/framework/intel",
};

# not sure how to get around referencing events via strings: can't use 
'any'
# to stick event values directly into the set, maybe that’s ok since we 
can
# at least detect lookup failures at bro_init time and emit 
errors/abort.
redef Cluster::manager_publications += {
["bro/event/framework/control/configuration_update_request",
 "Control::configuration_update_request"],
["bro/event/framework/intel/cluster_new_item",
 "Intel::cluster_new_item"],
};

Then subscriptions and auto-publications still get automatically set up by the 
cluster framework in bro_init().

Other Manual/Custom Topologies
--

I don’t see anything to do here as the Broker API already has enough to set up 
peerings and subscriptions in arbitrary ways.  The old “communication” 
framework scripts can just go away as most of its functions have direct 
corollaries in the new “broker” framework.

The one thing that is missing is the “Communication::nodes” table which acts as 
both a state-tracking structure and an API that users may use to have the comm. 
framework automatically set up connections between the nodes in the table.  I 
find this redundant — there’s two APIs to accomplish the same thing, with the 
table being an additional layer of indirection to the actual connect/listen 
functions a user can just as easily use themselves.  I also think it’s not 
useful for state-tracking as a user operating at the level of this use-case is 
can easily track nodes themselves or has some other notion of the state 
structures they need to track that is more intuitive for the particular problem 
they're solving.  Unless there’s arguments or I find it’s actually needed, I 
don’t plan to port this to Broker.


Broker-Based Data Distribution
==

Replacing  requires completely new APIs that script authors can 
easily use to work for both cluster and non-cluster use-cases and independently 
of a user’s choice of persistent storage backend.

Broker Framework API


const Broker::default_master_node = "manager" 

const Broker::default_backend = MEMORY 

# Setting a default dir will, for persistent backends that have not
# been given an explicit file path, automatically create a path within this
# dir that is based on the name of the data store.
const Broker::default_store_dir = "" 

type Broker::StoreInfo: record {
  name: string 
  store: opaque of Broker::Store 
  master_node: string =Broker::default_master_node;
  master: bool =F;
  backend: Broker::BackendType =default_backend;
  options: Broker::BackendOptions =Broker::BackendOptions();
};

# Primarily used by users to set up master store location and backend
# configuration, but also possible to lookup an existing/open store by name.
global Broker::stores: table[string] of StoreInfo =StoreInfo() 

# Set up data stores to properly function regardless of whether user is
# operating a cluster.  This also automatically sets up the store to
# be a clone or a master as is 

Re: [Bro-Dev] Performance Enhancements

[Azoff, Justin S]

> On Oct 6, 2017, at 12:10 AM, Clark, Gilbert  wrote:
> 
> I'll note that one of the challenges with profiling is that there are the bro 
> scripts, and then there is the bro engine.  The scripting layer has a 
> completely different set of optimizations that might make sense than the 
> engine does: turning off / turning on / tweaking different scripts can have a 
> huge impact on Bro's relative performance depending on the frequency with 
> which those script fragments are executed.  Thus, one way to look at speeding 
> things up might be to take a look at the scripts that are run most often and 
> seeing about ways to accelerate core pieces of them ... possibly by moving 
> pieces of those scripts to builtins (as C methods).
> 

Re: scripts, I have some code I put together to do arbitrary benchmarks of 
templated bro scripts.  I need to clean it up and publish it, but I found some 
interesting things.  Function calls are relatively slow.. so things like

ip in Site::local_nets

Is faster than calling

Site::is_local_addr(ip);

inlining short functions could speed things up a bit.

I also found that things like

port == 22/tcp || port == 3389/tcp

Is faster than checking if port in {22/tcp,3389/tcp}.. up to about 10 ports.. 
Having the hash class fallback to a linear search when the hash only contains 
few items could speed things up there.  Things like 'likely_server_ports' have 
1 or 2 ports in most cases.


> If I had to guess at one engine-related thing that would've sped things up 
> when I was profiling this stuff back in the day, it'd probably be rebuilding 
> the memory allocation strategy / management.  From what I remember, Bro does 
> do some malloc / free in the data path, which hurts quite a bit when one is 
> trying to make things go fast.  It also means that the selection of a memory 
> allocator and NUMA / per-node memory management is going to be important.  
> That's probably not going to qualify as something *small*, though ...

Ah!  This reminds me of something I was thinking about a few weeks ago.  I'm 
not sure to what extent bro uses memory allocation pools/interning for common 
immutable data structures.  Like for port objects or small strings.  There's no 
reason bro should be mallocing/freeing memory to create port objects when they 
are only 65536 times 2 (or 3?) port objects... but bro does things like

tcp_hdr->Assign(0, new PortVal(ntohs(tp->th_sport), TRANSPORT_TCP));
tcp_hdr->Assign(1, new PortVal(ntohs(tp->th_dport), TRANSPORT_TCP));

For every packet.  As well as allocating a ton of TYPE_COUNT vals for things 
like packet sizes and header lengths.. which will almost always be between 0 
and 64k.

For things that can't be interned, like ipv6 address, having an allocation pool 
could speed things up... Instead of freeing things like IPAddr objects they 
could just be returned to a pool, and then when a new IPAddr object is needed, 
an already initialized object could be grabbed from the pool and 'refreshed' 
with the new value.

https://golang.org/pkg/sync/#Pool

Talks about that sort of thing.

> On a related note, a fun experiment is always to try running bro with a 
> different allocator and seeing what happens ...

I recently noticed our boxes were using jemalloc instead of tcmalloc.. 
Switching that caused malloc to drop a few places down in 'perf top' output.


— 
Justin Azoff



___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev