Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-20 Thread Siwek, Jon 

> On Apr 20, 2017, at 12:40 PM, Robin Sommer  wrote:
> 
> I need to think about that argument ... Did you try reading from files
> while also doing communication (that would be pseudo-realtime mode),
> or was the pcap the only source of input?

Tested:

* pcap
* pcap + script doing DNS queries
* live interface
* live interface + script doing DNS queries 

Untested:

* anything with remote communication
* pseudo-realtime

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-20 Thread Robin Sommer 


On Fri, Apr 14, 2017 at 17:32 +, you wrote:

> Just mentioning it in case you didn’t account for the real fix also
> requiring the CAF-based loop being fully realized in addition to
> Broker

Yeah, true, I was thinking that eventually we will have this all solved.

>  (Also don’t have a sense of the frequency/urgency of the problem).

I think that's the main question. So far I haven't gotten the sense
that this really affects a lot of people, so I see the priority as
rather low given our limited cycles for development and testing. If
it's a more pressing problem, we can reconsider of course.

> So since I’ve been able to get the CAF-based loop working on offline
> pcap files (it does not rely on polling the FD of the open file since
> it didn't work anyway w/ CAF's epoll-based multiplexer on Linux), it
> may be fair to say that other packet sources that don’t
> require/support poll-ability should also be possible to integrate.

I need to think about that argument ... Did you try reading from files
while also doing communication (that would be pseudo-realtime mode),
or was the pcap the only source of input?

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-19 Thread Siwek, Jon 

> On Apr 19, 2017, at 2:55 AM, Robin Sommer  wrote:
> 
>> Just a quick comment here regarding FreeBSD: the native polling
>> mechanism is kqueue, and CAF still lacks support for it [1].
>> Fortunately, this is a rather straight-forward task.
> 
> Oh, sounds like that would be high-priority task then before we'd
> consider moving to a CAF-based loop?

It still falls back to ‘poll’ on non-Linux.  If more performance tests are to 
be done in realistic conditions (live traffic + cluster communication) on 
various platforms, you’d likely find out at that point whether it’s a 
high-priority task.  I expect ‘poll’ might still be usable if the common case 
is only going to max out around the order of 100s of peers.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-19 Thread Robin Sommer 


On Fri, Apr 14, 2017 at 07:56 -0700, you wrote:

> Just a quick comment here regarding FreeBSD: the native polling
> mechanism is kqueue, and CAF still lacks support for it [1].
> Fortunately, this is a rather straight-forward task.

Oh, sounds like that would be high-priority task then before we'd
consider moving to a CAF-based loop?

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-14 Thread Siwek, Jon 

> On Apr 14, 2017, at 8:32 AM, Robin Sommer  wrote:
> 
> - I don't think we should spend time anymore on improving the old
>  communication code. We're getting close to retire that now and a
>  number of its issues (like selects in the child process) will just
>  go away with that. Let's focus on the new setting where Broker/CAF
>  will be doing all communication.

If people are hitting the 1024 FD hard-limit in the old comm. code’s select(), 
that would indeed go away with the change to Broker.  But I think the way 
Broker is integrated in the parent’s main loop still relies on a select(), with 
the number of FDs it monitors scaling with the number of peers.  i.e. there may 
still be critical errors w/ large Bro clusters even using Broker as the 
communication system, just this time the problem manifests in the main loop.

Just mentioning it in case you didn’t account for the real fix also requiring 
the CAF-based loop being fully realized in addition to Broker — I’m less 
certain about the timeline of finishing up the CAF-based loop compared to just 
patching in a temporary stopgap of patching out the select() calls.  (Also 
don’t have a sense of the frequency/urgency of the problem).

> - Regarding optimizing for different use cases: I would prefer
>  avoiding having lots of knobs to configure the specifics of the
>  loop. We have these magic values in the current I/O loop where
>  nobody knows how to pick them because it's hard to understand their
>  impact; and where folks have played with them, it was always hard
>  conclude much about them beyond any specific setting. What we could
>  try instead is a loop that adjusts itself based on load patterns: if
>  the load is heavy on packets, build larger batches to process
>  between polls; if input comes from lots different sources, increase
>  the polling; etc.

That seems like a Good Idea.

>  it does pose the question if/how can
>  integrate packet sources that either don't need or don't support
>  select/poll

I think that’s just a matter of making sure the main loop “spins” at an 
appropriate frequency, which might change dynamically, dependent on loading 
pattern optimizations, as per the above idea.

Maybe you could even think of reading an offline pcap file as a source that 
doesn’t need select/poll.  Pedantically, regular files also don't “support” 
select(), at least not w/ the same intention (nonblocking IO), but it just 
happens to work fine in the current runloop implementation.

So since I’ve been able to get the CAF-based loop working on offline pcap files 
(it does not rely on polling the FD of the open file since it didn't work 
anyway w/ CAF's epoll-based multiplexer on Linux), it may be fair to say that 
other packet sources that don’t require/support poll-ability should also be 
possible to integrate.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-14 Thread Robin Sommer 
Nice, thanks for the doing these measurements! I haven't looked at the
code yet, but some quick thoughts on your results and some of the
other comments this thread, and then some suggested next steps at the
end.

- Agree that overall your numbers suggest that all these mechanisms
  are fine performancewise, assuming we keep the optimization to batch
  packets between polls/selects to avoid the
  one-system-call-per-packet overhead.

- I don't think we should spend time anymore on improving the old
  communication code. We're getting close to retire that now and a
  number of its issues (like selects in the child process) will just
  go away with that. Let's focus on the new setting where Broker/CAF
  will be doing all communication.

- Regarding optimizing for different use cases: I would prefer
  avoiding having lots of knobs to configure the specifics of the
  loop. We have these magic values in the current I/O loop where
  nobody knows how to pick them because it's hard to understand their
  impact; and where folks have played with them, it was always hard
  conclude much about them beyond any specific setting. What we could
  try instead is a loop that adjusts itself based on load patterns: if
  the load is heavy on packets, build larger batches to process
  between polls; if input comes from lots different sources, increase
  the polling; etc. Any heuristic here would need to stay quite simple
  (otherwise we'd again end up not being able to predict much), but I
  think that'd be worth a try.

- Gilbert's point on high-performance IPC is a good one. I don't think
  we want to switch to direct memory access as our main model for the
  time being at least, but it does pose the question if/how can
  integrate packet sources that either don't need or don't support
  select/poll. (Which, in a nod to history, accounts for some of the
  complexities of the current loop because many years ago some pcaps
  didn't support select)


In terms of next steps, we need to see if these results hold across
different OSs, and also with live traffic. The two questions are (1)
does the new loop function on all platforms with both low- and
high-volume live traffic (presumably it will but that needs double
checking, given the history of weird OS-specific effects); and (2)
does performance match the measurements shown so far? If we can
confirm that on at least Linux and FreeBSD for, say, the two most
recent major releases of each and also consider common alternative
capturing solutions (pfring, netmap, afnet?), I'd be pretty
comfortable switching.

Robin

-- 
Robin Sommer * ICSI/LBNL * ro...@icir.org * www.icir.org/robin
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-13 Thread Siwek, Jon 

> On Apr 12, 2017, at 10:19 PM, Azoff, Justin S  wrote:
> 
> Something that may be optimized for a worker dealing with 1 pktsrc and 2 
> peers may not be as optimal for a logger/manager that has no pktsrc but 100+ 
> worker connections.  I've often wondered if the event loop should have a hint 
> somewhere about which kind of process is running so it can optimize for 
> throughput vs multiplexing many peers.

Yeah, I’ve thought the same and related with takeaway (2) that I mention in the 
original post.  It seems like it would be nice to have a more well-defined 
system for specifying IOSource prioritization or at least between packet 
sources and other io sources.  Then, since it’s hard to nail down settings that 
are going to work for all deployments in general, it would also have ways to 
tune it via scripts so it would be open for a user to tweak settings that may 
improve for their particular manager, logger, worker, or whatever Bro node they 
have.

But at the moment, I don’t think there’s a whole lot of info on exactly what 
tweaks can be made to optimize for the “no pktsrc, but lots of remote comm.” 
case, and it may be best to wait for the broker integration to become fully 
realized before investigating that.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-13 Thread Siwek, Jon 

> On Apr 12, 2017, at 10:19 PM, Clark, Gilbert  wrote:
> 
> Also, relative overhead of packet ingest is going to vary based on the set of 
> loaded scripts in addition to the specific trace used to run the tests.  
> That's not trying to argue that these results are not useful / interesting, 
> but instead *only* that the specific percentages might not be representative 
> of the general case (just because I'm convinced that there really is not a 
> general case to objectively measure).

I agree, the specific numbers here aren’t generalizable, but I think that’s ok 
and we can still infer that the different runloop implementation doesn’t raise 
any obvious performance concern.  That being due to (1) with the tests using 
the default set of Bro scripts, I’d expect it to be more common for users to 
have more complicated scripts and highly customized deployments such that the 
relative overhead decreases further and becomes more irrelevant than the tests 
show and (2) even if the specific pcaps tested were at either end of the 
spectrum in terms of how much work is required to process them, it still shows 
that the relative overhead differences are minimal.

i.e. I think we’d only be in trouble interpreting the results if the tests 
showed a significant relative overhead difference.  Since then we don’t know if 
the given pcaps where just “easy” ones for Bro to process.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-13 Thread Slagell, Adam J 
That might be useful. I would like Robin’s thoughts, too.

On Apr 12, 2017, at 9:05 PM, Siwek, Jon 
> wrote:

In the near-term, I can make a totally separate code branch that simply 
replaces select() with epoll.  Then, if Justin were to test it and find it 
alleviates performance pains on the manager, it could potentially get merged 
into bro/master ahead of the any of the pending broker/caf/runloop projects 
since it should be a trivial and safe change to do.  Let me know.

--

Adam J. Slagell
Director, Cybersecurity & Networking Division
Chief Information Security Officer
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
www.slagell.info

"Under the Illinois Freedom of Information Act (FOIA), any written 
communication to or from University employees regarding University business is 
a public record and may be subject to public disclosure."








___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-12 Thread Clark, Gilbert 
$0.02 USD:


As I recall, Bro's per-packet processing overhead can vary significantly as a 
result of timers and triggers that execute on a situational basis.  Also, 
relative overhead of packet ingest is going to vary based on the set of loaded 
scripts in addition to the specific trace used to run the tests.  That's not 
trying to argue that these results are not useful / interesting, but instead 
*only* that the specific percentages might not be representative of the general 
case (just because I'm convinced that there really is not a general case to 
objectively measure).


Also ... if the overhead of the polling / ingest itself turns out to be a huge 
problem at high rates, one idea would be to separate that and pass packets (in 
bulk) through a ring / high-speed IPC to the process that needs to ingest them. 
 That's worked pretty well for me in DPDK, and has the benefit of being able to 
distribute packets from one ingest to multiple processors (which is something 
I've had to do for process-heavy workloads ... which I would argue is something 
that Bro tends to be).


Along those lines, rather than spending much time on packet ingest mechanics in 
bro (or pieces thereof), one idea might be to instead focus on integrating 
packet bricks as a standard ingest / distribution mechanic for everything 
packet-related in the general case.  The idea would be that fetching packets 
from bro (and its related processes) would become less about calls to epoll and 
select, and more about high-speed IPC that went out of its way to avoid 
kernel-space entirely.  The nice thing about that is that it'd be a little 
easier to standardize on the bro side of things, and would take a step toward 
separating bro as a scripting / event engine from bro as a (relative) monolith.


Of course, the down side is that packet bricks could add some serious 
(mandatory) complexity to bro, so maybe it's not the right answer ... but maybe 
a more lightweight, specialized distribution channel might be doable, or maybe 
there would be a way to embed packet bricks inside of an application in the 
event that folks didn't want to run the two separately, or ... etc.


As always, just for what it's worth :)


-Gilbert



From: bro-dev-boun...@bro.org <bro-dev-boun...@bro.org> on behalf of Siwek, Jon 
<jsi...@illinois.edu>
Sent: Tuesday, April 11, 2017 8:41:23 PM
To: <bro-dev@bro.org>
Subject: [Bro-Dev] early performance comparisons of CAF-based run loop

I recently got a minimal CAF-based run loop for Bro working, did crude 
performance comparisons, and wanted to share.

The approach was to measure average time between calls of net_packet_dispatch() 
and also the average time it takes to analyze a packet.  The former attempts to 
measure the overhead imposed by the loop implementation and the later just 
gives an idea of how significant a chunk of time that is in relation to Bro’s 
main workload.  I found that the overhead of the loop can be ~5-10% of the 
packet processing time, so it does seem worthwhile to try and keep the run loop 
overhead low.

Initial testing of the CAF-based loop showed the overhead increased by ~1.8x, 
but there was still a major difference in the implementations: the standard Bro 
loop only invokes its IOSource polling mechanism (select) once every 25 cycles 
of the loop, while the CAF implementation’s polling mechanism (actor/thread 
scheduling + messaging + epoll) is used for every cycle/packet.  As one would 
expect, by just trivially spinning the main process() function in a loop for 25 
iterations, the overhead of the CAF-based loop comes back into line with the 
standard run loop.

To try and better measure the actual differences related to the polling 
mechanism implementation, I quickly hacked Bro’s standard runloop to select() 
on every packet instead of once every 25th and found that the overhead measures 
+/- 10% within the 1.8x overhead increase of the initial CAF-based loop.  So is 
the cost of the extra system call for epoll/select per packet the main thing to 
avoid?  Sort of.  I again hacked Bro’s standard loop to be able to use either 
epoll or poll instead of select and found that those do better, with the 
overhead increase being about 1.3x (still doing one “poll” per packet) in 
relation to the standard run loop.  Meaning there is some measurable trend in 
polling mechanism performance (for sparse # of FDs/sources): poll comes in 
first, epoll second, with CAF and select about tied for third.

Takeaways:

(1) Regardless of runloop implementation or polling mechanism choices, 
performing the polling operation once per packet should probably be avoided.  
In concept, it’s an easy way to get a 2-5% speedup in relation to total packet 
processing time.

(2) Related to (1), but not in the sense of performance, is that even w/ a 
CAF-based loop it still seems somewhat difficult to reason about the reality of 
how IOSources are prioritized.  In

Re: [Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-12 Thread Siwek, Jon 

> On Apr 12, 2017, at 1:35 PM, Slagell, Adam J  wrote:
> 
> Justin asked an interesting question today, how does this affect performance 
> on the manager? That is where we are feeling a lot of pain with select().

If you mean the select() that’s in the process fork’d by the old 
RemoteSerializer code, you’d still see the same problems with the CAF-based 
runloop.  But that code is irrelevant once Broker takes its place. i.e. to 
answer that question, you need to design a communication stress test using 
Broker-based Bros as that’s more relevant than just changing the main loop.

Eventually, I can also imagine the Broker-based communication being more 
tightly integrated into the CAF-based runloop helping improve performance over 
the current Broker integration method.  Either way, what needs to be measured 
is how CAF’s multiplexer performs in relation to Bro’s communication patterns, 
but maybe still want to wait for the Broker improvements to wrap up before 
looking into doing those tests.

In the near-term, I can make a totally separate code branch that simply 
replaces select() with epoll.  Then, if Justin were to test it and find it 
alleviates performance pains on the manager, it could potentially get merged 
into bro/master ahead of the any of the pending broker/caf/runloop projects 
since it should be a trivial and safe change to do.  Let me know.

- Jon

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] early performance comparisons of CAF-based run loop

2017-04-11 Thread Siwek, Jon 
I recently got a minimal CAF-based run loop for Bro working, did crude 
performance comparisons, and wanted to share.

The approach was to measure average time between calls of net_packet_dispatch() 
and also the average time it takes to analyze a packet.  The former attempts to 
measure the overhead imposed by the loop implementation and the later just 
gives an idea of how significant a chunk of time that is in relation to Bro’s 
main workload.  I found that the overhead of the loop can be ~5-10% of the 
packet processing time, so it does seem worthwhile to try and keep the run loop 
overhead low.

Initial testing of the CAF-based loop showed the overhead increased by ~1.8x, 
but there was still a major difference in the implementations: the standard Bro 
loop only invokes its IOSource polling mechanism (select) once every 25 cycles 
of the loop, while the CAF implementation’s polling mechanism (actor/thread 
scheduling + messaging + epoll) is used for every cycle/packet.  As one would 
expect, by just trivially spinning the main process() function in a loop for 25 
iterations, the overhead of the CAF-based loop comes back into line with the 
standard run loop.

To try and better measure the actual differences related to the polling 
mechanism implementation, I quickly hacked Bro’s standard runloop to select() 
on every packet instead of once every 25th and found that the overhead measures 
+/- 10% within the 1.8x overhead increase of the initial CAF-based loop.  So is 
the cost of the extra system call for epoll/select per packet the main thing to 
avoid?  Sort of.  I again hacked Bro’s standard loop to be able to use either 
epoll or poll instead of select and found that those do better, with the 
overhead increase being about 1.3x (still doing one “poll” per packet) in 
relation to the standard run loop.  Meaning there is some measurable trend in 
polling mechanism performance (for sparse # of FDs/sources): poll comes in 
first, epoll second, with CAF and select about tied for third.

Takeaways:

(1) Regardless of runloop implementation or polling mechanism choices, 
performing the polling operation once per packet should probably be avoided.  
In concept, it’s an easy way to get a 2-5% speedup in relation to total packet 
processing time.

(2) Related to (1), but not in the sense of performance, is that even w/ a 
CAF-based loop it still seems somewhat difficult to reason about the reality of 
how IOSources are prioritized.  In the standard loop, the priority of an 
IOSource is a combination of its “idle” state, the polling frequency, and a 
timestamp, which it often chooses arbitrarily as the “time of last packet”, 
just so that it gets processed with higher priority than subsequent packets.  
Maybe the topic of making IOSource prioritization more explicit/well-defined 
could be another thread of discussion, but my initial thought is that the whole 
IOSource abstraction may be over-generalized and maybe not even needed.

(3) The performance overhead of a CAF-based loop doesn’t seem like a 
showstopper for proceeding with it as a choice for replacing the current loop.  
It’s not significantly worse than the current loop (provided we still throttle 
the polling ratio when packet sources are saturated), and even using the most 
minimal loop implementation of just poll() would only be about a 1% speedup in 
relation to the total packet processing workload.

Just raw data below, for those interested:

I tested against the pcaps from http://tcpreplay.appneta.com/wiki/captures.html
(I was initially going to use tcpreplay to test performance against a live 
interface, but decided reading from a file is easier and just as good for what 
I wanted to measure).
Numbers are measured in “ticks”, which are equivalent to nanoseconds on the 
test system.
Bro and CAF are both compiled w/ optimizations.

bigFlows.pcap, 1 “poll" per packet
--
poll
('avg overhead', 1018.886823998)
('avg process', 11664.4968147)

epoll
('avg overhead', 1114.216809699)
('avg process', 11680.6078816)

CAF
('avg overhead', 1515.993334396)
('avg process', 11914.89710923)

select
('avg overhead', 1792.814291095)
('avg process', 11863.30855041)

bigFlows.pcap, Polling Throttled to 1 per 25 packets
---
poll
('avg overhead', 772.611834799)
('avg process', 11504.2397625)

epoll
('avg overhead', 814.4771509)
('avg process', 11547.05839491)

CAF
('avg overhead', 847.6571822)
('avg process', 11681.37797272)

select
('avg overhead', 855.214749401)
('avg process', 11585.236)

smallFlows.pcap, 1 “poll" per packet

poll
('avg overhead', 1403.895028084)
('avg process', 22202.960570839998)

epoll
('avg overhead', 1470.0554376)
('avg process', 22210.3240474)

select
('avg overhead', 2305.627842926)
('avg process', 22549.29251384)

CAF
('avg overhead', 2405.140109335)
('avg process', 23401.66596454)

smallFlows.pcap, Polling