Re: [Bro-Dev] Performance Enhancements

2017-10-14 Thread Jim Mellander
Yeh, the lockless implementation has a bug: if (size) s/b if (size & 1) I ended up writing an checksum routine that sums 32 bits at a time into a 64 bit register, which avoids the need to check for overflow - it seems to be faster than the full 64 bit implementation - will test with Bro and

Re: [Bro-Dev] Performance Enhancements

2017-10-12 Thread Azoff, Justin S
> On Oct 6, 2017, at 5:59 PM, Jim Mellander wrote: > > I particularly like the idea of an allocation pool that per-packet > information can be stored, and reused by the next packet. > > There also are probably some optimizations of frequent operations now that > we're in

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Azoff, Justin S
> On Oct 9, 2017, at 4:57 PM, Jim Mellander wrote: > > Well, I found pathological behavior with BaseList::remove > > Instrumenting it with a printf of num_entries & i after the for loop, running > against a test pcap then summarizing with awk gives: > > Count,

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Azoff, Justin S
> On Oct 9, 2017, at 4:03 PM, Clark, Gilbert wrote: > > If you look in one of the subdirectories or another, in ages past there was a > little shell script to incrementally execute bro against a specific trace, > loading one script at a time to see the effect each of them

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Jim Mellander
gh: once > you hit about 10 - 15 Gbps, even a *cache miss* on a modern system is > enough to force a drop ... > > > For what it's worth ... > > > -Gilbert > > > -------------- > *From:* Azoff, Justin S <jaz...@illinois.edu> > *Sent:* Monday, October 9, 2017 10:

Re: [Bro-Dev] Performance Enhancements

2017-10-09 Thread Azoff, Justin S
> On Oct 6, 2017, at 5:59 PM, Jim Mellander wrote: > > I particularly like the idea of an allocation pool that per-packet > information can be stored, and reused by the next packet. Turns out bro does this most of the time.. unless you use the next_packet event. Normal

Re: [Bro-Dev] Performance Enhancements

2017-10-06 Thread Jim Mellander
Interesting info. The > order of magnitude difference in time between BaseList::remove & BaseList::removenth suggests the possibility that the for loop in BaseList::remove is falling off the end in many cases (i.e. attempting to remove an item that doesn't exist). Maybe thats whats broken. On

Re: [Bro-Dev] Performance Enhancements

2017-10-06 Thread Azoff, Justin S
> On Oct 6, 2017, at 5:59 PM, Jim Mellander wrote: > > I particularly like the idea of an allocation pool that per-packet > information can be stored, and reused by the next packet. > > There also are probably some optimizations of frequent operations now that > we're in

Re: [Bro-Dev] Performance Enhancements

2017-10-06 Thread Jim Mellander
I particularly like the idea of an allocation pool that per-packet information can be stored, and reused by the next packet. There also are probably some optimizations of frequent operations now that we're in a 64-bit world that could prove useful - the one's complement checksum calculation in

Re: [Bro-Dev] Performance Enhancements

2017-10-06 Thread Azoff, Justin S
> On Oct 6, 2017, at 12:10 AM, Clark, Gilbert wrote: > > I'll note that one of the challenges with profiling is that there are the bro > scripts, and then there is the bro engine. The scripting layer has a > completely different set of optimizations that might make sense

Re: [Bro-Dev] Performance Enhancements

2017-10-05 Thread Clark, Gilbert
y, October 5, 2017 3:45:21 PM To: bro-dev@bro.org Subject: [Bro-Dev] Performance Enhancements Hi all: One item of particular interest to me from Brocon was this tidbit from Packetsled's lightning talk: "Optimizing core loops (like net_run() ) with preprocessor branch prediction macros like

Re: [Bro-Dev] Performance Enhancements

2017-10-05 Thread Slagell, Adam J
On Oct 5, 2017, at 2:45 PM, Jim Mellander > wrote: 1. Obviously, branch prediction, as mentioned above. 3% speedup for (almost) free is nothing to sneeze at. 2. Profiling bro to identify other hot spots that could benefit from optimization. 3.

[Bro-Dev] Performance Enhancements

2017-10-05 Thread Jim Mellander
Hi all: One item of particular interest to me from Brocon was this tidbit from Packetsled's lightning talk: "Optimizing core loops (like net_run() ) with preprocessor branch prediction macros likely() and unlikely() for ~3% speedup. We optimize for maximum load." After conversing with Leo