Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?
On Wed, Jun 6, 2018 at 2:10 PM Azoff, Justin S wrote: > I haven't noticed a huge difference on a real multi process cluster, at least > not a 30x change, so this is odd that pcap processing is so much slower. > Especially since broker should be completely disabled when pcap files are > being read and caf shouldn't even be doing anything. It's not that simple. You could think of broker being enabled in this case of simply reading a pcap because it was querying the library for whether there were any peers (and that was actually a at least part of the perf. problem). Beyond that, you can still think of broker being enabled when you're reading a pcap and you are also using data stores (which the Known* scripts in Bro now do by default). Communication with a local master data store is still asynchronous communication (with local threads/actors) that ends up going through CAF messaging. There's also essentially a per-packet synchronization being done between Bro and Broker/CAF now to ensure that this type of asynchronous workload ends up producing the same results between any given Bro run. So there's now also that bit of extra overhead in pcap processing. - Jon ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?
On Wed, Jun 6, 2018 at 12:01 PM McMullan, Tim wrote: > Is the Bro development team still optimizing the Broker/Actor framework? Yes, in the sense that optimizations will be done according to feedback. No, in the sense that there was no one actively looking into it at the moment you asked. > It might be helpful to have a way to disable Broker for those of us who > haven’t migrated to it yet. That's unlikely at this point as scripts have been completely ported to Broker now and usages of the old communication system completely removed from them. No simple switch to it back, so your feedback is especially important/helpful. > # ~1GB file time (old) > > $ time /hostname/bro-devel/bin/bro -r > 20180606-1049-prodfilers-truncated_0_20180606104904.pcap master.bro > > > > real0m2.294s > > user0m1.862s > > sys 0m0.385s > > > > # ~1GB file time (new) > > $ time /hostname/bro-devel/bin/bro -r > 20180606-1049-prodfilers-truncated_0_20180606104904.pcap master.bro > > > > real1m11.458s > > user0m58.933s > > sys 1m34.074s Try pulling in the change I just did at [1], which was a big part of the problem in my own test: # 2.5.3 $ time bro -r 2009-M57-day11-18.trace real 0m16.187s user 0m16.312s sys 0m1.865s # master before [1] $ time bro -r ../testing/external/bro-testing/Traces/2009-M57-day11-18.trace real 1m31.434s user 1m44.925s sys 1m4.947s # master after [1] $ time bro -r 2009-M57-day11-18.trace real 0m24.595s user 0m25.574s sys 0m5.816s - Jon [1] https://github.com/bro/bro/commit/9822fc252d5e92208704df4a388ea31989869499 ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
Re: [Bro-Dev] Performance Issues after the fe7e1ee commit?
> On Jun 6, 2018, at 12:54 PM, McMullan, Tim wrote: > > We are running into performance issues (30x slower) since the Broker patch > (fe7e1ee) – > > We have 40G connections tapped from our storage filers feeding multiple Bro > instances which analyze specifically only NFS and SMB traffic; all other > analyzers are disabled. With the broker patch we are seeing processing times > for a ~1GB pcap jump from around 2 seconds to over 1 minute. Profiling Bro, > it looks like the culprit is the new Actor functions -- > > # Before patch > Overhead Shared Object Symbol > 14.57% [kernel] [k] copy_user_enhanced_fast_string >3.20% bro [.] EventHandler::operator bool >2.99% bro [.] _siphash >2.89% bro [.] Dictionary::Lookup > > # After patch > Overhead Shared Object Symbol >5.71% [kernel] [k] native_write_msr_safe >3.84% libcaf_core.so.0.15.7 [.] > caf::scheduler::worker::run >3.71% libcaf_core.so.0.15.7 [.] > caf::detail::double_ended_queue::take_head >3.29% [kernel] [k] _raw_spin_lock > > Is the Bro development team still optimizing the Broker/Actor framework? It > might be helpful to have a way to disable Broker for those of us who haven’t > migrated to it yet. > > # ~1GB file time (old) > $ time /hostname/bro-devel/bin/bro -r > 20180606-1049-prodfilers-truncated_0_20180606104904.pcap master.bro > > real0m2.294s > user 0m1.862s > sys 0m0.385s > > # ~1GB file time (new) > $ time /hostname/bro-devel/bin/bro -r > 20180606-1049-prodfilers-truncated_0_20180606104904.pcap master.bro > > real1m11.458s > user0m58.933s > sys 1m34.074s Well, I can at least reproduce that here on one of the standard test pcap files and no scripts loaded: $ /usr/local/bin/bro --version /usr/local/bin/bro version 2.5.2 $ time /usr/local/bin/bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace real0m2.317s user0m1.874s sys 0m0.327s $ . ~/src/bro/build/bro-path-dev.sh $ bro --version bro version 2.5-648 $ time bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace real1m5.523s user0m30.565s sys 0m32.661s and the same thing in perf: Samples: 270K of event 'cycles:u', Event count (approx.): 131984875188 Overhead Command Shared Object Symbol 8.95% bro libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_tail 5.90% bro libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_head 4.92% bro libcaf_core.so.0.15.7 [.] caf::scheduler::worker::run 2.67% bro libc-2.17.so [.] __GI___libc_nanosleep 1.98% bro libtcmalloc.so.4.4.5 [.] operator new[] 1.79% bro libc-2.17.so [.] __sleep 1.68% bro libtcmalloc.so.4.4.5 [.] tc_deletearray_nothrow 1.47% bro libc-2.17.so [.] __libc_disable_asynccancel 1.28% bro libstdc++.so.6.0.19 [.] std::this_thread::__sleep_for 1.10% bro libc-2.17.so [.] usleep 1.05% bro bro [.] ones_complement_checksum I haven't noticed a huge difference on a real multi process cluster, at least not a 30x change, so this is odd that pcap processing is so much slower. Especially since broker should be completely disabled when pcap files are being read and caf shouldn't even be doing anything. — Justin Azoff ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] Performance Issues after the fe7e1ee commit?
We are running into performance issues (30x slower) since the Broker patch (fe7e1ee) - We have 40G connections tapped from our storage filers feeding multiple Bro instances which analyze specifically only NFS and SMB traffic; all other analyzers are disabled. With the broker patch we are seeing processing times for a ~1GB pcap jump from around 2 seconds to over 1 minute. Profiling Bro, it looks like the culprit is the new Actor functions -- # Before patch Overhead Shared Object Symbol 14.57% [kernel] [k] copy_user_enhanced_fast_string 3.20% bro [.] EventHandler::operator bool 2.99% bro [.] _siphash 2.89% bro [.] Dictionary::Lookup # After patch Overhead Shared Object Symbol 5.71% [kernel] [k] native_write_msr_safe 3.84% libcaf_core.so.0.15.7 [.] caf::scheduler::worker::run 3.71% libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue::take_head 3.29% [kernel] [k] _raw_spin_lock Is the Bro development team still optimizing the Broker/Actor framework? It might be helpful to have a way to disable Broker for those of us who haven't migrated to it yet. # ~1GB file time (old) $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_0_20180606104904.pcap master.bro real0m2.294s user0m1.862s sys 0m0.385s # ~1GB file time (new) $ time /hostname/bro-devel/bin/bro -r 20180606-1049-prodfilers-truncated_0_20180606104904.pcap master.bro real1m11.458s user0m58.933s sys 1m34.074s Thanks! --Tim IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. ___ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
