> On Jun 6, 2018, at 12:54 PM, McMullan, Tim <tim.mcmul...@sig.com> wrote: > > We are running into performance issues (30x slower) since the Broker patch > (fe7e1ee) – > > We have 40G connections tapped from our storage filers feeding multiple Bro > instances which analyze specifically only NFS and SMB traffic; all other > analyzers are disabled. With the broker patch we are seeing processing times > for a ~1GB pcap jump from around 2 seconds to over 1 minute. Profiling Bro, > it looks like the culprit is the new Actor functions -- > > # Before patch > Overhead Shared Object Symbol > 14.57% [kernel] [k] copy_user_enhanced_fast_string > 3.20% bro [.] EventHandler::operator bool > 2.99% bro [.] _siphash > 2.89% bro [.] Dictionary::Lookup > > # After patch > Overhead Shared Object Symbol > 5.71% [kernel] [k] native_write_msr_safe > 3.84% libcaf_core.so.0.15.7 [.] > caf::scheduler::worker<caf::policy::work_stealing>::run > 3.71% libcaf_core.so.0.15.7 [.] > caf::detail::double_ended_queue<caf::resumable>::take_head > 3.29% [kernel] [k] _raw_spin_lock > > Is the Bro development team still optimizing the Broker/Actor framework? It > might be helpful to have a way to disable Broker for those of us who haven’t > migrated to it yet. > > # ~1GB file time (old) > $ time /hostname/bro-devel/bin/bro -r > 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro > > real 0m2.294s > user 0m1.862s > sys 0m0.385s > > # ~1GB file time (new) > $ time /hostname/bro-devel/bin/bro -r > 20180606-1049-prodfilers-truncated_00000_20180606104904.pcap master.bro > > real 1m11.458s > user 0m58.933s > sys 1m34.074s
Well, I can at least reproduce that here on one of the standard test pcap files and no scripts loaded: $ /usr/local/bin/bro --version /usr/local/bin/bro version 2.5.2 $ time /usr/local/bin/bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace real 0m2.317s user 0m1.874s sys 0m0.327s $ . ~/src/bro/build/bro-path-dev.sh $ bro --version bro version 2.5-648 $ time bro -b -r ~/src/bro/testing/external/bro-testing/2009-M57-day11-18.trace real 1m5.523s user 0m30.565s sys 0m32.661s and the same thing in perf: Samples: 270K of event 'cycles:u', Event count (approx.): 131984875188 Overhead Command Shared Object Symbol 8.95% bro libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue<caf::resumable>::take_tail 5.90% bro libcaf_core.so.0.15.7 [.] caf::detail::double_ended_queue<caf::resumable>::take_head 4.92% bro libcaf_core.so.0.15.7 [.] caf::scheduler::worker<caf::policy::work_stealing>::run 2.67% bro libc-2.17.so [.] __GI___libc_nanosleep 1.98% bro libtcmalloc.so.4.4.5 [.] operator new[] 1.79% bro libc-2.17.so [.] __sleep 1.68% bro libtcmalloc.so.4.4.5 [.] tc_deletearray_nothrow 1.47% bro libc-2.17.so [.] __libc_disable_asynccancel 1.28% bro libstdc++.so.6.0.19 [.] std::this_thread::__sleep_for 1.10% bro libc-2.17.so [.] usleep 1.05% bro bro [.] ones_complement_checksum I haven't noticed a huge difference on a real multi process cluster, at least not a 30x change, so this is odd that pcap processing is so much slower. Especially since broker should be completely disabled when pcap files are being read and caf shouldn't even be doing anything. — Justin Azoff _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
