Hi Jon, Thanks for looking into this! I tried this again after pulling from master and at least for our case it is looking good!
### old # /hostname/bro/bin/bro --version /hostname/bro/bin/bro version 2.5-519 # time /hostname/bro/bin/bro -r 20180606-1242-prodfilers.pcap master.bro real 0m32.772s user 0m28.821s sys 0m3.873s ### new # /hostname/bro-devel/bin/bro --version /hostname/bro-devel/bin/bro version 2.5-658 # time /hostname/bro-devel/bin/bro -r 20180606-1242-prodfilers.pcap master.bro real 0m34.684s user 0m31.502s sys 0m4.266s Thanks again! --Tim -----Original Message----- From: Jon Siwek [mailto:jsi...@corelight.com] Sent: Thursday, June 07, 2018 6:06 PM To: Robin Sommer <ro...@icir.org> Cc: McMullan, Tim <tim.mcmul...@msx.bala.susq.com>; <bro-dev@bro.org> <bro-dev@bro.org>; Wallior, Julien <julien.wall...@msx.bala.susq.com>; Trejo, Devin <devin.tr...@msx.bala.susq.com> Subject: Re: [Bro-Dev] Performance Issues after the fe7e1ee commit? On Thu, Jun 7, 2018 at 12:21 PM Robin Sommer <ro...@icir.org> wrote: > > Hmm, I'm still seeing much larger runtimes on that M57 trace using our > test configuration, even with yesterday's change: > > ; Master, pre-Broker > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 73.72user 7.90system 1:06.53elapsed 122%CPU (0avgtext+0avgdata > 199092maxresident) > > ; Current master > # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro > 2191.59user 1606.69system 12:39.92elapsed 499%CPU (0avgtext+0avgdata > 228400maxresident) > > Bro must still be blocking somewhere when reading from trace files. Thanks, if you pull master changes [1] again there's likely some improvement. You can also try comparing the timing of: # zcat 2009-M57-day11-18.trace.gz | time bro -r - tests/m57-long.bro Known::use_host_store=F Known::use_service_store=F Known::use_cert_store=F With that, I get the same timings as from before pre-Broker. At least a good chunk of the difference when using data stores is that, for every query, Bro will immediately block until getting a response back from the data store thread/actor. Not doing this type of synchronous data store access when reading pcaps leads to the possibility of results differing between runs (and I recall such differences being likely from experience in running the unit test suite). - Jon [1] https://github.com/bro/broker/commit/9b56fea4999d4e11a5cd2caaafd934759015fab5 ________________________________ IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
