> On Mar 8, 2018, at 5:46 PM, Jon Siwek <jsi...@corelight.com> wrote:
>
> On Thu, Mar 8, 2018 at 5:07 PM, Azoff, Justin S <jaz...@illinois.edu> wrote:
>
>> One thing I notice is that the traffic to/from the manager box and cpu has
>> increased quite a bit.
>>
>> Ignoring the large CPU spikes from building the new branch just before the
>> switch at 15:30, the overall cpu load on the manager box is about 3x higher.
>> The bandwidth isn't terribly excessive, but it increased from about 16mbit
>> to 40mbit (stupid graph is in mebibytes).
>>
>> Based on some capstats runs, it's all going to the logger port 47761. I
>> have another graph that shows the total log volume being written to disk and
>> that hasn't changed.
>> So it looks like this branch uses 2x the cpu and bandwidth to write the same
>> volume of logs.
>
> Interesting, I would have thought maybe the manager could be utilized
> more, though not the logger. Will have to think about that.
Yeah.. it's definitely worker -> logger:
logger=47761, manager=47762, proxy=47763
[root@bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47761' -n 5
1520553009.488934 pkts=1173 kpps=1.1 kbytes=2837 mbps=22.6 nic_pkts=1175
nic_drops=0 u=0 t=1173 i=0 o=0 nonip=0
1520553010.489053 pkts=1280 kpps=1.3 kbytes=3218 mbps=26.4 nic_pkts=2455
nic_drops=0 u=0 t=1280 i=0 o=0 nonip=0
1520553011.489177 pkts=1086 kpps=1.1 kbytes=2912 mbps=23.9 nic_pkts=3541
nic_drops=0 u=0 t=1086 i=0 o=0 nonip=0
1520553012.489310 pkts=1382 kpps=1.4 kbytes=3757 mbps=30.8 nic_pkts=4923
nic_drops=0 u=0 t=1382 i=0 o=0 nonip=0
1520553013.489471 pkts=1563 kpps=1.6 kbytes=3816 mbps=31.3 nic_pkts=6486
nic_drops=0 u=0 t=1563 i=0 o=0 nonip=0
[root@bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47762' -n 5
1520553018.874154 pkts=346 kpps=0.3 kbytes=92 mbps=0.7 nic_pkts=348 nic_drops=0
u=0 t=346 i=0 o=0 nonip=0
1520553019.875111 pkts=1044 kpps=1.0 kbytes=283 mbps=2.3 nic_pkts=1391
nic_drops=0 u=0 t=1044 i=0 o=0 nonip=0
1520553020.875448 pkts=759 kpps=0.8 kbytes=199 mbps=1.6 nic_pkts=2150
nic_drops=0 u=0 t=759 i=0 o=0 nonip=0
1520553021.875575 pkts=654 kpps=0.7 kbytes=196 mbps=1.6 nic_pkts=2804
nic_drops=0 u=0 t=654 i=0 o=0 nonip=0
1520553022.875920 pkts=470 kpps=0.5 kbytes=123 mbps=1.0 nic_pkts=3274
nic_drops=0 u=0 t=470 i=0 o=0 nonip=0
[root@bro-test ~]# capstats -i enp3s0f0 -I 1 -f 'port 47763' -n 5
1520553025.700919 pkts=26 kpps=0.0 kbytes=8 mbps=0.1 nic_pkts=31 nic_drops=0
u=0 t=26 i=0 o=0 nonip=0
1520553026.701047 pkts=27 kpps=0.0 kbytes=9 mbps=0.1 nic_pkts=58 nic_drops=0
u=0 t=27 i=0 o=0 nonip=0
1520553027.701185 pkts=24 kpps=0.0 kbytes=8 mbps=0.1 nic_pkts=82 nic_drops=0
u=0 t=24 i=0 o=0 nonip=0
1520553028.701306 pkts=55 kpps=0.1 kbytes=18 mbps=0.2 nic_pkts=137 nic_drops=0
u=0 t=55 i=0 o=0 nonip=0
1520553029.701434 pkts=9 kpps=0.0 kbytes=3 mbps=0.0 nic_pkts=146 nic_drops=0
u=0 t=9 i=0 o=0 nonip=0
[root@bro-test ~]#
On another cluster seeing the same traffic with 56 workers running 2.5.3ish
shows this for the logger port at the same time:
1520553725.875230 pkts=681 kpps=0.7 kbytes=1946 mbps=15.5 nic_pkts=768
nic_drops=0 u=0 t=681 i=0 o=0 nonip=0
1520553726.875374 pkts=508 kpps=0.5 kbytes=1605 mbps=13.1 nic_pkts=1392
nic_drops=0 u=0 t=508 i=0 o=0 nonip=0
1520553727.875520 pkts=554 kpps=0.6 kbytes=1681 mbps=13.8 nic_pkts=2247
nic_drops=0 u=0 t=554 i=0 o=0 nonip=0
1520553728.875713 pkts=536 kpps=0.5 kbytes=1708 mbps=14.0 nic_pkts=2902
nic_drops=0 u=0 t=536 i=0 o=0 nonip=0
1520553729.875845 pkts=518 kpps=0.5 kbytes=1669 mbps=13.7 nic_pkts=3641
nic_drops=0 u=0 t=518 i=0 o=0 nonip=0
[root@bro-test ~]# broctl top manager logger
Name Type Host Pid Proc VSize Rss Cpu Cmd
logger logger bro-test 2016 parent 938M 283M 81% bro
manager manager bro-test 2107 parent 619M 244M 25% bro
[root@bro-test ~]#
[root@bro-dev ~]# broctl top manager logger
Name Type Host Pid Proc VSize Rss Cpu Cmd
logger logger bro-dev 10704 child 492M 150M 6% bro
logger logger bro-dev 10571 parent 1G 221M 0% bro
manager manager bro-dev 10734 parent 909M 497M 0% bro
manager manager bro-dev 10782 child 482M 110M 0% bro
cpus are about the same, test has X5650 @ 2.67GHz and dev has E5-2420 @
1.90GHz
according to passmark, the older X5650 has a slightly faster single core
performance.
> An interesting experiment you could try is switching to an alternate
> implementation of the Known scripts. E.g. stick this in local.bro:
>
> redef Known::use_host_store = F;
> redef Known::use_cert_store = F;
> redef Known::use_device_store = F;
> redef Known::use_service_store = F;
I tried doing that for an unrelated reason.. but I'm not sure if it works right.
You can't redef Known::use_service_store before the script is loaded, because
otherwise you get
Can't document redef of Known::use_service_store, identifier lookup failed
but if you load the script first, the
@if ( Known::use_service_store )
block is already evaluated before you change the value.
[jazoff@bro-test ~]$ cat foo.bro
@load protocols/conn/known-services
redef Known::use_service_store=F;
event bro_init()
{
print Known::services;
}
[jazoff@bro-test ~]$ bro foo.bro
error in ./foo.bro, line 6: unknown identifier Known::services, at or near
"Known::services"
[jazoff@bro-test ~]$ cat foo.bro
redef Known::use_service_store=F;
@load protocols/conn/known-services
event bro_init()
{
print Known::services;
}
[jazoff@bro-test ~]$ bro foo.bro
error in ./foo.bro, line 1: "redef" used but not previously defined
(Known::use_service_store)
internal warning in ./foo.bro, line 1: Can't document redef of
Known::use_service_store, identifier lookup failed
error in ./foo.bro, line 1 and
/srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 34: already
defined (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 40:
value used but not set (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 40:
invalid expression in @if (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 93:
value used but not set (Known::use_service_store)
error in /srv/bro/share/bro/policy/protocols/conn/known-services.bro, line 93:
invalid expression in @if (Known::use_service_store)
—
Justin Azoff
_______________________________________________
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
