date:20180830

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Allow loading policy/protocols/smb once again (57a505b0e)

2018-08-30 Thread Johanna Amann

To pick up the idea that you mentioned before - do we also want to make 
the new policy/protocols/smb/__load__.bro trigger a reporter warning 
that it is deprecated?

Johanna

On 30 Aug 2018, at 14:07, Jonathan Siwek wrote:

> Repository : ssh://g...@bro-ids.icir.org/bro
> On branch  : master
> Link   : 
> https://github.com/bro/bro/commit/57a505b0e46d499644a6fb3b063cece0684240b8
>
>> ---
>
> commit 57a505b0e46d499644a6fb3b063cece0684240b8
> Author: Jon Siwek 
> Date:   Thu Aug 30 16:05:36 2018 -0500
>
> Allow loading policy/protocols/smb once again
>
> It just redirects to base/protocols/smb
>
>
>> ---
>
> 57a505b0e46d499644a6fb3b063cece0684240b8
>  CHANGES   | 4 
>  NEWS  | 8 ++--
>  VERSION   | 2 +-
>  scripts/policy/protocols/smb/__load__.bro | 1 +
>  scripts/test-all-policy.bro   | 1 +
>  5 files changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/CHANGES b/CHANGES
> index af31bdea0..15184aa4a 100644
> --- a/CHANGES
> +++ b/CHANGES
> @@ -1,4 +1,8 @@
>
> +2.5-947 | 2018-08-30 16:05:36 -0500
> +
> +  * Allow loading policy/protocols/smb once again (Jon Siwek, 
> Corelight)
> +
>  2.5-946 | 2018-08-30 09:51:16 -0500
>
>* Update NEWS with more info about runtime options (Daniel Thayer)
> diff --git a/NEWS b/NEWS
> index 0af51ef60..86839427b 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -267,8 +267,12 @@ New Functionality
>
>  - Added new NFS events: nfs_proc_symlink, nfs_proc_link, 
> nfs_proc_sattr.
>
> -- The SMB scripts in policy/protocols/smb are now moved into 
> base/protocols/smb
> -  and loaded/enabled by default.
> +- The SMB scripts in policy/protocols/smb are now moved into
> +  base/protocols/smb and loaded/enabled by default.  If you 
> previously
> +  loaded these scripts from their policy/ location (in local.bro or
> +  other custom scripts) you may now remove/change those although they
> +  should still work since policy/protocols/smb is simply a 
> placeholder
> +  script that redirects to the new base/ location.
>
>  - Added new SMB events: smb1_transaction_secondary_request,
>smb1_transaction2_secondary_request, smb1_transaction_response.
> diff --git a/VERSION b/VERSION
> index d522ba4d6..ecd34e707 100644
> --- a/VERSION
> +++ b/VERSION
> @@ -1 +1 @@
> -2.5-946
> +2.5-947
> diff --git a/scripts/policy/protocols/smb/__load__.bro 
> b/scripts/policy/protocols/smb/__load__.bro
> new file mode 100644
> index 0..8fd733d38
> --- /dev/null
> +++ b/scripts/policy/protocols/smb/__load__.bro
> @@ -0,0 +1 @@
> +@load base/protocols/smb
> diff --git a/scripts/test-all-policy.bro b/scripts/test-all-policy.bro
> index 11824c2c6..d31da6573 100644
> --- a/scripts/test-all-policy.bro
> +++ b/scripts/test-all-policy.bro
> @@ -82,6 +82,7 @@
>  @load protocols/modbus/track-memmap.bro
>  @load protocols/mysql/software.bro
>  @load protocols/rdp/indicate_ssl.bro
> +@load protocols/smb/__load__.bro
>  @load protocols/smb/log-cmds.bro
>  @load protocols/smtp/blocklists.bro
>  @load protocols/smtp/detect-suspicious-orig.bro
>
>
>
> ___
> bro-commits mailing list
> bro-comm...@bro.org
> http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-commits
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Bro 2.5 Packet Drop Issue

2018-08-30 Thread Rajput, Jawad (CONTR)

Thank you so much Justin, the solution worked. We were literally 
troubleshooting for more than a month and did not find anything online.  

Jawad Rajput 
System Administrator
U.S. Department of Energy 
IM-62 /Germantown Building
HQ Network Security Team
Email: jawad.raj...@hq.doe.gov
Office: 301-903-2176
Office: 301-903-3895
Cell: 301-795-5406

-Original Message-
From: Azoff, Justin S [mailto:jaz...@illinois.edu] 
Sent: Thursday, August 30, 2018 4:29 PM
To: Rajput, Jawad (CONTR) 
Cc: bro-dev@bro.org; Danis, Andrew (CONTR) 
Subject: Re: [Bro-Dev] Bro 2.5 Packet Drop Issue

> On Aug 30, 2018, at 4:11 PM, Rajput, Jawad (CONTR)  
> wrote:
> 
> Hello Everyone,
>  
> I am reaching out with the hope that someone will be able to help us with an 
> issue we are having with Bro upgrade from 2.4.1 to 2.5.X.
>  
> We have a system with  12 core (3Ghz) ,128GB RAM, and 10G NIC (Intel X520-SR2 
> 10GbE Dual-port), monitoring between 1.5 - 2.5 Gbps traffic.
>  
> Bro 2.4.1 is working great and periodically drops 2-5% when traffic peaks at 
> ~ 2.5. However, when we upgrade to Bro 2.5.3/4 on the same exact system the 
> drops go up to 90%.
>  
> We are using CentOS-7 and tired installing Bro and Pfring from both rpm and 
> source without any luck. I wonder if anyone has seen this issue and can give 
> some clues to resolve this issue.
>  
> Bro Node Conf: 
> [manager]
> type=manager
> host=localhost
> #
> [proxy-1]
> type=proxy
> host=localhost
>  
> #
> [worker-1]
> type=worker
> host=localhost
> interface=ens1f1
> lb_method=pf_ring
> lb_procs=11
> pin_cpus=1,2,3,4,5,6,7,8,9,10,11

You're missing a logger process, adding one will make the cluster run better:

[logger]
type=logger
host=localhost

> [root@bro-test ~]# cat /proc/net/pf_ring/info
> PF_RING Version  : 7.3.0 (unknown)
> Total rings  : 11

you should have 1, not 11...

> Standard (non ZC) Options
> Ring slots   : 65534
> Slot version : 17
> Capture TX   : No [RX only]
> IP Defragment: No
> Socket Mode  : Standard
> Cluster Fragment Queue   : 0
> Cluster Fragment Discard : 0

Looks like you are having the issue where bro is not actually use pf_ring load 
balancing if you installed it from rpms.
What you're effectively doing is running 11 workers that are all receiving 100% 
of the traffic, so you are doing 11 times the work.

You can further confirm that this is the problem you are having by running

broctl config | grep -i clusterid

and seeing if the id is set to 0:

pfringclusterid = 0

if so, edit /opt/bro/etc/broctl.cfg and add

PFRINGClusterID = 11

and broctl deploy to restart everything.

This is already fixed and won't happen again in bro >= 2.6... just keeps 
tripping people up on 2.5.x

You should also look into switching to the native bro pf_ring plugin or the bro 
af_packet plugin which are both better choices than using the pcap wrapper 
method.

— 
Justin Azoff

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Bro 2.5 Packet Drop Issue

2018-08-30 Thread Azoff, Justin S


> On Aug 30, 2018, at 4:11 PM, Rajput, Jawad (CONTR)  
> wrote:
> 
> Hello Everyone,
>  
> I am reaching out with the hope that someone will be able to help us with an 
> issue we are having with Bro upgrade from 2.4.1 to 2.5.X.
>  
> We have a system with  12 core (3Ghz) ,128GB RAM, and 10G NIC (Intel X520-SR2 
> 10GbE Dual-port), monitoring between 1.5 - 2.5 Gbps traffic.
>  
> Bro 2.4.1 is working great and periodically drops 2-5% when traffic peaks at 
> ~ 2.5. However, when we upgrade to Bro 2.5.3/4 on the same exact system the 
> drops go up to 90%.
>  
> We are using CentOS-7 and tired installing Bro and Pfring from both rpm and 
> source without any luck. I wonder if anyone has seen this issue and can give 
> some clues to resolve this issue.
>  
> Bro Node Conf: 
> [manager]
> type=manager
> host=localhost
> #
> [proxy-1]
> type=proxy
> host=localhost
>  
> #
> [worker-1]
> type=worker
> host=localhost
> interface=ens1f1
> lb_method=pf_ring
> lb_procs=11
> pin_cpus=1,2,3,4,5,6,7,8,9,10,11

You're missing a logger process, adding one will make the cluster run better:

[logger]
type=logger
host=localhost


> [root@bro-test ~]# cat /proc/net/pf_ring/info
> PF_RING Version  : 7.3.0 (unknown)
> Total rings  : 11

you should have 1, not 11...

> Standard (non ZC) Options
> Ring slots   : 65534
> Slot version : 17
> Capture TX   : No [RX only]
> IP Defragment: No
> Socket Mode  : Standard
> Cluster Fragment Queue   : 0
> Cluster Fragment Discard : 0

Looks like you are having the issue where bro is not actually use pf_ring load 
balancing if you installed it from rpms.
What you're effectively doing is running 11 workers that are all receiving 100% 
of the traffic, so you are doing 11 times the work.

You can further confirm that this is the problem you are having by running

broctl config | grep -i clusterid

and seeing if the id is set to 0:

pfringclusterid = 0

if so, edit /opt/bro/etc/broctl.cfg and add

PFRINGClusterID = 11

and broctl deploy to restart everything.

This is already fixed and won't happen again in bro >= 2.6... just keeps 
tripping people up on 2.5.x

You should also look into switching to the native bro pf_ring plugin or the bro 
af_packet plugin which are both better choices than using the pcap wrapper 
method.

— 
Justin Azoff

___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Bro 2.5 Packet Drop Issue

2018-08-30 Thread Rajput, Jawad (CONTR)

Hello Everyone,



I am reaching out with the hope that someone will be able to help us with an 
issue we are having with Bro upgrade from 2.4.1 to 2.5.X.



We have a system with  12 core (3Ghz) ,128GB RAM, and 10G NIC (Intel X520-SR2 
10GbE Dual-port), monitoring between 1.5 - 2.5 Gbps traffic.



Bro 2.4.1 is working great and periodically drops 2-5% when traffic peaks at ~ 
2.5. However, when we upgrade to Bro 2.5.3/4 on the same exact system the drops 
go up to 90%.



We are using CentOS-7 and tired installing Bro and Pfring from both rpm and 
source without any luck. I wonder if anyone has seen this issue and can give 
some clues to resolve this issue.



Bro Node Conf:

[manager]

type=manager

host=localhost

#

[proxy-1]

type=proxy

host=localhost



#

[worker-1]

type=worker

host=localhost

interface=ens1f1

lb_method=pf_ring

lb_procs=11

pin_cpus=1,2,3,4,5,6,7,8,9,10,11



[root@bro-test ~]# cat /proc/net/pf_ring/info

PF_RING Version  : 7.3.0 (unknown)

Total rings  : 11



Standard (non ZC) Options

Ring slots   : 65534

Slot version : 17

Capture TX   : No [RX only]

IP Defragment: No

Socket Mode  : Standard

Cluster Fragment Queue   : 0

Cluster Fragment Discard : 0





[root@bro-test ~]# tailf /opt/bro/logs/current/capture_loss.log

1535647921.339324   60.05   worker-1-8  318331  425005  
74.900531

1535647921.217853   60.00   worker-1-5  264716  349078  
75.832908

1535647921.241244   60.21   worker-1-9  265863  364089  
73.021432

1535647921.312567   60.02   worker-1-1  239036  315823  
75.686698

1535647922.188607   60.000420   worker-1-4  238192  322818  
73.785229

1535647922.760560   60.29   worker-1-11 250678  338188  74.12386

1535647922.864470   60.75   worker-1-3  232467  314963  
73.807717

1535647923.413121   60.24   worker-1-10 254241  345382  
73.611537

1535647923.205954   60.001556   worker-1-2  259932  354980  
73.224407





[root@bro-test ~]# less /opt/bro/logs/current/stats.log | bro-cut  ts  peer 
   mem pkts_proc   bytes_recv  pkts_dropped

1535644801.328981   worker-1-8  28543523252 2214563854  8841163

1535644801.235592   worker-1-9  28333422300 2135680645  9083143

1535644801.299138   worker-1-1  28013358673 2089659287  9059868

1535644802.177016   worker-1-4  27273262089 2027645336  9155838

1535644801.187590   worker-1-5  26403336190 2085853940  9332917

1535644802.750617   worker-1-11 27263432674 2153405372  9018943

1535644802.853617   worker-1-3  28163448836 2161753414  8929662

1535644803.186853   worker-1-2  26593387742 2116043509  9176871

1535644803.395256   worker-1-10 28713407486 2132043052  9049047

1535644803.403778   worker-1-7  28213278503 2023604941  9966347

1535644850.898433   manager 23400   0   -

1535644804.257320   proxy-1 73  0   0   -



[root@bro-test logs]# broctl netstats

worker-1-1: 1535651356.794609 recvd=3501813131 dropped=3589205826 
link=3501813131

worker-1-2: 1535651358.808626 recvd=4033892471 dropped=3057179730 
link=4033892471

worker-1-3: 1535651358.587316 recvd=3930325145 dropped=3160768660 
link=3930325145

worker-1-4: 1535651357.702299 recvd=3561053809 dropped=3530086444 
link=3561053809

worker-1-5: 1535651357.650359 recvd=3399338460 dropped=3691836209 
link=3399338460

worker-1-6: 1535651334.912244 recvd=3714154738 dropped=3376978237 
link=3714154738

worker-1-7: 1535651359.119492 recvd=3684804437 dropped=3406432666 
link=3684804437

worker-1-8: 1535651359.668621 recvd=4020016563 dropped=3071265083 
link=4020016563

worker-1-9: 1535651359.867601 recvd=3807658264 dropped=3283669188 
link=3807658264

worker-1-10: 1535651359.749253 recvd=3703077938 dropped=3388277853 
link=3703077938

worker-1-11: 1535651359.907420 recvd=4052516305 dropped=3038874387 
link=4052516305



nload output for capture NIC:

[cid:image001.png@01D4407C.0E3A9670]

Jawad Rajput

System Administrator

U.S. Department of Energy

IM-62 /Germantown Building

HQ Network Security Team

Email: jawad.raj...@hq.doe.gov

Office: 301-903-2176

Office: 301-903-3895


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Compatibilty script for policy/protocols/smb?

2018-08-30 Thread Azoff, Justin S


> On Aug 30, 2018, at 2:26 PM, Jon Siwek  wrote:
> 
> On Thu, Aug 30, 2018 at 9:50 AM Azoff, Justin S  wrote:
> 
>> fatal error in /bro/share/bro/site/local.bro, line 88: can't open 
>> /bro/share/bro/policy/protocols/smb/__load__.bro
>> 
>> I see in NEWS we have
>> 
>> - The SMB scripts in policy/protocols/smb are now moved into 
>> base/protocols/smb
>>  and loaded/enabled by default.
>> 
>> But should there be an empty script in there or something that does a 
>> reporter warning telling people to update local.bro?
> 
> Thanks for pointing that out.
> 
> I'll put a placeholder at the old policy/ location, but also call out
> in NEWS that such @loads can be removed from local.bro or other custom
> scripts.
> 
> Or let me know if there's other ideas.

Sounds good to me.

I was curious why this test didn't catch this:

testing/btest/scripts/site/local-compat.test

but the file as shipped was

# Uncomment the following line to enable the SMB analyzer.  The analyzer
# is currently considered a preview and therefore not loaded by default.
# @load policy/protocols/smb

so while 2.6 would have been compatible with the 2.5 config as it was 
distributed, it would have broken anyone that uncommented the line.



— 
Justin Azoff


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] Compatibilty script for policy/protocols/smb?

2018-08-30 Thread Jon Siwek

On Thu, Aug 30, 2018 at 9:50 AM Azoff, Justin S  wrote:

> fatal error in /bro/share/bro/site/local.bro, line 88: can't open 
> /bro/share/bro/policy/protocols/smb/__load__.bro
>
> I see in NEWS we have
>
> - The SMB scripts in policy/protocols/smb are now moved into 
> base/protocols/smb
>   and loaded/enabled by default.
>
> But should there be an empty script in there or something that does a 
> reporter warning telling people to update local.bro?

Thanks for pointing that out.

I'll put a placeholder at the old policy/ location, but also call out
in NEWS that such @loads can be removed from local.bro or other custom
scripts.

Or let me know if there's other ideas.

- Jon
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

[Bro-Dev] Compatibilty script for policy/protocols/smb?

2018-08-30 Thread Azoff, Justin S

Upgrading between master builds I just ran into this:

fatal error in /bro/share/bro/site/local.bro, line 88: can't open 
/bro/share/bro/policy/protocols/smb/__load__.bro

I see in NEWS we have

- The SMB scripts in policy/protocols/smb are now moved into base/protocols/smb
  and loaded/enabled by default.

But should there be an empty script in there or something that does a reporter 
warning telling people to update local.bro?




— 
Justin Azoff


___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev

Re: [Bro-Dev] [Bro-Commits] [git/bro] master: Allow loading policy/protocols/smb once again (57a505b0e)

Re: [Bro-Dev] Bro 2.5 Packet Drop Issue

Re: [Bro-Dev] Bro 2.5 Packet Drop Issue

[Bro-Dev] Bro 2.5 Packet Drop Issue

Re: [Bro-Dev] Compatibilty script for policy/protocols/smb?

Re: [Bro-Dev] Compatibilty script for policy/protocols/smb?

[Bro-Dev] Compatibilty script for policy/protocols/smb?

7 matches

Site Navigation

Mail list logo

Footer information