Re: [Ntop-misc] pfring.h

[James]

I'm still a linux novice, so I've just discovered the config.log, which
suggests the problem is permission related:

conftest.c:57:20: error: /usr/local/include/pfring.h: Permission denied

Current permissions on these files:
-rwxr-x---   1 root root 52817 Sep 13 14:37 pfring.h
-rwxr-x---   1 root root 12326 Sep 13 14:37 pfring_mod_sysdig.h
-rwxr-x---   1 root root 28511 Sep 13 14:37 pfring_zc.h

What should they be please? I'm running ./configure as a non-root user (as
I understand is best practice). I tried granting read to all on pfring.h
but that didn't help.

There's also a pf_ring.h in /usr/include/linux with these permissions:
-rwxr-x---   1 root root  39009 Sep 13 13:46 pf_ring.h

Thanks
James


On 19 September 2017 at 11:02, James <ntop-m...@cyclohexane.net> wrote:

> Hi,
>
> Yes I am, though I'm running this from the pfring-daq-module-zc directory
> and you're in pfring-daq-module, is that relevant? I do want to use ZC, but
> the drivers are not installed yet (that was my next task after the pfring
> DAQ).
>
> I've even tried putting a link file in /usr/include/pfring.h which points
> to /usr/local/include/pfring.h - no help.
>
> Thanks
> James
>
> On 18 September 2017 at 18:08, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> This is strange, actually if you have pfring installed in the standard
>> path there it should work
>> even without specifying the path. This is on a machine in our lab:
>>
>> $ pwd
>> /home/nbox/PF_RING-dev/userland/snort/pfring-daq-module
>>
>> $ autoreconf -ivf
>>
>> $ ./configure
>>
>> $ make
>>
>> $ ldd .libs/daq_pfring.so
>> linux-vdso.so.1 =>  (0x7ffce8f5f000)
>> libpfring.so => /usr/local/lib/libpfring.so (0x7f65d75be000)
>> libhiredis.so.0.13 => /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13
>> (0x7f65d73b1000)
>> libsfbpf.so.0 => /usr/lib/libsfbpf.so.0 (0x7f65d718a000)
>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f65d6dc)
>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x7f65d6ba3000)
>> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f65d699b000)
>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f65d6797000)
>> /lib64/ld-linux-x86-64.so.2 (0x7f65d7a45000)
>>
>> Are you still getting "configure: error: Could not find pfring.h!”?
>>
>> Alfredo
>>
>> On 18 Sep 2017, at 13:11, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Hi,
>>
>> This command still fails to find the pfring.h file:
>> ./configure --with-libpfring-includes=/usr/local/include
>> --with-pfring-kernel-includes=/usr/local/include
>> --with-libpfring-libraries=/usr/local/lib
>>
>>
>> On 18 September 2017 at 11:03, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>> Please specify all of them together with the proper paths (lib and
>>> include)
>>>
>>> Alfredo
>>>
>>> On 18 Sep 2017, at 10:56, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> Hi Alfredo,
>>>
>>> Thanks for helping me. I've tried all three of those but still get the
>>> same error:
>>> ./configure --with-libpfring-includes=/usr/local/include
>>> ./configure --with-pfring-kernel-includes=/usr/local/include
>>> ./configure --with-libpfring-libraries=/usr/local/include
>>>
>>> On 18 September 2017 at 09:19, Alfredo Cardigliano <cardigli...@ntop.org
>>> > wrote:
>>>
>>>> Hi James
>>>> the configure script currently checks for ${HOME}/PF_RING/ or installed
>>>> libraries specified with:
>>>>
>>>>  --with-libpfring-includes=
>>>>  --with-pfring-kernel-includes=
>>>>  --with-libpfring-libraries=
>>>>
>>>> Regards
>>>> Alfredo
>>>>
>>>> > On 15 Sep 2017, at 11:19, James <ntop-m...@cyclohexane.net> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I'm trying to install the pfring DAQ and when I run configure, am
>>>> getting the error:
>>>> >
>>>> > checking pfring.h usability... no
>>>> > checking pfring.h presence... no
>>>> > checking for pfring.h... no
>>>> > configure: error: Could not find pfring.h!
>>>> >
>>>> > I have installed /kernel and /userland/lib and the file exists here:
>>>> >
>>>> > /usr/local/src/PF_RING-dev/userland/lib/pfring.h
>>>> > /usr/local/include

Re: [Ntop-misc] pfring.h

[James]

Hi,

Yes I am, though I'm running this from the pfring-daq-module-zc directory
and you're in pfring-daq-module, is that relevant? I do want to use ZC, but
the drivers are not installed yet (that was my next task after the pfring
DAQ).

I've even tried putting a link file in /usr/include/pfring.h which points
to /usr/local/include/pfring.h - no help.

Thanks
James

On 18 September 2017 at 18:08, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> This is strange, actually if you have pfring installed in the standard
> path there it should work
> even without specifying the path. This is on a machine in our lab:
>
> $ pwd
> /home/nbox/PF_RING-dev/userland/snort/pfring-daq-module
>
> $ autoreconf -ivf
>
> $ ./configure
>
> $ make
>
> $ ldd .libs/daq_pfring.so
> linux-vdso.so.1 =>  (0x7ffce8f5f000)
> libpfring.so => /usr/local/lib/libpfring.so (0x7f65d75be000)
> libhiredis.so.0.13 => /usr/lib/x86_64-linux-gnu/libhiredis.so.0.13
> (0x7f65d73b1000)
> libsfbpf.so.0 => /usr/lib/libsfbpf.so.0 (0x7f65d718a000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f65d6dc)
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> (0x7f65d6ba3000)
> librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f65d699b000)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f65d6797000)
> /lib64/ld-linux-x86-64.so.2 (0x7f65d7a45000)
>
> Are you still getting "configure: error: Could not find pfring.h!”?
>
> Alfredo
>
> On 18 Sep 2017, at 13:11, James <ntop-m...@cyclohexane.net> wrote:
>
> Hi,
>
> This command still fails to find the pfring.h file:
> ./configure --with-libpfring-includes=/usr/local/include
> --with-pfring-kernel-includes=/usr/local/include
> --with-libpfring-libraries=/usr/local/lib
>
>
> On 18 September 2017 at 11:03, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> Please specify all of them together with the proper paths (lib and
>> include)
>>
>> Alfredo
>>
>> On 18 Sep 2017, at 10:56, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Hi Alfredo,
>>
>> Thanks for helping me. I've tried all three of those but still get the
>> same error:
>> ./configure --with-libpfring-includes=/usr/local/include
>> ./configure --with-pfring-kernel-includes=/usr/local/include
>> ./configure --with-libpfring-libraries=/usr/local/include
>>
>> On 18 September 2017 at 09:19, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>> Hi James
>>> the configure script currently checks for ${HOME}/PF_RING/ or installed
>>> libraries specified with:
>>>
>>>  --with-libpfring-includes=
>>>  --with-pfring-kernel-includes=
>>>  --with-libpfring-libraries=
>>>
>>> Regards
>>> Alfredo
>>>
>>> > On 15 Sep 2017, at 11:19, James <ntop-m...@cyclohexane.net> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I'm trying to install the pfring DAQ and when I run configure, am
>>> getting the error:
>>> >
>>> > checking pfring.h usability... no
>>> > checking pfring.h presence... no
>>> > checking for pfring.h... no
>>> > configure: error: Could not find pfring.h!
>>> >
>>> > I have installed /kernel and /userland/lib and the file exists here:
>>> >
>>> > /usr/local/src/PF_RING-dev/userland/lib/pfring.h
>>> > /usr/local/include/pfring.h
>>> >
>>> > Thanks
>>> > James
>>> > ___
>>> > Ntop-misc mailing list
>>> > Ntop-misc@listgateway.unipi.it
>>> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>>
>>>
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>>
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>>
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pfring.h

[James]

Hi,

This command still fails to find the pfring.h file:
./configure --with-libpfring-includes=/usr/local/include
--with-pfring-kernel-includes=/usr/local/include
--with-libpfring-libraries=/usr/local/lib


On 18 September 2017 at 11:03, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Please specify all of them together with the proper paths (lib and include)
>
> Alfredo
>
> On 18 Sep 2017, at 10:56, James <ntop-m...@cyclohexane.net> wrote:
>
> Hi Alfredo,
>
> Thanks for helping me. I've tried all three of those but still get the
> same error:
> ./configure --with-libpfring-includes=/usr/local/include
> ./configure --with-pfring-kernel-includes=/usr/local/include
> ./configure --with-libpfring-libraries=/usr/local/include
>
> On 18 September 2017 at 09:19, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> Hi James
>> the configure script currently checks for ${HOME}/PF_RING/ or installed
>> libraries specified with:
>>
>>  --with-libpfring-includes=
>>  --with-pfring-kernel-includes=
>>  --with-libpfring-libraries=
>>
>> Regards
>> Alfredo
>>
>> > On 15 Sep 2017, at 11:19, James <ntop-m...@cyclohexane.net> wrote:
>> >
>> > Hi,
>> >
>> > I'm trying to install the pfring DAQ and when I run configure, am
>> getting the error:
>> >
>> > checking pfring.h usability... no
>> > checking pfring.h presence... no
>> > checking for pfring.h... no
>> > configure: error: Could not find pfring.h!
>> >
>> > I have installed /kernel and /userland/lib and the file exists here:
>> >
>> > /usr/local/src/PF_RING-dev/userland/lib/pfring.h
>> > /usr/local/include/pfring.h
>> >
>> > Thanks
>> > James
>> > ___
>> > Ntop-misc mailing list
>> > Ntop-misc@listgateway.unipi.it
>> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pfring.h

[James]

Hi Alfredo,

Thanks for helping me. I've tried all three of those but still get the same
error:
./configure --with-libpfring-includes=/usr/local/include
./configure --with-pfring-kernel-includes=/usr/local/include
./configure --with-libpfring-libraries=/usr/local/include

On 18 September 2017 at 09:19, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Hi James
> the configure script currently checks for ${HOME}/PF_RING/ or installed
> libraries specified with:
>
>  --with-libpfring-includes=
>  --with-pfring-kernel-includes=
>  --with-libpfring-libraries=
>
> Regards
> Alfredo
>
> > On 15 Sep 2017, at 11:19, James <ntop-m...@cyclohexane.net> wrote:
> >
> > Hi,
> >
> > I'm trying to install the pfring DAQ and when I run configure, am
> getting the error:
> >
> > checking pfring.h usability... no
> > checking pfring.h presence... no
> > checking for pfring.h... no
> > configure: error: Could not find pfring.h!
> >
> > I have installed /kernel and /userland/lib and the file exists here:
> >
> > /usr/local/src/PF_RING-dev/userland/lib/pfring.h
> > /usr/local/include/pfring.h
> >
> > Thanks
> > James
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] pfring.h

[James]

Hi,

I'm trying to install the pfring DAQ and when I run configure, am getting
the error:

checking pfring.h usability... no
checking pfring.h presence... no
checking for pfring.h... no
configure: error: Could not find pfring.h!

I have installed /kernel and /userland/lib and the file exists here:

/usr/local/src/PF_RING-dev/userland/lib/pfring.h
/usr/local/include/pfring.h

Thanks
James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] HP 10G NIC

[James]

Hi Luke,

Sorry for slow reply but we're using HP 560FLR-SFP+ and they do work in
promisc mode.. despite the document you linked claiming they also don't.
Very odd.

James

On 23 February 2016 at 11:58, Whitworth, Luke <
luke.whitwo...@cranfield.ac.uk> wrote:

> Hi all,
>
> I'm trying to source a card for my HP server to allow me to capture from
> some 10G fiber using PF_RING ZC.  Was looking at HP Ethernet 10Gb 2-port
> 560SFP+ Adapter (
> http://www8.hp.com/uk/en/products/iss-adapters/product-detail.html?oid=5283369)
> which ticks all the boxes, but then stumbled upon
> https://www.intelethernet-hp.com/wp-content/uploads/2014/04/8954-HP_Intel-Ethernet-Data-SheetFINAL.pdf
> which on page 5 suggests that this NIC can't be used in promiscuous mode
> which makes it kind of useless if it's true!
>
> Does anyone have any experience with this NIC?  If so is the no
> promiscuous mode true?  If so does anyone have any alternative they'd
> recommend given that the card needs to be Intel 82599/X540/X710 based.
>
> Cheers,
>
> Luke
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] How can I output the traffic log from nDPI?

[James Cheng]

Dears,

I would like to output the traffic log from nDPI. Is it possible? and How
to do that?
The output log might include the source ip, destination ip, protocol,
destination URL/URI, etc.

Thanks for advise,
James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] How can I output the traffic log from nDPI?

[James Cheng]

Thanks Luca,
But we would like to extract the application, such as skype. Can ntopng or
nProbe do that?
Cheers,
James

On Tue, Dec 22, 2015 at 6:43 PM, Luca Deri <d...@ntop.org> wrote:

> James
> ntopng or nProbe do that
>
> Luca
>
> > On 22 Dec 2015, at 11:31, James Cheng <jih...@gmail.com> wrote:
> >
> > Dears,
> >
> > I would like to output the traffic log from nDPI. Is it possible? and
> How to do that?
> > The output log might include the source ip, destination ip, protocol,
> destination URL/URI, etc.
> >
> > Thanks for advise,
> > James
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pf-ring and snort

[James]

Hi Alfredo,

For the lists benefit, I have (with your help) solved my problem by
increasing $HUGEPAGES to 2048 in the load_driver script. I now have 16
instances of Snort running and alerts are being generated. I am surprised
by how few alerts are occurring though and wanted to verify what traffic
was being seen. I thought tcpdump would be a good way to do this, so
compiled the pf_ring version, but I can't get that to display any traffic
on either of my two interfaces. I've tried:
"tcpdump -i eth4" (or 5) which functions but shows no traffic
"tcpdump -i zc:eth4"
"tcpdump -i zc:eth4@0" both of which fail, telling me there is no such
device

Sorry for asking so many questions, but any ideas please?

Thanks
James

On 9 December 2015 at 14:40, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Hi James
> for debug purposes, do you see any improvement if you change the number of
> ring slots from 32k to 4k for instance? (i.e. ethtool -G ethX rx 4096)
>
> Alfredo
>
> On 07 Dec 2015, at 15:15, James <ntop-m...@cyclohexane.net> wrote:
>
> I'm another step forward now, thank you Alfredo. I can get 12 instances to
> start (for some reason it's always instances 0-7 and 12-15). Instances 8-11
> give the same "bus error" at startup though. There are no errors in
> /var/log/messages
>
>
>
> On 4 December 2015 at 11:03, James <ntop-m...@cyclohexane.net> wrote:
>
>> Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more
>> start and then back to command line. /var/log/messages no longer mentions
>> hugepages, so I guess that's not the problem any longer. The NIC's appear
>> to go up and down repeatedly (that might be correct?), come in and out of
>> promiscuous mode and then lots of these:
>>
>> ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied
>> ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied
>> ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied
>> etc
>>
>>
>> On 4 December 2015 at 09:46, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>> node means NUMA node (i.e. CPU)
>>>
>>> Alfredo
>>>
>>>
>>> On 04 Dec 2015, at 10:41, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> Hopefully my last stupid question - is node the same as processes and
>>> queues in this context? So I should do that all the way up to 15?
>>>
>>> On 4 December 2015 at 09:32, Alfredo Cardigliano <cardigli...@ntop.org>
>>> wrote:
>>>
>>>>
>>>> On 04 Dec 2015, at 10:27, James <ntop-m...@cyclohexane.net> wrote:
>>>>
>>>> Thanks Alfredo, even more that you reply so quickly! I respect the
>>>> "teach a man to fish.." method of helping, but that's a lot of parameters
>>>> and options and I'd be making complete guesses at which ones to change and
>>>> to what values. Would it be possible to recommend what you'd change based
>>>> on the spec of my system?
>>>>
>>>>
>>>> In essence if you have 4 nodes, you should set the number of huge pages
>>>> per node with:
>>>>
>>>>   $ echo 1024 >
>>>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>>>>   $ echo 1024 >
>>>> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
>>>>   $ echo 1024 >
>>>> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>>>>   $ echo 1024 >
>>>> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages
>>>>
>>>> I've also just noticed that the load_drive script should be changed to 
>>>> "insmod
>>>> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that
>>>> correct?
>>>>
>>>>
>>>> Correct
>>>>
>>>> Alfredo
>>>>
>>>>
>>>> Thank you again
>>>> J.
>>>>
>>>> On 4 December 2015 at 09:04, Alfredo Cardigliano <cardigli...@ntop.org>
>>>> wrote:
>>>>
>>>>> Please note the total amount of pages is divided by the nodes, please
>>>>> take a look at
>>>>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages
>>>>>
>>>>> Alfredo
>>>>>
>>>>> On 04 Dec 2015, at 10:00, James <ntop-m...@cyclohexane.net> wrote:
>>>>>
>>>>> In case it helps anyone else re

Re: [Ntop-misc] pf-ring and snort

[James]

I was and now that I'm not, tcpdump works. Obvious when you know. Thank you
again. :)

On 10 December 2015 at 14:04, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Are you running snort at the same time perhaps? Please note that running
> two applications on the same interface is not allowed with ZC (it is still
> possible to use zbalance_ipc to fanout the traffic).
> You said you are using the tcpdump part of pf_ring right?
>
> Alfredo
>
> On 10 Dec 2015, at 14:59, James <ntop-m...@cyclohexane.net> wrote:
>
> Hi Alfredo,
>
> For the lists benefit, I have (with your help) solved my problem by
> increasing $HUGEPAGES to 2048 in the load_driver script. I now have 16
> instances of Snort running and alerts are being generated. I am surprised
> by how few alerts are occurring though and wanted to verify what traffic
> was being seen. I thought tcpdump would be a good way to do this, so
> compiled the pf_ring version, but I can't get that to display any traffic
> on either of my two interfaces. I've tried:
> "tcpdump -i eth4" (or 5) which functions but shows no traffic
> "tcpdump -i zc:eth4"
> "tcpdump -i zc:eth4@0" both of which fail, telling me there is no such
> device
>
> Sorry for asking so many questions, but any ideas please?
>
> Thanks
> James
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pf-ring and snort

[James]

I'm another step forward now, thank you Alfredo. I can get 12 instances to
start (for some reason it's always instances 0-7 and 12-15). Instances 8-11
give the same "bus error" at startup though. There are no errors in
/var/log/messages



On 4 December 2015 at 11:03, James <ntop-m...@cyclohexane.net> wrote:

> Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more
> start and then back to command line. /var/log/messages no longer mentions
> hugepages, so I guess that's not the problem any longer. The NIC's appear
> to go up and down repeatedly (that might be correct?), come in and out of
> promiscuous mode and then lots of these:
>
> ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied
> ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied
> ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied
> etc
>
>
> On 4 December 2015 at 09:46, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> node means NUMA node (i.e. CPU)
>>
>> Alfredo
>>
>>
>> On 04 Dec 2015, at 10:41, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Hopefully my last stupid question - is node the same as processes and
>> queues in this context? So I should do that all the way up to 15?
>>
>> On 4 December 2015 at 09:32, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>>
>>> On 04 Dec 2015, at 10:27, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> Thanks Alfredo, even more that you reply so quickly! I respect the
>>> "teach a man to fish.." method of helping, but that's a lot of parameters
>>> and options and I'd be making complete guesses at which ones to change and
>>> to what values. Would it be possible to recommend what you'd change based
>>> on the spec of my system?
>>>
>>>
>>> In essence if you have 4 nodes, you should set the number of huge pages
>>> per node with:
>>>
>>>   $ echo 1024 >
>>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>>>   $ echo 1024 >
>>> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
>>>   $ echo 1024 >
>>> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>>>   $ echo 1024 >
>>> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages
>>>
>>> I've also just noticed that the load_drive script should be changed to 
>>> "insmod
>>> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that
>>> correct?
>>>
>>>
>>> Correct
>>>
>>> Alfredo
>>>
>>>
>>> Thank you again
>>> J.
>>>
>>> On 4 December 2015 at 09:04, Alfredo Cardigliano <cardigli...@ntop.org>
>>> wrote:
>>>
>>>> Please note the total amount of pages is divided by the nodes, please
>>>> take a look at
>>>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages
>>>>
>>>> Alfredo
>>>>
>>>> On 04 Dec 2015, at 10:00, James <ntop-m...@cyclohexane.net> wrote:
>>>>
>>>> In case it helps anyone else reading this, my startup script needed
>>>> some corrections, ending up with:
>>>>
>>>> for i in `seq 0 1 15`; do
>>>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c
>>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i
>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@
>>>> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i
>>>> done
>>>>
>>>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS
>>>> with multiqueue example does have this set in README.1st I linked earlier?)
>>>> and I needed to change my variable to be $i instead of $1.
>>>>
>>>> However when I run this script it only start's 4 "daemon child", then
>>>> gives a "bus error" on the next 4 and then returns to the command line with
>>>> no mention of the other 8. /var/log/messages tells me:
>>>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) -
>>>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error
>>>> 105)
>>>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot
>>>> allocate memory
>>>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB
>>>

Re: [Ntop-misc] pf-ring and snort

[James]

In case it helps anyone else reading this, my startup script needed some
corrections, ending up with:

for i in `seq 0 1 15`; do
snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c
/etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i
--daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@
$i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i
done

i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS with
multiqueue example does have this set in README.1st I linked earlier?) and
I needed to change my variable to be $i instead of $1.

However when I run this script it only start's 4 "daemon child", then gives
a "bus error" on the next 4 and then returns to the command line with no
mention of the other 8. /var/log/messages tells me:
snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) -
pfring_zc_daq_initialize: Cluster failed: No buffer space available (error
105)
ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot allocate
memory
ZC[4897]: error mmap'ing 128 hugepages of 2048 KB

If it's relevant, when I run the ZC load_driver script (I took the MQ=1,1,1,1
out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that
says:
Warning: 512 hugepages available, 1024 requested

Things I have checked are:
sudo more /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

sudo more /proc/sys/vm/nr_hugepages
1024

sudo more /proc/meminfo
MemTotal:   32748700 kB
MemFree:27563604 kB

AnonHugePages: 0 kB
HugePages_Total:1024
HugePages_Free: 1024
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:   2048 kB

Any ideas on how I can fix this please?

Thanks
J.

On 2 December 2015 at 15:10, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

>
> On 02 Dec 2015, at 16:08, James <ntop-m...@cyclohexane.net> wrote:
>
> Ah.. so if only 16 queues, I should go back to only 16 copies of snort?
>
>
> Yes, or you can consider using zbalance_ipc for load balance in software
> to more queues,
> but I do not what is the performance you can reach with 48 queues, you
> should run some test.
>
> Alfredo
>
>
> On 2 December 2015 at 15:06, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues.
>>
>> Alfredo
>>
>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>> It looks fine, you can omit MQ.
>>
>> Alfredo
>>
>> On 02 Dec 2015, at 16:04, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Many thanks for the help Alfredo. So I'll crank things up to use all
>> CPU's and that gives me (I've converted it to a for loop):
>>
>> for i in `seq 0 1 48`; do
>> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf
>> -l /logs/snort/eth4_eth5/instance-$1 --daq-dir=/usr/local/lib/daq --daq
>> pfring_zc --daq-mode passive -i zc:eth4@$1,zc:eth5@$1 --daq-var
>> clusterid=$1 --daq-var idsbridge=1 --daq-var bindcpu=$1
>> done
>>
>> Is this the correct load_driver.sh setting to match? I'm not sure about
>> the MQ values?
>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48
>>
>> On 2 December 2015 at 14:15, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>> Please use README.1st as reference.
>>> What you need to know:
>>> 1. Use --daq-var clusterid=K where K is a unique number per snort
>>> instance, used for resource allocation
>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, please
>>> ignore interrupts affinity with ZC
>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is
>>> used for IPS/IDS-bridge mode
>>> 4. We usually recommend using only the CPU where the NIC is connected,
>>> however since snort is (likely) the bottleneck, feel free to use all the
>>> cores available, setting RSS=N,N where N is the number of cores and the
>>> number of snort instances.
>>>
>>> Alfredo
>>>
>>> On 02 Dec 2015, at 15:08, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> Follow-up question - should I use the cluster-id parameter?
>>>
>>> This uses it:
>>>
>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st
>>>
>>> But this does not:
>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/
>>>
>>> On 2 December 2015 at 14:01, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I posted a few 

Re: [Ntop-misc] pf-ring and snort

[James]

Thanks Alfredo, even more that you reply so quickly! I respect the "teach a
man to fish.." method of helping, but that's a lot of parameters and
options and I'd be making complete guesses at which ones to change and to
what values. Would it be possible to recommend what you'd change based on
the spec of my system? I've also just noticed that the load_drive script
should be changed to "insmod ./ixgbe.ko RSS=16,16" because I'm only
monitoring two NIC's, is that correct?

Thank you again
J.

On 4 December 2015 at 09:04, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Please note the total amount of pages is divided by the nodes, please take
> a look at https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages
>
> Alfredo
>
> On 04 Dec 2015, at 10:00, James <ntop-m...@cyclohexane.net> wrote:
>
> In case it helps anyone else reading this, my startup script needed some
> corrections, ending up with:
>
> for i in `seq 0 1 15`; do
> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c
> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i
> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@
> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i
> done
>
> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS with
> multiqueue example does have this set in README.1st I linked earlier?) and
> I needed to change my variable to be $i instead of $1.
>
> However when I run this script it only start's 4 "daemon child", then
> gives a "bus error" on the next 4 and then returns to the command line with
> no mention of the other 8. /var/log/messages tells me:
> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) -
> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error
> 105)
> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot allocate
> memory
> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB
>
> If it's relevant, when I run the ZC load_driver script (I took the MQ=1,1,1,1
> out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that
> says:
> Warning: 512 hugepages available, 1024 requested
>
> Things I have checked are:
> sudo more /sys/kernel/mm/transparent_hugepage/enabled
> always madvise [never]
>
> sudo more /proc/sys/vm/nr_hugepages
> 1024
>
> sudo more /proc/meminfo
> MemTotal:   32748700 kB
> MemFree:27563604 kB
> 
> AnonHugePages: 0 kB
> HugePages_Total:1024
> HugePages_Free: 1024
> HugePages_Rsvd:0
> HugePages_Surp:    0
> Hugepagesize:   2048 kB
>
> Any ideas on how I can fix this please?
>
> Thanks
> J.
>
> On 2 December 2015 at 15:10, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>>
>> On 02 Dec 2015, at 16:08, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Ah.. so if only 16 queues, I should go back to only 16 copies of snort?
>>
>>
>> Yes, or you can consider using zbalance_ipc for load balance in software
>> to more queues,
>> but I do not what is the performance you can reach with 48 queues, you
>> should run some test.
>>
>> Alfredo
>>
>>
>> On 2 December 2015 at 15:06, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues.
>>>
>>> Alfredo
>>>
>>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <cardigli...@ntop.org>
>>> wrote:
>>>
>>> It looks fine, you can omit MQ.
>>>
>>> Alfredo
>>>
>>> On 02 Dec 2015, at 16:04, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> Many thanks for the help Alfredo. So I'll crank things up to use all
>>> CPU's and that gives me (I've converted it to a for loop):
>>>
>>> for i in `seq 0 1 48`; do
>>> snort -q --pid-path /var/run --create-pidfile -D -c
>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$1
>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@
>>> $1,zc:eth5@$1 --daq-var clusterid=$1 --daq-var idsbridge=1 --daq-var
>>> bindcpu=$1
>>> done
>>>
>>> Is this the correct load_driver.sh setting to match? I'm not sure about
>>> the MQ values?
>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48
>>>
>>> On 2 December 2015 at 14:15, Alfredo Cardigliano <cardigli...@ntop.org>
>>> wrote:
>>>
>>>> Please use README.1st as reference.
>>>> What you need to know:
>>>> 1. Use --daq-var clusterid=K w

Re: [Ntop-misc] pf-ring and snort

[James]

Hopefully my last stupid question - is node the same as processes and
queues in this context? So I should do that all the way up to 15?

On 4 December 2015 at 09:32, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

>
> On 04 Dec 2015, at 10:27, James <ntop-m...@cyclohexane.net> wrote:
>
> Thanks Alfredo, even more that you reply so quickly! I respect the "teach
> a man to fish.." method of helping, but that's a lot of parameters and
> options and I'd be making complete guesses at which ones to change and to
> what values. Would it be possible to recommend what you'd change based on
> the spec of my system?
>
>
> In essence if you have 4 nodes, you should set the number of huge pages
> per node with:
>
>   $ echo 1024 >
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>   $ echo 1024 >
> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
>   $ echo 1024 >
> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>   $ echo 1024 >
> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages
>
> I've also just noticed that the load_drive script should be changed to "insmod
> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that
> correct?
>
>
> Correct
>
> Alfredo
>
>
> Thank you again
> J.
>
> On 4 December 2015 at 09:04, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> Please note the total amount of pages is divided by the nodes, please
>> take a look at
>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages
>>
>> Alfredo
>>
>> On 04 Dec 2015, at 10:00, James <ntop-m...@cyclohexane.net> wrote:
>>
>> In case it helps anyone else reading this, my startup script needed some
>> corrections, ending up with:
>>
>> for i in `seq 0 1 15`; do
>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c
>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i
>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@
>> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i
>> done
>>
>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS with
>> multiqueue example does have this set in README.1st I linked earlier?) and
>> I needed to change my variable to be $i instead of $1.
>>
>> However when I run this script it only start's 4 "daemon child", then
>> gives a "bus error" on the next 4 and then returns to the command line with
>> no mention of the other 8. /var/log/messages tells me:
>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) -
>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error
>> 105)
>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot allocate
>> memory
>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB
>>
>> If it's relevant, when I run the ZC load_driver script (I took the MQ=1,1,1,1
>> out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that
>> says:
>> Warning: 512 hugepages available, 1024 requested
>>
>> Things I have checked are:
>> sudo more /sys/kernel/mm/transparent_hugepage/enabled
>> always madvise [never]
>>
>> sudo more /proc/sys/vm/nr_hugepages
>> 1024
>>
>> sudo more /proc/meminfo
>> MemTotal:   32748700 kB
>> MemFree:27563604 kB
>> 
>> AnonHugePages: 0 kB
>> HugePages_Total:1024
>> HugePages_Free: 1024
>> HugePages_Rsvd:0
>> HugePages_Surp:0
>> Hugepagesize:   2048 kB
>>
>> Any ideas on how I can fix this please?
>>
>> Thanks
>> J.
>>
>> On 2 December 2015 at 15:10, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>>
>>> On 02 Dec 2015, at 16:08, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> Ah.. so if only 16 queues, I should go back to only 16 copies of snort?
>>>
>>>
>>> Yes, or you can consider using zbalance_ipc for load balance in software
>>> to more queues,
>>> but I do not what is the performance you can reach with 48 queues, you
>>> should run some test.
>>>
>>> Alfredo
>>>
>>>
>>> On 2 December 2015 at 15:06, Alfredo Cardigliano <cardigli...@ntop.org>
>>> wrote:
>>>
>>>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues.
>>>>
>>>> Alfredo
>>>>
>>>> On 02 Dec 2015, at 16:06,

Re: [Ntop-misc] pf-ring and snort

[James]

Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more start
and then back to command line. /var/log/messages no longer mentions
hugepages, so I guess that's not the problem any longer. The NIC's appear
to go up and down repeatedly (that might be correct?), come in and out of
promiscuous mode and then lots of these:

ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied
ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied
ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied
etc


On 4 December 2015 at 09:46, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> node means NUMA node (i.e. CPU)
>
> Alfredo
>
>
> On 04 Dec 2015, at 10:41, James <ntop-m...@cyclohexane.net> wrote:
>
> Hopefully my last stupid question - is node the same as processes and
> queues in this context? So I should do that all the way up to 15?
>
> On 4 December 2015 at 09:32, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>>
>> On 04 Dec 2015, at 10:27, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Thanks Alfredo, even more that you reply so quickly! I respect the "teach
>> a man to fish.." method of helping, but that's a lot of parameters and
>> options and I'd be making complete guesses at which ones to change and to
>> what values. Would it be possible to recommend what you'd change based on
>> the spec of my system?
>>
>>
>> In essence if you have 4 nodes, you should set the number of huge pages
>> per node with:
>>
>>   $ echo 1024 >
>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
>>   $ echo 1024 >
>> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
>>   $ echo 1024 >
>> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages
>>   $ echo 1024 >
>> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages
>>
>> I've also just noticed that the load_drive script should be changed to 
>> "insmod
>> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that
>> correct?
>>
>>
>> Correct
>>
>> Alfredo
>>
>>
>> Thank you again
>> J.
>>
>> On 4 December 2015 at 09:04, Alfredo Cardigliano <cardigli...@ntop.org>
>> wrote:
>>
>>> Please note the total amount of pages is divided by the nodes, please
>>> take a look at
>>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages
>>>
>>> Alfredo
>>>
>>> On 04 Dec 2015, at 10:00, James <ntop-m...@cyclohexane.net> wrote:
>>>
>>> In case it helps anyone else reading this, my startup script needed some
>>> corrections, ending up with:
>>>
>>> for i in `seq 0 1 15`; do
>>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c
>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i
>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@
>>> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i
>>> done
>>>
>>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS
>>> with multiqueue example does have this set in README.1st I linked earlier?)
>>> and I needed to change my variable to be $i instead of $1.
>>>
>>> However when I run this script it only start's 4 "daemon child", then
>>> gives a "bus error" on the next 4 and then returns to the command line with
>>> no mention of the other 8. /var/log/messages tells me:
>>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) -
>>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error
>>> 105)
>>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot
>>> allocate memory
>>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB
>>>
>>> If it's relevant, when I run the ZC load_driver script (I took the 
>>> MQ=1,1,1,1
>>> out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that
>>> says:
>>> Warning: 512 hugepages available, 1024 requested
>>>
>>> Things I have checked are:
>>> sudo more /sys/kernel/mm/transparent_hugepage/enabled
>>> always madvise [never]
>>>
>>> sudo more /proc/sys/vm/nr_hugepages
>>> 1024
>>>
>>> sudo more /proc/meminfo
>>> MemTotal:   32748700 kB
>>> MemFree:27563604 kB
>>> 
>>> AnonHugePages: 0 kB
>>> HugePages_Total:

Re: [Ntop-misc] pf-ring and snort

[James]

Many thanks for the help Alfredo. So I'll crank things up to use all CPU's
and that gives me (I've converted it to a for loop):

for i in `seq 0 1 48`; do
snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf
-l /logs/snort/eth4_eth5/instance-$1 --daq-dir=/usr/local/lib/daq --daq
pfring_zc --daq-mode passive -i zc:eth4@$1,zc:eth5@$1 --daq-var
clusterid=$1 --daq-var idsbridge=1 --daq-var bindcpu=$1
done

Is this the correct load_driver.sh setting to match? I'm not sure about the
MQ values?
insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48

On 2 December 2015 at 14:15, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Please use README.1st as reference.
> What you need to know:
> 1. Use --daq-var clusterid=K where K is a unique number per snort
> instance, used for resource allocation
> 2. Use --daq-var bindcpu=K where K is the core id for affinity, please
> ignore interrupts affinity with ZC
> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is used
> for IPS/IDS-bridge mode
> 4. We usually recommend using only the CPU where the NIC is connected,
> however since snort is (likely) the bottleneck, feel free to use all the
> cores available, setting RSS=N,N where N is the number of cores and the
> number of snort instances.
>
> Alfredo
>
> On 02 Dec 2015, at 15:08, James <ntop-m...@cyclohexane.net> wrote:
>
> Follow-up question - should I use the cluster-id parameter?
>
> This uses it:
>
> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st
>
> But this does not:
> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/
>
> On 2 December 2015 at 14:01, James <ntop-m...@cyclohexane.net> wrote:
>
>> Hi all,
>>
>> I posted a few weeks ago and have since got pf_ring with ZC working. I'm
>> now trying to decide how best to configure snort (in IDS mode). My server
>> has 4 X 12 core CPU's and two NIC's which are being fed one half each of a
>> 10Gb connection.
>>
>> I have a few key questions:
>> - Within the ixgbe zc load_drive.sh script, would the default 16 queue
>> option do, or would you choose something different: insmod ./ixgbe.ko
>> MQ=1,1,1,1 RSS=16,16,16,16
>>
>> - Assuming the choice of 16 above, should I start 16 copies of Snort like
>> this (variation on the example from ntop website)?
>> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf
>> -l /var/log/snort/eth4_eth5/instance-1 --daq-dir=/usr/local/lib/daq --daq
>> pfring_zc --daq-mode passive -i zc:eth4@0+zc:eth5@0 --daq-var
>> idsbridge=1 --daq-var bindcpu=0
>>
>> The information on http://www.metaflows.com/features/pf_ring about CPU
>> affinity and interrupts has confused me somewhat.
>>
>> Thanks
>> J.
>>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pf-ring and snort

[James]

Ah.. so if only 16 queues, I should go back to only 16 copies of snort?

On 2 December 2015 at 15:06, Alfredo Cardigliano <cardigli...@ntop.org>
wrote:

> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues.
>
> Alfredo
>
> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
> It looks fine, you can omit MQ.
>
> Alfredo
>
> On 02 Dec 2015, at 16:04, James <ntop-m...@cyclohexane.net> wrote:
>
> Many thanks for the help Alfredo. So I'll crank things up to use all CPU's
> and that gives me (I've converted it to a for loop):
>
> for i in `seq 0 1 48`; do
> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf
> -l /logs/snort/eth4_eth5/instance-$1 --daq-dir=/usr/local/lib/daq --daq
> pfring_zc --daq-mode passive -i zc:eth4@$1,zc:eth5@$1 --daq-var
> clusterid=$1 --daq-var idsbridge=1 --daq-var bindcpu=$1
> done
>
> Is this the correct load_driver.sh setting to match? I'm not sure about
> the MQ values?
> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48
>
> On 2 December 2015 at 14:15, Alfredo Cardigliano <cardigli...@ntop.org>
> wrote:
>
>> Please use README.1st as reference.
>> What you need to know:
>> 1. Use --daq-var clusterid=K where K is a unique number per snort
>> instance, used for resource allocation
>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, please
>> ignore interrupts affinity with ZC
>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is used
>> for IPS/IDS-bridge mode
>> 4. We usually recommend using only the CPU where the NIC is connected,
>> however since snort is (likely) the bottleneck, feel free to use all the
>> cores available, setting RSS=N,N where N is the number of cores and the
>> number of snort instances.
>>
>> Alfredo
>>
>> On 02 Dec 2015, at 15:08, James <ntop-m...@cyclohexane.net> wrote:
>>
>> Follow-up question - should I use the cluster-id parameter?
>>
>> This uses it:
>>
>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st
>>
>> But this does not:
>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/
>>
>> On 2 December 2015 at 14:01, James <ntop-m...@cyclohexane.net> wrote:
>>
>>> Hi all,
>>>
>>> I posted a few weeks ago and have since got pf_ring with ZC working. I'm
>>> now trying to decide how best to configure snort (in IDS mode). My server
>>> has 4 X 12 core CPU's and two NIC's which are being fed one half each of a
>>> 10Gb connection.
>>>
>>> I have a few key questions:
>>> - Within the ixgbe zc load_drive.sh script, would the default 16 queue
>>> option do, or would you choose something different: insmod ./ixgbe.ko
>>> MQ=1,1,1,1 RSS=16,16,16,16
>>>
>>> - Assuming the choice of 16 above, should I start 16 copies of Snort
>>> like this (variation on the example from ntop website)?
>>> snort -q --pid-path /var/run --create-pidfile -D -c
>>> /etc/snort/snort.conf -l /var/log/snort/eth4_eth5/instance-1
>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@0
>>> +zc:eth5@0 --daq-var idsbridge=1 --daq-var bindcpu=0
>>>
>>> The information on http://www.metaflows.com/features/pf_ring about CPU
>>> affinity and interrupts has confused me somewhat.
>>>
>>> Thanks
>>> J.
>>>
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>>
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] pf-ring and snort

[James]

Hi all,

I posted a few weeks ago and have since got pf_ring with ZC working. I'm
now trying to decide how best to configure snort (in IDS mode). My server
has 4 X 12 core CPU's and two NIC's which are being fed one half each of a
10Gb connection.

I have a few key questions:
- Within the ixgbe zc load_drive.sh script, would the default 16 queue
option do, or would you choose something different: insmod ./ixgbe.ko
MQ=1,1,1,1 RSS=16,16,16,16

- Assuming the choice of 16 above, should I start 16 copies of Snort like
this (variation on the example from ntop website)?
snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf
-l /var/log/snort/eth4_eth5/instance-1 --daq-dir=/usr/local/lib/daq --daq
pfring_zc --daq-mode passive -i zc:eth4@0+zc:eth5@0 --daq-var idsbridge=1
--daq-var bindcpu=0

The information on http://www.metaflows.com/features/pf_ring about CPU
affinity and interrupts has confused me somewhat.

Thanks
J.
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pf-ring and snort

[James]

Follow-up question - should I use the cluster-id parameter?

This uses it:
https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st

But this does not:
http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/

On 2 December 2015 at 14:01, James <ntop-m...@cyclohexane.net> wrote:

> Hi all,
>
> I posted a few weeks ago and have since got pf_ring with ZC working. I'm
> now trying to decide how best to configure snort (in IDS mode). My server
> has 4 X 12 core CPU's and two NIC's which are being fed one half each of a
> 10Gb connection.
>
> I have a few key questions:
> - Within the ixgbe zc load_drive.sh script, would the default 16 queue
> option do, or would you choose something different: insmod ./ixgbe.ko
> MQ=1,1,1,1 RSS=16,16,16,16
>
> - Assuming the choice of 16 above, should I start 16 copies of Snort like
> this (variation on the example from ntop website)?
> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf
> -l /var/log/snort/eth4_eth5/instance-1 --daq-dir=/usr/local/lib/daq --daq
> pfring_zc --daq-mode passive -i zc:eth4@0+zc:eth5@0 --daq-var idsbridge=1
> --daq-var bindcpu=0
>
> The information on http://www.metaflows.com/features/pf_ring about CPU
> affinity and interrupts has confused me somewhat.
>
> Thanks
> J.
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] compiling pf_ring libcap

[James]

Hi,

I'm following this guide:
https://pradyumnajoshi.wordpress.com/2014/03/11/snort-daq-and-pf_ring-installation-on-centos/

And stuck at this stage, just before running ./configure in the libcap dir:
export LIBS=’-L/usr/local/lib -lpfring -lpthread’

I get this error:
bash: export: `-lpfring': not a valid identifier
bash: export: `-lpthread’': not a valid identifier

Any help gratefully received.

Thanks
James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] compiling pf_ring libcap

[James]

Answer my own question in case it helps anyone else - libpcap-devel was
missing because it isn't part of the standard RHEL packages, need to add
optional channel.

On 6 November 2015 at 15:29, James <ntop-m...@cyclohexane.net> wrote:

> Hi,
>
> I'm following this guide:
>
> https://pradyumnajoshi.wordpress.com/2014/03/11/snort-daq-and-pf_ring-installation-on-centos/
>
> And stuck at this stage, just before running ./configure in the libcap dir:
> export LIBS=’-L/usr/local/lib -lpfring -lpthread’
>
> I get this error:
> bash: export: `-lpfring': not a valid identifier
> bash: export: `-lpthread’': not a valid identifier
>
> Any help gratefully received.
>
> Thanks
> James
>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] pf_ring with snort

[James]

Hi all,

I sent this to the snort-users mailing list, but perhaps here is more
appropriate. Is anyone able to help please?

I'm attempting to make a set of instructions in advance of getting the
actual server to unleash it upon. The server will be RHEL 6.5 with a 10gb
intel nic, which I'd like to put snort in IDS mode on. I think I'm correct
that pf_ring is a "good thing", so I'd like to use that. I've spent days
trawling the web but have found lots of conflicting guides which have
confused as much as helped me. Could I ask someone to scan these steps and
tell me if I've missed something vital, done it in the wrong order or
otherwise done something stupid please? Your help is much appreciated.

The short version:
- Use yum to obtain a variety of things the subsequent steps depend on
- Use git to obtain pf_ring and install it
- Install the pf_ring ZC 10gb intel driver
- Get and install libdnet from source
- Get and install the snort daq from source
- Get and install snort from source
- Install the pf_ring daq module
- Start snort with some relevant pf_ring zc parameters
- If that works, next steps configuring snort and barnyard

The long version:

sudo yum -y install wget git kernel-devel libtool subversion automake make
autoconf pcre-devel libpcap-devel libpcap flex bison byacc gcc gcc-c++
zlib-devel numactl numactl-devel
sudo yum install "kernel-devel-uname-r == $(uname -r)"

git clone https://github.com/ntop/PF_RING.git
cd PF_RING/kernel
make
sudo make install
sudo insmod ./pf_ring.ko
cd ../userland
make
cd ../drivers/PF_RING_aware/intel/ixgbe/ixgbe-4.1.2-zc/src
make
./load_driver.sh

wget http://libdnet.googlecode.com/files/libdnet-1.12.tgz
tar zxvf libdnet-1.12.tgz
cd libdnet-1.12
./configure; make; sudo make install

wget https://www.snort.org/downloads/snort/daq-2.0.6.tar.gz
tar xvfz daq-2.0.6.tar.gz
cd daq-2.0.6
./configure; make; sudo make install

wget https://www.snort.org/downloads/snort/snort-2.9.7.6.tar.gz
tar xvfz snort-2.9.7.6.tar.gz
cd snort-2.9.7.6
./configure --enable-sourcefire -enable-reload; make; sudo make install

cd PF_RING/userland/snort/pfring-daq-module-zc
autoreconf -ivf
./configure
make
sudo make install

snort --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i
zc:eth1 --daq-var clusterid=1 -v -e

I've just re-read the pf_ring user manual and I think I've missed some
steps in the /userland dir. I've changed those steps to:

cd ../userland/lib
./configure
make
sudo make install
cd ../libpcap
./configure
make
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] PF_RING 5.4.1 initialization problem with e1000e DNA and non-1500 MTU

[James Hunter]

Hi,

Infact calling:

ifconfig dna0 mtu 1500

causes the problem also, even though the MTU is not altered..

James


Hi,

I use PF_RING to read packets from one port, and forward them to 
another port (and visa-versa) and have been unable to change the MTU 
value for these cards. The system works fine when the MTU is left 
1500, I can ping and see the VLAN traffic flow (without stripping of 
VLAN).


However when I start to flow TCP traffic which uses maximum size 
packets (1518 bytes with VLAN tag) I fail to receive them as the MTU 
is set too low.


I've been unable to change the MTU at all, no matter what I do PF_RING 
fails to open the dna0 port if I do:


ifconfig dna0 mtu 1504

I've tried many different ways to fix this, including different MTU 
values and various other things but it always fails (as for all 
PF_RING examples with Operation Not Permitted)..


Does anyone know how to fix this, its a customer facing problem at the 
minute and we don't easily have the ability to upgrade PF_RING.


Regards,

James

cat /proc/net/pf_ring/info(with MTU 9000)

PF_RING Version : 5.4.1 ($Revision: 5409M$)
Ring slots  : 32768
Slot version: 13
Capture TX  : No [RX only]
IP Defragment   : No
Socket Mode : Standard
Transparent mode: No (mode 2)
Total rings : 0
Total plugins   : 0

cat /proc/net/pf_ring/dev/dna0/info(with MTU 9000)

Name:  dna0
Index: 14
Address:   68:05:CA:09:10:B9
Polling Mode:  NAPI/TNAPI
Type:  Ethernet
Family:Standard NIC
# Bound Sockets:   0
Max # TX Queues:   1
# Used RX Queues:  1


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc




--

*James Hunter*
*JAR Technologies*
Mob. +44 (0) 7738 851417
Office. +44 (0) 2890 23 33 22

www.JarTechnologies.com http://www.jartechnologies.com/

JAR Technologies is a UK registered limited company, privately held.  
This email is from JAR Technologies Ltd. The email and any files 
transmitted with it are confidential and privileged and intended solely 
for the use of the individual or entity to whom they are addressed. If 
you have received the email in error please notify 
general.enquir...@jartechnologies.com 
mailto:general.enquir...@jartechnologies.com and delete the e-mail 
from your system.




___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] PF_RING 5.4.1 initialization problem with e1000e DNA and non-1500 MTU

[James Hunter]

Hi,

Hopefully this information may help some of the developers assist me, I 
found this in the dmesg output:


[ 4121.009662] [ cut here ]

[ 4121.009667] WARNING: at include/linux/skbuff.h:1458 skb_trim+0x38/0x40()

[ 4121.009670] Hardware name: G41MT-S2P

[ 4121.009673] Modules linked in: e1000e pf_ring atl1c [last unloaded: pf_ring]

[ 4121.009682] Pid: 8247, comm: ip Tainted: GW   3.0.12-rt30 #3

[ 4121.009685] Call Trace:

[ 4121.009690]  [8104290a] warn_slowpath_common+0x7a/0xb0

[ 4121.009696]  [81042955] warn_slowpath_null+0x15/0x20

[ 4121.009701]  [814501f8] skb_trim+0x38/0x40

[ 4121.009709]  [a09f50dd] e1000_alloc_rx_buffers_ps+0xbd/0x510 
[e1000e]

[ 4121.009715]  [810082d0] ? nommu_map_sg+0xf0/0xf0

[ 4121.009724]  [a09f8b4e] e1000_configure+0x47e/0x680 [e1000e]

[ 4121.009733]  [a09f9b41] e1000e_up+0x11/0xa0 [e1000e]

[ 4121.009742]  [a09fa288] e1000_change_mtu+0x118/0x200 [e1000e]

[ 4121.009747]  [8145adba] dev_set_mtu+0x3a/0x90

[ 4121.009753]  [8146ca17] do_setlink+0x1c7/0x990

[ 4121.009758]  [8158edb0] ? _raw_spin_unlock+0x10/0x40

[ 4121.009764]  [810f1c17] ? handle_pte_fault+0x517/0x830

[ 4121.009770]  [812b9c40] ? nla_parse+0x90/0xe0

[ 4121.009776]  [8146e815] rtnl_newlink+0x345/0x580

[ 4121.009783]  [8146e19c] rtnetlink_rcv_msg+0x15c/0x250

[ 4121.009788]  [8146e040] ? __rtnl_unlock+0x20/0x20

[ 4121.009793]  [81485129] netlink_rcv_skb+0xa9/0xd0

[ 4121.009798]  [8146c440] rtnetlink_rcv+0x20/0x30

[ 4121.009803]  [81484a10] netlink_unicast+0x2c0/0x310

[ 4121.009808]  [81484d2a] netlink_sendmsg+0x2ca/0x360

[ 4121.009815]  [81447796] sock_sendmsg+0x106/0x120

[ 4121.009820]  [81447615] ? sock_recvmsg+0x115/0x130

[ 4121.009827]  [81038b41] ? get_parent_ip+0x11/0x50

[ 4121.009832]  [81038b41] ? get_parent_ip+0x11/0x50

[ 4121.009838]  [810cf085] ? unlock_page+0x25/0x30

[ 4121.009844]  [810ef299] ? __do_fault+0x3d9/0x4b0

[ 4121.009849]  [81449b6c] ? move_addr_to_kernel+0x5c/0x70

[ 4121.009854]  [81454c4a] ? verify_iovec+0x4a/0xd0

[ 4121.009859]  [8144a00c] __sys_sendmsg+0x37c/0x390

[ 4121.009865]  [810f21a9] ? handle_mm_fault+0x139/0x240

[ 4121.009871]  [81591d7c] ? do_page_fault+0x18c/0x520

[ 4121.009877]  [810f716f] ? do_brk+0x21f/0x320

[ 4121.009881]  [8144a7a3] ? sys_getsockname+0xa3/0xb0

[ 4121.009887]  [8144ad94] sys_sendmsg+0x44/0x80

[ 4121.009892]  [81595abb] system_call_fastpath+0x16/0x1b

[ 4121.009896] ---[ end trace 139fb8e4d3539f0d ]---


James

Hi,

Infact calling:
ifconfig dna0 mtu 1500
causes the problem also, even though the MTU is not altered..

James


Hi,

I use PF_RING to read packets from one port, and forward them to 
another port (and visa-versa) and have been unable to change the MTU 
value for these cards. The system works fine when the MTU is left 
1500, I can ping and see the VLAN traffic flow (without stripping of 
VLAN).


However when I start to flow TCP traffic which uses maximum size 
packets (1518 bytes with VLAN tag) I fail to receive them as the MTU 
is set too low.


I've been unable to change the MTU at all, no matter what I do 
PF_RING fails to open the dna0 port if I do:


ifconfig dna0 mtu 1504

I've tried many different ways to fix this, including different MTU 
values and various other things but it always fails (as for all 
PF_RING examples with Operation Not Permitted)..


Does anyone know how to fix this, its a customer facing problem at 
the minute and we don't easily have the ability to upgrade PF_RING.


Regards,

James

cat /proc/net/pf_ring/info(with MTU 9000)

PF_RING Version : 5.4.1 ($Revision: 5409M$)
Ring slots  : 32768
Slot version: 13
Capture TX  : No [RX only]
IP Defragment   : No
Socket Mode : Standard
Transparent mode: No (mode 2)
Total rings : 0
Total plugins   : 0

cat /proc/net/pf_ring/dev/dna0/info(with MTU 9000)

Name:  dna0
Index: 14
Address:   68:05:CA:09:10:B9
Polling Mode:  NAPI/TNAPI
Type:  Ethernet
Family:Standard NIC
# Bound Sockets:   0
Max # TX Queues:   1
# Used RX Queues:  1


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc




--

*James Hunter*
*JAR Technologies*
Mob. +44 (0) 7738 851417
Office. +44 (0) 2890 23 33 22

www.JarTechnologies.com http://www.jartechnologies.com/

JAR Technologies is a UK registered limited company, privately held.  
This email is from JAR Technologies Ltd. The email and any files 
transmitted with it are confidential and privileged and intended 
solely for the use of the individual or entity to whom

Re: [Ntop-misc] PF_RING 5.4.1 initialization problem with e1000e DNA and non-1500 MTU

[James Hunter]

Hi Alfredo,

This is certainly something I can do to test, but unfortunately the 
customer is not running the latest SVN code on their hardware.


James

James
please update to latest PF_RING from svn and use the insmod parameter 
to set the mtu:


# modinfo e1000e.ko | grep mtu
parm:   mtu:Change the default Maximum Transmission Unit (uint)

Let us know.

Best Regards
Alfredo

On Mar 8, 2013, at 12:48 PM, James Hunter 
james.hun...@jartechnologies.com 
mailto:james.hun...@jartechnologies.com wrote:



Hi,

Hopefully this information may help some of the developers assist me, 
I found this in the dmesg output:


[ 4121.009662] [ cut here ]
[ 4121.009667] WARNING: at include/linux/skbuff.h:1458 skb_trim+0x38/0x40()
[ 4121.009670] Hardware name: G41MT-S2P
[ 4121.009673] Modules linked in: e1000e pf_ring atl1c [last unloaded: pf_ring]
[ 4121.009682] Pid: 8247, comm: ip Tainted: GW   3.0.12-rt30 #3
[ 4121.009685] Call Trace:
[ 4121.009690]  [8104290a] warn_slowpath_common+0x7a/0xb0
[ 4121.009696]  [81042955] warn_slowpath_null+0x15/0x20
[ 4121.009701]  [814501f8] skb_trim+0x38/0x40
[ 4121.009709]  [a09f50dd] e1000_alloc_rx_buffers_ps+0xbd/0x510 
[e1000e]
[ 4121.009715]  [810082d0] ? nommu_map_sg+0xf0/0xf0
[ 4121.009724]  [a09f8b4e] e1000_configure+0x47e/0x680 [e1000e]
[ 4121.009733]  [a09f9b41] e1000e_up+0x11/0xa0 [e1000e]
[ 4121.009742]  [a09fa288] e1000_change_mtu+0x118/0x200 [e1000e]
[ 4121.009747]  [8145adba] dev_set_mtu+0x3a/0x90
[ 4121.009753]  [8146ca17] do_setlink+0x1c7/0x990
[ 4121.009758]  [8158edb0] ? _raw_spin_unlock+0x10/0x40
[ 4121.009764]  [810f1c17] ? handle_pte_fault+0x517/0x830
[ 4121.009770]  [812b9c40] ? nla_parse+0x90/0xe0
[ 4121.009776]  [8146e815] rtnl_newlink+0x345/0x580
[ 4121.009783]  [8146e19c] rtnetlink_rcv_msg+0x15c/0x250
[ 4121.009788]  [8146e040] ? __rtnl_unlock+0x20/0x20
[ 4121.009793]  [81485129] netlink_rcv_skb+0xa9/0xd0
[ 4121.009798]  [8146c440] rtnetlink_rcv+0x20/0x30
[ 4121.009803]  [81484a10] netlink_unicast+0x2c0/0x310
[ 4121.009808]  [81484d2a] netlink_sendmsg+0x2ca/0x360
[ 4121.009815]  [81447796] sock_sendmsg+0x106/0x120
[ 4121.009820]  [81447615] ? sock_recvmsg+0x115/0x130
[ 4121.009827]  [81038b41] ? get_parent_ip+0x11/0x50
[ 4121.009832]  [81038b41] ? get_parent_ip+0x11/0x50
[ 4121.009838]  [810cf085] ? unlock_page+0x25/0x30
[ 4121.009844]  [810ef299] ? __do_fault+0x3d9/0x4b0
[ 4121.009849]  [81449b6c] ? move_addr_to_kernel+0x5c/0x70
[ 4121.009854]  [81454c4a] ? verify_iovec+0x4a/0xd0
[ 4121.009859]  [8144a00c] __sys_sendmsg+0x37c/0x390
[ 4121.009865]  [810f21a9] ? handle_mm_fault+0x139/0x240
[ 4121.009871]  [81591d7c] ? do_page_fault+0x18c/0x520
[ 4121.009877]  [810f716f] ? do_brk+0x21f/0x320
[ 4121.009881]  [8144a7a3] ? sys_getsockname+0xa3/0xb0
[ 4121.009887]  [8144ad94] sys_sendmsg+0x44/0x80
[ 4121.009892]  [81595abb] system_call_fastpath+0x16/0x1b
[ 4121.009896] ---[ end trace 139fb8e4d3539f0d ]---

James

Hi,

Infact calling:
ifconfig dna0 mtu 1500
causes the problem also, even though the MTU is not altered..

James


Hi,

I use PF_RING to read packets from one port, and forward them to 
another port (and visa-versa) and have been unable to change the 
MTU value for these cards. The system works fine when the MTU is 
left 1500, I can ping and see the VLAN traffic flow (without 
stripping of VLAN).


However when I start to flow TCP traffic which uses maximum size 
packets (1518 bytes with VLAN tag) I fail to receive them as the 
MTU is set too low.


I've been unable to change the MTU at all, no matter what I do 
PF_RING fails to open the dna0 port if I do:


ifconfig dna0 mtu 1504

I've tried many different ways to fix this, including different MTU 
values and various other things but it always fails (as for all 
PF_RING examples with Operation Not Permitted)..


Does anyone know how to fix this, its a customer facing problem at 
the minute and we don't easily have the ability to upgrade PF_RING.


Regards,

James

cat /proc/net/pf_ring/info(with MTU 9000)

PF_RING Version : 5.4.1 ($Revision: 5409M$)
Ring slots  : 32768
Slot version: 13
Capture TX  : No [RX only]
IP Defragment   : No
Socket Mode : Standard
Transparent mode: No (mode 2)
Total rings : 0
Total plugins   : 0

cat /proc/net/pf_ring/dev/dna0/info(with MTU 9000)

Name:  dna0
Index: 14
Address:   68:05:CA:09:10:B9
Polling Mode:  NAPI/TNAPI
Type:  Ethernet
Family:Standard NIC
# Bound Sockets:   0
Max # TX Queues:   1
# Used RX Queues:  1


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http

[Ntop-misc] Filtering rule stopped working after updating to recent revision

[Simon James]

Hi

Today I have updated from PF_RING revision 5441 to revision 5613 and I have an 
issue with a filtering rule.
This rule is the last rule in a list and is intended to reflect any incoming 
packets that I don't want to process to an egress interface.
With the previous revision, this worked as expected but with the latest 
revision, it's as if the rule is not matching any packets.
All packets are forwarded to userspace (unless I toggle the default filtering 
policy so that they are dropped).

   filtering_rule rule;
memset(rule, 0, sizeof(rule));
rule.rule_id = nextFilteringRuleId++;
rule.core_fields.proto = 0;
rule.rule_action = reflect_packet_and_stop_rule_evaluation;
snprintf(rule.reflector_device_name, REFLECTOR_NAME_LEN, %s,  eth2);

if (pfring_add_filtering_rule(ring, rule)  0) {
reportError
}

Earlier in the list I do have rules that reflect selected TCP packets to the 
same egress interface and these do continue to work as expected.

Is there any reason why the rule above should no longer be working?

Regards

Simon
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Filtering rule stopped working after updating to recent revision

[Simon James]

Following this up, I looked into pf_ring.c.
It seems to me it's only checking the first rule in the list.
It breaks out of the loop if that does not match.
I was able to get the expected behaviour by removing the break:

$ svn diff pf_ring.c
Index: pf_ring.c
===
--- pf_ring.c(revision 5613)
+++ pf_ring.c(working copy)
@@ -3658,7 +3658,7 @@
   if(unlikely(enable_debug))
 printk([PF_RING] Packet not matched\n);

-  break;
+  //break;
 }
   }  /* for */

$

Regards

Simon

On 02/08/2012 18:31, Simon James sja...@btisystems.com wrote:

Hi

Today I have updated from PF_RING revision 5441 to revision 5613 and I have an 
issue with a filtering rule.
This rule is the last rule in a list and is intended to reflect any incoming 
packets that I don't want to process to an egress interface.
With the previous revision, this worked as expected but with the latest 
revision, it's as if the rule is not matching any packets.
All packets are forwarded to userspace (unless I toggle the default filtering 
policy so that they are dropped).

   filtering_rule rule;
memset(rule, 0, sizeof(rule));
rule.rule_id = nextFilteringRuleId++;
rule.core_fields.proto = 0;
rule.rule_action = reflect_packet_and_stop_rule_evaluation;
snprintf(rule.reflector_device_name, REFLECTOR_NAME_LEN, %s,  eth2);

if (pfring_add_filtering_rule(ring, rule)  0) {
reportError
}

Earlier in the list I do have rules that reflect selected TCP packets to the 
same egress interface and these do continue to work as expected.

Is there any reason why the rule above should no longer be working?

Regards

Simon

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Link Aggregation Module

[Simon James]

Hi

To follow this up, I've looked inside the code and there is no explicit multi 
module listed in pfring_module_list (pfring.c).
But it does accept pfring_open(eth2,eth3, ...), so I am guessing the multi 
module is there by default and multi: does not need to be prepended to the 
device interface name(s).

Please let me know if that is NOT the case.

Thanks and regards

Simon


On 31/07/2012 15:52, Simon James sja...@btisystems.com wrote:

Luca

Thanks for the quick response!
Just to clarify, I was looking to do something like:

pfring_open(multi:eth2;eth3, ...);

But it is failing with Invalid argument.

Should it be possible to do this?

Regards

Simon


On 31/07/2012 15:08, Luca Deri d...@ntop.org wrote:

Simon,
all you need to do is -i interfaces. Example -i eth0,eth1 or use specific 
apps such as pfcount_bundle

Regards Luca

On Jul 31, 2012, at 2:47 PM, Simon James sja...@btisystems.com wrote:

Link Aggregation Module
Hi

I've been using pf_ring on single interfaces without any problem for a while, 
but I would like to experiment with the multi module to capture packets from 
multiple interfaces.
The user guide says these components are compiled inside the library according 
to the supports detected by the configure script.

How do I know whether the link aggregation module has been compiled and, if it 
has not, what does it depend on?

Please excuse me if I have missed something obvious.

Regards

Simon
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Link Aggregation Module

[Simon James]

Alfredo

Many thanks!

Simon


On 01/08/2012 19:20, Alfredo Cardigliano cardigli...@ntop.org wrote:

Simon
the multi module has been replaced by the kernel support for multiple 
interfaces, so there is no need to prepend multi: to the device list.

Alfredo

On Aug 1, 2012, at 12:04 PM, Simon James sja...@btisystems.com wrote:

Re: [Ntop-misc] Link Aggregation Module
Hi

To follow this up, I've looked inside the code and there is no explicit multi 
module listed in pfring_module_list (pfring.c).
But it does accept pfring_open(eth2,eth3, ...), so I am guessing the multi 
module is there by default and multi: does not need to be prepended to the 
device interface name(s).

Please let me know if that is NOT the case.

Thanks and regards

Simon


On 31/07/2012 15:52, Simon James sja...@btisystems.com 
x-msg://312/sja...@btisystems.com  wrote:

Luca

Thanks for the quick response!
Just to clarify, I was looking to do something like:

pfring_open(multi:eth2;eth3, ...);

But it is failing with Invalid argument.

Should it be possible to do this?

Regards

Simon


On 31/07/2012 15:08, Luca Deri d...@ntop.org x-msg://312/d...@ntop.org  
wrote:

Simon,
all you need to do is -i interfaces. Example -i eth0,eth1 or use specific 
apps such as pfcount_bundle

Regards Luca

On Jul 31, 2012, at 2:47 PM, Simon James sja...@btisystems.com 
x-msg://312/sja...@btisystems.com  wrote:

Link Aggregation Module
Hi

I've been using pf_ring on single interfaces without any problem for a while, 
but I would like to experiment with the multi module to capture packets from 
multiple interfaces.
The user guide says these components are compiled inside the library according 
to the supports detected by the configure script.

How do I know whether the link aggregation module has been compiled and, if it 
has not, what does it depend on?

Please excuse me if I have missed something obvious.

Regards

Simon
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it x-msg://312/Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] non DNA-Latency

[James Hunter]

Hi,

I've a very simple test application that takes in packets from Eth 1 and 
passes then to Eth 2, and visa versa  - the (real time / CPU affinity 
locked) threads spin on pf_recv() and does an immediate pf_send()


On a vanilla cable connection I see packet round trips of about 0.160 to 
0.200ms - with PF_Ring I see packet trips of 0.380 to 0.400 (an extra 
~200ms)


I have verified with DMA enabled PF_ring that there is no noticable 
increase in delay, but I'm curious as to why non-DNA pf_ring adds such 
high latency (+200ms is a eternity considering the thread spins on 
pf_recv() )


Is there anything I can do to reduce this? My application is latency 
sensitive but I don't want to use DMA (at least not at the minute)


Regards,

James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] non DMA-Latency

[James Hunter]

Apologies but the scale used in the previous e-mail was incorrect, Its 
+200 microseconds (so a little bit less latency than I first thought ;)


I'd still like to have information on reducing this further without 
resorting to DMA, is it possible?


Regards,

James

Hi,

I've a very simple test application that takes in packets from Eth 1 
and passes then to Eth 2, and visa versa  - the (real time / CPU 
affinity locked) threads spin on pf_recv() and does an immediate 
pf_send()


On a vanilla cable connection I see packet round trips of about 0.160 
to 0.200ms - with PF_Ring I see packet trips of 0.380 to 0.400 (an 
extra ~200ms)


I have verified with DMA enabled PF_ring that there is no noticable 
increase in delay, but I'm curious as to why non-DNA pf_ring adds such 
high latency (+200ms is a eternity considering the thread spins on 
pf_recv() )


Is there anything I can do to reduce this? My application is latency 
sensitive but I don't want to use DMA (at least not at the minute)


Regards,

James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] PF_RING, Zero copy and freeing packets

[James Hunter]

Hi,

I'm trying to understand the ownership of packet memory in PF_RING.

I don't see any explicit way of telling PF_RING to free or release the 
packet when I'm finished with it, does the packet memory remain valid 
until a) you call recv again, b) recv returns a new packet or c) you 
transmit the packet on?


What if I don't want to transmit the packet on, or receive another packet?

Regards,

James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] PF_RING, Zero copy and freeing packets

[James Hunter]

Thanks for the quick reply, I`ll definitely have to check out libzero 
when I get myself accustomed to PF_RING.


I've a similar question with transmitting packets:

When I transmit on a previously (zero copy) received packet the memory 
is freed or returned to the pool, however If I'm generating my own 
packets (using malloc) how is the freeing of this memory handled?


I can see two options, firstly pfring_send() free's the memory for me 
(either immediately or when its sent on the wire) or I can free the 
memory immediately after the pfring_send() function.


Regards,

James

James
with the classic (non-DNA) PF_RING the packet is valid until you move to the 
next one. This for RX and TX. So if you need to keep it you must copy it.

In DNA it's the same, but we're almost down with a new library called libzero 
that (it's not just limited to this) allows you to
1. keep packets in memory for later processing without copy (e.g. you need to 
collect all fragments and then process them at that point)
2. enlarge the buffer beyond the physical card memory size
3. switch/transmit/modify packets in zero-copy (as you can see we have 
published some results on 
http://www.ntop.org/pf_ring/pf_ring-dna-rfc-2544-benchmark/) across interfaces 
with minimal latency typical of switches

Libzero will be made available later this month, but in the meantime you can 
start testing it using the code that's currently in SVN

REgards Luca

On Apr 12, 2012, at 10:32 AM, James Hunter wrote:


Hi,

I'm trying to understand the ownership of packet memory in PF_RING.

I don't see any explicit way of telling PF_RING to free or release the packet 
when I'm finished with it, does the packet memory remain valid until a) you call recv 
again, b) recv returns a new packet or c) you transmit the packet on?

What if I don't want to transmit the packet on, or receive another packet?

Regards,

James
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] PF_RING and VLAN tagged packets

[Simon James]

Alfredo

As requested:
# ethtool -d eth4 | grep -i vlan
   VLAN mode: disabled
   VLAN filter:   enabled
0x00038: VET (VLAN Ether type)0x8100
0x05AC0: IMIRVP  (Immed interr rx VLAN priority)  0x

Simon

On 27/03/2012 11:01, Alfredo Cardigliano cardigli...@ntop.org wrote:

Simon
please let me see the output of
ethtool -d eth4 | grep -i vlan

Alfredo

On Mar 27, 2012, at 11:41 AM, Simon James wrote:

Re: [Ntop-misc] PF_RING and VLAN tagged packets
Alfredo

Thank you for the prompt response!

I'm using the igb driver:
# modinfo igb
filename:   
/lib/modules/2.6.40.4-5.local.fc15.x86_64/kernel/drivers/net/igb/igb.ko
version:3.0.6-k2

# dmesg | grep igb
[9.523397] igb :0b:00.0: eth4: (PCIe:2.5Gb/s:Width x4) 00:1b:21:6e:c4:d6
[9.523477] igb :0b:00.0: eth4: PBA No: E64750-004
[9.523480] igb :0b:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx 
queue(s)
[9.523515] igb :0b:00.1: PCI INT B - GSI 58 (level, low) - IRQ 58
[9.523530] igb :0b:00.1: setting latency timer to 64

I installed PF_RING, per the get-started/download page:

cd PF_RING/kernel

make

sudo insmod ./pf_ring.ko

Regards

Simon



On 27/03/2012 10:26, Alfredo Cardigliano cardigli...@ntop.org 
x-msg://100/cardigli...@ntop.org  wrote:

Simon
which driver/mode are you using with PF_RING? Please give us some more details 
about your configuration

Regards
Alfredo

On Mar 27, 2012, at 11:12 AM, Simon James wrote:

PF_RING and VLAN tagged packets
Hi

I want to use PF_RING to capture packets that are VLAN tagged, but it seems to 
me that the tag is getting stripped before the packets reach my application.

If I run a standard tcpdump, I see the tag:
listening on eth4, link-type EN10MB (Ethernet), capture size 65535 bytes
10:06:41.108489 ARP, Request who-has 172.27.8.1 tell 172.27.8.104, length 42
0x:     001b 215c 6d60 8100 0008   --- Tag: 8100 
0008
0x0010:  0806 0001 0800 0604 0001 001b 215c 6d60
0x0020:  ac1b 0868    ac1b 0801 
0x0030:       

But if I run the tcpdump from the userland directory, I don't see it:
listening on eth4, link-type EN10MB (Ethernet), capture size 8192 bytes
09:08:47.363752413 ARP, Request who-has 172.27.8.1 tell 172.27.8.104, length 42
0x:     001b 215c 6d60 0806 0001   --- Tag has been 
stripped
0x0010:  0800 0604 0001 001b 215c 6d60 ac1b 0868
0x0020:     ac1b 0801   
0x0030:     

Is this the expected behaviour?

I am using PF_RING SVN Revision 5271.

I am new to PF_RING so I apologize if I am missing something obvious (although 
I have checked the FAQ and the recent archives of this list).

Regards

Simon
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it x-msg://100/Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe 5 Pro SIP Plugin Issues

[James Jackson]

Thanks, 5.2.9 appears to work fine. I had tried specifying all those templates 
before so it was probably related to 5.2.7.

James

--- On Sat, 7/25/09, Luca Deri d...@ntop.org wrote:

 From: Luca Deri d...@ntop.org
 Subject: Re: [Ntop-misc] nProbe 5 Pro SIP Plugin Issues
 To: ntop-misc@listgateway.unipi.it
 Date: Saturday, July 25, 2009, 1:17 PM
 James
 0. use 5.2.9
 1. send me a .pcap file I can use for reproducing the
 problem
 2. your template is too small. You need at least IP
 src/dst, proto and ports. Example
 
 -T %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP
 %INPUT_SNMP %OUTPUT_SNMP %OUT_PKTS %OUT_BYTES
 %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT
 %PROTOCOL %SIP_CALL_ID
 
 Luca
 
 On Jul 25, 2009, at 5:35 AM, James Jackson wrote:
 
  
  
  I'm attempting to log some SIP stats locally as an
 initial test. Basic stats (IP address etc) are logged, but
 no SIP stats are logged. Note that if the debug level is
 increased above 1 with the SIP plugin enabled, there is a
 seg. fault. Also, all the traffic on this interface is SIP
 (udp port 5060).
  
  Thanks,
  James
  
  
  [r...@xxx nprobe_5.2.7_061609_pro]# nprobe -i eth0 -P
 /var/log/nprobe -D t -V 9 -T %SIP_CALL_ID -b 1
  24/Jul/2009 22:18:03 [nprobe.c:3104] Welcome to nprobe
 v.5.2.7 ($Revision: 977 $) for i686-redhat-linux-gnu with
 native PF_RING acceleration
  24/Jul/2009 22:18:03 [nprobe.c:3113] Tracing enabled
  24/Jul/2009 22:18:03 [nprobe.c:3152] Dumping flow
 files every 60 sec into directory /var/log/nprobe
  24/Jul/2009 22:18:03 [nprobe.c:3157] WARNING: -n
 parameter is missing. 127.0.0.1:2055 will be used.
  24/Jul/2009 22:18:03 [nprobe.c:1490] Exporting flows
 towards 127.0.0.1:2055 using UDP
  24/Jul/2009 22:18:03 [plugin.c:124] Loading
 plugins...
  24/Jul/2009 22:18:03 [plugin.c:139] Loading plugins
 from ./plugins
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/flowIdPlugin.so'
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/dumpPlugin.so'
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/rtpPlugin.so'
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/smtpPlugin.so'
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/l7Plugin.so'
  24/Jul/2009 22:18:03 [l7Plugin.c:398] L7 plugin
 disabled (missing library)
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/sipPlugin.so'
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/dbPlugin.so'
  24/Jul/2009 22:18:03 [plugin.c:396] Loaded
 './plugins/httpPlugin.so'
  24/Jul/2009 22:18:03 [flowIdPlugin.c:54] Initialized
 FlowId plugin
  24/Jul/2009 22:18:03 [dumpPlugin.c:75] Initialized
 dump plugin
  24/Jul/2009 22:18:03 [rtpPlugin.c:101] Initialized RTP
 plugin
  24/Jul/2009 22:18:03 [smtpPlugin.c:48] Initialized
 SMTP plugin
  24/Jul/2009 22:18:03 [sipPlugin.c:70] Initialized SIP
 plugin
  24/Jul/2009 22:18:03 [dbPlugin.c:156] WARNING: DB
 support is not enabled (disabled at compile time)
  24/Jul/2009 22:18:03 [httpPlugin.c:59] Initialized
 HTTP plugin
  24/Jul/2009 22:18:03 [plugin.c:170] 7 plugin(s) loaded
 [7 delete][7 packet].
  24/Jul/2009 22:18:03 [nprobe.c:3780] Welcome to nprobe
 v.5.2.7 for i686-redhat-linux-gnu
  24/Jul/2009 22:18:03 [nprobe.c:3987] Each flow is 50
 bytes long
  24/Jul/2009 22:18:03 [nprobe.c:3988] The # packets per
 flow has been set to 27
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 Flow Serial Identifier
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 dump
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 RTP
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 SMTP Protocol Dissector
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 SIP
  24/Jul/2009 22:18:03 [plugin.c:542] Enabling plugin
 SIP
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 MySQL DB
  24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin
 HTTP Protocol Dissector
  24/Jul/2009 22:18:03 [plugin.c:554] 2 plugin(s)
 enabled
  24/Jul/2009 22:18:03 [nprobe.c:4008] WARNING: Your
 template ignores IP addresses: your collector might ignore
 these flows.
  24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin
 Flow Serial Identifier (no template is using it)
  24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin
 dump (no template is using it)
  24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin
 RTP (no template is using it)
  24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin
 SMTP Protocol Dissector (no template is using it)
  24/Jul/2009 22:18:03 [plugin.c:502] Enabling plugin
 SIP
  24/Jul/2009 22:18:03 [plugin.c:502] Enabling plugin
 MySQL DB
  24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin
 HTTP Protocol Dissector (no template is using it)
  24/Jul/2009 22:18:03 [private/pf_ring.c:175]
 Succesfully open PF_RING v.3.9.4 on device eth0
  24/Jul/2009 22:18:03 [nprobe.c:4078] The flows hash
 has 4096 buckets
  24/Jul/2009 22:18:03 [nprobe.c:4080] Flows older than
 120 seconds will be exported
  24/Jul/2009 22:18:03 [nprobe.c:4086] Flows inactive
 for at least 30 seconds will be exported
  24

Re: [Ntop-misc] PF_RING build failure on Ubuntu 9.04

[James Jackson]

It compiles now, thanks ! PF_RING appears to be working.

I still don't get SIP stats. I'll start a new thread.

James

--- On Fri, 7/24/09, Luca Deri d...@ntop.org wrote:

 From: Luca Deri d...@ntop.org
 Subject: Re: [Ntop-misc] PF_RING build failure on Ubuntu 9.04
 To: ntop-misc@listgateway.unipi.it
 Date: Friday, July 24, 2009, 5:09 PM
 It should compile. Please report.
 
 Thanks Luca
 
 On Jul 24, 2009, at 11:40 PM, James Jackson wrote:
 
 
  Luca,
 
  Are you saying the current snapshot should now compile
 or we should  
  wait for an update ?
 
  Thanks !
  James
 
  --- On Fri, 7/24/09, Luca Deri d...@ntop.org
 wrote:
 
  From: Luca Deri d...@ntop.org
  Subject: Re: [Ntop-misc] PF_RING build failure on
 Ubuntu 9.04
  To: ntop-misc@listgateway.unipi.it
  Date: Friday, July 24, 2009, 4:21 PM
  James
  as I have this bug with TX, I'm reworking the code
 so this
  stuff has
  been temporarily commented out until I fix the
 problem.
 
  Luca
 
  On Jul 24, 2009, at 10:16 PM, James Jackson
 wrote:
 
 
 
  It seems like there was a kernel patch that
 moved
  HARD_TX_LOCK and
  HARD_TX_UNLOCK to a header file:
 
  http://www.mail-archive.com/net...@vger.kernel.org/msg47931.html
 
  We can give that a try.
 
  James
 
  --- On Fri, 7/24/09, James Jackson jamesjackso...@yahoo.com
  wrote:
 
  From: James Jackson jamesjackso...@yahoo.com
  Subject: Re: [Ntop-misc] PF_RING build
 failure on
  Ubuntu 9.04
  To: ntop-misc@listgateway.unipi.it
  Date: Friday, July 24, 2009, 2:51 PM
 
  Thanks ! It looks like we're now in a
 similar
  position:
 
  net/ring/ring_packet.c: In function
  'add_skb_to_ring':
  net/ring/ring_packet.c:2498: error:
 implicit
  declaration of
  function 'HARD_TX_LOCK'
  net/ring/ring_packet.c:2506: error:
 implicit
  declaration of
  function 'HARD_TX_UNLOCK'
  make[2]: *** [net/ring/ring_packet.o]
 Error 1
  make[1]: *** [net/ring] Error 2
  make: *** [net] Error 2
 
 
  --- On Fri, 7/24/09, Troy Jordan troy.jor...@gmail.com
  wrote:
 
  From: Troy Jordan troy.jor...@gmail.com
  Subject: Re: [Ntop-misc] PF_RING build
 failure
  on
  Ubuntu 9.04
  To: ntop-misc@listgateway.unipi.it
  Date: Friday, July 24, 2009, 8:15 AM
 
  James,
 
  The errors prior to HARD_TX_LOCK and
  HARD_TX_UNLCOCK
  can be
  remedied on
  RHEL5 by commenting out the #if block
 that
  contains
  those 4
  functions in
  ring_packet.c (before running
 mkpatch.sh) as
  mentioned
  here:
 
  http://www.gossamer-threads.com/lists/ntop/misc/22387
 
  - Troy
 
 
  I'm also getting a build failure
 on Red
  Hat 5.3
  after
  the ring module failed to load in Red
 Hat 4.4.
  Ubuntu
  was
  going to be the next on the list :)
 Can anyone
  suggest
  a
  current known good build environment
 ?
 
 
  net/ring/ring_packet.c:1311:
 error:
  redefinition
  of
  'ip_hdr'
  include/linux/ip.h:109: error:
 previous
  definition of
  'ip_hdr' was here
  net/ring/ring_packet.c:1317:
 error:
  redefinition
  of
  'skb_set_network_header'
  include/linux/skbuff.h:1021:
 error:
  previous
  definition of 'skb_set_network_header'
 was
  here
  net/ring/ring_packet.c:1322:
 error:
  redefinition
  of
  'skb_reset_network_header'
  include/linux/skbuff.h:1016:
 error:
  previous
  definition of
 'skb_reset_network_header' was
  here
  net/ring/ring_packet.c:1327:
 error:
  redefinition
  of
  'skb_reset_transport_header'
  include/linux/skbuff.h:995:
 error:
  previous
  definition
  of 'skb_reset_transport_header' was
 here
  net/ring/ring_packet.c: In
 function
  'add_skb_to_ring':
  net/ring/ring_packet.c:2496:
 error:
  implicit
  declaration of function
 'HARD_TX_LOCK'
  net/ring/ring_packet.c:2504:
 error:
  implicit
  declaration of function
 'HARD_TX_UNLOCK'
  make[2]: ***
 [net/ring/ring_packet.o]
  Error 1
  make[1]: *** [net/ring] Error 2
  make: *** [net] Error 2
 
 
  [root[at]xxx SPECS]# cat
  /etc/redhat-release
  Red Hat Enterprise Linux Server
 release
  5.3
  (Tikanga)
  [root[at]xxx SPECS]# uname -a
  Linux xxx 2.6.18-128.el5PAE #1 SMP
 Wed Dec
  17
  12:02:33
  EST 2008 i686 i686 i386 GNU/Linux
 
  I'm using a PF_RING snapshot from
  yesterday.
 
  Thanks,
  James
 
 
  --- On Thu, 7/23/09, troy jordan
  troy.jordan[at]gmail.com
 wrote:
 
  From: troy jordan
  troy.jordan[at]gmail.com
 
  Subject: [Ntop-misc] PF_RING
 build
  failure
  on
  Ubuntu 9.04
  To:
 ntop-misc[at]listgateway.unipi.it
  Date: Thursday, July 23, 2009,
 3:25
  PM
 
  I have encountered a new build
 error
  with
  the  pfring v.
  3.9.5 (build 3825) and am
 wondering if
  anyone
  else
  has seen
  this:
 
 
  % make-kpkg -initrd
  -revision=pfring3.9.5
  kernel_image
  snip
 
  CC [M] 
 net/ring/ring_packet.o
 
 
 net/ring/ring_packet.c:2500:26: error:
  macro
  HARD_TX_LOCK requires 3
 arguments,
  but only
  2
  given
  net/ring/ring_packet.c: In
 function
  ‘add_skb_to_ring’:
  net/ring/ring_packet.c:2496:
 error:
  ‘HARD_TX_LOCK’
  undeclared (first use in this
  function)
 
  net/ring/ring_packet.c:2496:
 error:
  (Each

[Ntop-misc] nProbe 5 Pro SIP Plugin Issues

[James Jackson]

I'm attempting to log some SIP stats locally as an initial test. Basic stats 
(IP address etc) are logged, but no SIP stats are logged. Note that if the 
debug level is increased above 1 with the SIP plugin enabled, there is a seg. 
fault. Also, all the traffic on this interface is SIP (udp port 5060).

Thanks,
James


[r...@xxx nprobe_5.2.7_061609_pro]# nprobe -i eth0 -P /var/log/nprobe -D t -V 9 
-T %SIP_CALL_ID -b 1
24/Jul/2009 22:18:03 [nprobe.c:3104] Welcome to nprobe v.5.2.7 ($Revision: 977 
$) for i686-redhat-linux-gnu with native PF_RING acceleration
24/Jul/2009 22:18:03 [nprobe.c:3113] Tracing enabled
24/Jul/2009 22:18:03 [nprobe.c:3152] Dumping flow files every 60 sec into 
directory /var/log/nprobe
24/Jul/2009 22:18:03 [nprobe.c:3157] WARNING: -n parameter is missing. 
127.0.0.1:2055 will be used.
24/Jul/2009 22:18:03 [nprobe.c:1490] Exporting flows towards 127.0.0.1:2055 
using UDP
24/Jul/2009 22:18:03 [plugin.c:124] Loading plugins...
24/Jul/2009 22:18:03 [plugin.c:139] Loading plugins from ./plugins
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/flowIdPlugin.so'
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/dumpPlugin.so'
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/rtpPlugin.so'
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/smtpPlugin.so'
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/l7Plugin.so'
24/Jul/2009 22:18:03 [l7Plugin.c:398] L7 plugin disabled (missing library)
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/sipPlugin.so'
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/dbPlugin.so'
24/Jul/2009 22:18:03 [plugin.c:396] Loaded './plugins/httpPlugin.so'
24/Jul/2009 22:18:03 [flowIdPlugin.c:54] Initialized FlowId plugin
24/Jul/2009 22:18:03 [dumpPlugin.c:75] Initialized dump plugin
24/Jul/2009 22:18:03 [rtpPlugin.c:101] Initialized RTP plugin
24/Jul/2009 22:18:03 [smtpPlugin.c:48] Initialized SMTP plugin
24/Jul/2009 22:18:03 [sipPlugin.c:70] Initialized SIP plugin
24/Jul/2009 22:18:03 [dbPlugin.c:156] WARNING: DB support is not enabled 
(disabled at compile time)
24/Jul/2009 22:18:03 [httpPlugin.c:59] Initialized HTTP plugin
24/Jul/2009 22:18:03 [plugin.c:170] 7 plugin(s) loaded [7 delete][7 packet].
24/Jul/2009 22:18:03 [nprobe.c:3780] Welcome to nprobe v.5.2.7 for 
i686-redhat-linux-gnu
24/Jul/2009 22:18:03 [nprobe.c:3987] Each flow is 50 bytes long
24/Jul/2009 22:18:03 [nprobe.c:3988] The # packets per flow has been set to 27
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin Flow Serial Identifier
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin dump
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin RTP
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin SMTP Protocol Dissector
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin SIP
24/Jul/2009 22:18:03 [plugin.c:542] Enabling plugin SIP
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin MySQL DB
24/Jul/2009 22:18:03 [plugin.c:532] Scanning plugin HTTP Protocol Dissector
24/Jul/2009 22:18:03 [plugin.c:554] 2 plugin(s) enabled
24/Jul/2009 22:18:03 [nprobe.c:4008] WARNING: Your template ignores IP 
addresses: your collector might ignore these flows.
24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin Flow Serial Identifier (no 
template is using it)
24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin dump (no template is using 
it)
24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin RTP (no template is using 
it)
24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin SMTP Protocol Dissector 
(no template is using it)
24/Jul/2009 22:18:03 [plugin.c:502] Enabling plugin SIP
24/Jul/2009 22:18:03 [plugin.c:502] Enabling plugin MySQL DB
24/Jul/2009 22:18:03 [plugin.c:498] Disabling plugin HTTP Protocol Dissector 
(no template is using it)
24/Jul/2009 22:18:03 [private/pf_ring.c:175] Succesfully open PF_RING v.3.9.4 
on device eth0
24/Jul/2009 22:18:03 [nprobe.c:4078] The flows hash has 4096 buckets
24/Jul/2009 22:18:03 [nprobe.c:4080] Flows older than 120 seconds will be 
exported
24/Jul/2009 22:18:03 [nprobe.c:4086] Flows inactive for at least 30 seconds 
will be exported
24/Jul/2009 22:18:03 [nprobe.c:4089] Expired flows will be checked every 30 
seconds
24/Jul/2009 22:18:03 [nprobe.c:4091] Expired flows will not be queued for more 
than 30 seconds
24/Jul/2009 22:18:03 [nprobe.c:4108] IP Protocol will be ignored and set to 0.
24/Jul/2009 22:18:03 [nprobe.c:4111] IP addresses will be ignored and set to 0.
24/Jul/2009 22:18:03 [nprobe.c:4114] UDP/TCP src/dst ports will be ignored and 
set to 0.
24/Jul/2009 22:18:03 [nprobe.c:4120] Flows ASs will not be computed
24/Jul/2009 22:18:03 [nprobe.c:4148] Flows will be emitted in NetFlow 9 format
24/Jul/2009 22:18:03 [nprobe.c:4177] Flow input interface index is dynamic 
(last two MAC address bytes)
24/Jul/2009 22:18:03 [nprobe.c:4183] Flow output interface index is dynamic 
(last two MAC address bytes)
24/Jul/2009 22:18:03 [nprobe.c:4224] Capturing packets from interface eth0
24/Jul/2009 22:18:03 [nprobe.c:4261] Starting 2 packet fetch

[Ntop-misc] PF_RING kernel module fails to load

[James Jackson]

I have purchased nProbe Pro and I'm attempting to configure it with PF_RING on 
Red Hat Enterprise Linux AS release 4 (Nahant Update 4). Without PF_RING nProbe 
is not reporting any SIP statistics. I assume that PF_RING is required for the 
SIP plugin to work. As such, I have built a new kernel RPM (following the 
directions here: 
http://synfulpacket.blogspot.com/2007/04/kernel-pfring-rpm.html).

After rebooting, the ring kernel module refuses to load:

root[xxx]:/lib/modules/2.6.9-42.0.10.EL.ring3smp uname -a
Linux dal00sip0001 2.6.9-42.0.10.EL.ring3smp #1 SMP Tue Jul 21 17:09:11 CDT 
2009 i686 i686 i386 GNU/Linux

root[xxx]:/lib/modules/2.6.9-42.0.10.EL.ring3smp/kernel/net/ring insmod ring.ko
insmod: error inserting 'ring.ko': -1 Unknown symbol in module

root[xxx]:/lib/modules/2.6.9-42.0.10.EL.ring3smp/kernel/net/ring dmesg

ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
bnx2: eth1: using MSI
bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
device eth1 entered promiscuous mode
eth1: no IPv6 routers present
device eth1 left promiscuous mode
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK
ring: Unknown symbol remap_pfn_range
ring: Unknown symbol HARD_TX_LOCK
ring: Unknown symbol HARD_TX_UNLOCK


nProbe has compiled completely (including PF_RING) but of course still fails.

Any help would be much appreciated.

Thanks,
James





  
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc