Re: [Ntop-misc] UDP flow collection / nprobe question

[Luca Deri]

Hi Peter
the problem is that your nProbe is receiving templates from multiple routers 
and they exceed the number of 256. You should see a log like

13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=257][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=1]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=258][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=2]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=259][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=3]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=260][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=4]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=261][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=5]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=262][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=6]
13/Nov/2020 09:43:22 [collect.c:1624] Added new flow template definition 
[id=263][flow_version=9][netflow_device=127.0.0.1:50509][observation_domain_id=115][total=7]

to show the templates defined.

We have introduced new checks as in some installations people was sending too 
many templates and nProbe did not operate properly,  So it might be this has 
broken something on your case.

I would appreciate if you can contact me directly and provide me information 
for troubleshooting this issue.

Thanks Luca


> On 12 Nov 2020, at 23:06, Peter Giles  wrote:
> 
> Hello ntop team,
> 
> I've been trying to troubleshoot a flow collection issue where an old version 
> of nprobe was collecting a high flow volume, but collecting from the same 
> stream of UDP flow packets with a newer version produced inconsistent flow 
> collection rates.  I was getting ready to write up a bug report, but decided 
> to update to the latest stable nprobe version and try again first.  Now, when 
> I run nprobe the log output is full of lines like these:
> 
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined 
> [bucket_id: 28][num: 256]: skipping
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined 
> [bucket_id: 29][num: 256]: skipping
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined 
> [bucket_id: 30][num: 256]: skipping
> 12/Nov/2020 13:50:57 [collect.c:1632] WARNING: Too many templates defined 
> [bucket_id: 25][num: 256]: skipping
> 
> I'm not sure I understand what this is telling me.  Wild guess: Are there 
> duplicate IPFIX templates for the same observation domain that nprobe isn't 
> sure what to do with?
> 
> My nprobe command line is like this:
> 
> /usr/bin/nprobe --collector-port=2155 --verbose 1 --max-log-lines=10 
> --dump-path=/u01/flow/raw/2056 --collector=none --disable-cache 
> --dump-format=t --dont-nest-dump-dirs --dont-drop-privileges 
> --smart-udp-frags -V 10
> 
> And the version I'm running:
> 
> $ nprobe --version
> 
> Welcome to nProbe v.9.2.201112 (r6993) for x86_64-pc-linux-gnu
> with native PF_RING acceleration.
> Copyright 2002-20 ntop.org 
> 
> Build OS:  Ubuntu 20.04.1 LTS
> ...
> 
> Thank you,
> Peter
> 
> -- 
> Peter Giles  |  Senior Developer & Data Analyst  |  Office of the CISO  |  
> University of Washington
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe dynamic blacklist / --max-num-flows

[Luca Deri]

Hi Peter
we have used Lua so far for other activities, even it could be potentially 
extended to what you need. The problem, is that executing a script during 
export slows down things a bit.
Please let us know if there is something you would like to implement

Cheers Luca

> On 22 Feb 2019, at 21:14,Peter Giles  wrote:
> 
> The -f tip is good to know.
> 
> Regarding --max-num-flows, while I could imagine different approaches, I 
> really just wanted to understand what the implemented behavior is.
> 
> Back to the wish for dynamic blacklisting, could Lua scripting be an option 
> for this? It seems there is no API for the Lua script to tell nProbe to 
> ignore a flow though.
> 
> Thanks again,
> Peter
> 
> On Fri, Feb 22, 2019 at 3:01 AM ntop-misc-requ...@listgateway.unipi.it 
> <mailto:ntop-misc-requ...@listgateway.unipi.it> 
>  <mailto:ntop-misc-requ...@listgateway.unipi.it>> wrote:
> Send Ntop-misc mailing list submissions to
> ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> or, via email, send a message with subject or body 'help' to
> ntop-misc-requ...@listgateway.unipi.it 
> <mailto:ntop-misc-requ...@listgateway.unipi.it>
> 
> You can reach the person managing the list at
> ntop-misc-ow...@listgateway.unipi.it 
> <mailto:ntop-misc-ow...@listgateway.unipi.it>
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Ntop-misc digest..."
> 
> 
> Today's Topics:
> 
>1. Re: nProbe dynamic blacklist / --max-num-flows (Luca Deri)
> 
> 
> --
> 
> Message: 1
> Date: Thu, 21 Feb 2019 11:39:48 +0100
> From: Luca Deri mailto:d...@ntop.org>>
> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> Subject: Re: [Ntop-misc] nProbe dynamic blacklist / --max-num-flows
> Message-ID: <80b34f5e-ef6d-41a7-ada9-ae4a6f64b...@ntop.org 
> <mailto:80b34f5e-ef6d-41a7-ada9-ae4a6f64b...@ntop.org>>
> Content-Type: text/plain; charset="us-ascii"
> 
> Hi Peter
> changing them on the fly is not supported. Better if you filter the host with 
> -f so avoid processing packets at all, instead of discarding egress flows
> 
> Yes of there is a DoS, flows exceeding the threshold are dropped, this to 
> avoid to DoS also nProbe. What is the algorithm you have in mind exactly?
> 
> Regards Luca
> 
> > On 21 Feb 2019, at 01:03, Peter Giles  > <mailto:gil...@uw.edu>> wrote:
> > 
> > Hi ntop team, I have a couple of nProbe questions for you:
> > 
> > We had an incident where a badly behaved host increased the number of flow 
> > records being generated by nProbe by a factor of 10 and really stressed our 
> > downstream processing.  I ended up restarting our nProbe processes with an 
> > added --black-list x.x.x.x/32 option to ignore that host.  That led me to 
> > wonder, is there any way to dynamically change the blacklist configuration 
> > so that in the future I could add a host or network without having to 
> > restart nProbe? Doing so without restarting would be preferable since 
> > restarting will result in some data loss across all the monitored traffic.  
> > I didn't see anything in the documentation, but thought it would be worth 
> > checking here.
> > 
> > On a related note, I wonder about the --max-num-flows option which limits 
> > the number of active flows in the case of DoS, etc. In the event that the 
> > maximum number of flows is exceeded, what flows will get discarded?  Any 
> > new flows above the limit, or is there a more selective algorithm?
> > 
> > Thank you!
> > Peter
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> -- next part --
> An HTML attachment was scrubbed...
> URL: 
> <http://listgateway.unipi.it/mailman/private/ntop-misc/attachments/20190221/2b2859e4/attachment-0001.htm
>  
> <http://listgateway.unipi.it/mailman/private/ntop-misc/attachments/20190221/2b2859e4/attachment-0001.htm>>
> 
> --
> 
> ___
> Ntop-misc mailing list
> Nt

Re: [Ntop-misc] nProbe dynamic blacklist / --max-num-flows

[Luca Deri]

Hi Peter
changing them on the fly is not supported. Better if you filter the host with 
-f so avoid processing packets at all, instead of discarding egress flows

Yes of there is a DoS, flows exceeding the threshold are dropped, this to avoid 
to DoS also nProbe. What is the algorithm you have in mind exactly?

Regards Luca

> On 21 Feb 2019, at 01:03, Peter Giles  wrote:
> 
> Hi ntop team, I have a couple of nProbe questions for you:
> 
> We had an incident where a badly behaved host increased the number of flow 
> records being generated by nProbe by a factor of 10 and really stressed our 
> downstream processing.  I ended up restarting our nProbe processes with an 
> added --black-list x.x.x.x/32 option to ignore that host.  That led me to 
> wonder, is there any way to dynamically change the blacklist configuration so 
> that in the future I could add a host or network without having to restart 
> nProbe? Doing so without restarting would be preferable since restarting will 
> result in some data loss across all the monitored traffic.  I didn't see 
> anything in the documentation, but thought it would be worth checking here.
> 
> On a related note, I wonder about the --max-num-flows option which limits the 
> number of active flows in the case of DoS, etc. In the event that the maximum 
> number of flows is exceeded, what flows will get discarded?  Any new flows 
> above the limit, or is there a more selective algorithm?
> 
> Thank you!
> Peter
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Send nprobe output to file - nprobe -P ?

[Luca Deri]

David
the dump to disk in text file has been done for batch DB import whereas JSON 
for export to apps. Like I have said you can use —tcp and create a small app 
(e.g. see https://pymotw.com/2/socket/tcp.html 
<https://pymotw.com/2/socket/tcp.html>) that can receive JSON as requested

Regards Luca

> On 2 Aug 2018, at 00:51, David Kraut  wrote:
> 
> Hi Luca, 
> 
> I got the output to text file working.  Use case is that we have a developer 
> trying to integrate nprobe output to an application they're working on. The 
> developer asked us to provide sample output in JSON format.  Any trick to do 
> that?  
> 
> Thanks!
> 
> David   
> 
> On Wednesday, August 1, 2018, 3:54:11 PM CDT, Luca Deri  wrote:
> 
> 
> David,
> you need to use -P , but when you start nprobe, the user is nobody so 
> this can be the problem. See also
> 
> --dont-drop-privileges  | Do not drop privileges changing to user 
> nobody
> --unprivileged-user   | Use  instead of nobody when 
> dropping privileges
> 
> This said with -P you have text file, no JSON that instead can be exported
> --json-to-syslog| Export flows in JSON format to syslog
> --json-labels   | In case JSON label is used (e.g. with 
> ZMQ)
> --tcp  | Deliver flows in JSON format to the 
> specified server via TCP.
> 
> to syslog, TCP or ELK.
> 
> What is the use case exactly?
> 
> Regards Luca
> 
> 
>> On 1 Aug 2018, at 22:26, David Kraut > <mailto:da...@rocketmail.com>> wrote:
>> 
>> I'm trying to send nprobe output to a file (preferably in JSON format). From 
>> the user guide, I'm guessing that's the -P option, but no matter what I do, 
>> I get an error stating - Sorry, the path you specified with -P is invalid.  
>> I created a directory with 777 permission.  Could someone please provide a 
>> sample of how to use nprobe -P ? or how to output flows to a file?  I only 
>> need a small sample of flows on this file for testing.
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Send nprobe output to file - nprobe -P ?

[Luca Deri]

David,
you need to use -P , but when you start nprobe, the user is nobody so this 
can be the problem. See also

--dont-drop-privileges  | Do not drop privileges changing to user 
nobody
--unprivileged-user   | Use  instead of nobody when 
dropping privileges

This said with -P you have text file, no JSON that instead can be exported
--json-to-syslog| Export flows in JSON format to syslog
--json-labels   | In case JSON label is used (e.g. with ZMQ)
--tcp  | Deliver flows in JSON format to the 
specified server via TCP.

to syslog, TCP or ELK.

What is the use case exactly?

Regards Luca


> On 1 Aug 2018, at 22:26, David Kraut  wrote:
> 
> I'm trying to send nprobe output to a file (preferably in JSON format). From 
> the user guide, I'm guessing that's the -P option, but no matter what I do, I 
> get an error stating - Sorry, the path you specified with -P is invalid.  I 
> created a directory with 777 permission.  Could someone please provide a 
> sample of how to use nprobe -P ? or how to output flows to a file?  I only 
> need a small sample of flows on this file for testing.
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] collector-sample-rate unrecognized option

[Luca Deri]

Simone
Can you please update the guide?

Luca

> On 22 Jul 2018, at 16:04, Ryan Gelobter  wrote:
> 
> That option is still in the current nprobe users guide. Also about a year ago 
> I wrote to the mailing list and Simone had recommended the setting 
> 
>> On Sat, Jul 21, 2018, 2:28 AM Luca Deri  wrote:
>> Ryan
>> that option has been reworked. Can you please tell me where you read of it 
>> that we need to fix the documentation?
>> 
>> This said this is the option you need to use.
>> 
>> Thanks Luca
>> 
>> [--sample-rate|-S] ::
>> | Packet capture sampling rate (-i only)
>> | and NetFlow collection/export sampling 
>> rate.
>> | If  starts with
>> | '@' it means that nprobe will report
>> | the specified sampling rate but will
>> | not sample itself as incoming packets
>> | are already sampled on the specified
>> | capture device at the specified rate.
>> | NOTE: in sFlow pkt rate is part of the 
>> packet.
>> | Flow collection rate specify the flow 
>> sampling
>> | rate of the flows being collected (-3 
>> only)). 
>> | Flow export rate (export only e.g. -n)
>> | specifies how many flows are exported 
>> | (i.e. it does not affect flow 
>> collection).
>> | Default: 1:1:1 [no sampling]
>> 
>> 
>>> On 20 Jul 2018, at 20:45, Ryan Gelobter  wrote:
>>> 
>>> Does --collector-sample-rate require a valid license? I'm running in demo 
>>> mode and get unrecognized option recently.
>>> 
>>> /usr/local/bin/nprobe -n 10.x.x.x:9996 -n 10.x.x.x:9996 --zmq 
>>> tcp://127.0.0.01:5556 -X 20 --collector-sample-rate 10
>>> 
>>> /usr/local/bin/nprobe: unrecognized option '--collector-sample-rate'
>>> 20/Jul/2018 18:44:33 [nprobe.c:5659] WARNING: Unrecognized option 
>>> '--collector-sample-rate'
>>> 
>>> /usr/local/bin/nprobe --version
>>> 
>>> Welcome to nProbe v.8.5.180719 (r6203) for x86_64-unknown-linux-gnu
>>> with native PF_RING acceleration.
>>> Copyright 2002-18 ntop.org
>>> 
>>> Build OS:  CentOS Linux release 7.5.1804 (Core)
>>> GIT rev:   dev:79f34d329f9dfd560495fc046d82634a15a9a13f:20180719
>>> License:   Invalid nProbe license (/etc/nprobe.license) [Missing 
>>> license file]
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>> 
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] collector-sample-rate unrecognized option

[Luca Deri]

Ryan
that option has been reworked. Can you please tell me where you read of it that 
we need to fix the documentation?

This said this is the option you need to use.

Thanks Luca

[--sample-rate|-S] ::
| Packet capture sampling rate (-i only)
| and NetFlow collection/export sampling 
rate.
| If  starts with
| '@' it means that nprobe will report
| the specified sampling rate but will
| not sample itself as incoming packets
| are already sampled on the specified
| capture device at the specified rate.
| NOTE: in sFlow pkt rate is part of the 
packet.
| Flow collection rate specify the flow 
sampling
| rate of the flows being collected (-3 
only)). 
| Flow export rate (export only e.g. -n)
| specifies how many flows are exported 
| (i.e. it does not affect flow collection).
| Default: 1:1:1 [no sampling]


> On 20 Jul 2018, at 20:45, Ryan Gelobter  wrote:
> 
> Does --collector-sample-rate require a valid license? I'm running in demo 
> mode and get unrecognized option recently.
> 
> /usr/local/bin/nprobe -n 10.x.x.x:9996 -n 10.x.x.x:9996 --zmq 
> tcp://127.0.0.01:5556  -X 20 
> --collector-sample-rate 10
> 
> /usr/local/bin/nprobe: unrecognized option '--collector-sample-rate'
> 20/Jul/2018 18:44:33 [nprobe.c:5659] WARNING: Unrecognized option 
> '--collector-sample-rate'
> 
> /usr/local/bin/nprobe --version
> 
> Welcome to nProbe v.8.5.180719 (r6203) for x86_64-unknown-linux-gnu
> with native PF_RING acceleration.
> Copyright 2002-18 ntop.org 
> 
> Build OS:  CentOS Linux release 7.5.1804 (Core)
> GIT rev:   dev:79f34d329f9dfd560495fc046d82634a15a9a13f:20180719
> License:   Invalid nProbe license (/etc/nprobe.license) [Missing license 
> file]
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe to hadoop/hdfs?

[Luca Deri]

Scott
no there are no plans as we’re not very skilled on Hadoop. But if you can tell 
me more in detail what you need and how, I will can see if I can accommodate 
that

Cheers Luca

> On 17 Jul 2018, at 22:10, Scott Bossi  wrote:
> 
> Hi,
>  
> I was wondering if there were any plans to output nprobe data to Hadoop/hdfs? 
>  
> Scott Bossi
> Cyber Threat Operations
> Cyber Operations Engineering
> Raytheon Company
>  
> +1.978.436.3750 business
> scott.v.bo...@raytheon.com 
>  
>  
> 880 Technology Park Drive
> Billerica, MA 01821-4164 USA
> www.raytheon.com 
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe and ntopng on same system?

[Luca Deri]

David
yes you can. When you want to do SPAN + collection you need to start two nprobe 
instances both exporting to ELK.

So 

nprobe -n none -i ethX … --elastic …
nprobe -n none -i none -3  … --elastic …

Regards Luca

> On 12 Jul 2018, at 05:51, David Kraut  wrote:
> 
> Is it possible to run nprobe and ntopng on the same box?  If so, what is the 
> correct config for Ubuntu 18 assuming I want to ingest both both SPAN and 
> Netflow and ultimately send stale data to ELK (or similar) to maintain ~30 
> day history.  I'm able to get ntopng working and connect via port 3000, but 
> when I try to install nprobe on the same box, I'm never able to get port 2055 
> to show as listening via netstat. 
> 
> Thanks! 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe performance, zbalance packet drops

[Luca Deri]

Hi David
your template is huge. Can you please omit (just for troubleshooting) 
"--flow-templ….” and report if you see changes in load?

Thanks Luca

> On 27 Jun 2018, at 08:43, David Notivol  wrote:
> 
> Hi,
> And now:
> - 1.log = scenario in your point 1, including top, zbalance output, and 
> nprobe stats.
> 
> El mié., 27 jun. 2018 a las 17:41, David Notivol ( >) escribió:
> Hi Alfredo,
> 
> Sorry, I forgot to attach the files as you said. I sent them awhile ago, but 
> it seems the mail size is over the limit and get held for approval. I'm 
> trying now deleting some info from my first email and pasting one file at a 
> time.
>> - 0.log = top output for the scenario in my fist email.
> 
> 
> El mié., 27 jun. 2018 a las 14:30, Alfredo Cardigliano ( >) escribió:
> Hi David
> 
>> On 27 Jun 2018, at 14:20, David Notivol > > wrote:
>> 
>> Hi Alfredo,
>> Thanks for  your recommendations.
>> 
>> I tested using core affinity as you suggested, and the in drops disappeared 
>> in zbalance. The output drops persist, but the absolute drops are less than 
>> before.
>> Actually I had tested the core affinity, but I didn't have in mind the 
>> physical cores. Now I put zbalance in one physical core, and 10 nprobe 
>> instances not sharing the physical core with zbalance.
>> 
>> About your point 2, by using zc drivers, how could I run several nprobe 
>> instances to share the load? I'm testing with one instance: -i 
>> zc:p2p1,zc:p2p2
> 
> You can keep using zbalance_ipc (-i zc:p2p1,zc:p2p2), or you can use RSS 
> (running nprobe on  -i zc:p2p1@,zc:p2p2@)
> 
>> Attached you can find:
>> - 0.log = top output for the scenario in my previous email.
>> - 1.log = scenario in your point 1, including top, zbalance output, and 
>> nprobe stats.
> 
> 
> I do not see the attachments, did you forget to enclose them?
> 
> Alfredo
> 
>> 
>> El mié., 27 jun. 2018 a las 12:13, Alfredo Cardigliano 
>> (mailto:cardigli...@ntop.org>>) escribió:
>> Hi David
>> it seems that you have packet loss both on zbalance and nprobe, 
>> I recommend you to:
>> 1. set the core affinity for both zbalance_ipc and the nprobe instances, 
>> trying to
>> use a different core for each (at least do not share the zbalance_ipc 
>> physical core
>> with nprobe instances)
>> 2. did you try using zc drivers for capturing traffic from the interfaces? 
>> (zc:p2p1,zc:p2p2)
>> Please also provide the top output (press 1 to see all cored) with the 
>> current configuration,
>> I guess kernel is using some of the available cpu with this configuration.
>> 
>> Alfredo
>> 
>>> On 26 Jun 2018, at 16:31, David Notivol >> > wrote:
>>> 
>>> Hi Alfredo,
>>> Thanks for replying.
>>> This is an excerpt of the zbalance and nprobe statistics:
>>> 
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:265] =
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:266] Absolute Stats: Recv 
>>> 1'285'430'239 pkts (1'116'181'903 drops) - Forwarded 1'266'272'285 pkts 
>>> (19'157'949 drops)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:305] p2p1,p2p2 RX 
>>> 1285430267 pkts Dropped 1116181981 pkts (46.5 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 0 RX 77050882 
>>> pkts Dropped 1127883 pkts (1.4 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 1 RX 70722562 
>>> pkts Dropped 756409 pkts (1.1 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 2 RX 76092418 
>>> pkts Dropped 1017335 pkts (1.3 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 3 RX 75088386 
>>> pkts Dropped 896678 pkts (1.2 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 4 RX 91991042 
>>> pkts Dropped 2114739 pkts (2.2 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 5 RX 81384450 
>>> pkts Dropped 1269385 pkts (1.5 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 6 RX 84310018 
>>> pkts Dropped 1801848 pkts (2.1 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 7 RX 84554242 
>>> pkts Dropped 1487329 pkts (1.7 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 8 RX 84090370 
>>> pkts Dropped 1482864 pkts (1.7 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 9 RX 73642498 
>>> pkts Dropped 732237 pkts (1.0 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 10 RX 76481026 
>>> pkts Dropped 1000496 pkts (1.3 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 11 RX 72496642 
>>> pkts Dropped 929049 pkts (1.3 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 12 RX 79386626 
>>> pkts Dropped 1122169 pkts (1.4 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 13 RX 79418370 
>>> pkts Dropped 1187172 pkts (1.5 %)
>>> 26/Jun/2018 17:29:58 [zbalance_ipc.c:319] Q 14 RX 80284162 
>>> pkts Dropped 1195559 pkts (1.5 %)
>>> 26/Jun/2018 17:29:58 

Re: [Ntop-misc] nprobe scaling...

[Luca Deri]

Hi Scott
thanks for using nProbe. A single instance should be able to collect 10-20k+ 
flows/core, this if you’re able to distribute flows across instances. Export to 
ElasticSearch has been improved (and extended to support the latest version) 
recently. What nProbe version are you using?

In order to assist you I would like you to send
- the exact command line you are using to start nprobe
- how do you balance traffic across the probes running on your system

Thanks Luca

> On 26 Jun 2018, at 07:51, Scott Bossi  wrote:
> 
> We are evaluating nprobe, and the results so far look very good.  We are 
> looking for advise on the best method to scale nprobe.  We have 3 reasonably 
> large linux systems -  32 cpu's/68gb of memory each.  We get about 150k flows 
> per second peak, with an avg of about 60k flows per second.  So far, we have 
> been running many nprobe instances (over 100)  on the same server to scale.   
> Nprobe is using very little cpu or memory, which makes me wonder if there is 
> a better way to scale this, so that one instance can take better advantage of 
> the resources on the server.
> Any advice is appreciated.
>  
>  
> We have also trying to export the data to Elastic, but it appears that the 
> nprobe can’t keep up with the data, as it’s exporting in very small batches, 
> in very small sizes.  Is there a way to fine-tune how the data is exported?
>  
> Thanks. 
>  
> Scott Bossi
> Cyber Threat Operations
> Cyber Operations Engineering
> Raytheon Company
>  
> +1.978.436.3750 business
> scott.v.bo...@raytheon.com 
>  
>  
> 880 Technology Park Drive
> Billerica, MA 01821-4164 USA
> www.raytheon.com 
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Running ntop on cisco nexus 3K/9K switch?

[Luca Deri]

Hi David
we’re not familiar with these devices but it would be nice to create a 
container you can use them. Please let us know more about this topic so we can 
build a ntopng container ready for this platform

Regards Luca

> On 19 Jun 2018, at 19:51, David Kraut  wrote:
> 
> Cisco Nexus 3K/9K switches allow you to run embedded linux containers/vm's.  
> Has anyone found a good use case for running ntop in this space?  Possibly 
> using the switch to aggregate span/taps from within the environment and 
> sending the data internally to ntop? An all-in-one aggregation/ntop box?  
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] Learning the ntopng Lua API

[Luca Deri]

Hi all,
we have finally written the documentation for the ntopng Lua API. You can read 
more here https://www.ntop.org/ntopng/learning-the-ntopng-lua-api/

Please let us know what you think and what is missing.

Thank you, Luca
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe -A option

[Luca Deri]

Pedro
yes we moved to geoid in recent versions. Please see 
https://blog.maxmind.com/2015/09/29/building-your-own-mmdb-database-for-fun-and-profit/
 
<https://blog.maxmind.com/2015/09/29/building-your-own-mmdb-database-for-fun-and-profit/>
 that explains you how to create your custom dat files

Lyca

> On 7 May 2018, at 14:06, PEDRO RODRIGUES TORRES <pedrr...@it.uc3m.es> wrote:
> 
> Hello Luca,
> 
> Possibly I read an old user guide: 
> https://www.ntop.org/wp-content/uploads/2013/03/nProbe_UserGuide.pdf 
> <https://www.ntop.org/wp-content/uploads/2013/03/nProbe_UserGuide.pdf>
> 
> I think the problem to use de BGP Plugin in my scenery is due I need to 
> process pcap files. I am not sure if the full BGP table will be loaded in 
> nProbe prior to process the pcap files.
> Other problem is that bgp_probe_client.pl <http://bgp_probe_client.pl/> is 
> not working for me... I need to debug it but if has some easy way to insert 
> prefix/ASN in the probes, let me know.
> 
> --
> Pedro
> 
> 
> 2018-05-07 12:51 GMT+02:00 Luca Deri <d...@ntop.org <mailto:d...@ntop.org>>:
> >
> > Pedro,
> > the file specified with -A has to be in GeoIP format, not text as you did
> >
> > Luca
> >
> > On 4 May 2018, at 18:20, PEDRO RODRIGUES TORRES <pedrr...@it.uc3m.es 
> > <mailto:pedrr...@it.uc3m.es>> wrote:
> >
> > Hello,
> >
> > I am using nProbe Pro v.8.5.180504 ($Revision: 6149 $) for
> > x86_64-pc-linux-gnu with native PF_RING acceleration.
> >
> > I am trying to use -A option with a data file like this:
> > $ head -n 5 ASList.dat
> > 2116:212.89.60.0/24 <http://212.89.60.0/24>
> > 2116:212.89.59.0/24 <http://212.89.59.0/24>
> > 2116:212.89.58.0/24 <http://212.89.58.0/24>
> > 2116:212.89.57.0/24 <http://212.89.57.0/24>
> > 2116:212.89.56.0/24 <http://212.89.56.0/24>
> >
> > The command line used to start nProbe is:
> > nprobe -V9 --pcap-file-list myfilelist.txt -A ASList.dat
> >
> > I got this output:
> > ...
> > 04/May/2018 13:00:58 [util.c:465] GeoIP: loaded AS config file ASList.dat
> > 04/May/2018 13:00:58 [util.c:479] WARNING: Unable to load AS IPv6 file
> > ASListv6.dat. AS IPv6 support disabled
> > ...
> > 04/May/2018 13:00:58 [nprobe.c:8925] nProbe started successfully
> >
> >
> > The problem is that the exported dump is not being modified to include
> > srcas, dstas, srcmsk, dstmsk.
> > (the mask is the most important for me). I double checked it in my
> > local collector (nfdump).
> > A strange thing happen when I stop the nProbe with ctrl-c. I lot of
> > messages like this are printed:
> >
> > Invalid database type GeoIP Country Edition, expected GeoIP Organization 
> > Edition
> >
> > Yes, I could try to use the BGP Plugin, but currently I do not have a
> > easy BGP full routing router. :(
> >
> > I appreciate any help,
> > Pedro
> >
> > --
> > PEDRO RODRIGUES TORRES
> > Universidad Carlos III de Madrid
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> >
> >
> >
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> 
> 
> 
> --
> PEDRO RODRIGUES TORRES
> Universidad Carlos III de Madrid
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe -A option

[Luca Deri]

Pedro,
the file specified with -A has to be in GeoIP format, not text as you did

Luca

> On 4 May 2018, at 18:20, PEDRO RODRIGUES TORRES  wrote:
> 
> Hello,
> 
> I am using nProbe Pro v.8.5.180504 ($Revision: 6149 $) for
> x86_64-pc-linux-gnu with native PF_RING acceleration.
> 
> I am trying to use -A option with a data file like this:
> $ head -n 5 ASList.dat
> 2116:212.89.60.0/24 
> 2116:212.89.59.0/24 
> 2116:212.89.58.0/24 
> 2116:212.89.57.0/24 
> 2116:212.89.56.0/24 
> 
> The command line used to start nProbe is:
> nprobe -V9 --pcap-file-list myfilelist.txt -A ASList.dat
> 
> I got this output:
> ...
> 04/May/2018 13:00:58 [util.c:465] GeoIP: loaded AS config file ASList.dat
> 04/May/2018 13:00:58 [util.c:479] WARNING: Unable to load AS IPv6 file
> ASListv6.dat. AS IPv6 support disabled
> ...
> 04/May/2018 13:00:58 [nprobe.c:8925] nProbe started successfully
> 
> 
> The problem is that the exported dump is not being modified to include
> srcas, dstas, srcmsk, dstmsk.
> (the mask is the most important for me). I double checked it in my
> local collector (nfdump).
> A strange thing happen when I stop the nProbe with ctrl-c. I lot of
> messages like this are printed:
> 
> Invalid database type GeoIP Country Edition, expected GeoIP Organization 
> Edition
> 
> Yes, I could try to use the BGP Plugin, but currently I do not have a
> easy BGP full routing router. :(
> 
> I appreciate any help,
> Pedro
> 
> -- 
> PEDRO RODRIGUES TORRES
> Universidad Carlos III de Madrid
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe and Andrisoft compatibility

[Luca Deri]

Benjaminall I did is this:I have started "nprobe nprobe.conf” (basically is your config file) and sent some flows to nprobe, then captured the emitted flows with wireshark. I enclose the pcap with such flows. If you open them with wireshark everything looks good with no decoding errors whatsoever.Please tell the Wansight folks to contact us and report the exact issue (so that we can reproduce it an fix it), so we can use to reproduce the issues they mentioned you.Regards Luca 

nprobe.tgz
Description: Binary data
On 12 Mar 2018, at 11:18, Benjamin Weik  wrote:Hi, I am trying to use nProbe as a flow filter & forwarder to filter out flows for customer prefixes and forward those flows to the customers Wansight but I am unable to get something useful on Wansight.Sometimes a few flows are recieved and a little bit is graphed but with each flow received, the timeout is increased until Wansight says the flow is too old and discards it...Andrisoft support says that nProbe is at fault: >If the flow exporter respects the RFC and it's configured to export long flows periodically, you only need to adjust the Flow Timeout(s) parameter from the Flow Sensor configuration window to the same value. >All flows will be accepted, even if the start time is very long in the past. >We don't have a nProbe license to be able to test it, but not even Wireshark can properly decode the start/end time of flows generated by it. So we can only conclude that it's a nProbe issue.>We do have customers that are monitoring their routers with Netflow v9 and IPFIX without any issues from Wanguard. Am I missing any parameters for nProbe? Am I misthinking something? ___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] About pf_ring licences (was nProbe Pro won't do more then 1Gb/s?)

[Luca Deri]

Marco
I suggest you to start reading the PF_RING articles on our blog as the 
discussion is pretty long. First of all think in terms of pps and not Gbps, 
then consider the configuration of nprobe, the nature of traffic (fragments?) 
etc. Packet capture is just one component 

Luca

> On 25 Jan 2018, at 21:54, Marco Teixeira <ma...@scom.uminho.pt> wrote:
> 
> Ok. And one can expect to reach more than 1Gb/s on vanilla drivers right? On 
> a somewhat decent server... Xeon with PCIe x8 NIC...
> 
> ​Regards
> Marco​
> 
> 
> 
> 2018-01-25 19:52 GMT+00:00 Luca Deri <d...@ntop.org <mailto:d...@ntop.org>>:
> 
> 
>> On 25 Jan 2018, at 20:29, Marco Teixeira <ma...@scom.uminho.pt 
>> <mailto:ma...@scom.uminho.pt>> wrote:
>> 
>> Hi Luca,
>> 
>> I have the details and was going to generate de license fo 10Gb/s... but I 
>> was expecting an option for "Standard NIC" and only see Intel, Myricom and 
>> Napatech/DAG/Fiberblaze...
>> Wich one should I choose to make for my "Standard NIC" that uses PF_RING in 
>> NAPI mode? 
>> ​Or maybe there was some misunderstanding here?​
> Yes there is some disconnection Marco
> 
> You only need a license for ZC drivers, otherwise you will use vanilla 
> drivers. So pfcount -i zc:ens2f0 requires a ZC drivers,  pfcount -i ens2f0 
> does not
> 
> Please see https://github.com/ntop/PF_RING/wiki 
> <https://github.com/ntop/PF_RING/wiki> for details
> 
> Regards Luca
> 
>> 
>> ​Regards, and thank you for your advice, and time, 
>> Marco​
>> 
>> 
>> 
>> 
>> 2018-01-25 12:05 GMT+00:00 Luca Deri <d...@ntop.org <mailto:d...@ntop.org>>:
>> Marco
>> our licenses don’t have a cap on speed, but they are per device family. So 
>> if these devices as 10G you need a 10G license
>> 
>> As you’re a university you can mail educat...@ntop.org 
>> <mailto:educat...@ntop.org> for free licenses
>> 
>> Regards Luca
>> 
>> 
>>> On 25 Jan 2018, at 12:52, Marco Teixeira <ma...@scom.uminho.pt 
>>> <mailto:ma...@scom.uminho.pt>> wrote:
>>> 
>>> Hello,
>>> 
>>> Regarding the output below, does one need a license to be able to run 
>>> 10Gb/s speeds (please mind, I'm only talking about pf_ring. nprobe is 
>>> already licensed with "enterprise pro")?
>>> 
>>> 
>>> ===
>>> [marco@nprobe ~]$ sudo pfcount -L -v1 | grep ens2f
>>> NameSystemName  Module  MAC BusID   
>>> NumaNodeStatus  License
>>> ens2f0  ens2f0  pf_ring D8:D3:85:A0:12:50   :13:00.0-1  
>>> Up  NotFound
>>> ens2f1  ens2f1  pf_ring D8:D3:85:A0:12:54   :13:00.1-1  
>>> DownNotFound
>>> [marco@nprobe ~]$ 
>>> [marco@nprobe ~]$ cat /proc/net/pf_ring/dev/ens2f0/info 
>>> Name: ens2f0
>>> Index:6
>>> Address:  D8:D3:85:A0:12:50
>>> Polling Mode: NAPI
>>> Type: Ethernet
>>> Family:   Standard NIC
>>> # Bound Sockets:  1
>>> TX Queues:1
>>> RX Queues:1
>>> ===
>>> ​
>>> 
>>> Regards,
>>> Marco
>>> 
>>> 
>>> 2018-01-18 10:35 GMT+00:00 Marco Teixeira <ma...@scom.uminho.pt 
>>> <mailto:ma...@scom.uminho.pt>>:
>>> Hi Afredo and anyone "listening",
>>> 
>>> I would like to rule out if pf_ring is the culprit here...
>>> What would be the correct way to have nprobe use libpcap (or AF_PACKET??) 
>>> mode of getting packets from the NIC?
>>> Blacklisting pf_ring module from loading?
>>> 
>>> Thank you
>>> Marco
>>> 
>>> 
>>> 
>>> 
>>> 2018-01-17 22:16 GMT+00:00 Alfredo Cardigliano <cardigli...@ntop.org 
>>> <mailto:cardigli...@ntop.org>>:
>>> “Absolute Stats” is the total / average number of packets/bytes
>>> “Actual Stats” is the current number of packets/bytes (last second)
>>> 
>>> Alfredo
>>> 
>>>> On 17 Jan 2018, at 21:13, Marco Teixeira <ma...@scom.uminho.pt 
>>>> <mailto:ma...@scom.uminho.pt>> wrote:
>>>> 
>>>> Hello list,
>>>> 
>>>> Any PFRING wizard that can offer clues on where to start troubleshooting 
>>>> this variance between "Absolut Stats" vs "Actual Stats"...?
>>>> 
>>>> ​==

Re: [Ntop-misc] About pf_ring licences (was nProbe Pro won't do more then 1Gb/s?)

[Luca Deri]

> On 25 Jan 2018, at 20:29, Marco Teixeira <ma...@scom.uminho.pt> wrote:
> 
> Hi Luca,
> 
> I have the details and was going to generate de license fo 10Gb/s... but I 
> was expecting an option for "Standard NIC" and only see Intel, Myricom and 
> Napatech/DAG/Fiberblaze...
> Wich one should I choose to make for my "Standard NIC" that uses PF_RING in 
> NAPI mode? 
> ​Or maybe there was some misunderstanding here?​
Yes there is some disconnection Marco

You only need a license for ZC drivers, otherwise you will use vanilla drivers. 
So pfcount -i zc:ens2f0 requires a ZC drivers,  pfcount -i ens2f0 does not

Please see https://github.com/ntop/PF_RING/wiki 
<https://github.com/ntop/PF_RING/wiki> for details

Regards Luca
> 
> ​Regards, and thank you for your advice, and time, 
> Marco​
> 
> 
> 
> 
> 2018-01-25 12:05 GMT+00:00 Luca Deri <d...@ntop.org <mailto:d...@ntop.org>>:
> Marco
> our licenses don’t have a cap on speed, but they are per device family. So if 
> these devices as 10G you need a 10G license
> 
> As you’re a university you can mail educat...@ntop.org 
> <mailto:educat...@ntop.org> for free licenses
> 
> Regards Luca
> 
> 
>> On 25 Jan 2018, at 12:52, Marco Teixeira <ma...@scom.uminho.pt 
>> <mailto:ma...@scom.uminho.pt>> wrote:
>> 
>> Hello,
>> 
>> Regarding the output below, does one need a license to be able to run 10Gb/s 
>> speeds (please mind, I'm only talking about pf_ring. nprobe is already 
>> licensed with "enterprise pro")?
>> 
>> 
>> ===
>> [marco@nprobe ~]$ sudo pfcount -L -v1 | grep ens2f
>> NameSystemName  Module  MAC BusID   
>> NumaNodeStatus  License
>> ens2f0  ens2f0  pf_ring D8:D3:85:A0:12:50   :13:00.0-1   
>>Up  NotFound
>> ens2f1  ens2f1  pf_ring D8:D3:85:A0:12:54   :13:00.1-1   
>>DownNotFound
>> [marco@nprobe ~]$ 
>> [marco@nprobe ~]$ cat /proc/net/pf_ring/dev/ens2f0/info 
>> Name: ens2f0
>> Index:6
>> Address:  D8:D3:85:A0:12:50
>> Polling Mode: NAPI
>> Type: Ethernet
>> Family:   Standard NIC
>> # Bound Sockets:  1
>> TX Queues:1
>> RX Queues:1
>> ===
>> ​
>> 
>> Regards,
>> Marco
>> 
>> 
>> 2018-01-18 10:35 GMT+00:00 Marco Teixeira <ma...@scom.uminho.pt 
>> <mailto:ma...@scom.uminho.pt>>:
>> Hi Afredo and anyone "listening",
>> 
>> I would like to rule out if pf_ring is the culprit here...
>> What would be the correct way to have nprobe use libpcap (or AF_PACKET??) 
>> mode of getting packets from the NIC?
>> Blacklisting pf_ring module from loading?
>> 
>> Thank you
>> Marco
>> 
>> 
>> 
>> 
>> 2018-01-17 22:16 GMT+00:00 Alfredo Cardigliano <cardigli...@ntop.org 
>> <mailto:cardigli...@ntop.org>>:
>> “Absolute Stats” is the total / average number of packets/bytes
>> “Actual Stats” is the current number of packets/bytes (last second)
>> 
>> Alfredo
>> 
>>> On 17 Jan 2018, at 21:13, Marco Teixeira <ma...@scom.uminho.pt 
>>> <mailto:ma...@scom.uminho.pt>> wrote:
>>> 
>>> Hello list,
>>> 
>>> Any PFRING wizard that can offer clues on where to start troubleshooting 
>>> this variance between "Absolut Stats" vs "Actual Stats"...?
>>> 
>>> ​===
>>> [marco@nprobe ~]$ sudo pfcount -i ens2f0 
>>> [sudo] password for marco: 
>>> Using PF_RING v.7.0.0
>>> Capturing from ens2f0 [mac: D8:D3:85:A0:12:50][if_index: 5][speed: 
>>> 1Mb/s]
>>> # Device RX channels: 1
>>> # Polling threads:1
>>> Dumping statistics on /proc/net/pf_ring/stats/3096-ens2f0.3
>>> =
>>> Absolute Stats: [136'980 pkts total][0 pkts dropped][0.0% dropped]
>>> [136'980 pkts rcvd][126'559'478 bytes rcvd]
>>> =
>>> 
>>> =
>>> Absolute Stats: [274'202 pkts total][0 pkts dropped][0.0% dropped]
>>> [274'202 pkts rcvd][254'708'653 bytes rcvd][274'163.89 pkt/sec][2'037.38 
>>> Mbit/sec]
>>> =
>>> Actual Stats: [137'222 pkts rcvd][1'000.13 ms][137'202.92 pps][1.03 Gbps]
>>> =
>>> 
>>> =
>>> Absolute Stats: [411'199

Re: [Ntop-misc] About pf_ring licences (was nProbe Pro won't do more then 1Gb/s?)

[Luca Deri]

Marco
our licenses don’t have a cap on speed, but they are per device family. So if 
these devices as 10G you need a 10G license

As you’re a university you can mail educat...@ntop.org 
 for free licenses

Regards Luca

> On 25 Jan 2018, at 12:52, Marco Teixeira  wrote:
> 
> Hello,
> 
> Regarding the output below, does one need a license to be able to run 10Gb/s 
> speeds (please mind, I'm only talking about pf_ring. nprobe is already 
> licensed with "enterprise pro")?
> 
> 
> ===
> [marco@nprobe ~]$ sudo pfcount -L -v1 | grep ens2f
> NameSystemName  Module  MAC BusID   
> NumaNodeStatus  License
> ens2f0  ens2f0  pf_ring D8:D3:85:A0:12:50   :13:00.0-1
>   Up  NotFound
> ens2f1  ens2f1  pf_ring D8:D3:85:A0:12:54   :13:00.1-1
>   DownNotFound
> [marco@nprobe ~]$ 
> [marco@nprobe ~]$ cat /proc/net/pf_ring/dev/ens2f0/info 
> Name: ens2f0
> Index:6
> Address:  D8:D3:85:A0:12:50
> Polling Mode: NAPI
> Type: Ethernet
> Family:   Standard NIC
> # Bound Sockets:  1
> TX Queues:1
> RX Queues:1
> ===
> ​
> 
> Regards,
> Marco
> 
> 
> 2018-01-18 10:35 GMT+00:00 Marco Teixeira  >:
> Hi Afredo and anyone "listening",
> 
> I would like to rule out if pf_ring is the culprit here...
> What would be the correct way to have nprobe use libpcap (or AF_PACKET??) 
> mode of getting packets from the NIC?
> Blacklisting pf_ring module from loading?
> 
> Thank you
> Marco
> 
> 
> 
> 
> 2018-01-17 22:16 GMT+00:00 Alfredo Cardigliano  >:
> “Absolute Stats” is the total / average number of packets/bytes
> “Actual Stats” is the current number of packets/bytes (last second)
> 
> Alfredo
> 
>> On 17 Jan 2018, at 21:13, Marco Teixeira > > wrote:
>> 
>> Hello list,
>> 
>> Any PFRING wizard that can offer clues on where to start troubleshooting 
>> this variance between "Absolut Stats" vs "Actual Stats"...?
>> 
>> ​===
>> [marco@nprobe ~]$ sudo pfcount -i ens2f0 
>> [sudo] password for marco: 
>> Using PF_RING v.7.0.0
>> Capturing from ens2f0 [mac: D8:D3:85:A0:12:50][if_index: 5][speed: 1Mb/s]
>> # Device RX channels: 1
>> # Polling threads:1
>> Dumping statistics on /proc/net/pf_ring/stats/3096-ens2f0.3
>> =
>> Absolute Stats: [136'980 pkts total][0 pkts dropped][0.0% dropped]
>> [136'980 pkts rcvd][126'559'478 bytes rcvd]
>> =
>> 
>> =
>> Absolute Stats: [274'202 pkts total][0 pkts dropped][0.0% dropped]
>> [274'202 pkts rcvd][254'708'653 bytes rcvd][274'163.89 pkt/sec][2'037.38 
>> Mbit/sec]
>> =
>> Actual Stats: [137'222 pkts rcvd][1'000.13 ms][137'202.92 pps][1.03 Gbps]
>> =
>> 
>> =
>> Absolute Stats: [411'199 pkts total][0 pkts dropped][0.0% dropped]
>> [411'199 pkts rcvd][382'383'683 bytes rcvd][205'575.03 pkt/sec][1'529.35 
>> Mbit/sec]
>> =
>> Actual Stats: [136'997 pkts rcvd][1'000.09 ms][136'983.43 pps][1.02 Gbps]
>> =
>> ===
>> 
>> Thankx
>> Marco
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it 
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> 
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe performance and packet drops

[Luca Deri]

David
sorry for the delay. What you can also do is the following
1. Enable RSS let’s say with two queues
2. start
nprobe -i eth1@0,eth2@0 -g 1 ...
nprobe -i eth1@1,eth2@1 -g 2 ...

If this is not enough you can increase the number of RSS queues so that each 
probe has less messages to process

Regards Luca

> On 22 Jan 2018, at 17:41, David Notivol  wrote:
> 
> Hello,
> 
> Sorry for replying to myself.
> 
> Just adding troubleshooting information. We've tested using zbalance to form 
> the virtual interface instead of our application, and we are getting the same 
> results; we keep having drops.
> Thanks.
> 
> Regards,
> David Notivol.
> 
> 2018-01-19 16:59 GMT+01:00 David Notivol  >:
> Hello list,
> 
> I'm testing nProbe listening from two different 10Gb interfaces (using i40e 
> pf_ring's driver). As we need to mix up the information of these two links, 
> we use a custom application (using pf_ring sources) that creates a virtual 
> interface with traffic from the original physical ones.
> 
> The total traffic is about 3Gbps (we expect to have more), but we are seeing 
> around 50-60% packet drops between our application and nProbe. When testing 
> with zcount/pfcount instead of nProbe, we see 0% drops.
> 
> I've made some tuning in the nProbe parameters (hash-size, max-num-flows, 
> idle-timeout, ...), but no significant changes has been noticed.
> 
> Drops are fewer when disabling the export to Kafka and disabling all plugins 
> (GTPv1, GTPv2 and HTTP), although we always have (around 1-2%).
> 
> I'm pasting below my nProbe configuration, and some traffic statistics.
> 
> Do you have any recommendation I could follow to improve this performance?
> Thanks a lot in advance.
> 
> 
> 
> -- System:
> 
> nProbe:   nprobe-8.0.171020-5797.x86_64
> System RAM: 64GB
> System CPU: 12 cores
> System OS:CentOS Linux release 7.4.1708 (Core) 
> Linux Kernel:   3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux
> 
> 
> -- nProbe configuration:
> -n=none
> -i=zc:1@0
> -s=128
> -t=60
> -d=30
> -a=0
> -e=1
> -B=10
> -w=4048000
> -M=1000
> -z=0
> -S=1:1
> -E=0:0
> -g=/var/run/nprobe-zc1-0.pid
> --vlanid-as-iface-idx=none
> -V=9
> -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES 
> %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL 
> %SRC_TOS %SRC_AS %DST_AS %L7_PROTO %L7_PROTO_NAME %SRC_IP_COUNTRY 
> %SRC_IP_CITY %SRC_IP_LONG %SRC_IP_LAT %DST_IP_COUNTRY %DST_IP_CITY 
> %DST_IP_LONG %DST_IP_LAT %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN 
> %DIRECTION %SSL_SERVER_NAME %SRC_AS_MAP %DST_AS_MAP %HTTP_METHOD 
> %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE 
> %UPSTREAM_TUNNEL_ID %UPSTREAM_SESSION_ID %DOWNSTREAM_TUNNEL_ID 
> %DOWNSTREAM_SESSION_ID %UNTUNNELED_PROTOCOL %UNTUNNELED_IPV4_SRC_ADDR 
> %UNTUNNELED_L4_SRC_PORT %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_L4_DST_PORT 
> %GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_C2S_S1U_GTPU_TEID 
> %GTPV2_C2S_S1U_GTPU_IP %GTPV2_S2C_S1U_GTPU_TEID %GTPV2_S5_S8_GTPC_TEID 
> %GTPV2_S2C_S1U_GTPU_IP %GTPV2_C2S_S5_S8_GTPU_TEID %GTPV2_S2C_S5_S8_GTPU_TEID 
> %GTPV2_C2S_S5_S8_GTPU_IP %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI 
> %GTPV2_END_USER_MSISDN %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC 
> %GTPV2_ULI_CELL_TAC %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE 
> %GTPV2_PDN_IP %GTPV2_END_USER_IMEI %GTPV2_C2S_S5_S8_GTPC_IP 
> %GTPV2_S2C_S5_S8_GTPC_IP %GTPV2_C2S_S5_S8_SGW_GTPU_TEID 
> %GTPV2_S2C_S5_S8_SGW_GTPU_TEID %GTPV2_C2S_S5_S8_SGW_GTPU_IP 
> %GTPV2_S2C_S5_S8_SGW_GTPU_IP %GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE 
> %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA 
> %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI 
> %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAT_TYPE 
> %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC 
> %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC 
> %GTPV1_RESPONSE_CAUSE %SRC_FRAGMENTS %DST_FRAGMENTS %CLIENT_NW_LATENCY_MS 
> %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_BYTES 
> %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_BYTES %RETRANSMITTED_OUT_PKTS 
> %OOORDER_IN_PKTS %OOORDER_OUT_PKTS %FLOW_ACTIVE_TIMEOUT 
> %FLOW_INACTIVE_TIMEOUT %MIN_TTL %MAX_TTL %IN_SRC_MAC %OUT_DST_MAC 
> %PACKET_SECTION_OFFSET %FRAME_LENGTH %SRC_TO_DST_MAX_THROUGHPUT 
> %SRC_TO_DST_MIN_THROUGHPUT %SRC_TO_DST_AVG_THROUGHPUT 
> %DST_TO_SRC_MAX_THROUGHPUT %DST_TO_SRC_MIN_THROUGHPUT 
> %DST_TO_SRC_AVG_THROUGHPUT %NUM_PKTS_UP_TO_128_BYTES 
> %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES 
> %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES 
> %NUM_PKTS_OVER_1514_BYTES %LONGEST_FLOW_PKT %SHORTEST_FLOW_PKT 
> %NUM_PKTS_TTL_EQ_1 %NUM_PKTS_TTL_2_5 %NUM_PKTS_TTL_5_32 %NUM_PKTS_TTL_32_64 
> %NUM_PKTS_TTL_64_96 %NUM_PKTS_TTL_96_128 %NUM_PKTS_TTL_128_160 
> 

Re: [Ntop-misc] nProbe Pro won't do more then 1Gb/s?

[Luca Deri]

Marco
how much traffic are you injecting and how (pfsend)? What does the sender say?

Luca


> On 16 Jan 2018, at 17:27, Marco Teixeira  wrote:
> 
> Hi Alfredo,
> 
> I think our emails got crossed... but nevertheless here is ethtool -S 
> output... Shows no drops.
> 
> ===
> [marco@nprobe ~]$ sudo ethtool --statistics ens2f0
> NIC statistics:
>  xmit_called: 0
>  xmit_finished: 0
>  rx_dropped: 0
>  tx_dropped: 0
>  csummed: 0
>  rx_pkts: 544668580
>  lro_pkts: 0
>  rx_bytes: 480924479057
>  tx_bytes: 0
> ===
> 
> Like i said... on my previous email, the problem might be with PFRING ?
> ===
> =
> Absolute Stats: [274'090 pkts total][0 pkts dropped][0.0% dropped]
> [274'090 pkts rcvd][252'023'439 bytes rcvd][274'048.34 pkt/sec][2'015.88 
> Mbit/sec]
> =
> Actual Stats: [137'201 pkts rcvd][1'000.15 ms][137'180.28 pps][0.99 Gbps]
> =
> ===
> 
> Cumprimentos,
> 
> Marco Teixeira
> 
> ---
> Serviços de Comunicações da Universidade do Minho
> Campus de Azurém, 4800-058 Guimarães - Portugal
> Tel.: +351 253510141, Fax: +351 253604021
> ma...@scom.uminho.pt  | 
> http://www.scom.uminho.pt 
> ---
> 
> 
> 2018-01-16 16:21 GMT+00:00 Alfredo Cardigliano  >:
> Hi Marco
> it seems there is no packet loss, did you check the interface stats with 
> ethtool -S to check if pats are getting lost at interface level?
> 
> Best Regards
> Alfredo
> 
>> On 16 Jan 2018, at 17:17, Marco Teixeira > > wrote:
>> 
>> Hello,
>> 
>> Is PF_RING the culprit here ?? How to tweak this?
>> Absolute stats showing around 2Gb/s and Actual stats near 1Gb/s...
>> 
>> ===
>> [marco@nprobe ~]$ sudo pfcount -i ens2f0
>> Using PF_RING v.7.0.0
>> Capturing from ens2f0 [mac: D8:D3:85:A0:12:50][if_index: 5][speed: 1Mb/s]
>> # Device RX channels: 1
>> # Polling threads:1
>> Dumping statistics on /proc/net/pf_ring/stats/2553-ens2f0.2
>> =
>> Absolute Stats: [136'889 pkts total][0 pkts dropped][0.0% dropped]
>> [136'889 pkts rcvd][128'561'396 bytes rcvd]
>> =
>> 
>> =
>> Absolute Stats: [274'090 pkts total][0 pkts dropped][0.0% dropped]
>> [274'090 pkts rcvd][252'023'439 bytes rcvd][274'048.34 pkt/sec][2'015.88 
>> Mbit/sec]
>> =
>> Actual Stats: [137'201 pkts rcvd][1'000.15 ms][137'180.28 pps][0.99 Gbps]
>> =
>> 
>> ===
>> 
>> Cumprimentos,
>> 
>> Marco Teixeira
>> 
>> ---
>> Serviços de Comunicações da Universidade do Minho
>> Campus de Azurém, 4800-058 Guimarães - Portugal
>> Tel.: +351 253510141, Fax: +351 253604021
>> ma...@scom.uminho.pt  | 
>> http://www.scom.uminho.pt 
>> ---
>> 
>> 
>> 2018-01-16 15:52 GMT+00:00 Marco Teixeira > >:
>> Hi list,
>> 
>> Do you know of any limitation (license wise) on the capture speed of nProbe?
>> Can't seem to go above 1Gb/s, but machine still has plenty of CPU available, 
>> and PCIe 10Gb/s NIC...
>> 
>> ===
>> Build OS:  CentOS Linux release 7.4.1708 (Core)
>> GIT rev:   8.2-stable:fe33351b54075fa76a242548fb830e2bdf1d9224:20180112
>> Edition:   nProbe Pro
>> License Type:  Permanent License 
>> === 
>> 
>> ===
>> [marco@nprobe ~]$ more /proc/net/pf_ring/stats/1361-ens2f0.1 
>> Duration: 0:00:33:05:185
>> Bytes:238825829235
>> Packets:  272020616
>> Dropped:  0
>> 
>> [marco@nprobe ~]$ more /proc/net/pf_ring/1361-ens2f0.1 
>> Bound Device(s): ens2f0
>> Active : 1
>> Breed  : Standard
>> Appl. Name : nProbe
>> Socket Mode: RX only
>> Capture Direction  : RX+TX
>> Sampling Rate  : 1
>> IP Defragment  : No
>> BPF Filtering  : Disabled
>> Sw Filt Hash Rules : 0
>> Sw Filt WC Rules   : 0
>> Sw Filt Hash Match : 0
>> Sw Filt Hash Miss  : 0
>> Hw Filt Rules  : 0
>> Poll Pkt Watermark : 8
>> Num Poll Calls : 0
>> Channel Id Mask: 0x
>> VLAN Id: 65535
>> Slot Version   : 16 [7.0.0]
>> Min Num Slots  : 4108
>> Bucket Len : 128
>> Slot Len   : 336 [bucket+header]
>> Tot Memory : 1388544
>> Tot Packets: 276360318
>> Tot Pkt Lost   : 0
>> Tot Insert   

Re: [Ntop-misc] Default values for nProbe settings

[Luca Deri]

)
> 08/Jan/2018 18:09:13 [nprobe.c:3035] Fragment queue length: 0
> 08/Jan/2018 18:09:13 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0 
> flows/0 pkts sent]
> 08/Jan/2018 18:09:13 [nprobe.c:3068] Flow collection:   [collected pkts: 
> 28566][processed flows: 765143]
> 08/Jan/2018 18:09:13 [nprobe.c:3071] Flow drop stats:   [0 bytes/0 pkts][0 
> flows]
> 08/Jan/2018 18:09:13 [nprobe.c:3076] Total flow stats:  [0 bytes/0 pkts][0 
> flows/0 pkts sent]
> 08/Jan/2018 18:09:13 [nprobe.c:3087] Kafka [flows exported=822][msgs 
> sent=822/1.0 flows/msg][send errors=0]
> 08/Jan/2018 18:09:43 [nprobe.c:3201] -
> 08/Jan/2018 18:09:43 [nprobe.c:3202] Average traffic: [0.00 pps][All Traffic 
> 0 b/sec][IP Traffic 0 b/sec][ratio -nan]
> 08/Jan/2018 18:09:43 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec]
> 08/Jan/2018 18:09:43 [nprobe.c:3216] Current flow export rate: [4333.8 
> flows/sec]
> 08/Jan/2018 18:09:43 [nprobe.c:3219] Flow drops: [export queue too 
> long=0][too many flows=0][ELK queue flow drops=0]
> 08/Jan/2018 18:09:43 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %]
> 08/Jan/2018 18:09:43 [nprobe.c:3229] Flow Buckets: 
> [active=96146][allocated=96146][toBeExported=0]
> 08/Jan/2018 18:09:43 [nprobe.c:3235] Kafka [flows exported=130835/4333.8 
> flows/sec][msgs sent=130835/1.0 flows/msg][send errors=0]
> 08/Jan/2018 18:09:43 [nprobe.c:3260] Collector Threads: [50988 pkts@0]
> 08/Jan/2018 18:09:43 [nprobe.c:3052] Processed packets: 0 (max bucket search: 
> 8)
> 08/Jan/2018 18:09:43 [nprobe.c:3035] Fragment queue length: 0
> 08/Jan/2018 18:09:43 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0 
> flows/0 pkts sent]
> 08/Jan/2018 18:09:43 [nprobe.c:3068] Flow collection:   [collected pkts: 
> 50988][processed flows: 1376945]
> 08/Jan/2018 18:09:43 [nprobe.c:3071] Flow drop stats:   [0 bytes/0 pkts][0 
> flows]
> 08/Jan/2018 18:09:43 [nprobe.c:3076] Total flow stats:  [0 bytes/0 pkts][0 
> flows/0 pkts sent]
> 08/Jan/2018 18:09:43 [nprobe.c:3087] Kafka [flows exported=130835][msgs 
> sent=130835/1.0 flows/msg][send errors=0]
> 
> 
> 
> On Mon, Jan 8, 2018 at 1:10 PM, Luca Deri <d...@ntop.org 
> <mailto:d...@ntop.org>> wrote:
> Mark
> the default is 1/1/1/1/1/1 but please note that depending on the template 
> some fields will be set to 0. Please pay attention to the nprobe startup log
> 
> Thanks Luca
> 
>> On 8 Jan 2018, at 19:01, Mark Petronic <markpetro...@gmail.com 
>> <mailto:markpetro...@gmail.com>> wrote:
>> 
>> Some indicate the default in the -h output and some do not. Can some please 
>> tell me the default value for --aggregation in v8.2? Thank you!
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Default values for nProbe settings

[Luca Deri]

Mark
the default is 1/1/1/1/1/1 but please note that depending on the template some 
fields will be set to 0. Please pay attention to the nprobe startup log

Thanks Luca

> On 8 Jan 2018, at 19:01, Mark Petronic  wrote:
> 
> Some indicate the default in the -h output and some do not. Can some please 
> tell me the default value for --aggregation in v8.2? Thank you!
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] General questions and documentation of nprobe internals

[Luca Deri]

Hi Mark,
sorry for the late reply but we;ve been in vacation lately

Please see below

> On 20 Dec 2017, at 13:25, Mark Petronic  wrote:
> 
> I am running with nprobe 8.2 in collector mode. I am currently designing a 
> collection infrastructure so I want to try to understand what nprobe is doing 
> internally as to better understand how data is being processed. I a number of 
> questions in regard to this. I have read the latest version of the user guide 
> PDF but still have some questions. I tried to organize my questions in blocks 
> to hopefully allow for easier commenting on each question. This is fairly 
> long but I figured asking this all together, in context, would be better. 
> Thanks in advance to whoever takes this on - I really appreciate it. :)
> 
> Is there any detailed documentation on what is going on internally with 
> nprobe. In particular, I am using it as a collector to forward UDP netflow v9 
> from our Cisco routers to Kafka. I am particularly interesting in 
> understanding some of these stats and what they "infer" is happening under 
> the hood:
> 
> 19/Dec/2017 13:36:09 [nprobe.c:3202] Average traffic: [0.00 pps][All Traffic 
> 0 b/sec][IP Traffic 0 b/sec][ratio -nan]
> 19/Dec/2017 13:36:09 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec]
> 19/Dec/2017 13:36:09 [nprobe.c:3216] Current flow export rate: [1818.5 
> flows/sec]
> 19/Dec/2017 13:36:09 [nprobe.c:3219] Flow drops: [export queue too 
> long=0][too many flows=0][ELK queue flow drops=0]
> 19/Dec/2017 13:36:09 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %]
> 19/Dec/2017 13:36:09 [nprobe.c:3229] Flow Buckets: 
> [active=92792][allocated=92792][toBeExported=0]
> 19/Dec/2017 13:36:09 [nprobe.c:3235] Kafka [flows exported=366299/1818.5 
> flows/sec][msgs sent=366299/1.0 flows/msg][send errors=0]
> 19/Dec/2017 13:36:09 [nprobe.c:3260] Collector Threads: [167203 pkts@0] 
> 19/Dec/2017 13:36:09 [nprobe.c:3052] Processed packets: 0 (max bucket search: 
> 8)
> 19/Dec/2017 13:36:09 [nprobe.c:3035] Fragment queue length: 0
> 19/Dec/2017 13:36:09 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0 
> flows/0 pkts sent]
> 19/Dec/2017 13:36:09 [nprobe.c:3068] Flow collection:   [collected pkts: 
> 167203][processed flows: 4561802]
> 19/Dec/2017 13:36:09 [nprobe.c:3071] Flow drop stats:   [0 bytes/0 pkts][0 
> flows]
> 19/Dec/2017 13:36:09 [nprobe.c:3076] Total flow stats:  [0 bytes/0 pkts][0 
> flows/0 pkts sent]
> 19/Dec/2017 13:36:09 [nprobe.c:3087] Kafka [flows exported=366299][msgs 
> sent=366299/1.0 flows/msg][send errors=0]
> 
> For these two stats:
> 
> Flow collection:   [collected pkts: 167203][processed flows: 4561802]
> Kafka [flows exported=366299][msgs sent=366299/1.0 flows/msg][send errors=0]
> 
> I am thinking they mean that 167203 UDP packets where received from routers 
> comprising a total of 4561802 individual flow records. However, is see only 
> 366299 flows exported to Kafka. So, am I correct in assuming that nprobe is 
> doing some internal aggregation of flow records that is essentially squashing 
> the 4561802 received flow records into 366299 aggregates?
Yes your assumption is correct. If you want to avoid that please use 
--disable-cache

> 
> A follow on question to this, then, is related to:
> 
> Flow Buckets: [active=92792][allocated=92792][toBeExported=0]
> 
> What are these and how are they utilized? Again, I am assuming these are hash 
> buckets used for internal aggregation per the user's guide. I have seen 
> warning indicating that the allotment of these buckets are too small and to 
> expect drops. So, my guess is, based on flows/sec ingested, these have to be 
> sized appropriately to support the flow volume. Is that a correct assumption? 

When you see these messages we need to investigate. This happens when too many 
flows fall into the same hash bucket for instance. Enlarging the hash (-w) can 
help if too small compared to the number of collected flows, but for replying 
more in detail I need some extra context
> 
> I also notice that, when I start up nprobe in collector mode publishing to 
> Kafka, it takes about 30 or more seconds before any flows actually are 
> published to Kafka. This leads me to believe internal aggregations are 
> occurring that are delaying publishing of data. If I crank up the --verbose 
> to 2, I can see UDP packets being processed and then, after some time, I 
> start to see log messages indicating flows are being exported to Kafka. It is 
> not as much the latency issue I am concerned with here but rather just 
> understanding what is happening so that I can properly monitor and 
> configure/size the system.
Yes correct. By default flows are aggregated in the cache and as you write 
below the minimum timeout is 30 sec
> 
> Do these parameters impact the utilization of the flow buckets in collector 
> mode or just when running in sniffer mode? I ask because, I know the routers 
> are already doing aggregations meaning, accumulation counts 

Re: [Ntop-misc] Few general question on using nprobe as a collector with Kafka

[Luca Deri]

Hi Mark
please see below, but first of all please move to 8.2 as we have fixes
many issues and many improvements in particular when collecting flows
https://www.ntop.org/category/nprobe/

On 12/12/2017 06:09 PM, Mark Petronic wrote:
> I am fairly new to nprobe and have been experimenting with the many
> commandline options. I have a few general questions that I would
> appreciate any clarification. 
>
> nprobe -v
>
> Welcome to nProbe v.8.0.171020 (r5797) for x86_64-unknown-linux-gnu
> with native PF_RING acceleration.
> Copyright 2002-17 ntop.org 
>
> Build OS:      CentOS Linux release 7.3.1611 (Core)
> SystemID:      68A2B43E76056A7E
> GIT rev:     
>  8.0-stable:478c52c6ce70feaf6c65fe4806be05f75fe0e196:20171020
> License:       Invalid nProbe license (/etc/nprobe.license) [Missing
> license file]
>
>
> Q1. When running on a multi-core host, will nprobe utilize all cores.
> Somewhere, I thought I saw something about it being single threaded
> but now cannot find that reference. This question goes to sizing my
> HW. I am seeing ~5% CPU load for one router's flow (about 2500 flow
> records/sec). I will ultimately need more than 20x this volume so I
> need to deploy N hosts eventually in full production setup. I just
> want to know if there are any settings needed to enable nprobe to
> fully utilize all cores on a given host.

nprobe will use one core because if you use RSS you can spawn an
instance per core. From our tests in collection mode we should be able
to handle ~20k flows/core
>
> Q2. I am running with this configuration:
>
> [root@vmwdnacollector01 ~]# cat /etc/nprobe/nprobe.conf 
> --interface=none
> --collector=none
> --collector-port=2055
> --verbose=1 
> --flow-version=9 
> --hash-size=262144
> --kafka="kafka01:9092;netflow-raw;1"
> --dump-stats=/var/log/nprobe/stats.txt
> --event-log=/var/log/nprobe/events.txt
> -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT
> %IPV4_SRC_MASK %IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES
> %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS
> %PROTOCOL %SRC_TOS %DIRECTION %EXPORTER_IPV4_ADDRESS"
>
> I am collecting netflow V9 records from a Cisco router. I was sort of
> expecting that the record would include the IP address of the router
> because I need that to know where the data came from for upstream
> enrichment.

> I have nprobe publishing to Kafka. But, looking at the raw flows
> coming from the router, there is no field that identifies the router
> IP. So, I experimented and added a -T  definition that
> matches the actual fields coming from the router. Then I added
> the %EXPORTER_IPV4_ADDRESS field (which is NOT in the raw record from
> the router) and voila, the IP address of the router shows up in that
> field. So, I assume that nprobe is simply adding the source IP address
> of each incoming flow record into that field, as well as mapping each
> field in the incoming flow record into the matching field in my
> defined template - sort of "cherry picking" the fields out of the
> source record and packing them into my template.
>
> So, my question on this point is, am I doing this correctly with
> defining my own template? Seems like the only way I can figure it out.
This is correct. But in >= 8.2 this is done automatically when using
ZMQ, but we'll also extend to kafka as of this email
>
> Q3. It appears, for the mode I am operating in, that no license is
> required to allow this to work. When I run in the mode where nprobe
> sniffs packets from my local interface, it will only produce 25K flows
> then stops if there is no license. However, in collector mode, where
> it just receives flows from a router and forwards them as JSON to
> Kafka, it runs for millions of flows. So, question here is, do I need
> a license for this sort of use case?
>
yes you need a license

> Q4. The Kafka producer has a boat load of configuration options but
> nprobe only exposes a couple basic options (topic, acks, brokers). Is
> that it or is there some way to provide additional configuration
> information to the embedded producer? For example, to properly
> aggregate data flows, I would like to partition the topic on the
> IPV4_SRC_ADDR. I am running in a multi-tenant environment where each
> tenant can have overlapping private IP addresses that we see in the
> flows. So, I need to aggregate the flows by TENANT_ID + IPV4_SRC_ADDR,
> for example. I see no way to configure this with nprobe + kafka mode.

This is not possible, but Simone is the kafka expert: If you can agree
on what type of extensions are needed, we'll implement them
>
> Q5. Is there any way to bind nprobe to specific interface when used as
> a collector in my use case? Meaning, I might need to run multiple
> instances on a single host but I want to be able to configure routers
> to direct their flow records to a specific IP address so that I can
> load-balance the flows over N instances of nprobe running on a single
> host. I cannot find any configuration 

Re: [Ntop-misc] Query regarding ZC license

[Luca Deri]

> On 11 Dec 2017, at 11:38, Chandrika Gautam <chandrika.iitd.r...@gmail.com> 
> wrote:
> 
> Hi all, 
> 
> It would be really helpful if you guys can reply since I need to communicate 
> same to our customer.
> 
> Regards,
> 
> On Mon, Dec 11, 2017 at 1:59 PM, Chandrika Gautam 
> <chandrika.iitd.r...@gmail.com <mailto:chandrika.iitd.r...@gmail.com>> wrote:
> 
> I completely understand the information over the link and below is the 
> understanding. Please verify this statement!
> 
> Let say If I buy a license in December 11, 2017 for PF_RING-ZC 6.2. So I can 
> upgrade from 6.2 to {any version released} until December 10, 2018 using the 
> same license.

Sort of. The count starts from the day you generate the license, not when ou 
buy it

Luca
> 
> Regards,
> 
> On Mon, Dec 11, 2017 at 1:19 PM, Luca Deri <d...@ntop.org 
> <mailto:d...@ntop.org>> wrote:
> Hi Chandrika,
> it is all explained here 
> http://www.ntop.org/support/faq/are-licenses-permanent-what-about-maintenance/
>  
> <http://www.ntop.org/support/faq/are-licenses-permanent-what-about-maintenance/>
> 
> In essence we enforce only the date you generate the license, so you can 
> upgrade for one year since license generation or if you so not upgrade 
> licenses are permanent
> 
> Regards Luca
> 
> > On 11 Dec 2017, at 06:06, Chandrika Gautam <chandrika.iitd.r...@gmail.com 
> > <mailto:chandrika.iitd.r...@gmail.com>> wrote:
> >
> > Hi all,
> >
> > I have a quick query and will really appreciate a quick response. Thanks in 
> > advance!
> >
> > If I am correct, I need to provide Hardware address of the ethernet port 
> > and PF_RING release label while configuring the PF_RING ZC license.
> > Let say if we are using PF_RING version 6.2.0 as of today and plan to 
> > upgrade to PF_RING 7.0 in coming 2-3 months, then we can use that  without 
> > any issue?
> > Since all future releases for next one year is included!
> >
> >
> > Regards,
> > Chandrika
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Query regarding ZC license

[Luca Deri]

Hi Chandrika,
it is all explained here 
http://www.ntop.org/support/faq/are-licenses-permanent-what-about-maintenance/

In essence we enforce only the date you generate the license, so you can 
upgrade for one year since license generation or if you so not upgrade licenses 
are permanent

Regards Luca

> On 11 Dec 2017, at 06:06, Chandrika Gautam  
> wrote:
> 
> Hi all, 
> 
> I have a quick query and will really appreciate a quick response. Thanks in 
> advance!
> 
> If I am correct, I need to provide Hardware address of the ethernet port and 
> PF_RING release label while configuring the PF_RING ZC license.
> Let say if we are using PF_RING version 6.2.0 as of today and plan to upgrade 
> to PF_RING 7.0 in coming 2-3 months, then we can use that  without any issue?
> Since all future releases for next one year is included!
> 
> 
> Regards,
> Chandrika
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] hw dimensioning for nprobe and ntopng

[Luca Deri]

Spiros
for 4 Gbit of traffic any modern server will work. You can use E3 processors up 
for nProbe, but if you add ELK you’re on an unchartered land as when indexes 
becomes large expect slow-downs and thus you probably need to spawn more ELK 
instances

Regards Luca

> On 8 Oct 2017, at 04:11, Spiros Papageorgiou  wrote:
> 
> Hi all,
> 
> I'll buy some new hardware for virtualization and I plan to run nprobe on it, 
> among other things. The plan is to use nprobe to send NF records to an ntopng 
> instance and also to an elasticsearch.
> 
> Does anyone have any core/Ghz recommendation for 4Gbps of traffic (with 
> pfringZC)?
> 
> Do you have a recommendation for the ntopng instance?
> 
> 
> Thank you,
> 
> Spiros
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe

[Luca Deri]

Eladio
I need to snd you a binary nProbe with debug symbols to figure out the problem.

Please open an issue on https://github.com/ntop/nProbe/issues 
 and specify the command line you have 
used to start nProbe as well add the output of nprobe --version

Regards Luca
> On 28 Sep 2017, at 01:38, Eladio Pérez - Dpto. Redes  
> wrote:
> 
> Hello,
> 
> we are testing nProbe in a system that receives cloned traffic from two 10G 
> interfaces connected to a Cisco router. After aprox. 1 hour of continuous 
> capturing, the nprobe daemon kills itself and send this message:
> 
> 27/Sep/2017 16:10:53 [nprobe.c:3022] Total flow stats:  [1016513907 
> bytes/522619508 pkts][41069671 flows/3495292 pkts sent]
> *** Error in `nprobe': free(): invalid next size (normal): 0x7f03cabd39d0 
> ***
> 
> ¿Anyone can help us in understanding why this happens?
> 
> Un saludo,
> 
> Eladio Pérez Nieto | Dinahosting, S.L.
> Dpto. Gestión de Red
> Email: epe...@dinahosting.com
> ---
> Voz:  900 854 000  | International: +34 981 040 200
> ---
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Getting the transmitted bytes src/dst in a flow

[Luca Deri]

Hi Gouveia
are you generating flows with nprobe or collecting them? Please explain the 
context

Thanks Luca

> On 19 Sep 2017, at 20:01, Henrique Nascimento Gouveia  
> wrote:
> 
> Hi,
> 
> I am having a hard time trying to realize how I could get the source and 
> destination traffic in bytes of a flow.
> I tried to use these elements of structs but with little success:
> 
> - packet_direction_counter
> - byte_counter
> 
> How can I get the amount of bytes coming from the source and coming from the 
> destination in a flow?
> 
> 
> Thanks!
> 
> Gouveia
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe sctp support

[Luca Deri]

You’re welcome

Luca

> On 3 Sep 2017, at 11:09, Felix Erlacher <felix.erlac...@uibk.ac.at> wrote:
> 
> Hello Luca,
> Downloaded and tested it: flows are arriving at my sctp collector :-)
> thank you very much!
> 
> regards
> Felix
> 
> 
> On 03/09/17 10:45, Luca Deri wrote:
>> Felix,
>> we have modified the Ubuntu 16 build. Please update the packages and let
>> me know
>> 
>> Regards Luca
>> 
>> 
>>> On 1 Sep 2017, at 18:09, Felix Erlacher <felix.erlac...@uibk.ac.at
>>> <mailto:felix.erlac...@uibk.ac.at>> wrote:
>>> 
>>> Hi all,
>>> 
>>> I am having trouble getting nprobe to export ipfix flows over SCTP. TCP
>>> and UDP work flawlessly but everytime I try with SCTP it says:
>>> 
>>> "ERROR: SCTP isn't supported on your system. Using UDP."
>>> 
>>> I am using "nProbe Pro v.8.1.170821 ($Revision: 5868 $) for
>>> x86_64-pc-linux-gnu" on Ubuntu 16.04.
>>> checksctp says "SCTP supported" and I have another network probe
>>> (vermont) that exports ipfix over sctp without complains.
>>> The manual says nprobe has to be compiled with sctp support, but I can
>>> only find binary packages for nprobe...
>>> 
>>> Any hints?
>>> 
>>> thanks and greets
>>> 
>>> Felix
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe sctp support

[Luca Deri]

Felix,
we have modified the Ubuntu 16 build. Please update the packages and let me know

Regards Luca


> On 1 Sep 2017, at 18:09, Felix Erlacher  wrote:
> 
> Hi all,
> 
> I am having trouble getting nprobe to export ipfix flows over SCTP. TCP
> and UDP work flawlessly but everytime I try with SCTP it says:
> 
> "ERROR: SCTP isn't supported on your system. Using UDP."
> 
> I am using "nProbe Pro v.8.1.170821 ($Revision: 5868 $) for
> x86_64-pc-linux-gnu" on Ubuntu 16.04.
> checksctp says "SCTP supported" and I have another network probe
> (vermont) that exports ipfix over sctp without complains.
> The manual says nprobe has to be compiled with sctp support, but I can
> only find binary packages for nprobe...
> 
> Any hints?
> 
> thanks and greets
> 
> Felix
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe biflows

[Luca Deri]

Felix
please see (-h) but in general the option below

[--biflows-export-policy|-N]   | Bi-directional flows export policy:
| 1 - export bi-directional flows only
| 2 - export mono-directional flows only

allows you to export only biflows or uniflows. THis is not what you want to do 
(export bi-directional flows). To do so please
1. in the -T use at lest the basic information elements such as protocols and 
bytes. nprobe should have reported this in the startup log
2. you need to use both IN and OUT as in the example below

 nprobe -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS 
%OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS 
%PROTOCOL ..."

Regards Luca

@Simone: please fix the nProbe manual


> On 23 Aug 2017, at 11:27, Felix Erlacher  wrote:
> 
> Dear ntop team,
> 
> I am using nprobe pro (8.1.170821) with the http plugin.
> The nprobe manual (8.1) states that to force flows to be bidirectional
> one should use the  "--bi-directional" switch.
> If I run:
> 
> sudo nprobe -n tcp://10.0.0.2:4740 -i /mynetworktrace.pcap
> --bi-directional -V10 -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %PROTOCOL
> %L4_SRC_PORT %L4_DST_PORT %FIRST_SWITCHED %LAST_SWITCHED %HTTP_URL
> %HTTP_METHOD"
> 
> it works fine but no IPFIX biflows are exported and the output says
> "nprobe: unrecognized option '--bi-directional'".
> I also tried adding the "--biflows-export-policy 2" switch to the above
> command, but still the above "unrecognized option" error appears.
> 
> Am I missing something obvious?
> Are there any other options to export IPFIX biflows?
> 
> thanks and regards
> 
> Felix
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] How to compile nDPI with JSON-C enabled?

[Luca Deri]

Marcel
please install it form source or the development package

Luca

> On 18 Jul 2017, at 17:12, Lüthi Marcel FUB  wrote:
> 
> Dear list
>  
> I downloaded the latest (development) version of nDPI from 
> https://github.com/ntop/nDPI .
> Now I can’t find out to compile it with JSON-C support enabled:
>  
> [user@server nDPI-dev]$ ./example/ndpiReader -i /home/user/test.pcapng -j 
> ./test.json
> WARNING: this copy of ndpiReader has been compiled without JSON-C: json 
> export disabled
>  
> What I do:
> ./autogen.sh
> ./configure --enable-json-c
> ./make
>  
> I have JSON-C installed:
> [user@server nDPI-dev]$ sudo yum list installed | grep json-c
> Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
> json-c.x86_64  0.11-4.el7_0
> @anaconda
>  
> Operating system:  CentOS 7.3
>  
> What am I doing wrong?
> Thank you very much for any help.
>  
> Regards,
> Marcel
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe not capturing traffic

[Luca Deri]

Hi Shahzada,
can you please submit an issue on ntopng’s github page so we can track this 
issue?

Luca

> On 1 Apr 2017, at 07:24, Shahzada Khurram  wrote:
> 
> hi Simone,
>  thanks for reply please find below detail configuration, all 
> configuration done by nbox web Gui mode. .
> 1. independently ntopng working fine and traffic capturing working fine  
> 2. When we configure nprobe ( probe) with ntopng ( collector ) its not 
> working.
> 
> (ntopng log )
> 
> 31/Mar/2017 22:17:32  Scripts/HTML pages directory: /usr/share/ntopng
> 31/Mar/2017 22:17:32  Welcome to ntopng x86_64 v.2.4.170215 - (C) 1998-2016 
> ntop.org 
> 31/Mar/2017 22:17:32  Built on Ubuntu 16.04.1 LTS
> 31/Mar/2017 22:17:32  Started periodic activities loop...
> 31/Mar/2017 22:17:32  Dumping alerts into syslog
> 31/Mar/2017 22:17:32  [LICENSE] ntopng systemId: 3BD34B1A00660F0E
> 31/Mar/2017 22:17:32  [LICENSE] ntopng license: 50FB086D8E0007E9944AAF3C6
> 31/Mar/2017 22:17:32  [LICENSE] Maintenance is available until Thu Mar 29 
> 01:48:45 2018 [362 days left]
> 31/Mar/2017 22:17:32  Started packet polling on interface 
> tcp://127.0.0.1:5556  [id: 4]...
> 31/Mar/2017 22:17:32  Collecting flows on tcp://127.0.0.1:5556 
>  [nprobe->ntopng]
> 
> (nprobe log)
> 
> 31/Mar/2017 22:19:07  Each flow is 63 bytes long
> 31/Mar/2017 22:19:07  The # packets per flow has been set to 22
> 31/Mar/2017 22:19:07  Non IPv4/v6 traffic is discarded according to the 
> template
> 31/Mar/2017 22:19:07  GeoIP: loaded AS config file 
> /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
> 31/Mar/2017 22:19:07  GeoIP: loaded AS IPv6 config file 
> /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
> 31/Mar/2017 22:19:07  WARNING: Your template ignores IP addresses: your 
> collector might ignore these flows.
> 31/Mar/2017 22:19:07  Using packet capture length 128
> 31/Mar/2017 22:19:07  Capturing packets from interface eth1 [snaplen: 128 
> bytes]
> 31/Mar/2017 22:19:07  nProbe changed user to 'nobody'
> 31/Mar/2017 22:19:07  nProbe started successfully
> 
> (nprobe-eth1-conf)
> 
> -n=tcp://127.0.0.1:5556 
> -i=eth1
> -s=128
> -t=60
> -d=60
> -a=0
> -e=1
> -B=10
> -w=128000
> -z=0
> -S=1:1
> -E=0:0
> -g=/var/run/nprobe-eth1.pid
> -p=1/0/0/0/0/1
> --zmq-probe-mode
> --vlanid-as-iface-idx=none
> -T=%IN_BYTES %IN_PKTS %PROTOCOL %SRC_TOS %TCP_FLAGS %L4_SRC_PORT 
> %IPV4_SRC_ADDR %INPUT_SNMP %L4_DST_PORT %IPV4_DST_ADDR %OUTPUT_SNMP 
> %LAST_SWITCHED %FIRST_SWITCHED
> -V=9
> --dump-stats=/var/log/nprobe/eth1-0_flows_stats.txt
> 
> 
> The scenario is   
> 
> eth1 > nprobe (probe-Packet capturing on eth1)  -->ntopng ( 
> collector)( all configuration on single machine)
> 
> Problem: nprobe not capturing traffic.
> 
> thanks in advance.. if you need further information letme know
> 
> 
> khurram
> 
> 
> 
> On Fri, Mar 31, 2017 at 3:51 PM, Simone Mainardi  > wrote:
> Khurram
> 
> Can you please post configurations used in both setups?
> 
> On Fri, Mar 31, 2017 at 8:46 AM, Shahzada Khurram  > wrote:
> Hi,
> I have installed both nprobe and ntopng at ubuntu 16.04. i want to 
> capture traffic on the same server on eth1 for research experimental purpose. 
> but the problem is when i run ntopng as interdependent its working fine and 
> capturing the packet. but when i run ntopng as collector with nprobe. nprobe 
> not capturing traffic. is there any special setting for nprobe. please help 
> me for this regard.
> 
> -- 
> Thanks & Regards,
> 
>  Khurram  
> 
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
> 
> 
> 
> -- 
> 
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Nprobe disable collector hostname resolution

[Luca Deri]

Morgan
can you please make an example?

Luca

> On 10 Mar 2017, at 04:09, Morgan Yang  wrote:
> 
> Hi All:
> 
> I'm using Nprobe 7.1 and it throws an error when the collector IP can not be 
> resolved via nslookup. Is it possible to disable hostname resolution for 
> collector? Maybe this is resolved in a newer version of nprobe already?
> 
> Much Thanks
> Morgan Yang
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Maximum of collectors for 1 session of nprobe

[Luca Deri]

Morgan
limit increased as requested. New build is in progress

Regards Luca

On 02/22/2017 04:12 AM, Morgan Yang wrote:
> Hi All:
>
> I see to be hitting a limit of 8 collectors per session of nprobe (I'm
> hoping for 16). I'm waiting for another person on my team to get the
> license, is that a limit that is dependent on the license?
>
>
> Much Thanks
> Morgan
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe: export IP in decimal format

[Luca Deri]

Please file an enhancement request on https://github.com/ntop/nProbe/issues 


Luca
> On 16 Feb 2017, at 17:51, Сяргей Башлыкевіч  wrote:
> 
> Hi,
> 
> is there any possibility to export IPs to file in decimal (not dotted) format?
> I want to get 2130706433, not 127.0.0.1.
> This is my nprobe's export params:
> --dump-path /opt/exports
> --collector none
> --dump-frequency 30
> --dump-format t 
> --csv-separator ;
> --flow-templ "%FLOW_START_SEC %FLOW_END_SEC %IPV4_SRC_ADDR %L4_SRC_PORT 
> %IPV4_DST_ADDR %L4_DST_PORT %IN_BYTES %OUT_BYTES"
> 
> Regards
> Serge
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] NTOPNG No Application Breakdown for local interface.

[Luca Deri]

Iain,
can you please file a bug on github (menu home "Report Issue") and
attach the complete URL as well a screendump?

Thanks Luca

On 12/13/2016 11:49 AM, Iain Bowker wrote:
> Hi,
>
> I've got a freshly installed linux NTOPNG (fully licensed) server with
> a locally configured sensor interface (Family PF_RING) and a second
> interface (Family zmq) from a remote server running nProbe.
>
> When I generate a report, I get no graphical representation of
> Applications or Application Breakdown for the local interface.  I do
> for the remote one however.  All the stats etc are available for both.
>
> Any thoughts?
>
> Thanks
>
> -- 
> Regards
> Dr B
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe proxy mode - not working with templates

[Luca Deri]

Troy,
the template you have used lacks core fields such as time, bytes and packets. 
This is the problem. Please add them to it

Regards Luca

> On 8 Dec 2016, at 04:10, Troy Jordan  wrote:
> 
> All,
> 
> Adding a -T template argument appears to break my nprobe in proxy mode.
> The setup is:
> 
>[nprobe-probe-mode1] --->   [nprobe-proxy-mode] -->  
> final_netflow_collector
> 
>[nprobe-probe-mode2] --^
> 
> When a template argument is added, such as: -T "%IPV4_SRC_ADDR
> %IPV4_DST_ADDR %PROTOCOL %L4_SRC_PORT %L4_DST_PORT", the proxy still
> receives netflow records, but doesn't pass them on to the final collector.
> 
> Any suggestions for troubleshooting this?
> 
> 
> nprobe commands used:
> 
> netflow generation w/o  template argument
> 
>nprobe -i myri:A1R1P0 -b 1 -n 127.0.0.1:3000
> 
> netfllow generation with template argument
> 
>nprobe -i myri:A1R1P0 -b 1 -n 127.0.0.1:3000 -T "%IPV4_SRC_ADDR
> %IPV4_DST_ADDR %PROTOCOL %L4_SRC_PORT %L4_DST_PORT "
> 
> nprobe proxy (unchanged)
> 
>nprobe -S 1:1 -i none --collector-port 3000 -n 10.1.1.1: -b 1 -V 9
> 
> 
> Thanks.
> 
> 
> - Troy
> 
> 
> 
> -- 
> 
> 
> Troy Jordan 
>   t r o y j @ m a i n e . e d u
>  GIAC GCIH,GCIA
> 
>Network Systems Security Analyst
> Information Technology Security Office
>University of Maine System
> 
> 233 Science Building   | voice: 207.561.3590
> Portland, ME 04103 | fax:   509.351.3650
> 
> 
> 
> "As you all know, Security Is Mortals chiefest Enemy"
> William Shakespeare, Macbeth
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] NPROBE CENTO Out of bounday error

[Luca Deri]

Hi Jesse,

> On 22 Nov 2016, at 20:24, Jesse Alexander <jes...@datafoundry.com> wrote:
> 
> Thank you Luca.  I ran the update and it seems to run without the error now.
> 
> It is odd though, since the upgrade the front end GUI now only shows traffic 
> on the dashboard for half of the interfaces.  However, when I look at our 
> graphs for the nbox host, I see normal traffic on all of the interfaces.  It 
> appears to be cosmetic, but just an observation I wish to share.
No it was not cosmetic, it was an error. Thanks for reporting it

> 
> Also, one quick question.  When we are exporting using Netflow v9, are we 
> also exporting IPv6 traffic?
Yes I do
> 
> Kind regards,
> 
Regards Luca
> Jesse
> 
> -Original Message-
> From: ntop-misc-boun...@listgateway.unipi.it 
> [mailto:ntop-misc-boun...@listgateway.unipi.it] On Behalf Of Luca Deri
> Sent: Tuesday, November 22, 2016 4:22 AM
> To: ntop-misc@listgateway.unipi.it
> Subject: Re: [Ntop-misc] NPROBE CENTO Out of bounday error
> 
> Hi Jesse,
> I have inspected the code, and I believe I have found a case where the
> bug you reported can be reproduced. I have fixed it and packages are
> being rebuilt. Please try again later today and let me know if the issue
> is gone.
> 
> Thanks Luca
> 
> On 11/21/2016 09:27 PM, Jesse Alexander wrote:
>> Hello,
>> 
>> Beginning today, we are receiving an error when running Cento that I haven't 
>> seen before.  I am very new to running Cento, so I suspect I am doing 
>> something wrong and hoping for some assistance.
>> 
>> I am running Cento ZC on eight 10G Myricom ports to export netflow v9.  Not 
>> long after I start running it, I see errors like the following, then it 
>> stops running:
>> 
>> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
>> [3493/1470][18952/22 flows]
>> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
>> [3557/1470][18953/22 flows]
>> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
>> [3621/1470][18954/22 flows]
>> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
>> [3685/1470][18955/22 flows]
>> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
>> [3749/1470][18956/22 flows]
>> 
>> I am running the following:
>> sudo cento -i myri1-1,myri1-2 -i myri2-1,myri2-2 -i myri3-1,myri3-2 -i 
>> myri4-1,myri4-2 -g 4,6,8,9,10,11,12,13,14,15,16,18 -G 2,3,7,17,19,20 --v9 
>> 10.10.10.2:2055
>> 
>> I know the Myricom naming convention is a bit different than default, which 
>> was intentional, but until today it has run great.  Only change today is I 
>> added two additional interfaces to export.  I am unable to find any online 
>> reference to CPP errors for Netflow v9 to point me in the right direction.
>> 
>> Thank you in advance,
>> 
>> Jesse
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] NPROBE CENTO Out of bounday error

[Luca Deri]

Jesse
what cento version are you using? This bug should have been solved a
while ago. I have tested the latest devel version and it works for me.
Please let me know.

Thanks Luca

On 11/21/2016 09:27 PM, Jesse Alexander wrote:
> Hello,
>
> Beginning today, we are receiving an error when running Cento that I haven't 
> seen before.  I am very new to running Cento, so I suspect I am doing 
> something wrong and hoping for some assistance.
>
> I am running Cento ZC on eight 10G Myricom ports to export netflow v9.  Not 
> long after I start running it, I see errors like the following, then it stops 
> running:
>
> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
> [3493/1470][18952/22 flows]
> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
> [3557/1470][18953/22 flows]
> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
> [3621/1470][18954/22 flows]
> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
> [3685/1470][18955/22 flows]
> 21/Nov/2016 02:26:30 [NetflowExporterV9.cpp:172] ERROR: *** Out of bounday 
> [3749/1470][18956/22 flows]
>
> I am running the following:
> sudo cento -i myri1-1,myri1-2 -i myri2-1,myri2-2 -i myri3-1,myri3-2 -i 
> myri4-1,myri4-2 -g 4,6,8,9,10,11,12,13,14,15,16,18 -G 2,3,7,17,19,20 --v9 
> 10.10.10.2:2055
>
> I know the Myricom naming convention is a bit different than default, which 
> was intentional, but until today it has run great.  Only change today is I 
> added two additional interfaces to export.  I am unable to find any online 
> reference to CPP errors for Netflow v9 to point me in the right direction.
>
> Thank you in advance,
>
> Jesse
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Ntopng chart time axis formatting problems

[Luca Deri]

No it should be the same (I assumed that you were using the community
edition)

please file a bug on github and assign it to Emanuele who is taking care
of these issues

luca

On 11/15/2016 10:19 AM, Peter Shute wrote:
> I'm using Pro Small Business Edition. Is that different to the Pro version 
> you're referring to?
>
> Peter Shute
>
> Sent from my iPad
>
>> On 15 Nov. 2016, at 6:55 pm, Luca Deri <d...@ntop.org> wrote:
>>
>> Peter
>> in the pro version we use a different library where the problem you
>> reported doesn't appear
>>
>> If you have a patch to share, please send us a pull request and we'll
>> consider its inclusion in ntopng
>>
>> Regards Luca
>>
>>> On 11/15/2016 01:33 AM, Peter Shute wrote:
>>> Charts are displayed in several places in ntopng's web interface. I find 
>>> these difficult to interpret beyond a one day date range because of several 
>>> problems:
>>> - It doesn't display the dates on the time axis.
>>> - The times are relative to the current time, so they aren't whole hours.
>>> - The chart appears to always be divided into the same number of segments, 
>>> so the interval between segments isn't a whole number of hours.
>>> - A cosmetic issue, but if I scale the display (in Firefox), at some scales 
>>> some of the divisions disappear.
>>>
>>> E.g I'm looking at a 1 week chart now, and the values on the axis are 
>>> 11:19:00, 21:13:20, 11:06:40, etc. It makes it hard to even tell where each 
>>> day begins.
>>>
>>> Looking at the webpage source, I found this function:
>>> function getTickFormat(diff_epoch) {
>>>   var tickFormat;
>>>
>>>   if(diff_epoch < 86400) {
>>>  tickFormat = "%H:%M:%S";
>>>   } else if(diff_epoch < 2*86400) {
>>>  tickFormat = "%b %e, %H:%M:%S";
>>>   } else {
>>>  tickFormat = "%b %e";
>>>   }
>>>
>>>   return(tickFormat);
>>> }
>>>
>>> That looks to me like it's supposed to display the date and time if the 
>>> range is over a day, and just the date if it's over 2 days. But the only 
>>> use of that function in the page source is:
>>> var tickFormat = getTickFormat(0);
>>>
>>> Is this variable scale format a feature that was never fully implemented?
>>>
>>> Peter Shute
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Ntopng chart time axis formatting problems

[Luca Deri]

Peter
in the pro version we use a different library where the problem you
reported doesn't appear

If you have a patch to share, please send us a pull request and we'll
consider its inclusion in ntopng

Regards Luca

On 11/15/2016 01:33 AM, Peter Shute wrote:
> Charts are displayed in several places in ntopng's web interface. I find 
> these difficult to interpret beyond a one day date range because of several 
> problems:
> - It doesn't display the dates on the time axis.
> - The times are relative to the current time, so they aren't whole hours.
> - The chart appears to always be divided into the same number of segments, so 
> the interval between segments isn't a whole number of hours.
> - A cosmetic issue, but if I scale the display (in Firefox), at some scales 
> some of the divisions disappear.
>
> E.g I'm looking at a 1 week chart now, and the values on the axis are 
> 11:19:00, 21:13:20, 11:06:40, etc. It makes it hard to even tell where each 
> day begins.
>
> Looking at the webpage source, I found this function:
> function getTickFormat(diff_epoch) {
>var tickFormat;
>
>if(diff_epoch < 86400) {
>   tickFormat = "%H:%M:%S";
>} else if(diff_epoch < 2*86400) {
>   tickFormat = "%b %e, %H:%M:%S";
>} else {
>   tickFormat = "%b %e";
>}
>
>return(tickFormat);
> }
>
> That looks to me like it's supposed to display the date and time if the range 
> is over a day, and just the date if it's over 2 days. But the only use of 
> that function in the page source is:
> var tickFormat = getTickFormat(0);
>
> Is this variable scale format a feature that was never fully implemented?
>
> Peter Shute
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc


___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64

[Luca Deri]

Derek,
this problem is odd because we run continue tests with docker (see 
https://github.com/ntop/packager) and we have never seen this problem.

I have now analysed the use on centos 6.8 and this is what I have

deri@centos6 205> ldd n2disk
linux-vdso.so.1 =>  (0x7fff75ddd000)
librt.so.1 => /lib64/librt.so.1 (0x0030e220)
libm.so.6 => /lib64/libm.so.6 (0x0030e1e0)
libdl.so.2 => /lib64/libdl.so.2 (0x0030e1a0)
libnuma.so.1 => /usr/lib64/libnuma.so.1 (0x7f05a2979000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0030e160)
libc.so.6 => /lib64/libc.so.6 (0x0030e120)
/lib64/ld-linux-x86-64.so.2 (0x0030e0e0)
deri@centos6 206> ls -l /lib64/libc.so.6
0 lrwxrwxrwx. 1 root root 12 Aug  9 10:14 /lib64/libc.so.6 -> libc-2.12.so*
deri@centos6 207> cat /etc/redhat-release 
CentOS release 6.8 (Final)

Can you please check how’s your system configured?

Regards Luca

> On 10 Nov 2016, at 21:37, Spransy, Derek <dspr...@emory.edu> wrote:
> 
> Hi Luca,
> 
> Yes, although I fully removed n2disk and pfring before reinstalling 
> everything from RPMs (via yum).
> 
> Thanks,
> Derek
> 
> 
> From: ntop-misc-boun...@listgateway.unipi.it 
> <ntop-misc-boun...@listgateway.unipi.it> on behalf of Luca Deri 
> <d...@ntop.org>
> Sent: Thursday, November 10, 2016 3:33 PM
> To: ntop-misc@listgateway.unipi.it
> Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 
> 2.6.32-642.4.2.el6.x86_64
>  
> Derek
> how did you update? Via yum?
> 
> Luca
> 
>> On 10 Nov 2016, at 20:11, Spransy, Derek <dspr...@emory.edu> wrote:
>> 
>> Hi Alfredo,
>> 
>> I updated, but now I've run into a different problem. It looks like the new 
>> version of n2disk10g requires glibc 2.14? Was that just changed in this 
>> version? I'm on RHEL 6 and have 2.12:
>> 
>> $ sudo /usr/local/bin/n2disk10g /etc/n2disk/n2disk-eth5.conf 
>> /usr/local/bin/n2disk10g: /lib64/libc.so.6: version `GLIBC_2.14' not found 
>> (required by /usr/local/bin/n2disk10g)
>> 
>> Thanks,
>> Derek
>> 
>> 
>> From: ntop-misc-boun...@listgateway.unipi.it 
>> <ntop-misc-boun...@listgateway.unipi.it> on behalf of Alfredo Cardigliano 
>> <cardigli...@ntop.org>
>> Sent: Thursday, November 10, 2016 1:07 PM
>> To: ntop-misc@listgateway.unipi.it
>> Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 
>> 2.6.32-642.4.2.el6.x86_64
>>  
>> Hi Derek
>> I tought I updated you on this, but it seems it is not the case,
>> could you try using -R with latest release?
>> 
>> Best Regards
>> Alfredo
>> 
>>> On 3 Nov 2016, at 18:53, Alfredo Cardigliano <cardigli...@ntop.org> wrote:
>>> 
>>> Hi Derek
>>> I need to check if it’s a bug in the n2disk version with -R. Please use it 
>>> without -R in the meantime.
>>> 
>>> Alfredo
>>> 
>>>> On 3 Nov 2016, at 18:52, Spransy, Derek <dspr...@emory.edu> wrote:
>>>> 
>>>> Hi Alfredo,
>>>> 
>>>> Commenting that line out in my config allowed n2disk to run properly. Is 
>>>> that an issue with my config issue or a bug?
>>>> 
>>>> Thanks,
>>>> Derek
>>>> 
>>>> From: ntop-misc-boun...@listgateway.unipi.it 
>>>> <ntop-misc-boun...@listgateway.unipi.it> on behalf of Alfredo Cardigliano 
>>>> <cardigli...@ntop.org>
>>>> Sent: Thursday, November 3, 2016 1:48 PM
>>>> To: ntop-misc@listgateway.unipi.it
>>>> Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 
>>>> 2.6.32-642.4.2.el6.x86_64
>>>>  
>>>> Derek
>>>> please try running n2disk without "-R 8,9,10”
>>>> 
>>>> Alfredo
>>>> 
>>>>> On 3 Nov 2016, at 18:27, Spransy, Derek <dspr...@emory.edu> wrote:
>>>>> 
>>>>> Sure, here's the config:
>>>>> 
>>>>> -i zc:eth5
>>>>> -o /data1/captures
>>>>> -b 3072
>>>>> -p 1024
>>>>> -q 1
>>>>> -S 0
>>>>> -c 6
>>>>> -R 8,9,10
>>>>> -w 7
>>>>> -z 12
>>>>> -I
>>>>> -A /data1/index
>>>>> --max-num-files 43000
>>>>> -P=/var/run/n2disk.pid
>>>>> --event-log /var/log/n2disk.log
>>>>> --index-on-compressor-threads
>>>>> --pcap-compression
>>>>> 

Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 2.6.32-642.4.2.el6.x86_64

[Luca Deri]

Derek
how did you update? Via yum?

Luca

> On 10 Nov 2016, at 20:11, Spransy, Derek  wrote:
> 
> Hi Alfredo,
> 
> I updated, but now I've run into a different problem. It looks like the new 
> version of n2disk10g requires glibc 2.14? Was that just changed in this 
> version? I'm on RHEL 6 and have 2.12:
> 
> $ sudo /usr/local/bin/n2disk10g /etc/n2disk/n2disk-eth5.conf 
> /usr/local/bin/n2disk10g: /lib64/libc.so.6: version `GLIBC_2.14' not found 
> (required by /usr/local/bin/n2disk10g)
> 
> Thanks,
> Derek
> 
> 
> From: ntop-misc-boun...@listgateway.unipi.it 
>  on behalf of Alfredo Cardigliano 
> 
> Sent: Thursday, November 10, 2016 1:07 PM
> To: ntop-misc@listgateway.unipi.it
> Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 
> 2.6.32-642.4.2.el6.x86_64
>  
> Hi Derek
> I tought I updated you on this, but it seems it is not the case,
> could you try using -R with latest release?
> 
> Best Regards
> Alfredo
> 
>> On 3 Nov 2016, at 18:53, Alfredo Cardigliano > > wrote:
>> 
>> Hi Derek
>> I need to check if it’s a bug in the n2disk version with -R. Please use it 
>> without -R in the meantime.
>> 
>> Alfredo
>> 
>>> On 3 Nov 2016, at 18:52, Spransy, Derek >> > wrote:
>>> 
>>> Hi Alfredo,
>>> 
>>> Commenting that line out in my config allowed n2disk to run properly. Is 
>>> that an issue with my config issue or a bug?
>>> 
>>> Thanks,
>>> Derek
>>> 
>>> From: ntop-misc-boun...@listgateway.unipi.it 
>>>  
>>> >> > on behalf of Alfredo 
>>> Cardigliano >
>>> Sent: Thursday, November 3, 2016 1:48 PM
>>> To: ntop-misc@listgateway.unipi.it 
>>> Subject: Re: [Ntop-misc] n2disk dumping with RHEL6 Kernel 
>>> 2.6.32-642.4.2.el6.x86_64
>>>  
>>> Derek
>>> please try running n2disk without "-R 8,9,10”
>>> 
>>> Alfredo
>>> 
 On 3 Nov 2016, at 18:27, Spransy, Derek > wrote:
 
 Sure, here's the config:
 
 -i zc:eth5
 -o /data1/captures
 -b 3072
 -p 1024
 -q 1
 -S 0
 -c 6
 -R 8,9,10
 -w 7
 -z 12
 -I
 -A /data1/index
 --max-num-files 43000
 -P=/var/run/n2disk.pid
 --event-log /var/log/n2disk.log
 --index-on-compressor-threads
 --pcap-compression
 --remove-ahead
 --unprivileged-user n2disk
 --verbose
 
 And starting verbose:
 
 $ sudo /usr/local/bin/n2disk10g /etc/n2disk/n2disk-eth5.conf 
 03/Nov/2016 13:26:19 [n2disk.c:4808] Welcome to n2disk10g v.2.6.160917 
 (r4666) [SandyBridge]
 03/Nov/2016 13:26:19 [n2disk.c:4835] Running on 2 node(s) system with 24 
 core(s). NUMA affinity set to node 1.
 03/Nov/2016 13:26:19 [n2disk.c:4864] Using PF_RING for packet capture
 03/Nov/2016 13:26:19 [n2disk.c:4890] WARNING: If you are using standard 
 drivers (packet capture via kernel) please disable time-pulse thread
 03/Nov/2016 13:26:19 [n2disk.c:4893] Multithread support enabled
 03/Nov/2016 13:26:19 [n2disk.c:5007] Dump files max size is set to 1024 MB
 03/Nov/2016 13:26:19 [n2disk.c:5024] Buffer memory is set to 3 GB (x 3 
 pcap files)
 03/Nov/2016 13:26:19 [n2disk.c:5059] Using directory /data1/captures for 
 dump files
 03/Nov/2016 13:26:19 [n2disk.c:5064] No sub-directories will be created
 03/Nov/2016 13:26:19 [n2disk.c:5069] Up to 43000 files will be written 
 before overwriting
 03/Nov/2016 13:26:19 [n2disk.c:5079] Dump files max duration is set to 600 
 sec
 03/Nov/2016 13:26:19 [n2disk.c:5095] Dumping data in 0.1 MB chunks
 03/Nov/2016 13:26:19 [n2disk.c:5138] Index processing memory is set to 847 
 MB (x 3 index files)
 03/Nov/2016 13:26:22 [n2disk.c:5328] Memory allocated successfully
 03/Nov/2016 13:26:22 [n2disk.c:3597] Using time pulse timestamps
 03/Nov/2016 13:26:22 [n2disk.c:3630] Started PF_RING packet reader thread 
 for device zc:eth5
 $
>>> 
>>> 
>>> 
>>> This e-mail message (including any attachments) is for the sole use of
>>> the intended recipient(s) and may contain confidential and privileged
>>> information. If the reader of this message is not the intended
>>> recipient, you are hereby notified that any dissemination, distribution
>>> or copying of this message (including any attachments) is strictly
>>> prohibited.
>>> 
>>> If you have received this message in error, please contact
>>> the sender by reply e-mail message and destroy all copies of the
>>> original message (including attachments).
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it 

Re: [Ntop-misc] PFRING not giving callback for fragmented packets if frag enabled in PFRING

[Luca Deri]

Hi Chandrika
in PF_RING we rely on kernel defragmentation routines, it’s not our code. As 
the kernel code is designed to work per socket and not in in the wild, this 
could be the issue. I suggest to defragment in user-space as your traffic seems 
to be heavily fragmented (BTW why don’t you increase the MTU?)

Luca


> On 9 Nov 2016, at 08:20, Chandrika Gautam  
> wrote:
> 
> Hi Alfredo, 
> 
> Let me rephrase the problem I am facing some problem when fragmentation is 
> enabled in PFRING.
> 
> Please find the attachment trace where there are total 18 fragmented 
> packets(9 complete packets), out of which PFRING is assembling only 1.
> 
> These packets having some padding bytes which PFRING seems not able to decode 
> and assemble correctly.
> 
> Regards,
> Chandrika
> 
> On Thu, Nov 3, 2016 at 2:34 PM, Alfredo Cardigliano  > wrote:
> Hi Chandrika
> a few quesitons:
> - are you talking about defragmentation?
> - what do you mean with "callback”?
> - what drivers are you using?
> Please note that in case of defragmentation, the pf_ring kernel module is 
> using the linux ip_defrag() functionality.
> 
> Alfredo
> 
> > On 3 Nov 2016, at 06:36, Chandrika Gautam  > > wrote:
> >
> > Hi,
> >
> > I am using PFRING 6.2.0 and enabled fragmentation in PFRING but PFRING is 
> > not giving a callback on the attached packets having two fragments (first 
> > fragment is smaller and latter one is bigger).
> > Can you please help us to understand how does fragmentation work in PFRING.
> >
> > How many simultaneous ip fragments it can cater(hash size) and what is the 
> > timeout for fragments?
> >
> > Regards,
> > C.G.
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it 
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] PF_RING sees DAG but nprobe does not

[Luca Deri]

John
our PF_RING packages binaries do not include DAG support yet. We’ve just 
received DAG drivers and working at this. Please file an issue on 
https://github.com/ntop/PF_RING/issues  
so we can track the problem more effectively than via email

Regards Luca

> On 28 Oct 2016, at 02:04, John Marshall  wrote:
> 
> Current situation.
> When trying to run nprobe against dag0 I get the following:
> -
> 28/Oct/2016 12:57:02 [util.c:4371] ERROR: Cannot get hw addr for dag0
> 28/Oct/2016 12:57:02 [pro/pf_ring.c:377] Initializing PF_RING socket on 
> device dag0..
> 28/Oct/2016 12:57:02 [nprobe.c:5573] ERROR: Unable to open interface dag0.
> 28/Oct/2016 12:57:02 [nprobe.c:7401] ERROR: Unable to open interface dag0 
> (dag0: No such device exists (SIOCGIFHWADDR: No such device))
> ---
> 
> Even though the PF_RING pcount utility sees it and can get packet stats from 
> it.
> 
> [root@localhost ~]# /usr/local/src/pfring/userland/examples/pcount -h
> pcount
> (C) 2003-14 Deri Luca 
> -h  [Print help]
> -i  [Device name]
> -f  [pcap filter]
> -l [Capture length]
> -S  [Do not strip hw timestamps (if present)]
> -v[Verbose [1: verbose, 2: very verbose (print packet payload)]]
> 
> Available devices (-i):
> 0. ens3
> 1. ens4
> 2. any
> 3. lo
> 4. dag0
> 5. dag0:0
> 6. nflog
> 7. nfqueue
> 8. usbmon1
> [root@localhost ~]#
> ---
> What now?
> 
> How I got here:
> 1)compiled PF_RING with dag libraries and confirmed it worked with the pcount 
> utility.
> 2)installed nprobe using yum.  (I did not see how to compile nprobe)
> 
> --
> John Marshall
> Senior Consultant
> 
> T +64 9 355 4818 (extn 94818)
> M +64 27 819 8366
> E john.marsh...@spark.co.nz
> 
> Level 1 Green, Spark City | 167 Victoria Street West | Private Bag 92028, 
> Auckland 1010
> www.sparkdigital.co.nz
> --
> This communication, including any attachments, is confidential. If you are 
> not the intended recipient, you should not read it - please contact me 
> immediately, destroy it, and do not copy or use any part of this 
> communication or disclose anything about it. Thank you. Please note that this 
> communication does not designate an information system for the purposes of 
> the Electronic Transactions Act 2002.
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Access to historical charts very slow in ntopng

[Luca Deri]

Peter
it is very likely that your MySQL is not fast enough.

Try to run the query below on your DB to see how long it took (MySQL 5.6 or 
later)

Luca

mysql> select * from  events_statements_summary_by_digest order by 
MAX_TIMER_WAIT desc limit 1 \G
*** 1. row ***
SCHEMA_NAME: ntopng
 DIGEST: 79669e73b0e9bcf17c7ebc9c5ba6b8de
DIGEST_TEXT: SELECT COUNT ( * ) AS `TOT_FLOWS` , SUM ( 
`IN_BYTES` + `OUT_BYTES` ) AS `TOT_BYTES` , SUM ( `PACKETS` ) AS `TOT_PACKETS` 
FROM `flowsv4` WHERE `FIRST_SWITCHED` <= ? AND `FIRST_SWITCHED` >= ? AND ( 
`NTOPNG_INSTANCE_NAME` = ? OR `NTOPNG_INSTANCE_NAME` IS NULL ) AND ( 
`INTERFACE_ID` = ? ) AND `L7_PROTO` = ? AND ( `IP_SRC_ADDR` = `INET_ATON` (?) 
OR `IP_DST_ADDR` = `INET_ATON` (?) )
 COUNT_STAR: 37
 SUM_TIMER_WAIT: 247554300
 MIN_TIMER_WAIT: 2207200
 AVG_TIMER_WAIT: 66906567000
 MAX_TIMER_WAIT: 47117300
  SUM_LOCK_TIME: 1040700
 SUM_ERRORS: 0
   SUM_WARNINGS: 0
  SUM_ROWS_AFFECTED: 0
  SUM_ROWS_SENT: 37
  SUM_ROWS_EXAMINED: 817254
SUM_CREATED_TMP_DISK_TABLES: 0
 SUM_CREATED_TMP_TABLES: 0
   SUM_SELECT_FULL_JOIN: 0
 SUM_SELECT_FULL_RANGE_JOIN: 0
   SUM_SELECT_RANGE: 1
 SUM_SELECT_RANGE_CHECK: 0
SUM_SELECT_SCAN: 34
  SUM_SORT_MERGE_PASSES: 0
 SUM_SORT_RANGE: 0
  SUM_SORT_ROWS: 0
  SUM_SORT_SCAN: 0
  SUM_NO_INDEX_USED: 34
 SUM_NO_GOOD_INDEX_USED: 0
 FIRST_SEEN: 2016-10-26 16:11:04
  LAST_SEEN: 2016-10-26 16:43:11
1 row in set (0.00 sec)



> On 27 Oct 2016, at 01:26, Peter Shute  wrote:
> 
> In the charts tab of the Interfaces section, we can choose time ranges 
> between 5 minutes and 1 year to display the traffic levels on a chart. I 
> haven't been able to get the chart to display more than one week of data. If 
> I ask it to display two weeks, it waits for a very long time then seems to 
> give up.
> 
> I assume it's having trouble querying the mysql database for that much data. 
> Are there any tests I can do to prove this, and is there anything I can do to 
> speed it up?
> 
> The mysql I installed still has all the default configuration settings. It's 
> running on a recent version of Ubuntu server.
> 
> Peter Shute
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] trying to get PF_RING to recognize DAG

[Luca Deri]

Hi John,
we are in touch with Endace and we’ll refresh DAG support in PF_RING in the 
near future to make sure we support all their latest products.

This said, you need to compile PF_RING from source on a host where you have the 
DAG tools installed and it should work. Did you do that perhaps?

Regards Luca

> On 19 Oct 2016, at 03:34, John Marshall  wrote:
> 
> I've installed nProbe and, consequently, PF_RING.  The only interface I can 
> currently use is ens3.  I'm trying to get it to work with the Endace dag0.
> 
> I've done the install from packages.
> 
> Does anyone have any experience with this you can share?
> 
> --
> -John
> 
> This communication, including any attachments, is confidential. If you are 
> not the intended recipient, you should not read it - please contact me 
> immediately, destroy it, and do not copy or use any part of this 
> communication or disclose anything about it. Thank you. Please note that this 
> communication does not designate an information system for the purposes of 
> the Electronic Transactions Act 2002.
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Different reported packet rates in cento

[Luca Deri]

This looks like a bug. Please file an issue on 
https://github.com/ntop/PF_RING/issues <https://github.com/ntop/PF_RING/issues>

Luca

> On 3 Oct 2016, at 19:06, Jeremy Ashton <jeremy.ash...@shopify.com> wrote:
> 
> Here is the deal.  If I configure zbalance_ipc listen on 4x interfaces
> and then map those queues to dummy interfaces; Cento shows the
> bandwidth 100x more than reality.  i.e. it shows 27.91Gbps whereas it
> is actually ~279 Mbit/s.
> 
> pfcount on dummy0 interface:
> $ sudo pfcount -i dummy0 -c 100
> 
> =
> Absolute Stats: [152'733 pkts total][0 pkts dropped][0.0% dropped]
> [152'733 pkts rcvd][108'744'822 bytes rcvd][50'907.23 pkt/sec][289.96 
> Mbit/sec]
> =
> Actual Stats: [41'132 pkts rcvd][1'000.05 ms][41'129.73 pps][0.25 Gbps]
> =
> 
> =
> Absolute Stats: [189'340 pkts total][0 pkts dropped][0.0% dropped]
> [189'340 pkts rcvd][134'926'703 bytes rcvd][47'331.02 pkt/sec][269.83 
> Mbit/sec]
> =
> Actual Stats: [36'607 pkts rcvd][1'000.11 ms][36'602.82 pps][0.21 Gbps]
> =
> 
> 
> cento on dummy0 interface (should be 1/4 of total bandwidth):
> $ sudo /usr/local/bin/zbalance_ipc -i
> zc:,zc:,zc:,zc: -c10 -n4 -m1 -g0 -u
> /mnt/hugepages/ -r 0:dummy0 -r 1:dummy1 -r 2:dummy2 -r 3:dummy3
> $ sudo cento -i dummy0
> 03/Oct/2016 16:58:26 [NetworkInterface.cpp:990] [dummy0] [12'458'091
> pps/27.91 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
> pkt drops][0 TX pps]
> 03/Oct/2016 16:58:27 [NetworkInterface.cpp:990] [dummy0] [12'447'559
> pps/27.88 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
> pkt drops][0 TX pps]
> 03/Oct/2016 16:58:28 [NetworkInterface.cpp:990] [dummy0] [12'753'627
> pps/28.57 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
> pkt drops][0 TX pps]
> 03/Oct/2016 16:58:29 [NetworkInterface.cpp:990] [dummy0] [12'385'380
> pps/27.74 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
> pkt drops][0 TX pps]
> 03/Oct/2016 16:58:30 [NetworkInterface.cpp:990] [dummy0] [12’261'717
> pps/27.47 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
> pkt drops][0 TX pps]
> 
> cento on raw interface with zc:
> $ sudo cento -i zc: -i zc: -i zc: -i zc: -g
> 1,2,3,4 -C 100 -H
> 03/Oct/2016 16:59:38 [NetworkInterface.cpp:990] [zc:] [0
> pps/0.00 Gbps][3’966/0/0/512’000 act/exp/drop/max flows][137’492/0
> RX/TX pkt drops][0 TX pps]
> 03/Oct/2016 16:59:38 [NetworkInterface.cpp:990] [zc:] [0
> pps/0.00 Gbps][1’960/0/0/512’000 act/exp/drop/max flows][34’004/0
> RX/TX pkt drops][0 TX pps]
> 03/Oct/2016 16:59:38 [NetworkInterface.cpp:990] [zc:] [0
> pps/0.00 Gbps][9’583/0/0/512’000 act/exp/drop/max flows][125’445/0
> RX/TX pkt drops][0 TX pps]
> 03/Oct/2016 16:59:38 [NetworkInterface.cpp:990] [zc:] [0
> pps/0.00 Gbps][1/0/0/512’000 act/exp/drop/max flows][0/0 RX/TX put
> drops][0 TX pps]
> 
> On Mon, Oct 3, 2016 at 10:14 AM, Luca Deri <d...@ntop.org> wrote:
>> Jeremy,
>> I annoy sure I understand. in essence you have attached cento to 
>> zbalance_ipc queues, and the traffic rates are not correct? What you see 
>> with pfcount instead?
>> 
>> This said, what do you want to do exactly? Perhaps use cento as 
>> flow-generator and attach bro to it on balanced egress queues?
>> 
>> Regards luca
>> 
>>> On 3 Oct 2016, at 16:04, Jeremy Ashton <jeremy.ash...@shopify.com> wrote:
>>> 
>>> After following these instructions:
>>> http://www.ntop.org/pf_ring/best-practices-for-using-bro_ids-with-pf_ring-zc-reliably/
>>> 
>>> The actual command was like the following:
>>> 
>>> zbalance_ipc -i zc:,zc:,zc:,zc: -n 4 -m 1
>>> -c 2 -r 0:dummy0 -r 1:dummy1 -r 2:dummy2 -r 3:dummy3
>>> 
>>> 
>>> I found it was reporting something like the following:
>>> 
>>> 27/Sep/2016 18:17:56 [NetworkInterface.cpp:990] [dummy0] [8'854'686
>>> pps/19.83 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
>>> pkt drops][0 TX pps]
>>> 
>>> 27/Sep/2016 18:17:56 [NetworkInterface.cpp:990] [dummy1] [18’428'232
>>> pps/41.28 Gbps][29'412/0/0/512'000 act/exp/drop/max flows][0/0 RX/TX
>>> pkt drops][0 TX pps]
>>> 
>>> 
>>> If I configure cento to listen to the interfaces directly, I see that
>>> there is <5Gbit aggregate bandwidth.  Is there something strange with
>>> the way cento attempts to listen to dummy interfaces?
>>> ___
>>> Ntop-misc mailing list
>>> N

Re: [Ntop-misc] Trouble with Zero Copy Performance

[Luca Deri]

Michael
95% of CPU load is already too much. I would look at the nProbe traces to see 
if the number of slots, fragments etc are ok. If you do not decrease the Cpu 
load in case of traffic spikes what you describe is reasonable although not 
desirable. Please let me know if you can see anything strange in logs. As of 
the warning about the MAC address is already fixed in the development version. 
I suggest to use RSS to virtualise the interface and start multiple nprobe's 
one per virtual queue.

This said, have you tried nProbe cento? I believe this is the app you need for 
20G+. Give it a try and let me know how it goes. You can read more about it 
here 
http://www.ntop.org/nprobe/flow-based-monitoring-nprobe-cento-vs-standardpro/ 


Regards Luca


> On 30 Sep 2016, at 17:05, Lang, Michael  wrote:
> 
> Hello,
>  
> I’m having high CPU issues with a nProbe/Zero Copy setup on a system that is 
> intended to receive up to 20 Gbps of traffic on an x520-DA2 NIC.  The traffic 
> is being split up by a network switch so that each interface on the 520 is 
> receiving the same amount.  Currently the traffic load is 500 kpps / 3.5 Gbps 
> on each interface and the CPU is right around 95% utilized on each core the 
> interfaces are pinned to.  If I startup nProbe without zero copy the 
> performance is about the same, maybe a bit better.  I’ve gone through 
> documentation, six months of listserv, and tried just about every setting I 
> can think of.  I’m stuck and could use some help or hints on where to go to 
> troubleshoot this.  When the traffic gets higher than the stated above 
> (happens every day for hours during peak) the two CPU cores are totally 
> pegged.
>  
> Here are some system/setup specifics:
>  
> Intel(R) Xeon(R) CPU   X3440  @ 2.53GHz (hyperthreading enabled)
> Intel Corporation Ethernet Server Adapter X520-2 (p1p1 & p1p2)
>  
> [root@flowgen log]# cat /etc/redhat-release
> CentOS Linux release 7.2.1511 (Core)
> [root@flowgen log]# uname -a
> Linux flowgen 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 
> x86_64 x86_64 x86_64 GNU/Linux
>  
> Using PFRING_ZC v.6.4.1.160615
> Welcome to nProbe v.7.4.160928 (r5333) for x86_64-unknown-linux-gnu
> Sep 28 12:24:00 Installed: ixgbe-zc-4.1.5.859-dkms.noarch
> RSS=1,1
> IRQ for p1p1 pinned to CPU3
> IRQ for p1p2 pinned to CPU4
> LRO OFF, GRO OFF, RXVLAN OFF, RING=32768, HUGEPAGES=1204
>  
> [root@flowgen log]# cat /proc/net/pf_ring/dev/p1p1/info
> Name:  p1p1
> Index: 10
> Address:   
> Polling Mode:  NAPI/ZC
> Type:  Ethernet
> Family:Intel ixgbe 82599
> Max # TX Queues:   1
> # Used RX Queues:  1
> Num RX Slots:  32768
> Num TX Slots:  32768
>  
> /usr/local/bin/nprobe --daemon-mode --pid-file /var/nprobe_p1p1.pid 
> --interface zc:p1p1 --as-list /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 
> --lifetime-timeout 300 --verbose 1 --syslog nProbe_p1p1 --collector 
> udp://redacted.uconn.edu:21583  
> --in-iface-idx 1 --out-iface-idx 1
> /usr/local/bin/nprobe --daemon-mode --pid-file /var/nprobe_p1p2.pid 
> --interface zc:p1p2 --as-list /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat 
> --lifetime-timeout 300 --verbose 1 --syslog nProbe_p1p2 --collector 
> udp://redacted.uconn.edu:21583  
> --in-iface-idx 2 --out-iface-idx 2
>  
> Licenses look OK:
>  
> [root@flowgen pf_ring]# zcount -i zc:p1p1 -C
> License Ok
>  
> Doing nprobe -v shows nProbe Standard and doing nprobe --help shows nProbe 
> Pro, both with valid licenses
>  
> One other thing, not sure if it is related, I get the following error and 
> warning when starting the above in zero copy only (goes away with zero copy 
> mode):
>  
> 29/Sep/2016 14:14:31 [util.c:4371] ERROR: Cannot get hw addr for zc:p1p1
> 29/Sep/2016 14:14:32 [nprobe.c:5620] WARNING: Unable to set pcap capture 
> direction
>  
> Thanks in advance,
>  
> -  Mike
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Even there are 16 child processes, ntopng only uses 2

[Luca Deri]

Sacha,
can you please file a ticket on github and attach the current status (# of 
hosts and flows etc) so we can trck this issue?

Luca
> On 13 Sep 2016, at 18:32, Sacha Yunusic  wrote:
> 
> Hi!
> I’m starting to use ntopng that is receiving flows from a Cisco switch 4507 
> thru netflow.
> I start nprobe and ntop in the same server (Dell R720, 24 cores, 128GB RAM, 
> 1TB HD), and I can enter to the GUI, but as soon as I try to look into one 
> specific host (http://server:4000/lua/host_details.lua?host=192.168.200.104 
> ), two child 
> processes of ntopng takes 100% of one core each (so, two out of 24) and it 
> takes forever this simple task.
> Now, after 45 minutes since I click on that link, nothing happened and the 
> browser is still “thinking”.
> So, my questios are: 
> -  Why is taking so much CPU for that simple task
> -  Why it doesn’t use more CPU if there are 16 child processes and is 
> only using two
>  
> I’m using ntopng Pro [Small Business Edition] v.2.5.160816, running on Centos 
> 7.1 x64 installed with yum using /etc/yum.repos.d/ntop.repo. 
>  
> This is how I run nprobe:
> # nprobe --collector-port 2055 --zmq "tcp://*:5888 " --redis 
> 127.0.0.1:6379 -n none
>  
> This is how I run ntopng:
> # ntopng -i tcp://127.0.0.1:5888  --redis 
> 127.0.0.1:6379 -w 4000 -m 192.168.0.0/16
>  
>  
> This is what I see in stdout where I wun ntopng: 
> 13/Sep/2016 13:02:09 [Lua.cpp:5420] WARNING: Script failure 
> [/usr/share/ntopng/scripts/lua/find_host.lua][attempt to index a userdata 
> value]
>  
> 
> Sacha Yunusic | Gerente Técnico | Pentagon Security & Akainix
> Av. Kennedy 4700, Piso 10, Of. 1002, Edificio New Century, Vitacura | Código 
> Postal (ZIP Code) 7630454
> Central: (56-2) 2246 1050 | Directo: (56-2) 2246 2620 | Cel: (56-9) 9883 4752 
> | www.penta-sec.com  & www.akainix.com 
> 
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe delay.

[Luca Deri]

Gabriel
can you please provide a pcap with the flow so I can see what you mean?

Luca
> On 13 Sep 2016, at 10:22, Gabriel Zamorski  wrote:
> 
> Hello,
> 
> I’m using nProbe from yesterday with my WanGuard Flow Sensor. There are logs 
> on it like this: "Received flow from 142 seconds ago on interface "eth5". 
> Adjusting flow delay from 141 to 142”
> 
> I asked the WanGuard Support and I got the answer:
> 
> "The flow delay is the amount of time between the start time of the flow and 
> the time the flow was received by the Sensor.
> You can tune the flow export time (delay) in the flow exporter. “
> 
> So, the question is - what can I do to minimize this delay?
> 
> 
> Regards,
> 
> Gabriel
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Analysing just inbound internet traffic with ntopng

[Luca Deri]

Peter
analysing only one traffic direction will break (most of) nDPI. Please don;t do 
that.

As you’re asking several questions, I suggest you to file individual issues on 
https://github.com/ntop/ntopng/issues so we can answer one by one

Luca

> On 29 Aug 2016, at 23:36, Peter Shute  wrote:
> 
> I've now got NetFlow data being logged in MySQL via nprobe and ntopng. I'm 
> mostly interested in analysing the inbound traffic from the internet to help 
> me find out why we're going over our ISP's download quota. For example, I'd 
> like to find out which device here downloaded the most from the internet 
> yesterday.
> 
> I assumed I must use the Historical Data Explorer, but I can't see any way to 
> filter out all the other flows - ie internal and outgoing. I think I need to 
> look at just the flows where the src ip address is not 192.168.x.y and the 
> dst ip address is 192.168.x.y. 
> 
> I've defined a Traffic Profile called "Incoming only" as "dst net 192.168 and 
> not src net 192.168", but the only place I can see to use this is to click on 
> Interfaces, then select my interface, then click on the funny little symbol 
> that I think is a doctor with a stethoscope, and then on the chart symbol 
> beside the "Incoming only" profile name.   (Can I suggest tool tips for all 
> the symbols so one doesn't have to click on them to find out what they are?)
> 
> But then what? I'd like to be able to select a data range that covers, say, 
> yesterday from midnight to midnight, and see which address downloaded the 
> most data. I can choose a one day range, but it will end at the current time. 
> And I can't see how to get a list of top downloaders for that whole day. If I 
> hover over the chart, it shows a list which I think is for that minute only. 
> And it lists senders and receivers - how can there be both if my filter only 
> matches external sources and internal destinations?
> 
> If I choose a week for the chart length, it still ends at the current time, 
> and I think it still shows the top senders and receivers for one minute 
> periods. I can't tell for sure which day I've chosen because it only displays 
> times, not dates. (Could I suggest that dates are also shown, or at least a 
> clear vertical line for each midnight?)
> 
> Am I looking in the wrong place for the data I want? Or do I need to query 
> the MySQL database myself?
> 
> Peter Shute
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] 100GbE Network Adapters

[Luca Deri]

Josiah,
yes we tested that with Napatech NIC and ti works at line rate

Luca

> On 24 Aug 2016, at 23:29, Josiah White <jos...@protrafsolutions.com> wrote:
> 
> Hello,
> 
> Is it possible for PF_RING to receive 100Gbps line-rate? (150Mpps)
> Has anyone tested this?
> 
> Thanks
> 
> On Wed, Aug 24, 2016 at 2:03 PM, Luca Deri <d...@ntop.org 
> <mailto:d...@ntop.org>> wrote:
> Robert
> we currently support
> 1. Napatech 100G
> 2. Accolade 100G
> 3. Intel FM10K (see 
> http://www.silicom-usa.com/100_Gigabit_Dual_Port_NIC_PE3100G2DQiRL_96 
> <http://www.silicom-usa.com/100_Gigabit_Dual_Port_NIC_PE3100G2DQiRL_96>)
> 4. InveaTech/Netcope
> 
> We have limited mellanox support (see 
> https://github.com/ntop/PF_RING/blob/dev/userland/lib/pfring_mod_mlx.c 
> <https://github.com/ntop/PF_RING/blob/dev/userland/lib/pfring_mod_mlx.c>) as 
> we have tested only at 10G.
> 
> Cheers Luca
> 
> > On 24 Aug 2016, at 12:36, Finze, Robert <robert.fi...@uni-tuebingen.de 
> > <mailto:robert.fi...@uni-tuebingen.de>> wrote:
> >
> > Hi,
> >
> > we're planning on upgrading to 100GbE in near future.
> > Currently I'm running nProbe and am happy with it. On the software side
> > seems to be the next step for 100GbE Cento.
> >
> > On the hardware side on the top of my list is currently the ConnectX-4
> > from Mellanox. Intel has announce a card but it is still not available.
> >
> > Is there a list of supported 100GbE NICs for Cento? Or are there
> > recommended adapters?
> >
> > Cheers
> > Robert
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] 100GbE Network Adapters

[Luca Deri]

Robert
we currently support
1. Napatech 100G
2. Accolade 100G
3. Intel FM10K (see 
http://www.silicom-usa.com/100_Gigabit_Dual_Port_NIC_PE3100G2DQiRL_96)
4. InveaTech/Netcope

We have limited mellanox support (see 
https://github.com/ntop/PF_RING/blob/dev/userland/lib/pfring_mod_mlx.c) as we 
have tested only at 10G.

Cheers Luca

> On 24 Aug 2016, at 12:36, Finze, Robert  wrote:
> 
> Hi,
> 
> we're planning on upgrading to 100GbE in near future.
> Currently I'm running nProbe and am happy with it. On the software side
> seems to be the next step for 100GbE Cento.
> 
> On the hardware side on the top of my list is currently the ConnectX-4
> from Mellanox. Intel has announce a card but it is still not available.
> 
> Is there a list of supported 100GbE NICs for Cento? Or are there
> recommended adapters?
> 
> Cheers
> Robert
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] What does the nprobe --nf parameter do?

[Luca Deri]

Thanks for reporting the problem. This needs to be fixed as you pointed out

Regards Luca

Sent from my iPad

> On 18 Aug 2016, at 01:10, Peter Shute  wrote:
> 
> I believe I have solved this myself. To elaborate, page 44 in the current 
> nProbe_UserGuide.pdf lists sample commands for three modes of nProbe use. The 
> command listed for collector mode is "nprobe -nf-collector-port 2055"
> 
> Because there is no --nf-collector-port parameter described in the manual, I 
> assumed that given there is an em dash used in place of a double dash, 
> perhaps it was a misprint, and that it meant "--nf --collector-port 2055". I 
> tried adding a --nf parameter to my command, and nProbe behaved differently 
> (appeared to override the "-i none" parameter).
> 
> I have found this parameter description in an older version of the user guide 
> (nProbe and nBox User's Guide v.4.0):
> [--nf-collector-port|-3]  | NetFlow collector port
> and this appears to have been renamed in the latest user guide (v.6.16):
> [--collector-port|-3]  | NetFlow/IPFIX/sFlow collector flows port
> 
> I will submit an issue requesting this be corrected in the user guide.
> 
> How the presence of the erroneous --nf parameter was interpreted by nProbe 
> can remain a mystery.
> 
>> -Original Message-
>> From: ntop-misc-boun...@listgateway.unipi.it [mailto:ntop-misc-
>> boun...@listgateway.unipi.it] On Behalf Of Peter Shute
>> Sent: Wednesday, 17 August 2016 12:55 PM
>> To: 'ntop-misc@listgateway.unipi.it' 
>> Subject: [Ntop-misc] What does the nprobe --nf parameter do?
>> 
>> Appendix C of the nprobe user guide specifies the --nf parameter in a couple
>> of the example commands. What does it do? I can't see it described
>> anywhere in the guide.
>> 
>> Peter Shute
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Collecting NetFlow data with nprobe

[Luca Deri]

Peter,
for dumping packets please use tcodump -s 0 -w my.pcap … or wireshark. 

Luca
> On 17 Aug 2016, at 11:28, Peter Shute <psh...@nuw.org.au> wrote:
> 
> Thanks, should I generate the pcap file with the --dump-pkts parameter? I 
> suspect with -i none that there will be nothing dumped, but I'll check.
> 
> Sent from my iPad
> 
>> On 17 Aug 2016, at 6:54 PM, Luca Deri <d...@ntop.org> wrote:
>> 
>> Peter
>> please file an issue on https://github.com/ntop/nProbe/issues and attach a 
>> pcap file. I need to see what nprobe is receiving before commenting. Please 
>> make sure you also add “-i none”
>> 
>> Thanks Luca
>> 
>>> On 17 Aug 2016, at 04:17, Peter Shute <psh...@nuw.org.au> wrote:
>>> 
>>> I still haven't made any progress with this. I've now installed Wireshark, 
>>> and followed these instructions to prove to myself that the NetFlow data is 
>>> arriving at my PC:
>>> https://communities.ca.com/docs/DOC-231149629
>>> 
>>> So why does this command collect no data?
>>> nprobe /c -i none -n none --collector-port 9996 -V9 -P c:\temp\nprobe
>>> 
>>>> -Original Message-
>>>> From: ntop-misc-boun...@listgateway.unipi.it [mailto:ntop-misc-
>>>> boun...@listgateway.unipi.it] On Behalf Of Peter Shute
>>>> Sent: Monday, 15 August 2016 4:00 PM
>>>> To: 'ntop-misc@listgateway.unipi.it' <ntop-misc@listgateway.unipi.it>
>>>> Subject: [Ntop-misc] Collecting NetFlow data with nprobe
>>>> 
>>>> Our ISP has configured several internet routers to send NetFlow data on 
>>>> port
>>>> 9996 to a particular machine. I have successfully configured PRTG to get 
>>>> the
>>>> data to see lists of top recipients, etc, so I know this machine is 
>>>> receiving the
>>>> NetFlow data ok, but it doesn't store the flows for later analysis, so I've
>>>> disabled it. How do I configure nprobe to get the flow into a file I can
>>>> analyse?
>>>> 
>>>> I'm confused about which mode nprobe needs to be used in to collect the
>>>> data. I've tried this:
>>>> nprobe /c --collector 192.168.0.203:9996  -V9 -P c:\temp\nprobe but it 
>>>> seems
>>>> to be collecting local traffic. In among it, I can see that there are 
>>>> flows from
>>>> the router to this machine on port 9996. What I need is the flow 
>>>> information
>>>> inside those packets.
>>>> 
>>>> I tried this:
>>>> nprobe /c -i none -n none --collector-port 9996 -V9 -P c:\temp\nprobe but 
>>>> it
>>>> collects nothing.
>>>> 
>>>> Where am I going wrong? I'm not sure if I usderstand the differences
>>>> between probe mode, collector mode and proxy mode. I need collector
>>>> mode, don't I?
>>>> 
>>>> Peter Shute
>>>> ___
>>>> Ntop-misc mailing list
>>>> Ntop-misc@listgateway.unipi.it
>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>> 
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Collecting NetFlow data with nprobe

[Luca Deri]

Peter
please file an issue on https://github.com/ntop/nProbe/issues and attach a pcap 
file. I need to see what nprobe is receiving before commenting. Please make 
sure you also add “-i none”

Thanks Luca

> On 17 Aug 2016, at 04:17, Peter Shute  wrote:
> 
> I still haven't made any progress with this. I've now installed Wireshark, 
> and followed these instructions to prove to myself that the NetFlow data is 
> arriving at my PC:
> https://communities.ca.com/docs/DOC-231149629
> 
> So why does this command collect no data?
> nprobe /c -i none -n none --collector-port 9996 -V9 -P c:\temp\nprobe
> 
>> -Original Message-
>> From: ntop-misc-boun...@listgateway.unipi.it [mailto:ntop-misc-
>> boun...@listgateway.unipi.it] On Behalf Of Peter Shute
>> Sent: Monday, 15 August 2016 4:00 PM
>> To: 'ntop-misc@listgateway.unipi.it' 
>> Subject: [Ntop-misc] Collecting NetFlow data with nprobe
>> 
>> Our ISP has configured several internet routers to send NetFlow data on port
>> 9996 to a particular machine. I have successfully configured PRTG to get the
>> data to see lists of top recipients, etc, so I know this machine is 
>> receiving the
>> NetFlow data ok, but it doesn't store the flows for later analysis, so I've
>> disabled it. How do I configure nprobe to get the flow into a file I can
>> analyse?
>> 
>> I'm confused about which mode nprobe needs to be used in to collect the
>> data. I've tried this:
>> nprobe /c --collector 192.168.0.203:9996  -V9 -P c:\temp\nprobe but it seems
>> to be collecting local traffic. In among it, I can see that there are flows 
>> from
>> the router to this machine on port 9996. What I need is the flow information
>> inside those packets.
>> 
>> I tried this:
>> nprobe /c -i none -n none --collector-port 9996 -V9 -P c:\temp\nprobe but it
>> collects nothing.
>> 
>> Where am I going wrong? I'm not sure if I usderstand the differences
>> between probe mode, collector mode and proxy mode. I need collector
>> mode, don't I?
>> 
>> Peter Shute
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] cento flow template

[Luca Deri]

Issue closed: implemented

Regards Luca

> On 20 Jul 2016, at 14:00, Jeremy Ashton <jeremy.ash...@shopify.com> wrote:
> 
> Issue created.  https://github.com/ntop/nProbe/issues/85
> 
> Thanks.
> 
> On Wed, Jul 20, 2016 at 4:06 AM, Luca Deri <d...@ntop.org> wrote:
>> Jeremy
>> the idea of cento is to make it fast and thus have some limited NetFlow
>> configurability, to avoid spending time handling custom templates. If adding
>> Mac address is all you need, we can do that as it’s relatively simple.
>> Please file an issue on https://github.com/ntop/nProbe/issues
>> 
>> Thank you
>> 
>> Luca
>> 
>> On 19 Jul 2016, at 19:03, Jeremy Ashton <jeremy.ash...@shopify.com> wrote:
>> 
>> I have been testing out cento a bit and was wondering if there are any
>> ways to get it to include IN_SRC_MAC and OUT_DST_MAC values?  With
>> nprobe one could historically define a custom template.  Any thoughts
>> on this?
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>> 
>> 
>> 
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] How to Include Agent ID Field in MySQL SFlow Records?

[Luca Deri]

Dennis
please add to your template (-T) these IEs (or just IPv4 if your exporter is 
via IPv4)

[130] %EXPORTER_IPV4_ADDRESS  %exporterIPv4Address  Exporter IPv4 
Address
[131] %EXPORTER_IPV6_ADDRESS  %exporterIPv6Address  Exporter IPv6 
Address

Cheers Luca

> On 15 Jul 2016, at 16:57, Dennis Rardin  wrote:
> 
> All/Any, <>
>  
> I currently have nprobe collecting sflows; handing them to ntopng via zmq, 
> and ntopng writing them to MySQL. The data is fine, but I really would like 
> to have the IP Address of the agent reporting the flow data in the database. 
> Any suggestions?
>  
> Thanks,
> Dennis
> Pico Quantitative Trading LLC ("PQT"). This e-mail (including any 
> attachments) is intended only for use by the addressee(s) named above, and 
> may contain confidential, proprietary or legally privileged information. If 
> you are not the intended recipient of this e-mail, any review, use, 
> disclosure, dissemination, distribution, printing or copying of this e-mail 
> or any attachment is strictly prohibited. If you have received this e-mail in 
> error, please notify Pico immediately by return e-mail and permanently delete 
> the original from your system and any hard copy printout thereof. E-mails are 
> not encrypted and cannot be guaranteed to be secure or error-free and, as 
> with all Internet communications, information could be intercepted, 
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. 
> Accordingly, Pico accepts no liability for any errors or omissions in the 
> content contained herein. In compliance with applicable laws, rules and 
> regulations and/or at its discretion, Pico may review and archive incoming 
> and outgoing e-mail communications, copies of which may be produced at the 
> request of regulators. ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] cento flow template

[Luca Deri]

Jeremy
the idea of cento is to make it fast and thus have some limited NetFlow 
configurability, to avoid spending time handling custom templates. If adding 
Mac address is all you need, we can do that as it’s relatively simple. Please 
file an issue on https://github.com/ntop/nProbe/issues 


Thank you

Luca

> On 19 Jul 2016, at 19:03, Jeremy Ashton  wrote:
> 
> I have been testing out cento a bit and was wondering if there are any
> ways to get it to include IN_SRC_MAC and OUT_DST_MAC values?  With
> nprobe one could historically define a custom template.  Any thoughts
> on this?
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] cento netflow v9 generation problems

[Luca Deri]

Jeremy
this has been fixed already: please update your cento copy

Regards Luca

> On 19 Jul 2016, at 18:53, Jeremy Ashton  wrote:
> 
> Wanted to start playing more with cento, but have been running into
> the following problem:
> 
> sudo /usr/local/bin/cento -i zc: -C 100 -9 127.0.0.1:1 -v5
> 19/Jul/2016 16:50:45 [cento.cpp:901] Welcome to nProbe cento v.1.1.160715
> 19/Jul/2016 16:50:45 [cento.cpp:902] Copyright 2015-16 - ntop
> 19/Jul/2016 16:50:45 [cento.cpp:904] SystemId: 
> 19/Jul/2016 16:50:45 [cento.cpp:916] WARNING: Invalid license
> /etc/nprobe.cento: Missing license file
> 19/Jul/2016 16:50:45 [cento.cpp:919] WARNING: Running in demo mode (5 min)
> 19/Jul/2016 16:50:45 [cento.cpp:923] Built on Ubuntu 14.04.4 LTS
> 19/Jul/2016 16:50:45 [cento.cpp:924] Available CPU Cores: 20
> 19/Jul/2016 16:50:45 [cento.cpp:925] Max flow duration set to 120 sec
> 19/Jul/2016 16:50:45 [cento.cpp:926] Max flow idleness set to 30 sec
> 19/Jul/2016 16:50:45 [cento.cpp:927] Flow hash size set to 512000 buckets
> 19/Jul/2016 16:50:45 [cento.cpp:928] Hashes will have up to 200 buckets
> 19/Jul/2016 16:50:45 [cento.cpp:750] Setting up interface zc:
> 19/Jul/2016 16:50:46 [cento.cpp:1113] Initialized global ZC cluster 100
> 19/Jul/2016 16:50:46 [Utils.cpp:502] Interface zc: has speed = 1
> 19/Jul/2016 16:50:47 [ZCInterface.cpp:97] Reading packets from
> interface zc:...
> 19/Jul/2016 16:50:47 [cento.cpp:1182] Created interface zc:
> 19/Jul/2016 16:50:47 [FlowCollector.cpp:160] Exporting flows towards
> 127.0.0.1:1 using UDP
> 19/Jul/2016 16:50:47 [NetflowExporterV9.cpp:108] ERROR: INTERNAL
> ERROR: Invalid # flows/packet [actual: 19][expected: 16]
> 
> It seems net flow v5 and ipfix work fine, but v9 throws the error.  Any ideas?
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] NT_InfoOpen() failed: NT Service is not started

[Luca Deri]

Jeremy
are you using Napatech NICs perhaps? If so, you need to start the napatech 
services (/opt/napatech….)

Cheers Luca

> On 13 Jul 2016, at 19:27, Jeremy Ashton  wrote:
> 
> Recently when I tried to configure an additional interface I got the
> following error:
> 
> [zbalance_ipc] NT_ExplainError: Code=0x2013 Error="NT Service is
> not started"
> NT_Init() failed: NT Service is not started
> [zbalance_ipc] NT_ExplainError: Code=0x2013 Error="NT Service is
> not started"
> NT_InfoOpen() failed: NT Service is not started
> 13/Jul/2016 17:21:25 [zbalance_ipc.c:566] ERROR: pfring_zc_open_device
> error [Socket operation on non-socket] Please check that zc: is
> up and not already used
> 
> I get this error when running:
> sudo  /usr/local/bin/zbalance_ipc -i zc: -c 50 -n 2 -m 2 -g 0 -p
> 
> which works perfectly fine for 4 other interfaces.  The only
> difference being that each interface gets assigned its own clusterid.
> 
> The box is Supermicro with XL170 NICs running Ubuntu 14.04.
> 
> Any thoughts on how I might approach this?
> 
> Cheers.
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Ouch! Wiped Out My Old OS

[Luca Deri]

On 06/06/2016 11:02 PM, Avery Rozar wrote:
> Thank you Luca and all the people at ntop for the time you spend on
> these "free" applications and programs, AND for making them available
> in package form...
>
What do you mean exactly? nBox is available in source format (so you can
inspect what we do) at no cost, and it's packaged as all the other sw we
make.

We can discuss if factory reset is something different than what you
think, but this has nothing to do with these arguments. If you want to
move to the next level, stop complaining and start contributing by
improving what we have done.

> ;)
>
> On Mon, Jun 6, 2016 at 4:38 PM, Annoyed User
> <annoyedntopu...@gmail.com <mailto:annoyedntopu...@gmail.com>> wrote:
>
> I appreciate the quick response, however...
>
> When you install any software package on a server and do "factory
> reset" of the software, you would expect it to reset JUST itself,
> not the entire server where the package was installed.
>
> I have never seen any installed application on Windows or Linux
> that ever wipes out all user folders and logfiles when you ask the
> program to repair or reset itself. 
>
> It's one thing when you install the application from a USB or CD
> and it wipes the entire system before installing itself (like
> pfSense or m0m0wall), but not when you have an existing, running
> server and you are just doing an apt-get install.
>
> You stated in your reply below that it just resets the IP address
> "etc".  It does quite a bit more than just reset the IP.
>
> It should be made a little more clear what exactly the "Factory
> Reset" option is doing to include wiping out /root and all /home user
>

This is what we have done yesterday after you reported the issue. Please
report if it's now enough.

Luca
>
> folders!
>
>
> On Mon, Jun 6, 2016 at 1:22 PM, Luca Deri <d...@ntop.org
> <mailto:d...@ntop.org>> wrote:
>
> Dear annoyed user,
> the nbox is a physical probe that people build using our tools
> and our GUI (i.e. the nbox package). It is not designed to be
> installed on an existing computer with users etc but to
> replicate a physical probe using commodity hardware like those
> we have on our website. The script factory reset that you have
> executed resets the nbox to its initial state, similar to what
> happens with routers etc. 
>
> So what to expect for a factory reset? In our understanding we
> reset the nbox to the initial state, but we do not wipe the OS
> (Linux is still there), just reset system to the initial state
> IP etc. including removing users because they are not part of
> the nBox hardware probe.
>
> I am sorry if you have deleted your users, but on the other
> hand ntop is the factory so I believe we should be free to
> decide what factory means, and you have been warned before
> doing that
>
> We’ve added an extra warning in addition to the one that is
> already exiting to warn you. Packages are currently being rebuilt.
>
> Regards Luca
>
>> On 06 Jun 2016, at 18:36, Annoyed User
>> <annoyedntopu...@gmail.com
>> <mailto:annoyedntopu...@gmail.com>> wrote:
>>
>> Luca, your reply is complete BullSH*T
>>
>> Following up on this thread, the exact same thing happened to me!
>>
>> The factory default script provided by one of these packages
>> wiped out all /home folders and /var/log.
>>
>> Someone needs to fix this.
>>
>> After running the apt-get install, the instructions state
>> "IMPORTANT IMPORTANT IMPORT IMPORTANT IMPORTANT
>>
>> You can now point your browser to https://localhost
>> <https://localhost/>
>>
>> The default user is nbox with password nbox
>>
>> Please run a factory reset by GUI (System -> Factory Reset )
>>
>> IMPORTANT IMPORTANT IMPORT IMPORTANT IMPORTANT"
>>
>>
>> After you do this, you will completely screw up your system.
>>
>> Looking at the CGI/HTML code, that button runs
>> /usr/local/bin/factory_reset
>>
>> This script looks like something NTOP installed, since it was
>> created the same time I did my apt-get install of
>> ntopng/nbox/etc.
>>
>> Inside that script, there are several HIGHLY DESTRUCTIVE
>> com

Re: [Ntop-misc] Nprobe black list network

[Luca Deri]

Loic
please file an enhhancement issue for Office365: in essence you don't
want to emit flows for protocol X,Y,Z that in your case is Office365?
Luca




On 05/24/2016 09:24 AM, Loic CRUCHADE wrote:
>
> Hello,
>
>  
>
> Thanks for the reply.
>
> I reached the same goal with the « collection-filter » argument.
>
> But i had some problems too. The « ! » was returning a shell fuction i
> think. Whan use it put some « yum install… » instead of the « ! ».
>
> I solved this by using a configuration file for nprobe.
>
> Now, the last thing i have to do is to filter only Office365 flows,
> but it’s tricky because of there is almost 1000 IP to filter.
>
> If anybody have an idea.
>
> Thanks again.
>
>  
>
> CRUCHADE Loïc
>
> 05.82.52.22.02
>
> Service Exploitation Informatique
>
> Direction des Systèmes d’information
>
> logo
>
>  
>
> *De :*ntop-misc-boun...@listgateway.unipi.it
> [mailto:ntop-misc-boun...@listgateway.unipi.it] *De la part de* Luca Deri
> *Envoyé :* mardi 24 mai 2016 09:02
> *À :* ntop-misc@listgateway.unipi.it
> *Objet :* Re: [Ntop-misc] Nprobe black list network
>
>  
>
> Loïc
>
> I have just tested and it seems to work for me. What nprobe version
> are you using? I have tested the latest 7.3 release.
>
>  
>
> Please add a “ “ between the blacklist parameter to make sure the
> shell does not mess-up. If still not working, please file a bug
> at https://github.com/ntop/nProbe/issues
>
>  
>
> Regards Luca
>
>  
>
>  
>
> On 23 May 2016, at 10:10, Loic CRUCHADE <loic.cruch...@consuel.com
> <mailto:loic.cruch...@consuel.com>> wrote:
>
>  
>
> Hello,
>
>  
>
> I recently bought Nprobe pro. I collect Netflow V9 and then sent
> it back in V5 to a server.
>
> I need to blacklist some networks, so i used the « --black-list »
> argument, but it does not seems to works.
>
>  
>
> Here is the command i use :
>
> nprobe -n udp://10.11.1.140:2055 -i none -t 20 -d 20 -a 0 -e 1 -b
> 2 -w 128000 -z 0 -S 1:1 -u 1 -Q 1 -3 9995
> --zmq tcp://127.0.0.1:5556 -V5 -G --black-list
> 10.7.0.0/16,10.1.0.0/16,10.11.0.0/16,192.168.0.0/16
>
>  
>
> And here is somes logs of networks that i dont want to send back
> to my server :
>
> 23/May/2016 09:55:43 [engine.c:2541] Emitting Flow:
> [->][icmp] *10.1.1.104:2048 -> 10.2.1.41:0 *[1 pkt/60 bytes][ifIdx
> 22273->111][0.0 sec][ECHO REPLY][init Unknown][AS: 0 -> 0]
>
> 23/May/2016 09:55:46 [engine.c:2568] Emitting Flow:
> [<-][icmp] *10.2.1.42:0 -> 10.1.1.48:2048* [2 pkt/120 bytes][ifIdx
> 111->22273][0.0 sec][AS: 0 -> 0]
>
> 23/May/2016 09:55:42 [engine.c:2361] New Flow:
> [icmp] *10.1.1.104:2048 -> 10.2.1.1:0* [00:00:00:00:00:00 ->
> 00:00:00:00:00:00][vlan 65535][tos 0][ifIdx: 22273 ->
> 111][subflowId: 0/0x][idx=69225]
>
>  
>
> What did i do wrong ?
>
>  
>
> Thanks for you help !
>
>  
>
> CRUCHADE Loïc
>
> 05.82.52.22.02
>
> Service Exploitation Informatique
>
> Direction des Systèmes d’information
>
> 
>
>  
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>  
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Nprobe black list network

[Luca Deri]

Loïc
I have just tested and it seems to work for me. What nprobe version are you 
using? I have tested the latest 7.3 release.

Please add a “ “ between the blacklist parameter to make sure the shell does 
not mess-up. If still not working, please file a bug at 
https://github.com/ntop/nProbe/issues

Regards Luca


> On 23 May 2016, at 10:10, Loic CRUCHADE  wrote:
> 
> Hello,
>  
> I recently bought Nprobe pro. I collect Netflow V9 and then sent it back in 
> V5 to a server.
> I need to blacklist some networks, so i used the « --black-list » argument, 
> but it does not seems to works.
>  
> Here is the command i use :
> nprobe -n udp://10.11.1.140:2055  -i none -t 20 -d 20 
> -a 0 -e 1 -b 2 -w 128000 -z 0 -S 1:1 -u 1 -Q 1 -3 9995 --zmq 
> tcp://127.0.0.1:5556  -V5 -G --black-list 
> 10.7.0.0/16,10.1.0.0/16,10.11.0.0/16,192.168.0.0/16
>  
> And here is somes logs of networks that i dont want to send back to my server 
> :
> 23/May/2016 09:55:43 [engine.c:2541] Emitting Flow: [->][icmp] 
> 10.1.1.104:2048 -> 10.2.1.41:0 [1 pkt/60 bytes][ifIdx 22273->111][0.0 
> sec][ECHO REPLY][init Unknown][AS: 0 -> 0]
> 23/May/2016 09:55:46 [engine.c:2568] Emitting Flow: [<-][icmp] 10.2.1.42:0 -> 
> 10.1.1.48:2048 [2 pkt/120 bytes][ifIdx 111->22273][0.0 sec][AS: 0 -> 0]
> 23/May/2016 09:55:42 [engine.c:2361] New Flow: [icmp] 10.1.1.104:2048 -> 
> 10.2.1.1:0 [00:00:00:00:00:00 -> 00:00:00:00:00:00][vlan 65535][tos 0][ifIdx: 
> 22273 -> 111][subflowId: 0/0x][idx=69225]
>  
> What did i do wrong ?
>  
> Thanks for you help !
>  
> CRUCHADE Loïc
> 05.82.52.22.02
> Service Exploitation Informatique
> Direction des Systèmes d’information
> 
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Ouch! Wiped Out My Old OS

[Luca Deri]

Kevin
for the nBox a factory reset means to set things like IP address etc. not to 
wipe the OS. Pur tools are just packages not an OS, so you do noted to modify 
the OS

Regard Luca

> On 09 May 2016, at 22:53, Kevin Kleinfelter  wrote:
> 
> I didn't RTFM closely enough.  I installed ntopng and nbox.  It said to run a 
> Factory Reset, so I ran a Factory Reset.  I wasn't planning on wiping out my 
> old OS, but I did.
> 
> I'd like to not repeat that mistake, but I'd also like to get all the 
> goodness of web-based analysis of rflow data.
> 
> Was my key mistake in installing nbox?  Can I safely install ntopng after 
> rebuilding my machine and not have it wipe out my OS?
> thanks,
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe and ntopng - Collecting sFlow Data

[Luca Deri]

Karl,
can you please add “-b 2” to nProbe to see if flows are properly collected? 
Please note that we need packet samples in sflows (not just bytes and counters)

Regard Luca

> On 06 May 2016, at 09:35, Karl van der Schyff  wrote:
> 
> Hello
> 
> I have been trying to get these two products to collect sFlow data from our 
> Dell switches, but just can't get it working. I am currently running nProbe 
> with the following arguments:
> 
> nprobe --zmq tcp://127.0.0.1:5556  -i none -n none 
> --collector-port 2055 -G
> 
> and ntopng with these arguments:
> 
> ntopng -G /var/tmp/ntopng.pid -e -i tcp://127.0.0.1:5556 
>  -w 3000 -m 192.168.1.0/24  -n 
> 1 -d /var/tmp/ntopng -q
> 
> When I log into ntopng I get absolutely nothing. When i run tcpdump -v I also 
> seem to only see traffic actually destined for the machine I am running 
> nProbe on. Assuming that the switch is correctly configured should tcpdump be 
> showing me the sflow traffic? If so, my assumption is incorrect.
> 
> Any help would be appreciated.
> 
> Thanks
> Karl
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Nprobe and Omniswitch

[Luca Deri]

Timo,
please file a bug on github and attach a pcap file with some flows so we can 
see what’s the problem

Thanks Luca

> On 23 Apr 2016, at 19:04, Timo Ylikännö  wrote:
> 
> Hi
> 
> After many days of tweaking it seems to me that nprobe does not work with 
> Alcatel Omniswitch switches. If I try with other sflow analyzer (for example 
> sFlowTrend) everything works fine and I can monitor my network traffic. 
> 
> Does anyone got nprobe and Omniswitch combo work? 
> 
> Timo
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] pf_ring hardware filter question

[Luca Deri]

Chris
you can set rules via the PF_RING API: did you see 
https://github.com/ntop/PF_RING/blob/dev/userland/examples/pffilter_test.c ?

Regards Luca

> On 30 Mar 2016, at 21:12, Clark, Erik J  wrote:
> 
> All;
>   I am trying to filter out tcp and udp traffic at the kernel level via 
> pf_ring, but can not find any documentation as to how to actually craft a 
> rule, or how you would make one persist. The only reference I can find is to
>  
> /proc/net/pf_ring/dev/${interface}/rules
>  
> Which would not be persistent. If I wanted to filter out all tcp 443 traffic 
> before handing it off to the application layer, say for Snort or Bro, how do 
> I do that at the pf_ring level persistently? Thanks much!
>  
> Erik
>  
>  
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Current most production-ready PF_RING

[Luca Deri]

Andrew
yes it is the stable branch

Luca

On 03/29/2016 12:44 PM, Andrew Howard wrote:
>
> Hi All,
>
> We've been using a snapshot of the git master head (taken 12th Nov
> 2015), and would now like to deploy the current most stable and
> production-ready version.
>
> Would this be the 6.2.0-stable branch updated 7th March (commit b95d67f) ?
>
> Thanks,
> A.
> --
> Andrew Howard
> a...@andrew-howard.me.uk
>
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe packaging - missing dependency

[Luca Deri]

Carsten
can you please try again with the new package just built?

Luca

On 03/22/2016 05:31 PM, InterNetX - Carsten Schoene wrote:
> Hi,
>
> nprobe package from rpm stable repo is missing dependency to 
> libnetfilter_queue on Centos 7.
>
> host:~$ nprobe --version
> nprobe: error while loading shared libraries: libnetfilter_queue.so.1: cannot 
> open shared object
> file: No such file or directory
>
> host:~$ rpm -q nprobe
> nprobe-7.2.160315-4472.x86_64
>
> host:~$ rpm -q nprobe --requires
> pfring = 6.2.0-487
> glibc >= 2.3.4
> mariadb-libs >= 5.5
> tcp_wrappers-libs >= 7.6
> GeoIP >= 1.4.8
> redis >= 2.4.10
> hiredis >= 0.10.1
> mysql
> mysql-libs
> hiredis
> redis
> zeromq
> numactl
> coreutils
> ntopng-data
> /bin/sh
> rpmlib(FileDigests) <= 4.6.0-1
> rpmlib(PayloadFilesHavePrefix) <= 4.0-1
> rpmlib(CompressedFileNames) <= 3.0.4-1
> rpmlib(PayloadIsXz) <= 5.2-1
>
> host:~$ cat /etc/redhat-release
> CentOS Linux release 7.2.1511 (Core)
>
> running 'yum -y install libnetfilter_queue' fixes this, but i think 
> libnetfilter_queue should be in
> Requires list of the nprobe rpm package.
>
> Regards
> Carsten Schöne
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe install on centos

[Luca Deri]

Hi Eduardo
the library should be installed as dependency. What is the version you are 
installing the stable or dev?

Luca

> On 09 Mar 2016, at 16:35, Eduardo  wrote:
> 
> Hi Folks,
> 
> I am on centos 6.7 64bit, installed the nprobe binary but it's missing
> a library:
> 
> nprobe: error while loading shared libraries: libnetfilter_queue.so.1:
> cannot open shared object file: No such file or directory
> 
> ldd confirmed ... it's the only one missing ...
> 
> then I tried to compile ... autogen finishes with this error:
> 
> 8. Downloading nDPI...
> nDPI already available
> 8.1 Compiling nDPI...
> ./autogen.sh: line 478: ./configure: No such file or directory
> make: *** No targets specified and no makefile found.  Stop.
> nDPI compilation failed: please check errors
> 
> Any ideias ?
> 
> thanks!
> 
> -Ed
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Support for Python

[Luca Deri]

Ajit
for ntopng we support only Lua. ntop is no longer supported

Luca

On 02/25/2016 10:26 AM, Ajit Sarnaik wrote:
> Hello Luca,
>
> There are C APIs, similarly any python interfaces. Which is the right
> link to the ntop and interface script please. The once in the document
> return 404.
>
> Regards,
>
> Ajit
>
> On Thu, Feb 25, 2016 at 1:12 AM, Luca Deri <d...@ntop.org
> <mailto:d...@ntop.org>> wrote:
>
> Ajit
> support for Python where?
>
> Luca
>
>
> On 02/25/2016 10:07 AM, Ajit Sarnaik wrote:
>> Hello,
>>
>> New to this list. Any body know if there is support for python
>> please.
>>
>> Regards,
>>
>> -- 
>> Ajit Sarnaik
>>
>>
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it
>> <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
>
> -- 
> Ajit Sarnaik
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Is it possible to disable pf_ring.ko from being loaded by ntopng or nprobe upon invokation

[Luca Deri]

> On 28 Jan 2016, at 03:48, Morgan Yang  wrote:
> 
> Hi:
> 
> Ntopng and Nprobe would "insmod pf_ring.ko" upon starting. Is there a command 
> line or configuration option to disable that?
> 
> Currently, we wish to disable until a pending feature requests are 
> implemented.https://github.com/ntop/ntopng/issues/265 
> 
> https://github.com/ntop/nProbe/issues/5 
> Implemented. packages will be 
> updated later today

Luca
> 
> I have been removing the actual pf_ring.ko file from the file system, but 
> everysingle time our customers perform "yum update", it comes back. 
> Blacklisting the module only blocks it from being loaded upon boot up.
> 
> Much Thanks
> Morgan Yang
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Nprobe Dump max file size of 1.6 MB

[Luca Deri]

Hi Ohad,
did you try this

max-log-lines| Maximum number of lines on a dump file. 
Default: 1.
  

Regards Luca

> On 19 Jan 2016, at 06:42, Ohad Kleinman  wrote:
> 
> We are utilizing nProbe Pro v.7.3.160104 ($Revision: 4767 $) on Ubuntu, the 
> main purpose is dump text flows and analyzing them offline.
> The dump files frequency are every 60 seconds, currently our test environment 
> is generating massive flows and as a result of that the dump files size are 
> above 1.6 MB. It seems that this is the max size of dump files and the nprobe 
> is generating above that size an additional file under the same 60 seconds 
> duration.
>  
> Is there a way to define a larger file size for nprobe?
>  
> Thank you.
>  
>  
> Ohad
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Suricata and PF_RING ZC

[Luca Deri]

Hi Mark,
the problem you reported should have been fixed in the current PF_RING that is 
in git: please update.

We have sent to OISF people various patches some of which have been included in 
their repository and others are pending since months (e.g. we have implemented 
IPS mode over PF_RING, https://github.com/inliniac/suricata/pull/1587). 
Unfortunately like you have seen those guys are unresponsive sometime, so all 
we can do is on the PF_RING side

Regards Luca

> On 15 Jan 2016, at 08:54, Mark Stingley  wrote:
> 
> I posted this to the OISF list, but thought I would check here to see
> if anyone has solved this already.
> 
> To me, the below looks like Suricata is looking for old style DNA and
> not the new PF_RING ZC way of doing things.
> 
> Opinions?
> 
> Thanks.
> 
> -
> 
> I just tried this on the latest git of pf_ring and Suricata 2.0.11,
> but had the same problem with Suricata 2.0.8 and pf_ring 6.0.3.  Error
> output and configuration data below.
> 
> Has anyone gotten Suricata to compile and work with pf_ring ZC?
> 
> Please advise.
> 
> Thanks.
> 
> -
> 
> gcc -DHAVE_CONFIG_H -I. -I..   -I./../libhtp/
> -I/usr/local/pfring/include -I/usr/include/nspr  -I/usr/include/nss
> -I/usr/include/nspr  -I/usr/include/luajit-2.0
> -DLOCAL_STATE_DIR=\"/var\" -g -O2 -Wextra
> -Werror-implicit-function-declaration -fno-tree-pre -Wall
> -Wno-unused-parameter -std=gnu99 -march=native -DHAVE_LIBNET11
> -D_BSD_SOURCE -D__BSD_SOURCE -D__FAVOR_BSD -DHAVE_NET_ETHERNET_H
> -DHAVE_LIBNET_ICMPV6_UNREACH -DHAVE_PFRING  -I/usr/local/include
> -DLIBPCAP_VERSION_MAJOR=1 -DHAVE_PCAP_SET_BUFF -DHAVE_LIBCAP_NG -MT
> runmode-erf-dag.o -MD -MP -MF .deps/runmode-erf-dag.Tpo -c -o
> runmode-erf-dag.o runmode-erf-dag.c
> In file included from source-pfring.h:31:0,
> from runmode-erf-dag.c:25:
> /usr/local/pfring/include/pfring.h:90:0: warning: "likely" redefined
> #define likely(x)   __builtin_expect((x),1)
> ^
> In file included from flow.h:31:0,
> from detect.h:29,
> from detect-engine-alert.h:29,
> from suricata-common.h:321,
> from runmode-erf-dag.c:18:
> util-optimize.h:32:0: note: this is the location of the previous definition
> #define likely(expr) __builtin_expect(!!(expr), 1)
> ^
> In file included from source-pfring.h:31:0,
> from runmode-erf-dag.c:25:
> /usr/local/pfring/include/pfring.h:91:0: warning: "unlikely" redefined
> #define unlikely(x) __builtin_expect((x),0)
> ^
> In file included from flow.h:31:0,
> from detect.h:29,
> from detect-engine-alert.h:29,
> from suricata-common.h:321,
> from runmode-erf-dag.c:18:
> util-optimize.h:35:0: note: this is the location of the previous definition
> #define unlikely(expr) __builtin_expect(!!(expr), 0)
> ^
> In file included from source-pfring.h:31:0,
> from runmode-erf-dag.c:25:
> /usr/local/pfring/include/pfring.h:184:5: error: unknown type name 
> ‘dna_device’
> dna_device dna_dev;
> ^
> /usr/local/pfring/include/pfring.h:185:5: error: unknown type name 
> ‘dna_indexes’
> dna_indexes *indexes_ptr;
> ^
> /usr/local/pfring/include/pfring.h:188:5: error: unknown type name
> ‘dna_device_operation’
> dna_device_operation last_dna_operation;
> ^
> Makefile:1379: recipe for target 'runmode-erf-dag.o' failed
> make[3]: *** [runmode-erf-dag.o] Error 1
> make[3]: Leaving directory '/usr/local/src/suricata-2.0.11/src'
> Makefile:925: recipe for target 'all' failed
> make[2]: *** [all] Error 2
> make[2]: Leaving directory '/usr/local/src/suricata-2.0.11/src'
> Makefile:446: recipe for target 'all-recursive' failed
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory '/usr/local/src/suricata-2.0.11'
> Makefile:375: recipe for target 'all' failed
> make: *** [all] Error 2
> 
> 
> CONFIGURE OUTPUT---
> suricata-2.0.11# LIBS="-lrt -lnuma" ./configure --prefix=/usr
> --sysconfdir=/etc --localstatedir=/var --enable-luajit --enable-pfring
> --with-libpfring-includes=/usr/local/pfring/include
> --with-libpfring-libraries=/usr/local/pfring/lib
> 
> Suricata Configuration:
>  AF_PACKET support:   yes
>  PF_RING support: yes
>  NFQueue support: no
>  NFLOG support:   no
>  IPFW support:no
>  DAG enabled: no
>  Napatech enabled:no
>  Unix socket enabled: yes
>  Detection enabled:   yes
> 
>  libnss support:  yes
>  libnspr support: yes
>  libjansson support:  yes
>  Prelude support: no
>  PCRE jit:yes
>  LUA 

Re: [Ntop-misc] nprobe and Cisco 4948E Netflow-Lite

[Luca Deri]

Hi Andrey
you do not need to do "--interpret-flow-packets --debug” as this are only for 
debugging.

For NFlite you need to use the NFlite plugin (as NFLite flows are called 
NetFlow…. but the name is misleading), so something like

nprobe -i none --nflite 2055 -b 2

Cheers Luca
 

> On 11 Jan 2016, at 16:03, Andrzej Miesiak  wrote:
> 
> Hello,
> 
> I try to use nprobe as Cisco Netflow-Lite aggregator with C4849E
> switch; nprobe is receiving templates and packet samples, but no flows
> are exported. Is there anyone using this feature? My switch config and
> nprobe debug output:
> 
> !
> netflow-lite exporter nprobe
> transport udp 2055
> template data timeout 60
> options sampler-table timeout 60
> options interface-table timeout 60
> source 
> destination 
> netflow-lite sampler sampl
> packet-rate 32
> !
> interface GigabitEthernet1/1
> netflow-lite monitor 1
>   sampler sampl
>   exporter nprobe
> 
> 
> # nprobe --nflite 2055 -3 2055 -i none -n 127.0.0.1:9996 -V 10
> --interpret-flow-packets --debug -b 2 -t 15
> (...)
> 11/Jan/2016 21:11:25 [nflitePlugin.c:900] [NFLite] Created UDP socket
> [# sockets: 1]
> 11/Jan/2016 21:11:25 [nflitePlugin.c:904] [NFLite] Listening on port
> range 2055-2055 (1)
> 11/Jan/2016 21:11:25 [nflitePlugin.c:914] [NFLite] Initialized
> NetFlow-Lite plugin
> (...)
> 11/Jan/2016 21:12:08 [collect.c:1081] > Defined flow template
> [id=304][flowLen=104][fieldCount=11]
> 11/Jan/2016 21:12:08 [collect.c:1096] Moving 44 bytes forward: new
> offset is 72 [stillToProcess=0]
> 11/Jan/2016 21:12:11 [collect.c:1750] NETFLOW_DEBUG: Received 128 bytes flow
> 11/Jan/2016 21:12:11 [collect.c:833] [displ=20][01 30 00]
> 11/Jan/2016 21:12:11 [collect.c:1109] Found FlowSet [displ=20]
> 11/Jan/2016 21:12:11 [collect.c:1154] > Rcvd flow with known
> template 304 [24...108]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=24/108][template=304][fieldId=312][fieldLen=2][isPenField=0][field=0/11]
> [24...128] [accum_len=0] [00 D4 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=26/108][template=304][fieldId=319][fieldLen=8][isPenField=0][field=1/11]
> [26...128] [accum_len=2] [00 00 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=34/108][template=304][fieldId=318][fieldLen=8][isPenField=0][field=2/11]
> [34...128] [accum_len=10] [00 00 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=42/108][template=304][fieldId=277][fieldLen=2][isPenField=0][field=3/11]
> [42...128] [accum_len=18] [00 01 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=44/108][template=304][fieldId=138][fieldLen=4][isPenField=0][field=4/11]
> [44...128] [accum_len=20] [00 00 00 01]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=48/108][template=304][fieldId=302][fieldLen=2][isPenField=0][field=5/11]
> [48...128] [accum_len=24] [00 01 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=50/108][template=304][fieldId=167][fieldLen=4][isPenField=0][field=6/11]
> [50...128] [accum_len=26] [00 00 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=54/108][template=304][fieldId=10][fieldLen=4][isPenField=0][field=7/11]
> [54...128] [accum_len=30] [00 00 00 01]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=58/108][template=304][fieldId=14][fieldLen=4][isPenField=0][field=8/11]
> [58...128] [accum_len=34] [00 00 00 00]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=62/108][template=304][fieldId=103][fieldLen=2][isPenField=0][field=9/11]
> [62...128] [accum_len=38] [00 40 33 33]
> 11/Jan/2016 21:12:11 [collect.c:1208] > Dissecting flow field
> [optionTemplate=0][displ=64/108][template=304][fieldId=104][fieldLen=64][isPenField=0][field=10/11]
> [64...128] [accum_len=40] [33 33 00 00]
> 11/Jan/2016 21:12:14 [collect.c:1750] NETFLOW_DEBUG: Received 72 bytes flow
> ^C
> 11/Jan/2016 21:13:32 [cache.c:1210] Redis Cache [0 total/0.0
> get/sec][0 total/0.0 set/sec]
> 11/Jan/2016 21:13:32 [nprobe.c:394] Received shutdown request... [signal: 2]
> (...)
> 11/Jan/2016 21:13:35 [plugin.c:285] Terminating Netflow-Lite Plugin
> (...)
> 11/Jan/2016 21:13:35 [nprobe.c:2505] Processed packets: 33 (max bucket
> search: 0)
> 11/Jan/2016 21:13:35 [nprobe.c:2488] Fragment queue length: 0
> 11/Jan/2016 21:13:35 [nprobe.c:2514] Flow export stats: [0 bytes/0
> pkts][0 flows/0 pkts sent]
> 11/Jan/2016 21:13:35 [nprobe.c:2521] Flow collection: [collected pkts:
> 148][processed flows: 0]
> 11/Jan/2016 21:13:35 [nprobe.c:2524] Flow drop stats:   [0 bytes/0
> pkts][0 flows]
> 11/Jan/2016 21:13:35 [nprobe.c:2529] Total flow stats:  [0 

Re: [Ntop-misc] User Agent ntopng

[Luca Deri]

Spencer,
this is something that will happen soon, but that is not yet implemented. 
Likely next week we will integrate HTTP (URL and Host) and DNS (query) support. 
I have opened an issue you can track
- https://github.com/ntop/ntopng/issues/346
- https://github.com/ntop/ntopng/issues/347
- https://github.com/ntop/ntopng/issues/348

Regards Luca

> On 08 Jan 2016, at 00:56, Spencer Lowe  wrote:
> 
> I am looking at purchasing the "nProbe Pro with Plugin Support" and sending 
> data to ntopng. I am looking at also getting the HTTP Plugin for nProbe, my 
> question is does ntopng show data pertaining to what was gathered using the 
> HTTP plugin like user agent and domain?
> 
> Thanks!
> Spencer   
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] n2disk

[Luca Deri]

Hi Mark,
we have fixed the dependency and a new package is currently being build (it 
will be available within 20 mins)

Regards Luca

> On 31 Dec 2015, at 04:03, Mark Stingley  wrote:
> 
> I attempted to install n2disk on an AMD64 Debian Jessie system via:
> 
> n2disk_2.3.151230-4473_amd64.deb
> 
> Installation failed with the following error:
> 
> n2disk depends on libnl1; however:
>  Package libnl1 is not installed.
> 
> Please note that libnl1 is no longer in distribution.  The only
> packages listed for libnl are in the "libnl-3" series.  libnl1 last
> appeared in the wheezy distribution.
> 
> I'm open to suggestions on how to proceed.
> 
> Thanks.
> 
> Mark
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nprobe support for Cisco WLC netflow export

[Luca Deri]

Hi Yasser,
please file an issue request on https://github.com/ntop/nProbe/issues, attach a 
pcap file (flows + templates) full packet size, and we’ll see what we can do.

Please also specify also the whole command line you have used to start nProbe

Thanks Luca

> On 29 Dec 2015, at 10:39, Yasser Slarmie  wrote:
> 
> Hello everyone
> 
> After reading through the following blog post:
> 
> http://mrncciew.com/2013/02/13/who-really-support-wlc-netflow/ 
> 
> 
> It seems that Cisco sends these unique fields in their Wireless LAN 
> Controller netflow v9 exports:
> 
> • applicationTag
> • ipDiffServCodePoint
> • octetDeltaCount
> • packetDeltaCount
> • postIpDiffServCodePoint
> • staIPv4Address
> • staMacAddress
> • wlanSSID
> • wtpMacAddress
> 
> How do I go about getting these fields incorporated into nprobe? Currently, 
> ntopng shows zero data for the received netflow packets coming from the WLC.
> 
> Regards,
> Yasser
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] How can I output the traffic log from nDPI?

[Luca Deri]

Yes
> On 22 Dec 2015, at 11:50, James Cheng <jih...@gmail.com> wrote:
> 
> Thanks Luca,
> But we would like to extract the application, such as skype. Can ntopng or 
> nProbe do that?
> Cheers,
> James
> 
> On Tue, Dec 22, 2015 at 6:43 PM, Luca Deri <d...@ntop.org 
> <mailto:d...@ntop.org>> wrote:
> James
> ntopng or nProbe do that
> 
> Luca
> 
> > On 22 Dec 2015, at 11:31, James Cheng <jih...@gmail.com 
> > <mailto:jih...@gmail.com>> wrote:
> >
> > Dears,
> >
> > I would like to output the traffic log from nDPI. Is it possible? and How 
> > to do that?
> > The output log might include the source ip, destination ip, protocol, 
> > destination URL/URI, etc.
> >
> > Thanks for advise,
> > James
> > ___
> > Ntop-misc mailing list
> > Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] How can I output the traffic log from nDPI?

[Luca Deri]

James
ntopng or nProbe do that

Luca

> On 22 Dec 2015, at 11:31, James Cheng  wrote:
> 
> Dears,
> 
> I would like to output the traffic log from nDPI. Is it possible? and How to 
> do that?
> The output log might include the source ip, destination ip, protocol, 
> destination URL/URI, etc.
> 
> Thanks for advise,
> James
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] run nprobe as a linux service

[Luca Deri]

Dieter
I believe you are using a custom startup script

Please use our init.d file that it expects a config files with the following 
format

-T=
-n=none


If you want to avoid headache, please use the nbox configuration GUI

Regards Luca

Sent from my iPad

> On 21 Dec 2015, at 15:30, Dieter Gerhard  wrote:
> 
> Hi Luca,
> 
> i try to run nprobe as a service.
> From the command line, i start it with the parameters:
> 
> /usr/local/bin/nprobe -W -T " %EXPORTER_IPV4_ADDRESS %IPV4_SRC_ADDR 
> %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES 
> %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %PROTOCOL %SRC_AS 
> %DST_AS %IPV4_SRC_MASK 
> %IPV4_DST_MASK" -nnone --collector-port 2055 -B 1 -V 9 --max-num-flows 
> 5242880 --in-iface-idx -1 --out-iface-idx -1 
> --mysql=127.0.0.1:netflow:raw:netflow:blabla &
> 
> It works fine.
> 
> If i insert this in the "options=" section from the /etc.Iinit.d/nprobe.sh 
> file, it dont works.
> 
> if i try a /etc/init.d/nprobe restart
> 
> i see  this:
> 
> /etc/init.d/nprobe.sh: line 21: fg: no job control
> Restarting nProbe Done.
> 
> the nprobe service is not running.
> 
> The OS is  centos 6.7
> 
> 
> you have any idea ?
> 
> 
> 
> Mit freundlichen Grüßen / Kind regards
> 
> Dieter Gerhard
> 
> Senior Network Engineer Network Analysis
> MCITP,WCNA,CCNA
> Datacenter Network Aviation Industry Services
> FRA AI/D-L, LIC-B ,Room 6.1.52
> Phone:+49 (0)69 / 696-96963IBM Deutschland Aviation Industry 
> Services GmbH
> 
> Fax:  +49 (0)69 / 696- 98 96963Am Weiher 24
> Mobile:   +49 (0)151 58921919  65451 Kelsterbach
> Email:dieter.gerh...@de.ibm.comGermany
> IBM Deutschland Aviation Industry Services GmbH / Geschäftsführung: Claus 
> Rohde, Jürgen Schwarz, Dirk Weigel
> Sitz der Gesellschaft: Kelsterbach / Registergericht: Amtsgericht Darmstadt, 
> HRB 94232
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Complex Proxy Config

[Luca Deri]

> On 18 Dec 2015, at 17:04, Erik Schmersal  wrote:
> 
> I am interested in ways to set up a complex proxy config as follows:
> 
> Router 1 exports flows on port 2055, I would like to proxy them to collector 
> A as version 5, collector B as version 9 and collector C as version 10
this is not possible. You need to use as collector a tool such as flow-tools 
that can fanout flows to three different nprobe instances each configured for 
your collectors.
> 
> Router 2 exports flows on port 2060. I would like to proxy them in the same 
> manner.
same as above
> 
> I need to keep the source address/port of each routers' flow data unique.
> 
you can on Linux (see nprobe -h)

Regards Luca

> Thanks,
> 
> Erik
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Tunnel option applied only for 25% of packets

[Luca Deri]

Hi Gregoire,
please file a bug on https://github.com/ntop/nProbe 
 and attach a pcap file for reproducing it

Regards Luca

> On 17 Dec 2015, at 15:21, gregoire.le...@retenodus.net wrote:
> 
> Hello,
> 
> I want to test nprobe stable on CentOS6 (v.7.2.151211) and I have an issue 
> with nprobe and L2TP tunnelled traffic. Here is the command I launch :
> 
> [root@netflow-linux ~]# nprobe -i eth1 -d 60 -P /tmp/flows -D t -I sfr -T 
> "%IN_BYTES %IN_PKTS %L4_SRC_PORT %PROTOCOL %IPV4_SRC_ADDR %L4_DST_PORT 
> %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_MASK %UPSTREAM_TUNNEL_ID 
> %DOWNSTREAM_TUNNEL_ID" -V 9 --smart-udp-frags -N 0 --tunnel
> 
> I'd expect to get records like
> "122|1|53|17|IP_IN_TUNNEL|13217|IP_IN_TUNNEL|::|0|54B5|B5AB|
> 117|2|443|6|IP_IN_TUNNEL|53820|IP_IN_TUNNEL|::|0|6304|BB56|
> "
> I get some of them, but most of my records are not correctly decapsulated and 
> I usually get records like that :
> 
> 52|1|30753|17|L2TP_IP|49752|L2TP_IP|::|0|||
> 52|1|4560|17|L2TP_IP|34232|L2TP_IP|::|0|||
> 
> As you can see, L4_SRC_PORT and L4_DST_PORT are correctly decapsulated. 
> However, I neither get the tunneled IP address or the tunnel informations (I 
> obfuscated IP informations, replacing them with IP_IN_TUNNEL and L2TP_IP). 
> ~75% of flows are concerned.
> 
> I am pretty sure the problem comes from the decapsulation and it's not a 
> false positive as if it was, src port and dest port would be 1701.
> 
> When I try to use it in debug mode I get a segfault (which I don't get 
> without the --tunnel option) :
> 
> [root@netflow-linux ~]# nprobe -i eth1 -d 60 -P /tmp/flows -D t -I sfr -T 
> "%IN_BYTES %IN_PKTS %L4_SRC_PORT %PROTOCOL %IPV4_SRC_ADDR %L4_DST_PORT 
> %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_MASK %UPSTREAM_TUNNEL_ID 
> %DOWNSTREAM_TUNNEL_ID %UNTUNNELED_IPV4_SRC_ADDR" -V 9 --smart-udp-frags -N 0 
> --debug --tunnel
> 17/Dec/2015 16:19:38 [nprobe.c:3114] ERROR: Invalid nProbe license 
> (/etc/nprobe.license) [Missing license file]
> 17/Dec/2015 16:19:38 [nprobe.c:3121] ERROR: 
> *
> 17/Dec/2015 16:19:38 [nprobe.c:3122] ERROR: **
>  **
> 17/Dec/2015 16:19:38 [nprobe.c:3123] ERROR: **  Switching to DEMO MODE 
> (missing valid license) **
> 17/Dec/2015 16:19:38 [nprobe.c:3124] ERROR: **
>  **
> 17/Dec/2015 16:19:38 [nprobe.c:3125] ERROR: **  Create your nProbe license at 
>  **
> 17/Dec/2015 16:19:38 [nprobe.c:3126] ERROR: **   
> http://www.nmon.net/mklicense/**
> 17/Dec/2015 16:19:38 [nprobe.c:3127] ERROR: **
>  **
> 17/Dec/2015 16:19:38 [nprobe.c:3128] ERROR: 
> *
> 17/Dec/2015 16:19:38 [nprobe.c:6508] ERROR: 
> ***
> 17/Dec/2015 16:19:38 [nprobe.c:6509] ERROR: * NOTE: This is a DEMO version 
> limited to 25000 flows export.  *
> 17/Dec/2015 16:19:38 [nprobe.c:6510] ERROR: 
> ***
> 17/Dec/2015 16:19:38 [plugin.c:166] No plugins found in ./plugins
> 17/Dec/2015 16:19:38 [plugin.c:174] Loading 22 plugins [.so] from 
> /usr/local/lib/nprobe/plugins
> datagramSourceIP 0.0.0.0
> datagramSize 48
> unixSecondsUTC 1450365578
> datagramVersion 5
> agentSubId 0
> agent 192.168.1.1
> packetSequenceNo 1084445
> sysUpTime 2429093100
> samplesInPacket 4
> startSample --
> sampleType_tag 0:2
> sampleType COUNTERSSAMPLE
> sampleSequenceNo 187645
> sourceId 0:1
> counterBlock_tag 2176:0
> skipping unknown counters_sample_element: 2176:0 len=0
> counterBlock_tag 568615:598
> skipping unknown counters_sample_element: 568615:598 len=0
> endSample   --
> unexpected end of datagram after sample 1 of 4
> datagramSourceIP 0.0.0.0
> datagramSize 48
> unixSecondsUTC 1450365578
> datagramVersion 5
> agentSubId 0
> agent 192.168.1.1
> packetSequenceNo 1084446
> sysUpTime 2429093100
> samplesInPacket 10
> startSample --
> sampleType_tag 0:1
> sampleType FLOWSAMPLE
> sampleSequenceNo 11443
> sourceId 0:2
> meanSkipCount 50
> samplePool 8912896
> dropEvents 0
> inputPort multiple 181563990
> outputPort 0
> flowBlock_tag 0:0
> skipping unknown flow_sample_element: 0:0 len=-2147483648
> Segmentation fault
> 
> When I compare with what I get in a pcap, I can see that in my pcap file I 
> almost don't get any packet
> 
> Is there a performance issue (it doesn't seem so, CPU stays low) ? Is there a 
> fix somewhere, or did I miss something ?
> 
> Thank you very much,
> Regards,
> Grégoire
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___

Re: [Ntop-misc] how to correct the direction of long-lived tcp connection

[Luca Deri]

Alan,
unfortunately nprobe does not offer other solutions, and it’s your app to do 
that

Regards Luca

> On 15 Dec 2015, at 11:06, Wang  wrote:
> 
> Dear all,
> nprobe will consider the first packet it observes with the direction of src 
> --> dst if no packet with SYN flag observed for this connection. So, there is 
> some possibility for nprobe to export flow with wrong direction. This is not 
> a big deal for short-lived connections, but it would be a different story for 
> long-lived connections. Is there any way to correct this if the ip address of 
> servers are already known?
> 
> Alan
> 
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] ntopng active flows for network element says No Results Found

[Luca Deri]

Ohad,
(said that you better move this issue to github for tracking issues) can you 
please check the browser javascript console and see if the .lua file that 
returns the JSON to the page does not contain errors?

Regards Luca

> On 14 Dec 2015, at 16:59, Ohad Kleinman  wrote:
> 
> We are using the latest version of ntopng v.2.3.151213, when trying to view 
> any of the host active flows I get   No Results Found.
> All other tabs with info is working properly, is this a bug or I am missing 
> anything? I know this option used to work in the previous versions.
> I have attached a screen shot of the problem we encounter.
> https://dl.dropboxusercontent.com/u/27973370/ntopngactiveflows.jpg 
> 
>  
>  
>  
> Thanks.
>  
> Ohad
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Ohad
I am unable to see such file on my setup. Can you please send me a
portion of this log? Are you sure nprobe is creating it?

Regards Luca

On 02/12/2015 08:47, Ohad Kleinman wrote:
>
> Hi Luca,
>
> The log file that I am referring to is nprobe-e...@0.log
> <mailto:nprobe-e...@0.log> located in /var/log/nprobe directory.
>
>  
>
> Yes currently we are using both dumping files in text format and also
> to the elastic search.
>
>  
>
> I hope this helps.
>
>  
>
> Ohad
>
>  
>
> *From:*ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>
> [mailto:ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] *On Behalf Of *Luca Deri
> *Sent:* Wednesday, December 02, 2015 9:37 AM
> *To:* ntop-misc@listgateway.unipi.it
> <mailto:ntop-misc@listgateway.unipi.it>
> *Subject:* Re: [Ntop-misc] nProbe big log file with elastic search
>
>  
>
> Hi Ohad,
>
> using the latest nProbe I have been unable to reproduce the issue you
> reported. I have even started nprobe with strace just to make sure I
> didn’t miss anything but the .log file you mention is not created.
>
>  
>
> Instead using -P you are telling nprobe to dump flows in text format
> (in addition to pushing them to ES): is this what you want? 
>
>  
>
> Where is this .log file created? (path I mean)
>
>  
>
> Regards Luca
>
>  
>
> On 29 Nov 2015, at 13:04, Ohad Kleinman <oh...@vglnt.com
> <mailto:oh...@vglnt.com>> wrote:
>
>  
>
> Luca,
>
> Please see attached the configuration file that we are using.
>
>  
>
> Ohad
>
>  
>
> *From:* ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] *On Behalf
> Of *Luca Deri
> *Sent:* Sunday, November 29, 2015 12:22 PM
> *To:* ntop-misc@listgateway.unipi.it
> <mailto:ntop-misc@listgateway.unipi.it>
> *Subject:* Re: [Ntop-misc] nProbe big log file with elastic search
>
>  
>
> Ohad,
>
> nProbe should not write to this log. I think it is a combination
> of options we do not handle properly. Can you please let send me
> the complete command line you are using to start nProbe so I can
> analyse it?
>
>  
>
> Thanks Luca
>
>  
>
> On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com
> <mailto:oh...@vglnt.com>> wrote:
>
>  
>
> Hi Luca,
>
> Can you confirm if there is a way to make the nprobe
> to *not* write to the log file each flow that is being
>     exported to elastic?
>
>  
>
> Thanks
>
>  
>
> Ohad
>
>  
>
>  
>
> *From:* ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] *On Behalf
> Of *Luca Deri
> *Sent:* Monday, November 23, 2015 2:59 PM
> *To:* ntop-misc@listgateway.unipi.it
> <mailto:ntop-misc@listgateway.unipi.it>
> *Subject:* Re: [Ntop-misc] nProbe big log file with elastic search
>
>  
>
> Hi Ohad,
>
> is this file on the ElasticSearch side right?
>
>  
>
> Regards Luca
>
>  
>
> On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com
> <mailto:oh...@vglnt.com>> wrote:
>
>  
>
> Hi,
>
> We are using nProbe with the option of writing all flows
> into elastic search, the nprobe-e...@0.log
> <mailto:nprobe-e...@0.log> file is becoming large as each
> flow that is written into the elastic search is also being
> written into the log file.
>
>  
>
> Is there any parameter that can disable this?
>
>  
>
> Thanks.
>
>  
>
> Ohad
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>  
>
> __

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Hi Ohad,
using the latest nProbe I have been unable to reproduce the issue you reported. 
I have even started nprobe with strace just to make sure I didn’t miss anything 
but the .log file you mention is not created.

Instead using -P you are telling nprobe to dump flows in text format (in 
addition to pushing them to ES): is this what you want? 

Where is this .log file created? (path I mean)

Regards Luca

> On 29 Nov 2015, at 13:04, Ohad Kleinman <oh...@vglnt.com> wrote:
> 
> Luca,
> Please see attached the configuration file that we are using.
>  
> Ohad
>  
> From: ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] On Behalf Of Luca Deri
> Sent: Sunday, November 29, 2015 12:22 PM
> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> Subject: Re: [Ntop-misc] nProbe big log file with elastic search
>  
> Ohad,
> nProbe should not write to this log. I think it is a combination of options 
> we do not handle properly. Can you please let send me the complete command 
> line you are using to start nProbe so I can analyse it?
>  
> Thanks Luca
>  
>> On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com 
>> <mailto:oh...@vglnt.com>> wrote:
>>  
>> Hi Luca,
>> Can you confirm if there is a way to make the nprobe to not write to the log 
>> file each flow that is being exported to elastic?
>>  
>> Thanks
>>  
>> Ohad
>>  
>>  
>> From: ntop-misc-boun...@listgateway.unipi.it 
>> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
>> [mailto:ntop-misc-boun...@listgateway.unipi.it 
>> <mailto:ntop-misc-boun...@listgateway.unipi.it>] On Behalf Of Luca Deri
>> Sent: Monday, November 23, 2015 2:59 PM
>> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
>> Subject: Re: [Ntop-misc] nProbe big log file with elastic search
>>  
>> Hi Ohad,
>> is this file on the ElasticSearch side right?
>>  
>> Regards Luca
>>  
>>> On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com 
>>> <mailto:oh...@vglnt.com>> wrote:
>>>  
>>> Hi,
>>> We are using nProbe with the option of writing all flows into elastic 
>>> search, the nprobe-e...@0.log <mailto:nprobe-e...@0.log> file is becoming 
>>> large as each flow that is written into the elastic search is also being 
>>> written into the log file.
>>>  
>>> Is there any parameter that can disable this?
>>>  
>>> Thanks.
>>>  
>>> Ohad
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>>  
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Ohad,
nProbe should not write to this log. I think it is a combination of options we 
do not handle properly. Can you please let send me the complete command line 
you are using to start nProbe so I can analyse it?

Thanks Luca

> On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com> wrote:
> 
> Hi Luca,
> Can you confirm if there is a way to make the nprobe to not write to the log 
> file each flow that is being exported to elastic?
>  
> Thanks
>  
> Ohad
>  
>  
> From: ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] On Behalf Of Luca Deri
> Sent: Monday, November 23, 2015 2:59 PM
> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> Subject: Re: [Ntop-misc] nProbe big log file with elastic search
>  
> Hi Ohad,
> is this file on the ElasticSearch side right?
>  
> Regards Luca
>  
>> On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com 
>> <mailto:oh...@vglnt.com>> wrote:
>>  
>> Hi,
>> We are using nProbe with the option of writing all flows into elastic 
>> search, the nprobe-e...@0.log <mailto:nprobe-e...@0.log> file is becoming 
>> large as each flow that is written into the elastic search is also being 
>> written into the log file.
>>  
>> Is there any parameter that can disable this?
>>  
>> Thanks.
>>  
>> Ohad
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] ntopng remove syslog

[Luca Deri]

Morgen
clear now. please file an issue on https://github.com/ntop/ntopng/issues

Luca

> On 27 Nov 2015, at 23:04, Morgan Yang <morgan.yang1...@gmail.com> wrote:
> 
> Hi Luca:
> 
> Sorry my statement was incorrect. I've been using the "-D" option of ntopng 
> to write to a log locally on the system. But I wish to configure it to log to 
> a syslog collector elsewhere. For nprobe, there is a "--syslog" option, but I 
> don't see in the man page for ntopng.
> 
> Morgan
> 
> On Fri, Nov 27, 2015 at 10:42 AM, Luca Deri <d...@ntop.org 
> <mailto:d...@ntop.org>> wrote:
> Morgan
> if you start ntopng as a daemon messages go to syslog, then I think you can 
> configure a remote syslog, correct?
> 
> Regards Luca
> 
>> On 26 Nov 2015, at 08:20, Morgan Yang <morgan.yang1...@gmail.com 
>> <mailto:morgan.yang1...@gmail.com>> wrote:
>> 
>> That was a terrible typo, i meant to say "remote" syslog listener/collector.
>> 
>> On Wed, Nov 25, 2015 at 11:05 AM, Morgan Yang <morgan.yang1...@gmail.com 
>> <mailto:morgan.yang1...@gmail.com>> wrote:
>> Is there an option in CLI or via web interface to configure ntopng to log to 
>> a remove syslog listener?
>> 
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
> 
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Hi Ohad,
is this file on the ElasticSearch side right?

Regards Luca

> On 18 Nov 2015, at 15:34, Ohad Kleinman  wrote:
> 
> Hi,
> We are using nProbe with the option of writing all flows into elastic search, 
> the nprobe-e...@0.log  file is becoming large as 
> each flow that is written into the elastic search is also being written into 
> the log file.
>  
> Is there any parameter that can disable this?
>  
> Thanks.
>  
> Ohad
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Steps required to compile nDPI for MIPS

[Luca Deri]

Prateek,
I compile as usual ./configure + make on MIPS. I am not familiar with 
cross-compilers that you seem to use instead.

Regards Luca

> On 19 Nov 2015, at 13:32, PRATEEK MOHANTY  
> wrote:
> 
> Hi,
> 
> I am trying to compile nDPI package for MIPS processor. But facing issues at 
> the time of configuration. Can you please provide the steps(commands), it 
> will be very helpful.
> 
> I can see from the configuration log, it say cross-compile mode is not 
> selected. What is the parameter, I need to set with configure command so that 
> it will compile for mips not for X86.
> 
> command I am giving:
> 
> ./configure --target=mips-linux-uclibc CC=mips-linux-uclibc-gcc
> (Please tell me where I am doing wrong)
> 
> regards
> Prateek
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] The new guy on the block

[Luca Deri]

Sacha,
if nprobe is used as collector do

nprobe -n none -i none -3 2055 --zmq “tcp://*:5888 ”

Regards Luca

> On 15 Nov 2015, at 21:13, Sacha Yunusic  wrote:
> 
> Hi there, 
> I’m starting using ntopng and nprobe and we want to use it in production, so 
> I’m in the learning process.
> The lab I’m running has some boxes that send NetFlow v9 to the server where 
> I’ve running nprobe and ntopng, thru udp-2055:
> [root~]# tcpdump port 2055 -nnn
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
> 16:58:22.508489 IP 192.168.xxx.yyy.58136 > 192.168.zzz.www.2055: UDP, length 
> 1368
> 16:58:22.508529 IP 192.168.xxx.yyy.58136 > 192.168.zzz.www.2055: UDP, length 
> 692
>  
> 192.168.xxx.yyy is the box that sends Netflow, and 192.168.zzz.www is the 
> server we’re running nprobe and ntopng.
>  
> What I want is to capture that Netflow v9 traffic, send it to ntopng, so, 
> this is what I’m doing:
> # nprobe -n 127.0.0.1:2055 -i em1 --zmq "tcp://*:5888 " --redis 
> 127.0.0.1:6379 --flow-version 9
> I’m not sure how usefull/needed is to have Redis in here… but still…
> In this case, I see traffic, but only traffic I see in em1 (eth0) that is 
> sent directly to my probe server (not the netflow data), so I tried this:
>  
> # nprobe -n 127.0.0.1:2055 -i none --zmq "tcp://*:5888 " 
> --redis 127.0.0.1:6379 --flow-version 9
>  
> And there I don’t see any flows nor anything.
>  
> At the nprobeng part, this is what I do:
> # ntopng -i tcp://127.0.0.1:5888  --redis 
> 127.0.0.1:6379 --http-port 4000
>  
> What I’m doing bad?
>  
> Sacha Yunusic | Gerente Técnico | Pentagon Security & Akainix
> Av. Kennedy 4700, Piso 10, Of. 1002, Edificio New Century, Vitacura | Código 
> Postal (ZIP Code) 7630454
> Central: (56-2) 2246 1050 | Directo: (56-2) 2246 2620 | Cel: (56-9) 9883 4752 
> | www.penta-sec.com  & www.akainix.com 
> 
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] help on plugging in nDPI to Yet Another Flowmeter YAF

[Luca Deri]

Hi Manickam,
we're not familiar (from the programming point of view) of YAF, but if
you have questions on nDPI please feel free to ask

Regards Luca

On 10/26/2015 08:27 AM, Manickam wrote:
> Hi,
>
> I am using YAF as a flow generator and figured out that DPI engine
> inbuilt in YAF is not as powerful as nDPI. Hence i am thinking of
> making YAF code changes to interact with nDPI APIs. I am familiar with
> ndpiReader example which can run on PCAP store and detect the
> application. Just wondering how easy or difficult it will be to make
> YAF code changes.
>
> Anyone in the forum tried it before? If yes, any pointers on how to
> hook at a higher level will be of great help. Really appreciate your help.
>
> Thanks N Regards,
> Manickam
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Nprobe http plugin dump

[Luca Deri]

Ohad,
this is not possible as the format of the http dump is fixes. Please send me a 
pcap dump as example, file a bug on github, as we’ll follow up

Regards Luca

> On 11 Oct 2015, at 14:58, Ohad Kleinman  wrote:
> 
> Hi,
> We have installed nProbe v.7.2.150922 (r4468) on ubuntu 14.04 (64 but) and we 
> are trying to monitor a network with IP cameras.
> We are using the nProbe to send network information into elastic search along 
> with dump files into folder and to analyze the information, with the http 
> plugin we also dump logs into a folder 
>  
> Our config file contains the following parameters:
>  
> -n=none
> -i=eth1
> -s=128
> -t=60
> -d=60
> -a=0
> -e=1
> -B=10
> -w=128000
> -z=0
> -S=1:1
> -E=0:0
> -g=/var/run/nprobe-eth1.pid
> --vlanid-as-iface-idx=none
> -V=5
> --dump-stats=/var/log/nprobe/eth1-0_flows_stats.txt
> -T=%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_SRC_MAC %OUT_DST_MAC %L4_SRC_PORT 
> %L4_DST_PORT %IN_BYTES %OUT_BYTES %IN_PKTS %OUT_PKTS %FIRST_SWITCHED 
> %LAST_SWITCHED %L7_PROTO_NAME %PROTOCOL
> -D=t
> -P=/var/log/nprobe
> --http-dump-dir=/var/log/nprobe
> --elastic=nProbe;nprobe;http://10.0.1.64:9200/_bulk 
> 
> --timestamp-format 2
> --dont-nest-dump-dirs
>  
>  
> I could not have find one thing, a way to dump into the http log also the 
> actual data and not just the existing info. 
> The relevant information that we can see in the pcap files is located in the 
> envelope section when dealing with http/xml format and in the line-base text 
> data.
> Is this possible with the nProbe software?
>  
>  
> Thanks
>  
> Best Regards,
> Ohad Kleinman
>  
>  
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] Cisco ASA V9 flows into elasticsearch

[Luca Deri]

Victor
inserting them in ELK is not different from collector to probe mode. The
thing is that we transform ASA flows into the template specified by -T
and thus you will not see a 1:1 correspondence between collected and
stored flows in ELK

Luca

On 10/07/2015 03:20 PM, Victor Castro wrote:
> Hello,
>
> I'm looking for assistance in what I think is a simple nProbe
> configuration.
>
> I would like to export Cisco ASA NetFlow V9 flows from the ASA,
> through nProbe and into elasticsearch.  I've tried a number of
> combinations but I cannot seem to get a working configuration.
>
>
> ASA:
> IP: 10.1.1.1
> Netflow collector: 10.2.2.2:20555 
>
> nProbe:
> IP: 10.2.2.2
> Collector mode
> Collector port: 2055
>
> elasticsearch:
> IP: 10.2.2.2:9200 
>
>
> I have been able to get interface flows from eth0 on the nProbe box
> into elasticseearch.
> My issue is with nProbe listening on port 2055 and transforming the
> netflow v9 packets for export into elasticsearch.
>
>
> Can someone lend some assistance on how I would configure nprobe in
> collector or proxy mode to read the ASA V9 flows and export them to
> elasticsearch?
>
> Thanks
>
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc