Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-10 Thread Steve Clark 

Hi Paolo,

could you send me the output of ldd pmacctd so I can see the versions of the 
libraries being used?
also the ouput of pmacctd -V


Thanks,
Steve

On 07/09/2020 03:06 PM, Paolo Lucente wrote:

I did test on a Debian 10:

4.19.0-8-686-pae #1 SMP Debian 4.19.98-1 (2020-01-26) i686 GNU/Linux

As i was suspecting, passing the pcap you sent me through a daemon
compiled on this box went fine (that is, i can't reproduce the issue).
From what i see, by the way, this is not something related to nDPI.

Paolo

On 09/07/2020 18:19, Steve Clark wrote:


Thanks for checking, could you tell what distro and version you tested on?

Also when I compile on 32 bit I get a lot of warning of redefines
between ndpi.h and pmacct.h
do you get those also?




On 07/09/2020 11:55 AM, Paolo Lucente wrote:


Hi Steve,

I do have avail of a i686-based VM. I can't say everything is tested on
i686 but i tend to check every now and then that nothing fundamental is
broken. I took the example config you used, compiled master code with
the same config switches as you did (essentially --enable-ndpi) and had
no joy reproducing the issue.

You could send me privately your capture and i may try with that one
(although i am not highly positive it will be a successful test); or you
could arrange me access to your box to read the pcap. Let me know.

Paolo

On 09/07/2020 14:54, Steve Clark wrote:


Hi Paolo,

I have compiled master with nDPI on both 32bit and 64bit CentOS 6
systems. The 64 bit pmacctd seems
to work fine. But I get bogus byte counts when I run the 32bit version
against the same pcap file.

Just wondered if you have done any testing on 32bit intel system with
the above combination.

below is the output when using 32bit pmacctd - first the pmacctd
invocation then the nfacctd output
pmacct/src/pmacctd -f ./mypaolo.conf -I v1.7.5_v9_ndpi_class_paolo.pcap
INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
1.7.6-git (20200707-01)
INFO ( default/core ):  '--enable-ndpi'
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/var/lib/pgsql/sclark/mypaolo.conf'.
INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on
softflowd 0.9.7 software, Copyright 2002 Damien Miller 

All rights reserved.
INFO ( p4p1/nfprobe ):   TCP timeout: 3600s
INFO ( p4p1/nfprobe ):  TCP post-RST timeout: 120s
INFO ( p4p1/nfprobe ):  TCP post-FIN timeout: 300s
INFO ( p4p1/nfprobe ):   UDP timeout: 300s
INFO ( p4p1/nfprobe ):  ICMP timeout: 300s
INFO ( p4p1/nfprobe ):   General timeout: 3600s
INFO ( p4p1/nfprobe ):  Maximum lifetime: 604800s
INFO ( p4p1/nfprobe ):   Expiry interval: 60s
INFO ( default/core ): PCAP capture file, sleeping for 2 seconds
INFO ( p4p1/nfprobe ): Exporting flows to [172.24.109.157]:rrac
WARN ( p4p1/nfprobe ): Shutting down on user request.
INFO ( default/core ): OK, Exiting ...

src/nfacctd -f examples/nfacctd-print.conf.example
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.6-git
(20200623-00)
INFO ( default/core ):  '--enable-ndpi'
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/var/lib/pgsql/sclark/pmacct/examples/nfacctd-print.conf.example'.
INFO ( default/core ): waiting for NetFlow/IPFIX data on :::5678
INFO ( foo/print ): cache entries=16411 base cache memory=56322552 bytes
WARN ( foo/print ): no print_output_file and no print_output_lock_file
defined.
INFO ( foo/print ): *** Purging cache - START (PID: 21926) ***
CLASS SRC_IP
DST_IP SRC_PORT  DST_PORT
PROTOCOLPACKETS   BYTES
NetFlow   172.24.110.104
172.24.109.247 41900 2055
udp 26 1576253010996
NetFlow   172.24.110.104
172.24.109.247 58131 2055
udp 211576253008620
INFO ( foo/print ): *** Purging cache - END (PID: 21926, QN: 2/2, ET:
0) ***
^CINFO ( foo/print ): *** Purging cache - START (PID: 21559) ***
INFO ( foo/print ): *** Purging cache - END (PID: 21559, QN: 0/0, ET:
X) ***
INFO ( default/core ): OK, Exiting ...

Now the output when using and the same .pcap file 64bit version of
pmacctd

sudo /root/pmacctd-176 -f ./mypaolo.conf -I
v1.7.5_v9_ndpi_class_paolo.pcap
INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
1.7.6-git (20200623-00)
INFO ( default/core ):  '--enable-ndpi'
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/var/lib/pgsql/sclark/mypaolo.conf'.
INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on
softflowd 0.9.7 software,

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Marc Sune 
Steve,

Try running it with valgrind and copy the warnings, if any

./valgrind pmacct/src/pmacctd -f ./mypaolo.conf -I
v1.7.5_v9_ndpi_class_paolo.pcap

marc

Missatge de Paolo Lucente  del dia dj., 9 de jul. 2020 a
les 21:08:

>
> I did test on a Debian 10:
>
> 4.19.0-8-686-pae #1 SMP Debian 4.19.98-1 (2020-01-26) i686 GNU/Linux
>
> As i was suspecting, passing the pcap you sent me through a daemon
> compiled on this box went fine (that is, i can't reproduce the issue).
>  From what i see, by the way, this is not something related to nDPI.
>
> Paolo
>
> On 09/07/2020 18:19, Steve Clark wrote:
> > Thanks for checking, could you tell what distro and version you tested
> on?
> >
> > Also when I compile on 32 bit I get a lot of warning of redefines
> > between ndpi.h and pmacct.h
> > do you get those also?
> >
> >
> >
> >
> > On 07/09/2020 11:55 AM, Paolo Lucente wrote:
> >> Hi Steve,
> >>
> >> I do have avail of a i686-based VM. I can't say everything is tested on
> >> i686 but i tend to check every now and then that nothing fundamental is
> >> broken. I took the example config you used, compiled master code with
> >> the same config switches as you did (essentially --enable-ndpi) and had
> >> no joy reproducing the issue.
> >>
> >> You could send me privately your capture and i may try with that one
> >> (although i am not highly positive it will be a successful test); or you
> >> could arrange me access to your box to read the pcap. Let me know.
> >>
> >> Paolo
> >>
> >> On 09/07/2020 14:54, Steve Clark wrote:
> >>> Hi Paolo,
> >>>
> >>> I have compiled master with nDPI on both 32bit and 64bit CentOS 6
> >>> systems. The 64 bit pmacctd seems
> >>> to work fine. But I get bogus byte counts when I run the 32bit version
> >>> against the same pcap file.
> >>>
> >>> Just wondered if you have done any testing on 32bit intel system with
> >>> the above combination.
> >>>
> >>> below is the output when using 32bit pmacctd - first the pmacctd
> >>> invocation then the nfacctd output
> >>> pmacct/src/pmacctd -f ./mypaolo.conf -I v1.7.5_v9_ndpi_class_paolo.pcap
> >>> INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
> >>> 1.7.6-git (20200707-01)
> >>> INFO ( default/core ):  '--enable-ndpi'
> >>> '--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
> >>> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> >>> '--enable-st-bins'
> >>> INFO ( default/core ): Reading configuration file
> >>> '/var/lib/pgsql/sclark/mypaolo.conf'.
> >>> INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on
> >>> softflowd 0.9.7 software, Copyright 2002 Damien Miller <
> d...@mindrot.org>
> >>> All rights reserved.
> >>> INFO ( p4p1/nfprobe ):   TCP timeout: 3600s
> >>> INFO ( p4p1/nfprobe ):  TCP post-RST timeout: 120s
> >>> INFO ( p4p1/nfprobe ):  TCP post-FIN timeout: 300s
> >>> INFO ( p4p1/nfprobe ):   UDP timeout: 300s
> >>> INFO ( p4p1/nfprobe ):  ICMP timeout: 300s
> >>> INFO ( p4p1/nfprobe ):   General timeout: 3600s
> >>> INFO ( p4p1/nfprobe ):  Maximum lifetime: 604800s
> >>> INFO ( p4p1/nfprobe ):   Expiry interval: 60s
> >>> INFO ( default/core ): PCAP capture file, sleeping for 2 seconds
> >>> INFO ( p4p1/nfprobe ): Exporting flows to [172.24.109.157]:rrac
> >>> WARN ( p4p1/nfprobe ): Shutting down on user request.
> >>> INFO ( default/core ): OK, Exiting ...
> >>>
> >>> src/nfacctd -f examples/nfacctd-print.conf.example
> >>> INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.6-git
> >>> (20200623-00)
> >>> INFO ( default/core ):  '--enable-ndpi'
> >>> '--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
> >>> '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
> >>> '--enable-st-bins'
> >>> INFO ( default/core ): Reading configuration file
> >>> '/var/lib/pgsql/sclark/pmacct/examples/nfacctd-print.conf.example'.
> >>> INFO ( default/core ): waiting for NetFlow/IPFIX data on :::5678
> >>> INFO ( foo/print ): cache entries=16411 base cache memory=56322552
> bytes
> >>> WARN ( foo/print ): no print_output_file and no print_output_lock_file
> >>> defined.
> >>> INFO ( foo/print ): *** Purging cache - START (PID: 21926) ***
> >>> CLASS SRC_IP
> >>> DST_IP SRC_PORT  DST_PORT
> >>> PROTOCOLPACKETS   BYTES
> >>> NetFlow   172.24.110.104
> >>> 172.24.109.247 41900 2055
> >>> udp 26 1576253010996
> >>> NetFlow   172.24.110.104
> >>> 172.24.109.247 58131 2055
> >>> udp 211576253008620
> >>> INFO ( foo/print ): *** Purging cache - END (PID: 21926, QN: 2/2, ET:
> >>> 0) ***
> >>> ^CINFO ( foo/print ): *** Purging cache - START (PID: 21559) ***
> >>> INFO ( foo/print ): *** Purging cache - END (PID: 21559, QN: 0/0, ET:
> >>> X) ***
> >>> INFO ( default/core ): OK, Exiting ...
> >>>
> >>> Now the output when using and the same .pcap file 64bit

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Paolo Lucente 


I did test on a Debian 10:

4.19.0-8-686-pae #1 SMP Debian 4.19.98-1 (2020-01-26) i686 GNU/Linux

As i was suspecting, passing the pcap you sent me through a daemon 
compiled on this box went fine (that is, i can't reproduce the issue).

From what i see, by the way, this is not something related to nDPI.

Paolo

On 09/07/2020 18:19, Steve Clark wrote:

Thanks for checking, could you tell what distro and version you tested on?

Also when I compile on 32 bit I get a lot of warning of redefines 
between ndpi.h and pmacct.h

do you get those also?




On 07/09/2020 11:55 AM, Paolo Lucente wrote:

Hi Steve,

I do have avail of a i686-based VM. I can't say everything is tested on
i686 but i tend to check every now and then that nothing fundamental is
broken. I took the example config you used, compiled master code with
the same config switches as you did (essentially --enable-ndpi) and had
no joy reproducing the issue.

You could send me privately your capture and i may try with that one
(although i am not highly positive it will be a successful test); or you
could arrange me access to your box to read the pcap. Let me know.

Paolo

On 09/07/2020 14:54, Steve Clark wrote:

Hi Paolo,

I have compiled master with nDPI on both 32bit and 64bit CentOS 6
systems. The 64 bit pmacctd seems
to work fine. But I get bogus byte counts when I run the 32bit version
against the same pcap file.

Just wondered if you have done any testing on 32bit intel system with
the above combination.

below is the output when using 32bit pmacctd - first the pmacctd
invocation then the nfacctd output
pmacct/src/pmacctd -f ./mypaolo.conf -I v1.7.5_v9_ndpi_class_paolo.pcap
INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
1.7.6-git (20200707-01)
INFO ( default/core ):  '--enable-ndpi'
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/var/lib/pgsql/sclark/mypaolo.conf'.
INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on
softflowd 0.9.7 software, Copyright 2002 Damien Miller 
All rights reserved.
INFO ( p4p1/nfprobe ):   TCP timeout: 3600s
INFO ( p4p1/nfprobe ):  TCP post-RST timeout: 120s
INFO ( p4p1/nfprobe ):  TCP post-FIN timeout: 300s
INFO ( p4p1/nfprobe ):   UDP timeout: 300s
INFO ( p4p1/nfprobe ):  ICMP timeout: 300s
INFO ( p4p1/nfprobe ):   General timeout: 3600s
INFO ( p4p1/nfprobe ):  Maximum lifetime: 604800s
INFO ( p4p1/nfprobe ):   Expiry interval: 60s
INFO ( default/core ): PCAP capture file, sleeping for 2 seconds
INFO ( p4p1/nfprobe ): Exporting flows to [172.24.109.157]:rrac
WARN ( p4p1/nfprobe ): Shutting down on user request.
INFO ( default/core ): OK, Exiting ...

src/nfacctd -f examples/nfacctd-print.conf.example
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.6-git
(20200623-00)
INFO ( default/core ):  '--enable-ndpi'
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/var/lib/pgsql/sclark/pmacct/examples/nfacctd-print.conf.example'.
INFO ( default/core ): waiting for NetFlow/IPFIX data on :::5678
INFO ( foo/print ): cache entries=16411 base cache memory=56322552 bytes
WARN ( foo/print ): no print_output_file and no print_output_lock_file
defined.
INFO ( foo/print ): *** Purging cache - START (PID: 21926) ***
CLASS SRC_IP
DST_IP SRC_PORT  DST_PORT
PROTOCOL    PACKETS   BYTES
NetFlow   172.24.110.104
172.24.109.247 41900 2055
udp 26 1576253010996
NetFlow   172.24.110.104
172.24.109.247 58131 2055
udp 21    1576253008620
INFO ( foo/print ): *** Purging cache - END (PID: 21926, QN: 2/2, ET: 
0) ***

^CINFO ( foo/print ): *** Purging cache - START (PID: 21559) ***
INFO ( foo/print ): *** Purging cache - END (PID: 21559, QN: 0/0, ET: 
X) ***

INFO ( default/core ): OK, Exiting ...

Now the output when using and the same .pcap file 64bit version of 
pmacctd


sudo /root/pmacctd-176 -f ./mypaolo.conf -I 
v1.7.5_v9_ndpi_class_paolo.pcap

INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd
1.7.6-git (20200623-00)
INFO ( default/core ):  '--enable-ndpi'
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2'
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins'
'--enable-st-bins'
INFO ( default/core ): Reading configuration file
'/var/lib/pgsql/sclark/mypaolo.conf'.
INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on
softflowd 0.9.7 software, Copyright 2002 Damien Miller 
All rights reserved.
INFO ( default/core ): PCAP capture file, sleeping for 2 seconds
INFO ( p4p1/nfprobe ):   TCP timeout: 3600s
INFO ( p4p1/nfprobe ):  TCP post-RST timeout: 120s

Re: [pmacct-discussion] master - ndpi on 32bit CentOS 6

2020-07-09 Thread Paolo Lucente 


Hi Steve,

I do have avail of a i686-based VM. I can't say everything is tested on 
i686 but i tend to check every now and then that nothing fundamental is 
broken. I took the example config you used, compiled master code with 
the same config switches as you did (essentially --enable-ndpi) and had 
no joy reproducing the issue.


You could send me privately your capture and i may try with that one 
(although i am not highly positive it will be a successful test); or you 
could arrange me access to your box to read the pcap. Let me know.


Paolo

On 09/07/2020 14:54, Steve Clark wrote:

Hi Paolo,

I have compiled master with nDPI on both 32bit and 64bit CentOS 6 
systems. The 64 bit pmacctd seems
to work fine. But I get bogus byte counts when I run the 32bit version 
against the same pcap file.


Just wondered if you have done any testing on 32bit intel system with 
the above combination.


below is the output when using 32bit pmacctd - first the pmacctd 
invocation then the nfacctd output

pmacct/src/pmacctd -f ./mypaolo.conf -I v1.7.5_v9_ndpi_class_paolo.pcap
INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd 
1.7.6-git (20200707-01)
INFO ( default/core ):  '--enable-ndpi' 
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2' 
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' 
'--enable-st-bins'
INFO ( default/core ): Reading configuration file 
'/var/lib/pgsql/sclark/mypaolo.conf'.
INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on 
softflowd 0.9.7 software, Copyright 2002 Damien Miller  
All rights reserved.

INFO ( p4p1/nfprobe ):   TCP timeout: 3600s
INFO ( p4p1/nfprobe ):  TCP post-RST timeout: 120s
INFO ( p4p1/nfprobe ):  TCP post-FIN timeout: 300s
INFO ( p4p1/nfprobe ):   UDP timeout: 300s
INFO ( p4p1/nfprobe ):  ICMP timeout: 300s
INFO ( p4p1/nfprobe ):   General timeout: 3600s
INFO ( p4p1/nfprobe ):  Maximum lifetime: 604800s
INFO ( p4p1/nfprobe ):   Expiry interval: 60s
INFO ( default/core ): PCAP capture file, sleeping for 2 seconds
INFO ( p4p1/nfprobe ): Exporting flows to [172.24.109.157]:rrac
WARN ( p4p1/nfprobe ): Shutting down on user request.
INFO ( default/core ): OK, Exiting ...

src/nfacctd -f examples/nfacctd-print.conf.example
INFO ( default/core ): NetFlow Accounting Daemon, nfacctd 1.7.6-git 
(20200623-00)
INFO ( default/core ):  '--enable-ndpi' 
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2' 
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' 
'--enable-st-bins'
INFO ( default/core ): Reading configuration file 
'/var/lib/pgsql/sclark/pmacct/examples/nfacctd-print.conf.example'.

INFO ( default/core ): waiting for NetFlow/IPFIX data on :::5678
INFO ( foo/print ): cache entries=16411 base cache memory=56322552 bytes
WARN ( foo/print ): no print_output_file and no print_output_lock_file 
defined.

INFO ( foo/print ): *** Purging cache - START (PID: 21926) ***
CLASS SRC_IP 
DST_IP SRC_PORT  DST_PORT  
PROTOCOL    PACKETS   BYTES
NetFlow   172.24.110.104 
172.24.109.247 41900 2055  
udp 26 1576253010996
NetFlow   172.24.110.104 
172.24.109.247 58131 2055  
udp 21    1576253008620

INFO ( foo/print ): *** Purging cache - END (PID: 21926, QN: 2/2, ET: 0) ***
^CINFO ( foo/print ): *** Purging cache - START (PID: 21559) ***
INFO ( foo/print ): *** Purging cache - END (PID: 21559, QN: 0/0, ET: X) ***
INFO ( default/core ): OK, Exiting ...

Now the output when using and the same .pcap file 64bit version of pmacctd

sudo /root/pmacctd-176 -f ./mypaolo.conf -I v1.7.5_v9_ndpi_class_paolo.pcap
INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd 
1.7.6-git (20200623-00)
INFO ( default/core ):  '--enable-ndpi' 
'--with-ndpi-static-lib=/usr/local/lib/' '--enable-l2' 
'--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' 
'--enable-st-bins'
INFO ( default/core ): Reading configuration file 
'/var/lib/pgsql/sclark/mypaolo.conf'.
INFO ( p4p1/nfprobe ): NetFlow probe plugin is originally based on 
softflowd 0.9.7 software, Copyright 2002 Damien Miller  
All rights reserved.

INFO ( default/core ): PCAP capture file, sleeping for 2 seconds
INFO ( p4p1/nfprobe ):   TCP timeout: 3600s
INFO ( p4p1/nfprobe ):  TCP post-RST timeout: 120s
INFO ( p4p1/nfprobe ):  TCP post-FIN timeout: 300s
INFO ( p4p1/nfprobe ):   UDP timeout: 300s
INFO ( p4p1/nfprobe ):  ICMP timeout: 300s
INFO ( p4p1/nfprobe ):   General timeout: 3600s
INFO ( p4p1/nfprobe ):  Maximum lifetime: 604800s
INFO ( p4p1/nfprobe ):   Expiry interval: 60s
INFO ( p4p1/nfprobe ): Exporting flows to [172.24.109.157]:rrac
WARN ( p4p1/nfprobe ): Shutting down on user request.
INFO ( default/core