from:"Yann Belin"

Re: [pmacct-discussion] New to pmacct - Need help with Netflow

2017-01-18 Thread Yann Belin

Hi Luc,

Did you try to enable debug mode on nfacctd (-d)? It will show you
when the flows are received, as well any potential errors when sending
it to db.

Also, keep in mind that if you use NetflowV9/IPfix, nfacctd wont be
able to process incoming flows until a template is received.

Cheers,

Yann

On Thu, Jan 19, 2017 at 4:51 AM, Luc Perreau  wrote:
> Hi all,
>
> I am fairly new to pmacct and have been struggling for a while to get it to
> do what i want.
>
> I have it setup and logging to a mysql db.
>
> All i want is to send netflow traffic to it so that i know which IP accessed
> what and at what time.
>
> Basically i am interested in src ip, dst ip, src port, dst port, and time
>
> I have tried using nfacct but when i query the db, i do not see time entries
> :(
>
> I know flows are hitting the hitting box of the right port as i have done a
> tcpdump and i see the flows.
>
> Can someone please help me out?
>
> Thanks,
>
> Luc
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] nfacctd and NBAR

2016-12-14 Thread Yann Belin

Thanks Paolo,

The class field was showing up as "unknown" for me, but by using
aggregate_primitive I was indeed able to extract the field I need
(#95). Cool stuff!

Cheers,

Yann

On Wed, Dec 14, 2016 at 2:38 AM, Paolo Lucente <pa...@pmacct.net> wrote:
>
> Hi Yann,
>
> You should use the 'class' aggregation primitive for that - or are you
> already doing so ant it's not working? To your other question: yes, you
> can extend, within some limits, the set of natively supported primitives
> with custom ones: please look at the aggregate_primitives framework (in
> CONFIG-KEYS which, in turn, points you to an example).
>
> Cheers,
> Paolo
>
> On Mon, Dec 12, 2016 at 01:38:29PM +0100, Yann Belin wrote:
>> Hello,
>>
>> I am trying to use the NBAR "application ID" field (#95) in nfacctd
>> aggregation but I cannot figure out how to do that. My situation is
>> very similar to what Olaf encountered a couple of years ago (see link
>> below) but unfortunately that thread did not reach a conclusion (at
>> least on its public part).
>>
>> https://www.mail-archive.com/pmacct-discussion@pmacct.net/msg01831.html
>>
>> This is the template sent by my Cisco router, the field I am
>> interested in is "95". Is there a way to have nfacctd aggregate on
>> primitives that are not explicitly listed under "nfacctd -a"?
>>
>> DEBUG ( default/core ): NfV10 agent : x.x.x.x:1792
>> DEBUG ( default/core ): NfV10 template type : flow
>> DEBUG ( default/core ): NfV10 template ID   : 274
>> DEBUG ( default/core ):
>> -
>> DEBUG ( default/core ): |pen | field type |
>> offset |  size  |
>> DEBUG ( default/core ): | 0  | IPv4 src addr  [8] |
>>   0 |  4 |
>> DEBUG ( default/core ): | 0  | IPv4 dst addr  [12   ] |
>>   4 |  4 |
>> DEBUG ( default/core ): | 0  | tos[5] |
>>   8 |  1 |
>> DEBUG ( default/core ): | 0  | L4 protocol[4] |
>>   9 |  1 |
>> DEBUG ( default/core ): | 0  | L4 src port[7] |
>>  10 |  2 |
>> DEBUG ( default/core ): | 0  | L4 dst port[11   ] |
>>  12 |  2 |
>> DEBUG ( default/core ): | 0  | input snmp [10   ] |
>>  14 |  4 |
>> DEBUG ( default/core ): | 0  | 95 [95   ] |
>>  18 |  4 |
>> DEBUG ( default/core ): | 0  | direction  [61   ] |
>>  22 |  1 |
>> DEBUG ( default/core ): | 0  | in bytes   [1] |
>>  23 |  4 |
>> DEBUG ( default/core ): | 0  | in packets [2] |
>>  27 |  4 |
>> DEBUG ( default/core ): | 0  | first switched [22   ] |
>>  31 |  4 |
>> DEBUG ( default/core ): | 0  | last switched  [21   ] |
>>  35 |  4 |
>> DEBUG ( default/core ):
>> -
>> DEBUG ( default/core ): Netflow V9/IPFIX record size : 39
>> (...)
>> DEBUG ( default/core ): NfV10 agent : x.x.x.x:6
>> DEBUG ( default/core ): NfV10 template type : options
>> DEBUG ( default/core ): NfV10 template ID   : 259
>> DEBUG ( default/core ): 
>> DEBUG ( default/core ): | field type | offset |  size  |
>> DEBUG ( default/core ): | app id [95   ] |  0 |  4 |
>> DEBUG ( default/core ): | app name   [96   ] |  4 | 24 |
>> DEBUG ( default/core ): | app desc   [94   ] | 28 | 55 |
>> DEBUG ( default/core ): 
>> DEBUG ( default/core ): Netflow V9/IPFIX record size : 83
>>
>> Kind regards,
>>
>> Yann
>>
>> ___
>> pmacct-discussion mailing list
>> http://www.pmacct.net/#mailinglists
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Configure with MySQL / MariaDB support

2016-12-13 Thread Yann Belin

Hi Mehul,

Yes I thought it could be the issue too but the softlinks look fine on FS level.

[root@ pmacct]# ll /usr/lib64/libmysqlclient.*
-rw-r--r-- 1 root root 4386606 Oct 27 14:54 /usr/lib64/libmysqlclient.a
lrwxrwxrwx 1 root root  20 Aug 29 22:25
/usr/lib64/libmysqlclient.so -> libmysqlclient.so.18
lrwxrwxrwx 1 root root  24 Aug 29 22:25
/usr/lib64/libmysqlclient.so.15 -> libmysqlclient.so.15.0.0
-rwxr-xr-x 1 root root 2043496 Aug 24 12:38 /usr/lib64/libmysqlclient.so.15.0.0
lrwxrwxrwx 1 root root  24 Aug 29 22:25
/usr/lib64/libmysqlclient.so.16 -> libmysqlclient.so.16.0.0
-rwxr-xr-x 1 root root 2016648 Aug 24 12:38 /usr/lib64/libmysqlclient.so.16.0.0
lrwxrwxrwx 1 root root  24 Aug 29 22:25
/usr/lib64/libmysqlclient.so.18 -> libmysqlclient.so.18.0.0
-rwxr-xr-x 1 root root 6232959 Aug 24 12:30 /usr/lib64/libmysqlclient.so.18.0.0

[root@ pmacct]# file /usr/lib64/libmysqlclient.so.18.0.0
/usr/lib64/libmysqlclient.so.18.0.0: ELF 64-bit LSB shared object,
x86-64, version 1 (SYSV), dynamically linked,
BuildID[sha1]=8b103aaafd60c23addd177a4fd254affb3abc916, not stripped

[root@ pmacct]# md5sum /usr/lib64/libmysqlclient.so.18.0.0
817b141e90cdb66f63432fda22f8db6f  /usr/lib64/libmysqlclient.so.18.0.0


Kind regards,

Yann

On Tue, Dec 13, 2016 at 10:47 AM, Mehul Prajapati
<mehul.prajap...@mobileinternet.com> wrote:
> Hi,
>
> In my Ubuntu machine where pmacct is working with MySQL
>
> /usr/lib/x86_64-linux-gnu/libmysqlclient.so -> libmysqlclient.so.18
> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18 -> libmysqlclient.so.18.0.0
> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
>
> In your machine,
> libmysqlclient.so (libc6,x86-64) => /lib64/libmysqlclient.so
>
> Can you check soft links, whether it's pointing to proper .so file ?
>
>
> -Original Message-
> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On 
> Behalf Of Yann Belin
> Sent: Tuesday, December 13, 2016 3:07 PM
> To: pmacct-discussion@pmacct.net
> Subject: Re: [pmacct-discussion] Configure with MySQL / MariaDB support
>
> Hi Mehul,
>
> It didn't help. could it be a mysql-specific issue (e.g. version)? For 
> instance ./configure is able to find the pcap libraries under the same 
> location (/usr/lib64/) with no problems.
>
> [root@ pmacct-1.6.1]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/
> [root@ pmacct-1.6.1]# echo $LD_LIBRARY_PATH :/usr/lib64/ [root@ 
> pmacct-1.6.1]# ./configure --enable-mysql
> (...)
> checking default locations for pcap.h... found in /usr/include checking 
> default locations for libpcap... no checking for pcap_dispatch in -lpcap... 
> yes checking for pcap_setnonblock in -lpcap... yes checking for bpf_filter in 
> -lpcap... yes checking packet capture type... linux checking whether to 
> enable MySQL support... yes checking for mysql_config... mysql_config 
> checking for mysql_init in -lmysqlclient... no
> configure: error: ERROR: missing MySQL client library [root@ 
> pmacct-1.6.1]#
>
> Kind regards,
>
> Yann
>
> On Tue, Dec 13, 2016 at 10:22 AM, Mehul Prajapati 
> <mehul.prajap...@mobileinternet.com> wrote:
>> Hi,
>>
>> I think your environment variable LD_LIBRARY_PATH is not pointing to 
>> /usr/lib64/ directory.
>>
>> Please run following command and try.
>>
>> $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH: /usr/lib64/
>>
>> -Original Message-
>> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
>> On Behalf Of Yann Belin
>> Sent: Tuesday, December 13, 2016 2:47 PM
>> To: pmacct-discussion@pmacct.net
>> Subject: [pmacct-discussion] Configure with MySQL / MariaDB support
>>
>> Hello,
>>
>> I am trying to install pmacct with MySQL / MariaDB support but [./configure 
>> --enable-mysql] fails with the message below.
>>
>> *-*-*-*-*
>> (...)
>> checking for mysql_init in -lmysqlclient... no
>> configure: error: ERROR: missing MySQL client library
>> *-*-*-*-*
>>
>> I have proper libraries and devel packages installed, any idea what may be 
>> causing this?
>>
>> [root@ pmacct-1.6.1]# locate libmysqlclient
>> /usr/lib64/libmysqlclient.so
>> /usr/lib64/libmysqlclient.so.15
>> /usr/lib64/libmysqlclient.so.15.0.0
>> /usr/lib64/libmysqlclient.so.16
>> /usr/lib64/libmysqlclient.so.16.0.0
>> /usr/lib64/libmysqlclient.so.18
>> /usr/lib64/libmysqlclient.so.18.0.0
>> /usr/lib64/libmysqlclient_r.so
>> /usr/lib64/libmysqlclient_r.so.15
>> /usr/lib64/libmysqlclient_r.so.15.0.0
>> /usr/lib64/libmysqlclient_r.so.16
>> /usr/lib64/libmysqlclient_r.so.16.0.0
>> /usr/lib64/libmysqlclient_r.so

[pmacct-discussion] Configure with MySQL / MariaDB support

2016-12-13 Thread Yann Belin

Hello,

I am trying to install pmacct with MySQL / MariaDB support but
[./configure --enable-mysql] fails with the message below.

*-*-*-*-*
(...)
checking for mysql_init in -lmysqlclient... no
configure: error: ERROR: missing MySQL client library
*-*-*-*-*

I have proper libraries and devel packages installed, any idea what
may be causing this?

[root@ pmacct-1.6.1]# locate libmysqlclient
/usr/lib64/libmysqlclient.so
/usr/lib64/libmysqlclient.so.15
/usr/lib64/libmysqlclient.so.15.0.0
/usr/lib64/libmysqlclient.so.16
/usr/lib64/libmysqlclient.so.16.0.0
/usr/lib64/libmysqlclient.so.18
/usr/lib64/libmysqlclient.so.18.0.0
/usr/lib64/libmysqlclient_r.so
/usr/lib64/libmysqlclient_r.so.15
/usr/lib64/libmysqlclient_r.so.15.0.0
/usr/lib64/libmysqlclient_r.so.16
/usr/lib64/libmysqlclient_r.so.16.0.0
/usr/lib64/libmysqlclient_r.so.18
/usr/lib64/libmysqlclient_r.so.18.0.0

[root@ pmacct-1.6.1]# ldconfig -p | grep mysqlclient
libmysqlclient_r.so.16 (libc6,x86-64) => /lib64/libmysqlclient_r.so.16
libmysqlclient_r.so.15 (libc6,x86-64) => /lib64/libmysqlclient_r.so.15
libmysqlclient.so.18 (libc6,x86-64) => /lib64/libmysqlclient.so.18
libmysqlclient.so.16 (libc6,x86-64) => /lib64/libmysqlclient.so.16
libmysqlclient.so.15 (libc6,x86-64) => /lib64/libmysqlclient.so.15
libmysqlclient.so (libc6,x86-64) => /lib64/libmysqlclient.so

[root@ pmacct-1.6.1]# rpm -qa | grep -i maria
MariaDB-devel-10.0.28-1.el7.centos.x86_64
MariaDB-client-10.0.27-1.el7.centos.x86_64
MariaDB-compat-10.0.27-1.el7.centos.x86_64
MariaDB-connect-engine-10.0.28-1.el7.centos.x86_64
MariaDB-common-10.0.27-1.el7.centos.x86_64
MariaDB-server-10.0.27-1.el7.centos.x86_64
MariaDB-shared-10.0.27-1.el7.centos.x86_64
[root@scrutinizer01 pmacct-1.6.1]#

Thanks in advance!


Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Configure with MySQL / MariaDB support

2016-12-13 Thread Yann Belin

Solved :)

The configure script output was misleading and reported libmysqlclient to
be missing, while it was in fact libz, libssl and libcrypto (see below).
After installing openssl-devel I was able to configure/build successfully.

configure:13308: checking for mysql_init in -lmysqlclient
configure:1: gcc -o conftest -O2 -g -O2  -Wl,--export-dynamic
 conftest.c -lmysqlclient -L/usr/lib64 -lmysqlclient -lpthread -lz -lm
-lssl -lcrypto -ldl -lpcap  -ldl >&5
/usr/bin/ld: cannot find -lz
/usr/bin/ld: cannot find -lssl
/usr/bin/ld: cannot find -lcrypto
collect2: error: ld returned 1 exit status

Kind regards,

Yann

On Tue, Dec 13, 2016 at 11:01 AM Yann Belin <y.belin...@gmail.com> wrote:

> Hi Mehul,
>
> Yes I thought it could be the issue too but the softlinks look fine on FS
> level.
>
> [root@ pmacct]# ll /usr/lib64/libmysqlclient.*
> -rw-r--r-- 1 root root 4386606 Oct 27 14:54 /usr/lib64/libmysqlclient.a
> lrwxrwxrwx 1 root root  20 Aug 29 22:25
> /usr/lib64/libmysqlclient.so -> libmysqlclient.so.18
> lrwxrwxrwx 1 root root  24 Aug 29 22:25
> /usr/lib64/libmysqlclient.so.15 -> libmysqlclient.so.15.0.0
> -rwxr-xr-x 1 root root 2043496 Aug 24 12:38
> /usr/lib64/libmysqlclient.so.15.0.0
> lrwxrwxrwx 1 root root  24 Aug 29 22:25
> /usr/lib64/libmysqlclient.so.16 -> libmysqlclient.so.16.0.0
> -rwxr-xr-x 1 root root 2016648 Aug 24 12:38
> /usr/lib64/libmysqlclient.so.16.0.0
> lrwxrwxrwx 1 root root  24 Aug 29 22:25
> /usr/lib64/libmysqlclient.so.18 -> libmysqlclient.so.18.0.0
> -rwxr-xr-x 1 root root 6232959 Aug 24 12:30
> /usr/lib64/libmysqlclient.so.18.0.0
>
> [root@ pmacct]# file /usr/lib64/libmysqlclient.so.18.0.0
> /usr/lib64/libmysqlclient.so.18.0.0: ELF 64-bit LSB shared object,
> x86-64, version 1 (SYSV), dynamically linked,
> BuildID[sha1]=8b103aaafd60c23addd177a4fd254affb3abc916, not stripped
>
> [root@ pmacct]# md5sum /usr/lib64/libmysqlclient.so.18.0.0
> 817b141e90cdb66f63432fda22f8db6f  /usr/lib64/libmysqlclient.so.18.0.0
>
>
> Kind regards,
>
> Yann
>
> On Tue, Dec 13, 2016 at 10:47 AM, Mehul Prajapati
> <mehul.prajap...@mobileinternet.com> wrote:
> > Hi,
> >
> > In my Ubuntu machine where pmacct is working with MySQL
> >
> > /usr/lib/x86_64-linux-gnu/libmysqlclient.so -> libmysqlclient.so.18
> > /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18 ->
> libmysqlclient.so.18.0.0
> > /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18.0.0
> >
> > In your machine,
> > libmysqlclient.so (libc6,x86-64) => /lib64/libmysqlclient.so
> >
> > Can you check soft links, whether it's pointing to proper .so file ?
> >
> >
> > -Original Message-
> > From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
> On Behalf Of Yann Belin
> > Sent: Tuesday, December 13, 2016 3:07 PM
> > To: pmacct-discussion@pmacct.net
> > Subject: Re: [pmacct-discussion] Configure with MySQL / MariaDB support
> >
> > Hi Mehul,
> >
> > It didn't help. could it be a mysql-specific issue (e.g. version)? For
> instance ./configure is able to find the pcap libraries under the same
> location (/usr/lib64/) with no problems.
> >
> > [root@ pmacct-1.6.1]# export
> LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/
> > [root@ pmacct-1.6.1]# echo $LD_LIBRARY_PATH :/usr/lib64/ [root@
> pmacct-1.6.1]# ./configure --enable-mysql
> > (...)
> > checking default locations for pcap.h... found in /usr/include checking
> default locations for libpcap... no checking for pcap_dispatch in -lpcap...
> yes checking for pcap_setnonblock in -lpcap... yes checking for bpf_filter
> in -lpcap... yes checking packet capture type... linux checking whether to
> enable MySQL support... yes checking for mysql_config... mysql_config
> checking for mysql_init in -lmysqlclient... no
> > configure: error: ERROR: missing MySQL client library [root@
> pmacct-1.6.1]#
> >
> > Kind regards,
> >
> > Yann
> >
> > On Tue, Dec 13, 2016 at 10:22 AM, Mehul Prajapati <
> mehul.prajap...@mobileinternet.com> wrote:
> >> Hi,
> >>
> >> I think your environment variable LD_LIBRARY_PATH is not pointing to
> /usr/lib64/ directory.
> >>
> >> Please run following command and try.
> >>
> >> $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH: /usr/lib64/
> >>
> >> -Original Message-
> >> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
> >> On Behalf Of Yann Belin
> >> Sent: Tuesday, December 13, 2016 2:47 PM
> >> To: pmacct-discussion@pmacct.net
> >> Subject: [pmacct-discussion] Configure with

Re: [pmacct-discussion] Configure with MySQL / MariaDB support

2016-12-13 Thread Yann Belin

Hi Mehul,

It didn't help. could it be a mysql-specific issue (e.g. version)? For
instance ./configure is able to find the pcap libraries under the same
location (/usr/lib64/) with no problems.

[root@ pmacct-1.6.1]# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/
[root@ pmacct-1.6.1]# echo $LD_LIBRARY_PATH
:/usr/lib64/
[root@ pmacct-1.6.1]# ./configure --enable-mysql
(...)
checking default locations for pcap.h... found in /usr/include
checking default locations for libpcap... no
checking for pcap_dispatch in -lpcap... yes
checking for pcap_setnonblock in -lpcap... yes
checking for bpf_filter in -lpcap... yes
checking packet capture type... linux
checking whether to enable MySQL support... yes
checking for mysql_config... mysql_config
checking for mysql_init in -lmysqlclient... no
configure: error: ERROR: missing MySQL client library
[root@ pmacct-1.6.1]#

Kind regards,

Yann

On Tue, Dec 13, 2016 at 10:22 AM, Mehul Prajapati
<mehul.prajap...@mobileinternet.com> wrote:
> Hi,
>
> I think your environment variable LD_LIBRARY_PATH is not pointing to 
> /usr/lib64/ directory.
>
> Please run following command and try.
>
> $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH: /usr/lib64/
>
> -Original Message-
> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On 
> Behalf Of Yann Belin
> Sent: Tuesday, December 13, 2016 2:47 PM
> To: pmacct-discussion@pmacct.net
> Subject: [pmacct-discussion] Configure with MySQL / MariaDB support
>
> Hello,
>
> I am trying to install pmacct with MySQL / MariaDB support but [./configure 
> --enable-mysql] fails with the message below.
>
> *-*-*-*-*
> (...)
> checking for mysql_init in -lmysqlclient... no
> configure: error: ERROR: missing MySQL client library
> *-*-*-*-*
>
> I have proper libraries and devel packages installed, any idea what may be 
> causing this?
>
> [root@ pmacct-1.6.1]# locate libmysqlclient /usr/lib64/libmysqlclient.so
> /usr/lib64/libmysqlclient.so.15
> /usr/lib64/libmysqlclient.so.15.0.0
> /usr/lib64/libmysqlclient.so.16
> /usr/lib64/libmysqlclient.so.16.0.0
> /usr/lib64/libmysqlclient.so.18
> /usr/lib64/libmysqlclient.so.18.0.0
> /usr/lib64/libmysqlclient_r.so
> /usr/lib64/libmysqlclient_r.so.15
> /usr/lib64/libmysqlclient_r.so.15.0.0
> /usr/lib64/libmysqlclient_r.so.16
> /usr/lib64/libmysqlclient_r.so.16.0.0
> /usr/lib64/libmysqlclient_r.so.18
> /usr/lib64/libmysqlclient_r.so.18.0.0
>
> [root@ pmacct-1.6.1]# ldconfig -p | grep mysqlclient
> libmysqlclient_r.so.16 (libc6,x86-64) => /lib64/libmysqlclient_r.so.16
> libmysqlclient_r.so.15 (libc6,x86-64) => /lib64/libmysqlclient_r.so.15
> libmysqlclient.so.18 (libc6,x86-64) => /lib64/libmysqlclient.so.18
> libmysqlclient.so.16 (libc6,x86-64) => /lib64/libmysqlclient.so.16
> libmysqlclient.so.15 (libc6,x86-64) => /lib64/libmysqlclient.so.15
> libmysqlclient.so (libc6,x86-64) => /lib64/libmysqlclient.so
>
> [root@ pmacct-1.6.1]# rpm -qa | grep -i maria
> MariaDB-devel-10.0.28-1.el7.centos.x86_64
> MariaDB-client-10.0.27-1.el7.centos.x86_64
> MariaDB-compat-10.0.27-1.el7.centos.x86_64
> MariaDB-connect-engine-10.0.28-1.el7.centos.x86_64
> MariaDB-common-10.0.27-1.el7.centos.x86_64
> MariaDB-server-10.0.27-1.el7.centos.x86_64
> MariaDB-shared-10.0.27-1.el7.centos.x86_64
> [root@scrutinizer01 pmacct-1.6.1]#
>
> Thanks in advance!
>
>
> Yann
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Reporting on additional primitives

2016-12-15 Thread Yann Belin

Hello,

I am looking for a way to report on additional primitives with nfacct.
It does by default with bytes and packets but I cannot find how to add
extra fields. Am I missing something?

What I want to include in reports is application performance
information (rtd, packet-loss, etc.) from Cisco devices, as described
on the link below. For a given flow, I would like to store an
aggregated version (e.g. average) of this data.

http://www.cisco.com/c/en/us/td/docs/ios/media_monitoring/configuration/guide/15_1m_and_t/mm_15_1m_and_t/mm_pasv_mon.html.


Thanks,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Reporting on additional primitives

2016-12-21 Thread Yann Belin

Hi Paolo,

Sorry for the late reply, I was busy rebuilding our test lab and had to
leave my Netlfow experiments aside for a while.

Thanks for the your response, I understand the constraints. I will use perf
counters as an aggregator for now, it will probably add some I/O load and
storage needs but all-in-all it is still better than non-aggregating
solutions and - as you said - I can do the aggregation myself.

Cheers,

Yann




On Sat, Dec 17, 2016 at 4:45 PM Paolo Lucente <pa...@pmacct.net> wrote:

>
> Hi Yann,
>
> You remember i was saying of the current limitations of the
> aggregate_primitives framework. That's it: you can add key primitives
> to the aggregation method but you can't add non-key ones on which, for
> exxample, you want to perform operations (ie. sum like in the case of
> bytes and packets). It is on my todo list to add such feature although
> priority is not high. This all said, what i must be realistic is i was
> not thinking to include 'advanced' operators like average (as that would
> introduce a whole new behaviour, ie. cache entries de-aggregated in
> memory then consolidte on purge [to make the average work]). What i may
> suggest as workaround is to use the aggregate_primitives framework you
> have today and make the averages yourself by post-processing the output.
>
> Cheers,
> Paolo
>
> On Thu, Dec 15, 2016 at 01:44:16PM +0100, Yann Belin wrote:
> > Hello,
> >
> > I am looking for a way to report on additional primitives with nfacct.
> > It does by default with bytes and packets but I cannot find how to add
> > extra fields. Am I missing something?
> >
> > What I want to include in reports is application performance
> > information (rtd, packet-loss, etc.) from Cisco devices, as described
> > on the link below. For a given flow, I would like to store an
> > aggregated version (e.g. average) of this data.
> >
> >
> http://www.cisco.com/c/en/us/td/docs/ios/media_monitoring/configuration/guide/15_1m_and_t/mm_15_1m_and_t/mm_pasv_mon.html
> .
> >
> >
> > Thanks,
> >
> > Yann
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] pretag_map with multiple option records

2016-12-22 Thread Yann Belin

I might have been overthinking this, it seems that no explicit tag/filter
is needed as long as I aggregate on different fields (see below).

It makes sense that nfacctd will not aggregate on primitives that do not
exist (duh), and it provides the kind of filtering that I was looking for,
but is it the proper method to filter or does it just "happen to work" this
way?

pre_tag_filter[options_nbar]:   200
aggregate[options_nbar]:peer_src_ip, nbar_id, nbar_name,
nbar_desc
sql_table[options_nbar]:pmacct_options_nbar
!(...)

pre_tag_filter[options_iface]:  200
aggregate[options_iface]:   peer_src_ip, in_iface, iface_short,
iface_long
sql_table[options_iface]:   pmacct_options_iface
!(...)

Cheers,

Yann


On Wed, Dec 21, 2016 at 4:54 PM Yann Belin <y.belin...@gmail.com> wrote:

> Hello,
>
> After following the examples in pmacct documentation, I was able to
> assign different tags to flow and option records (respectively 100 and
> 200) , but I cannot figure out how to assign different tags to
> different "types" of option records in order to store their data in
> different SQL tables.
>
> For instance, I receive the option records below. What I would like to
> do is to assign tag #200 to "application" records and tag #201 to
> "interface" records but I cannot figure out a way to do it.
>
> I cannot use the template/flowset ID because it bound to change
> occasionally... I thought about checking the presence of a field
> instead (e.g. 10 v.s. 95) but there is nothing in the pretag_map
> documentation about this. Any ideas?
>
>
> DEBUG ( default/core ): NfV10 agent : 172.16.2.1:6
> DEBUG ( default/core ): NfV10 template type : options
> DEBUG ( default/core ): NfV10 template ID   : 256
> DEBUG ( default/core ): 
> DEBUG ( default/core ): | field type | offset |  size  |
> DEBUG ( default/core ): | 10 [10   ] |  0 |  4
> |   Interface input snmp
> DEBUG ( default/core ): | 82 [82   ] |  4 | 32
> |   Interface name short
> DEBUG ( default/core ): | 83 [83   ] | 36 | 64
> |   Interface name long
> DEBUG ( default/core ): 
> DEBUG ( default/core ): Netflow V9/IPFIX record size : 100
>
>
> DEBUG ( default/core ): NfV10 agent : 172.16.2.1:6
> DEBUG ( default/core ): NfV10 template type : options
> DEBUG ( default/core ): NfV10 template ID   : 257
> DEBUG ( default/core ): 
> DEBUG ( default/core ): | field type | offset |  size  |
> DEBUG ( default/core ): | app id [95   ] |  0 |  4
> |   Application ID
> DEBUG ( default/core ): | app name   [96   ] |  4 | 24
> |   Application name
> DEBUG ( default/core ): | app desc   [94   ] | 28 | 55
> |   Application description
> DEBUG ( default/core ): 
> DEBUG ( default/core ): Netflow V9/IPFIX record size : 83
>
>
> Thanks in advance,
>
> Yann
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] pretag_map with multiple option records

2016-12-21 Thread Yann Belin

Hello,

After following the examples in pmacct documentation, I was able to
assign different tags to flow and option records (respectively 100 and
200) , but I cannot figure out how to assign different tags to
different "types" of option records in order to store their data in
different SQL tables.

For instance, I receive the option records below. What I would like to
do is to assign tag #200 to "application" records and tag #201 to
"interface" records but I cannot figure out a way to do it.

I cannot use the template/flowset ID because it bound to change
occasionally... I thought about checking the presence of a field
instead (e.g. 10 v.s. 95) but there is nothing in the pretag_map
documentation about this. Any ideas?


DEBUG ( default/core ): NfV10 agent : 172.16.2.1:6
DEBUG ( default/core ): NfV10 template type : options
DEBUG ( default/core ): NfV10 template ID   : 256
DEBUG ( default/core ): 
DEBUG ( default/core ): | field type | offset |  size  |
DEBUG ( default/core ): | 10 [10   ] |  0 |  4
|   Interface input snmp
DEBUG ( default/core ): | 82 [82   ] |  4 | 32
|   Interface name short
DEBUG ( default/core ): | 83 [83   ] | 36 | 64
|   Interface name long
DEBUG ( default/core ): 
DEBUG ( default/core ): Netflow V9/IPFIX record size : 100


DEBUG ( default/core ): NfV10 agent : 172.16.2.1:6
DEBUG ( default/core ): NfV10 template type : options
DEBUG ( default/core ): NfV10 template ID   : 257
DEBUG ( default/core ): 
DEBUG ( default/core ): | field type | offset |  size  |
DEBUG ( default/core ): | app id [95   ] |  0 |  4
|   Application ID
DEBUG ( default/core ): | app name   [96   ] |  4 | 24
|   Application name
DEBUG ( default/core ): | app desc   [94   ] | 28 | 55
|   Application description
DEBUG ( default/core ): 
DEBUG ( default/core ): Netflow V9/IPFIX record size : 83


Thanks in advance,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Matching data to options with nfacctd

2017-06-06 Thread Yann Belin

Hi,

I was reading trough recent issues on GitHub, and #137 [see link
below] got my attention. The last comment from Paolo leads me to think
that nfacctd can be configured to (try to) automatically match flow
data to option table(s).

Is it the case, or am I misreading something? Until now, I have been
collecting data and options separately (using nfacctd_account_options)
and had to match it afterwards via a script; such feature could make
my life quite easier.

Ref. https://github.com/pmacct/pmacct/issues/137

Thanks,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] nfacctd and amqp_multi_values

2017-09-19 Thread Yann Belin

Thanks Paolo,

>From your opinion, would such feature be difficult (possible) to implement?
I didn't look into the code yet, I didn't program in C since high school
and my skills are quite rusty.

Cheers,

Yann

On Fri, Sep 15, 2017 at 6:38 PM Paolo Lucente <pa...@pmacct.net> wrote:

>
> Hi Yann,
>
> I confirm you can't do that with AMQP as the only knob pmacct gives
> you is the size-based amqp_multi_values. Although not part of your
> question, with Kafka and you may choose not to leverage the pmacct
> knob, kafka_multi_values, and use instead batch.num.messages (ie.
> amount of messages you want to batch before sending to the broker) or
> queue.buffering.max.ms (queue for some given amount of time, if for
> example latency is the constraint) offered by librdkafka.
>
> Paolo
>
> On Thu, Sep 14, 2017 at 04:12:45PM +0200, Yann Belin wrote:
> > Hello,
> >
> > Does anyone know if there is a way to control the maximum number of
> > rows sent in an AMQP message?
> >
> > amqp_multi_values allows me to do that in an approximate way
> > (row_size/message_size) but I need to have a finer control on that,
> > which is not dependent of variations of row size.
> >
> > Thanks in advance,
> >
> > Yann
> >
> > ___
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] nfacctd and amqp_multi_values

2017-09-14 Thread Yann Belin

Hello,

Does anyone know if there is a way to control the maximum number of
rows sent in an AMQP message?

amqp_multi_values allows me to do that in an approximate way
(row_size/message_size) but I need to have a finer control on that,
which is not dependent of variations of row size.

Thanks in advance,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Load balancing nfacctd

2017-08-21 Thread Yann Belin

Hello,

I have been looking into solutions to achieve reliable load balancing
of my incoming flows across multiple nfacctd servers / daemons.

Basic load balancing is relatively easy (see Nginx configuration
below), but *reliable* load balancing (only sending flows to servers
that have a running nfacctd daemon) is quite more complicated. For
instance, Nginx normally monitors UDP responses from the remote
servers to determine if those servers are health, but this approach
will not work in the case of netflow or ipfix.

Did anybody already managed to solve this? Or has a suggestion perhaps?

Thanks in advance!

*-*-*-*-*-*-*-*
stream {
upstream ipfix_traffic {
hash $binary_remote_addr;
server 10.20.10.10:9055;
server 10.20.10.20:9055;
}

server {
listen 9055 udp;
proxy_responses 0;
proxy_pass ipfix_traffic;
proxy_bind $remote_addr transparent;
error_log /var/log/nginx/ipfix_traffic.error.log;
}
}
*-*-*-*-*-*-*-*

Kind regards,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] Best way to output ip addresses as integers

2017-09-04 Thread Yann Belin

Hello,

I need to run some checks / manipulate source/destination IP addresses
that I am getting from nfacctd, and for that purpose it makes much
more sense to output hose IPs in their inetger form, rather than in
their human-readable (x.x.x.x) form.

In order to do that, I created custom primitives that read the same
fields than the native ones, but interpret it as unsigned integers
instead.

name=src_host_intfield_type=8 len=4   semantics=u_int
name=dst_host_intfield_type=12len=4   semantics=u_int

Is it the best method? Or is there a built-in way to achieve the same result.

Kind regards,

Yann

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Load balancing nfacctd

2017-09-04 Thread Yann Belin

Hi all,

Updating on this, in case someone is interested.

Consul was indeed the way to go:

* nginx is doing the actual UDP load balancing, based on source IP
hash (to optimize aggregation).
* consul keeps track of nfacctd collectors, of their health, and of
the health of their dependencies (rabbitmq in my case).
* consul-template uses the information provided by consul (servers +
health) to generate nginx configuration files, and reloads nginx
service if needed; if a collector becomes unhealthy (e.g. rabbitmq
crashes), it will be removed from nginx configuration and will stop
receiving flows.

The great thing with consul is that you can write your own checks. For
now my checks are relatively basic (process + port binding checks) but
I am working on a more advanced one for rabbitmq (e.g. queue length /
ram usage). I'm still thinking about more advanced ways to check
nfacctd health, if anyone has a suggestion.

Cheers,

Yann

On Mon, Aug 21, 2017 at 4:02 PM, Aaron Finney <aaron.fin...@openx.com> wrote:
> Hi Yann
>
> We use Consul for this, it works very well.
>
> https://www.consul.io
>
>
> Aaron
>
>
>
> On Aug 21, 2017 6:44 AM, "Yann Belin" <y.belin...@gmail.com> wrote:
>
> Hello,
>
> I have been looking into solutions to achieve reliable load balancing
> of my incoming flows across multiple nfacctd servers / daemons.
>
> Basic load balancing is relatively easy (see Nginx configuration
> below), but *reliable* load balancing (only sending flows to servers
> that have a running nfacctd daemon) is quite more complicated. For
> instance, Nginx normally monitors UDP responses from the remote
> servers to determine if those servers are health, but this approach
> will not work in the case of netflow or ipfix.
>
> Did anybody already managed to solve this? Or has a suggestion perhaps?
>
> Thanks in advance!
>
> *-*-*-*-*-*-*-*
> stream {
> upstream ipfix_traffic {
> hash $binary_remote_addr;
> server 10.20.10.10:9055;
> server 10.20.10.20:9055;
> }
>
> server {
> listen 9055 udp;
> proxy_responses 0;
> proxy_pass ipfix_traffic;
> proxy_bind $remote_addr transparent;
> error_log /var/log/nginx/ipfix_traffic.error.log;
> }
> }
> *-*-*-*-*-*-*-*
>
> Kind regards,
>
> Yann
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
>
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] ipv4 conversion to int

2018-04-19 Thread Yann Belin

As far as I know it doesn't but if you use nfacctd, you can easily define
your own primitives to do the same job:

name=src_host_intfield_type=8 len=4   semantics=u_int
name=dst_host_intfield_type=12len=4   semantics=u_int

Then, you can use those primitives instead of the standard ones in your
config.

On Thu, Apr 19, 2018 at 12:14 AM Anthony Caiafa <2600...@gmail.com> wrote:

> Does this feature currently exist? Having the ability to convert the ipv4
> key field to an int?
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] New to pmacct - Need help with Netflow

Re: [pmacct-discussion] nfacctd and NBAR

Re: [pmacct-discussion] Configure with MySQL / MariaDB support

[pmacct-discussion] Configure with MySQL / MariaDB support

Re: [pmacct-discussion] Configure with MySQL / MariaDB support

Re: [pmacct-discussion] Configure with MySQL / MariaDB support

[pmacct-discussion] Reporting on additional primitives

Re: [pmacct-discussion] Reporting on additional primitives

Re: [pmacct-discussion] pretag_map with multiple option records

[pmacct-discussion] pretag_map with multiple option records

[pmacct-discussion] Matching data to options with nfacctd

Re: [pmacct-discussion] nfacctd and amqp_multi_values

[pmacct-discussion] nfacctd and amqp_multi_values

[pmacct-discussion] Load balancing nfacctd

[pmacct-discussion] Best way to output ip addresses as integers

Re: [pmacct-discussion] Load balancing nfacctd

Re: [pmacct-discussion] ipv4 conversion to int

17 matches

Site Navigation

Mail list logo

Footer information