Re: [pmacct-discussion] Only packets from router to netflow server

[Jentsch, Mario]

Hi Mattias,

do you have a drawing of your setup? I have to admit that it is unclear to me…

Thanks,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mattias Larsson
Sent: Thursday, August 18, 2016 1:36 PM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] Only packets from router to netflow server


I use a SPAN port on my switch to capture all netflow (udp 2055) packets and 
send it to a interface where my pmacct server has one extra interface connected 
to.

But when I look on the traffic/packets that pmacctd genereates it seems only be 
the IP packets between my router and netflow server. It seems it not decodes 
the cisco netflow payload/data.

When I do a tcpdump on the interface and look at it with wireshark I can see 
see the flows.

Any suggestion what I'm doing wrong?

Thanks in advance!

Mattias
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] collecting large number of netflows

[Jentsch, Mario]

Sounds like you have already have the DB server hardware.
It may be a good idea to simulate the data flows to and from your DB. Some 
scripts that insert data at different constant rates and/or intermittent as it 
comes from nfacctd normally generate the input to the DB. At the same time you 
prepare the next steps of processing with the fake data. This should reveal 
bottlenecks and give you the chance to address them before they appear in the 
live system.
E.g. using multiple Netflow collectors that write to the same tablespace may 
lock each other and decreasing insert performance. Same applies for reading the 
written data for further processing. Reducing the locks can be challenging, 
splitting the tablespace with partitioning or per collector separated inbound 
tables can help.

Good luck!
Mario

> -Original Message-
> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
> On Behalf Of Stephen Clark
> Sent: Thursday, August 18, 2016 2:24 PM
> To: pmacct-discussion@pmacct.net
> Subject: Re: [pmacct-discussion] collecting large number of netflows
> 
> On 08/17/2016 08:38 AM, Jentsch, Mario wrote:
> > Hey Steve,
> >
> > that question can't be answered without a lot of assumptions about the
> details of your project and we made the experience that even with project
> details it is a hard thing to predict due to the nature of network traffic
> patterns. Pmacct (namely nfacctd) can handle that number of flows - even
> with only one instance - and is most probably not the bottleneck. If it is
> possible what you plan to do, depends on questions like "how many records
> per timebin do you have after aggregation in nfacctd" - this is what your
> backend DB has to handle and "how is this data processed later on?" - this
> has more or less impact on DB performance and the time it takes to create
> reports or feed any user interfaces.
> >
> > Regards,
> > Mario
> Hi Mario,
> 
> Thanks for the response. We will be collecting data from about 200 probes.
> This
> is a new endeavor so I guess we be learning on the fly. We are planning on
> using
> fsrc sampling feature set at 20 flows per minute with inserts only into a
> postgresql 9.4 DB running on CentOS 6.8 in VMware on a hefty Cisco UCS
> system.
> 
> Regards,
> Steve
> >> -Original Message-
> >> From: pmacct-discussion [mailto:pmacct-discussion-
> boun...@pmacct.net]
> >> On Behalf Of Stephen Clark
> >> Sent: Thursday, August 04, 2016 5:01 PM
> >> To: pmacct-discussion@pmacct.net
> >> Subject: [pmacct-discussion] collecting large number of netflows
> >>
> >> Hi List,
> >>
> >> I am looking to collect a large number of netflow records, on the order of
> a
> >> 100
> >> million a day,
> >> and store them in a postgres DB. Has anyone done this or something
> similar
> >> using
> >> pmacct?
> >>
> >> Thanks,
> >> Steve
> >>
> >>
> 
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Dynamic filtering of packets

[Jentsch, Mario]

Hi Mehul

> Is there any configuration to start/stop accounting at run time ?

In Pmacct only by using maps + filtering and recreating + reloading the maps 
for changes at runtime. For all I know - there is no API provided by Pmacct for 
that and there is no way implemented to trigger actions based on decoded 
packets.

Regards,
Mario


From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Wednesday, July 27, 2016 8:30 AM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mario,

I want to make configuration in PMacct for my requirement.
Let me reframe this question.

-I get triggering message in PMacct (e.g. from TCP/UDP port).
-I decode the message.
-I get an IP address and database logging on/off information.

If I get logging ON for an IP address then I want to make its entry in MySQL 
and start accounting.
If I get logging OFF for an IP address then I want to stop accounting for that 
IP.

I will ignore accounting for all other packets for which logging ON information 
is not received.

Is there any configuration to start/stop accounting at run time ?


Mehul Prajapati
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] sfacct feature suggestion - traffic in/out direction

[Jentsch, Mario]

Hi Jordan,

not sure what you mean with “equipment that cannot separate inbound/outbound 
traffic” but as long as you have direction in your flow data you can add a 
pre-tag map like

!
! tag=1  - inbound IPv4 traffic
! tag=2  - outbound IPv4 traffic
! tag=3  - inbound IPv6 traffic
! tag=4  - outbound IPv6 traffic
!
set_tag=1 ip=0.0.0.0/0 direction=0 filter='ip'
set_tag=2 ip=0.0.0.0/0 direction=1 filter='ip'
set_tag=3 ip=0.0.0.0/0 direction=0 filter='ip6'
set_tag=4 ip=0.0.0.0/0 direction=1 filter='ip6'
set_tag=0 ip=0.0.0.0/0
!

and filter e.g. the ingress flows with

!
pre_tag_filter[ingress]: 1,3
aggregate[ingress]: …
!

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jordan
Sent: Wednesday, July 27, 2016 5:06 PM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] sfacct feature suggestion - traffic in/out 
direction

Hello,

We're having issues with equipment that cannot separate inbound/outbound 
traffic using sflow V5.

Looking at the sflow V5 protocol it's having the following fields. Usually they 
match the snmp interface indexes.
source_id
interface input
interface output


What I suggest as a new feature are the following cases:

Match_all_traffic(by default) - matches all packets (as it currently works)
Match_input_only - (ifsource_id==interface inputpermit, else drop the 
rest of the samples)
Match_output_only - (ifsource_id==interface outputpermit, else drop the 
rest of the samples)


Please let me know if such feature would be possible?
If there is any other already implemented solution I would be glad to know.

Thank you in advance.

Best Regards,


--
---
Jordan

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Dynamic filtering of packets

[Jentsch, Mario]

The mentioned user groups need to be build when the user authenticates and get 
the IP address assigned. This is most probably done in your Radius 
configuration and not in pmacct.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 3:21 PM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi,

RADIUS packets are received on port 1812 & 1813.
I am decoding RADIUS packet in core process and then I get Accounting ON/OFF 
information in payload of packet.

I like your suggestion of user groups.

How can I configure user groups in PMacct ?
Is there any provision such that this Accounting ON/OFF requests configuration 
can be changes at run time ?




From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jentsch, Mario
Sent: Tuesday, July 26, 2016 6:18 PM
To: pmacct-discussion@pmacct.net<mailto:pmacct-discussion@pmacct.net>
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mehul.

the way you detect "Accounting ON/OFF" in pmacct affects possible 
recommendations. Same applies to other preconditions/requirements you may have.

If we don't need to care about them I would think you can filter out all IPs 
that you don't want accounting enabled for - this assumes there is no dynamic 
assignment from users to IP addresses.

With such a dynamic assignment a solution could be: assign the user groups 
("accounting on" vs "accounting off") different IP blocks and have pmacct 
collect the data only for the IP block for users with "accounting on".

In case your situation is more complicated and you can't take one of these ways 
it is helpful to know how you "receive some event i.e. Accounting ON in PMacct" 
and process it. Depending on if you're allowed to keep data about users with 
"accounting off" you also may solve it on MySQL side (presuming this is your 
data store). Receiving an "accounting on" you put the user/IP address for it in 
table1. Accounting data for this IP address is only stored in table2 if an 
appropriate entry in table1 is there. Receiving an "accounting off" you remove 
user/IP address for it in table1 and if required all appropriate entries from 
table2. This behavior should be possible with DB triggers / helper table.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 11:48 AM
To: pmacct-discussion@pmacct.net<mailto:pmacct-discussion@pmacct.net>
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mario,

Remote Authentication Dial-In User Service (RADIUS) is a networking 
protocol<https://en.wikipedia.org/wiki/Communications_protocol> that provides 
centralized Authentication, Authorization, and Accounting 
(AAA<https://en.wikipedia.org/wiki/AAA_protocol>) management for users who 
connect and use a network service.

You can ignore the RADIUS decoding part.

Requirement:
I want to do accounting for only selective users/IP addresses.

Let say, I receive some event i.e. Accounting ON in PMacct. Now, processing 
this event, I want to start accounting for only this IP address/user.
After some time, I receive Accounting OFF event in PMacct. Now, processing this 
event, I want to stop accounting for only this IP address/user.

Is there any mechanism to achieve it ?



From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jentsch, Mario
Sent: Tuesday, July 26, 2016 2:55 PM
To: pmacct-discussion@pmacct.net<mailto:pmacct-discussion@pmacct.net>
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mehul,

> I have explored about "refresh_maps" config key.
> If I use it, then I need to make changes in map file at run time.

that is working fine in one of our setups. We detect changes in the network, 
re-create the map file and have the daemon reload it without restart.

> But, I want to make filtering such that changes reside in memory only.

That sounds like you need to patch pmacct what IMHO is the least best solution.

> I am decoding RADIUS packet in PMacct at run-time. Therefore, I
> want to make account filtering after decoding RADIUS packet data.

Can you show us your configuration for that?

> I have looked into code and there is not handler for DELETE query in mysql.
> I want to delete records from code itself when Accounting OFF is received.

I think you need to create your own plugin for such actions based on collected 
data.

> Would you suggest any other suitable way?

Right now I can't - have to admit that I don't understand your use case and 
what your RADIUS packets are :\

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2

Re: [pmacct-discussion] Dynamic filtering of packets

[Jentsch, Mario]

Hi Mehul,

> I have explored about "refresh_maps" config key.
> If I use it, then I need to make changes in map file at run time.

that is working fine in one of our setups. We detect changes in the network, 
re-create the map file and have the daemon reload it without restart.

> But, I want to make filtering such that changes reside in memory only.

That sounds like you need to patch pmacct what IMHO is the least best solution.

> I am decoding RADIUS packet in PMacct at run-time. Therefore, I
> want to make account filtering after decoding RADIUS packet data.

Can you show us your configuration for that?

> I have looked into code and there is not handler for DELETE query in mysql.
> I want to delete records from code itself when Accounting OFF is received.

I think you need to create your own plugin for such actions based on collected 
data.

> Would you suggest any other suitable way?

Right now I can't - have to admit that I don't understand your use case and 
what your RADIUS packets are :\

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 10:27 AM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi,

Thanks for your inputs.


1)  I have explored about "refresh_maps" config key.
If I use it, then I need to make changes in map file at run time.

But, I want to make filtering such that changes reside in memory only.

I am decoding RADIUS packet in PMacct at run-time. Therefore, I want to make 
account filtering after decoding RADIUS packet data.


2)  I have looked into code and there is not handler for DELETE query in 
mysql.
I want to delete records from code itself when Accounting OFF is received.

Would you suggest any other suitable way?


From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jentsch, Mario
Sent: Tuesday, July 26, 2016 12:46 PM
To: pmacct-discussion@pmacct.net<mailto:pmacct-discussion@pmacct.net>
Subject: Re: [pmacct-discussion] Dynamic filtering of packets

Hi Mehul,

> Is there any mechanism available such that I can apply tagging and
> filtering at run time after decoding of RADIUS packet ?

Have a look at the "refresh_maps" config key. You can update your map at run 
time and have pmacct reload it by sending SIGUSR2.

> After decoding, is there any way to remove records from database
> at run time ?

Depends on the used database. With an SQL one you can, with a memory table not. 
I would consider to evaluate the "Accounting ON/OFF" flag when creating the 
report.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 8:43 AM
To: pmacct-discussion@pmacct.net<mailto:pmacct-discussion@pmacct.net>
Subject: [pmacct-discussion] Dynamic filtering of packets

Hi,

I have one Query regarding to Dynamic filtering and aggregation

Requirements:

1)  Account for only those IP addresses/users for which Accounting ON 
request is received in RADIUS packet

2)  Purge records from database for which Accounting OFF request is 
received in RADIUS packet

I have explored pre-tagging section of PMacct.
According to my understanding, it takes filtering from configuration file once 
and afterwards filtering remains same at run time.

I am decoding and processing RADIUS packet at run time.


1.   Is there any mechanism available such that I can apply tagging and 
filtering at run time after decoding of RADIUS packet ?

2.   After decoding, is there any way to remove records from database at 
run time ?



Regards,
Mehul


Mehul Prajapati

Mehul Prajapati
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Dynamic filtering of packets

[Jentsch, Mario]

Hi Mehul,

> Is there any mechanism available such that I can apply tagging and
> filtering at run time after decoding of RADIUS packet ?

Have a look at the "refresh_maps" config key. You can update your map at run 
time and have pmacct reload it by sending SIGUSR2.

> After decoding, is there any way to remove records from database
> at run time ?

Depends on the used database. With an SQL one you can, with a memory table not. 
I would consider to evaluate the "Accounting ON/OFF" flag when creating the 
report.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Mehul Prajapati
Sent: Tuesday, July 26, 2016 8:43 AM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] Dynamic filtering of packets

Hi,

I have one Query regarding to Dynamic filtering and aggregation

Requirements:

1)  Account for only those IP addresses/users for which Accounting ON 
request is received in RADIUS packet

2)  Purge records from database for which Accounting OFF request is 
received in RADIUS packet

I have explored pre-tagging section of PMacct.
According to my understanding, it takes filtering from configuration file once 
and afterwards filtering remains same at run time.

I am decoding and processing RADIUS packet at run time.


1.   Is there any mechanism available such that I can apply tagging and 
filtering at run time after decoding of RADIUS packet ?

2.   After decoding, is there any way to remove records from database at 
run time ?



Regards,
Mehul


Mehul Prajapati
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] MySQL Timezone handling

[Jentsch, Mario]

> I just feel that running the server clock in UTC will be a bit
> uncomfortable having to constantly translate the dates from UTC to
> UTC+2/3 while working with logs etc.

Sounds like you don't have server in different time zones... believe me, 
merging log data stored with local time from different time zones is no fun. 
Another problem is the daylight saving time changes where data from 2 hours is 
pushed into one or results in a one hour gap of data when some may be expected.

> How do you propose to work with UTC? Should the OS' clock run in UTC or
> only the MySQL server?

I recommend to work on backend systems with UTC only. Converting from and to 
local or user time zone on the frontend only.

> Would running only MySQL in UTC timezone be sufficient?

I would say yes as long as client connections don't set their own time zone.

Regards,
Mario

> -Original Message-
> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
> On Behalf Of Vaggelis Koutroumpas
> Sent: Tuesday, May 31, 2016 12:41 PM
> To: pmacct-discussion@pmacct.net
> Subject: Re: [pmacct-discussion] MySQL Timezone handling
> 
> Thanks for your suggestion guys,
> 
> How do you propose to work with UTC? Should the OS' clock run in UTC or
> only the MySQL server?
> 
> I just feel that running the server clock in UTC will be a bit
> uncomfortable having to constantly translate the dates from UTC to
> UTC+2/3 while working with logs etc.
> 
> Would running only MySQL in UTC timezone be sufficient?
> 
> 
> 
> On 31/5/2016 11:40 πμ, raf wrote:
> >
> >
> >> +1 to storing your application data in UTC.
> >>
> >>
> >
> > +1. Your server and storage backend should be set to UTC. This is the
> > only way to assure consistency of timestamped data.
> > Zoning should be make on the frontend side.
> >
> >
> 
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Jentsch, Mario]

Please note that for template based Netflow versions the destinations need them 
to process data flowsets. Using the samplicator not all destinations get all 
sent template flowsets - it will take some time and re-sending of these 
template flowsets till all destinations received them and understand the data 
flowsets.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Jordan Grigorov (Neterra NMT)
Sent: Wednesday, February 10, 2016 10:35 AM
To: pmacct-discussion@pmacct.net
Subject: Re: [pmacct-discussion] Question about teeing and sampling

Hello Pau,

You can try samplicate tool (https://github.com/sleinen/samplicator) to forward 
netflow data to multiple IPs/ports.

Just install it and issue:

samplicate -s 88.22.33.99 -p 9996 127.0.0.1/9995 127.0.0.1/ -f

Best Regards,


---
Jordan

On 8.02.2016 16:27, KA PDE wrote:
Hi all,

I've recently discovered pmacct and I'm evaluating it to forward netflow data 
for security purposes to a set of collectors, some of them requiring less 
amount of data sent.

I have a simple configuration using the tee plugin. I've managed to send flow 
information to NFsen but I'm unable to find a way of sampling to the other 
destination.Is this achievable with pmacct?

! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995
tee_receiver[pmacct]: 127.0.0.1:
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau




___

pmacct-discussion mailing list

http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Question about teeing and sampling

[Jentsch, Mario]

Hi Pau,

it depends on the Netflow version. With versions that use templates it may be 
the easiest way to ignore the data at the end points that is “too much”. The 
problem is that without the templates the receiver can’t process the data.

For versions without templates have a look at the ‘balance-alg’ option in the 
tee_receivers.lst example. You may send some data into “blackhole destinations” 
to get rid of it.

I don’t know how Paolo handles balancing for packets that contain Template 
FlowSet(s) – if they are forwarded to all pool destinations or not – didn’t 
test this myself or checked the code trying to find out. In case the templates 
are forwarded to all destinations in exception to the balancing method, it 
looks like you can use it for Netflow v9 etc too.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of KA PDE
Sent: Monday, February 08, 2016 3:28 PM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] Question about teeing and sampling

Hi all,

I've recently discovered pmacct and I'm evaluating it to forward netflow data 
for security purposes to a set of collectors, some of them requiring less 
amount of data sent.

I have a simple configuration using the tee plugin. I've managed to send flow 
information to NFsen but I'm unable to find a way of sampling to the other 
destination.Is this achievable with pmacct?

! nfacctd configuration
!
!
!
daemonize: true
pidfile: /var/run/nfacctd.pid
syslog: daemon

nfacctd_port: 9996
nfacctd_ip: 88.22.33.99
plugin_pipe_size: 1024
plugin_buffer_size: 10240

plugins: tee[nfsen], tee[pmacct]
tee_receiver[nfsen]: 127.0.0.1:9995
tee_receiver[pmacct]: 127.0.0.1:
! sampling_rate[pmacct]: 4096
tee_transparent: true

Thanks in advance and best regards,

Pau
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] nfacct total bytes inconsistencies

[Jentsch, Mario]

Hi Vaggelis,

do the SNMP OIDs are you monitoring for these traffic numbers include packets 
that are not exported via Netflow (broadcast, multicast etc)?

Regards,
Mario

> -Original Message-
> From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
> On Behalf Of Vaggelis Koutroumpas
> Sent: Sunday, November 29, 2015 12:23 AM
> To: Paolo Lucente; pmacct-discussion@pmacct.net
> Subject: Re: [pmacct-discussion] nfacct total bytes inconsistencies
> 
> It seems that the new server shows the same behavior after all :(
> 
> 
> mysql> SELECT (
> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB')
> as bytesFROM hourlyWHERE ip_dst = '0.0.0.0' AND stamp_inserted
> BETWEEN  '2015-11-28 20:00:00'  AND  '2015-11-28 23:59:59'
> -> ) as total_out, (
> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB')
> as bytesFROM hourlyWHERE ip_src = '0.0.0.0' AND stamp_inserted
> BETWEEN  '2015-11-28 20:00:00'  AND  '2015-11-28 23:59:59'
> -> ) as total_in;
> +---+--+
> | total_out | total_in |
> +---+--+
> | 101.03GB  | 15.43GB  |
> +---+--+
> 1 row in set (0.05 sec)
> 
> While at the same time-frame observium reports higher 'total out' and
> less 'total in' http://prntscr.com/983ers
> 
> I guess the 'total in' discrepancy is acceptable. But the 'total out' is
> over 6Gbytes off!
> 
> If I increase the time-frame then the totals are more off.
> 
> mysql> SELECT (
> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB')
> as bytesFROM hourlyWHERE ip_dst = '0.0.0.0' AND stamp_inserted
> BETWEEN  '2015-11-28 19:00:00'  AND  '2015-11-28 23:59:59'
> -> ) as total_out, (
> -> SELECT concat(truncate((sum(bytes)/1024/1024/1024),2), 'GB')
> as bytesFROM hourlyWHERE ip_src = '0.0.0.0' AND stamp_inserted
> BETWEEN  '2015-11-28 19:00:00'  AND  '2015-11-28 23:59:59'
> -> ) as total_in;
> +---+--+
> | total_out | total_in |
> +---+--+
> | 129.60GB  | 19.46GB  |
> +---+--+
> 1 row in set (0.02 sec)
> 
> Observium: http://prntscr.com/983nxa
> 
> Here the 'total out' is 8GBytes off.
> While 'total in' seems to be a little off but in acceptable range.
> 
> 
> There are no drops AFAICT.
> 
> root@netflow:~# netstat -s | grep Udp\: -A 5
> Udp:
> 817211 packets received
> 688 packets to unknown port received.
> 122 packet receive errors
> 14971 packets sent
> RcvbufErrors: 122
> 
> Those 122 errors are there for hours (before 20:00:00 of my query).
> 
> root@netflow:~# cat /proc/net/udp
>   sl  local_address rem_address   st tx_queue rx_queue tr tm->when
> retrnsmt   uid  timeout inode ref pointer drops
>   696: :0044 : 07 : 00:
>  00 10611 2 88007b36c780 0
>   751: :307B : 07 : 00:
>  00 10580 2 88007b36cb00 0
> 
> 
> I've also installed munin to monitor the performance of the server.
> MySQL does on average 40 queries/s.
> The server load is steadily 0.1
> The avg incoming packets are ~40pps
> 
> So the server is pretty much idle to lose any data.
> 
> Any ideas what else to check?
> What would be an acceptable 'off percentage' of the bytes in comparison
> with SNMP measurements?
> 
> 
> Thanks.
> 
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Fortigate netflow inaccurate?

[Jentsch, Mario]

Hi Thomas,

I guess you use sampled Netflow on the Cisco router and the renormalization 
isn't working. In that case you may need to use pmacct's sampling_map directive 
to tell it the sample rate. I didn't check (yet) why it is not working in our 
system too, we just apply the renormalization after pmacct ourself.

Regards,
Mario

-Original Message-
From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Thomas M Steenholdt
Sent: Sunday, November 01, 2015 4:56 PM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] Fortigate netflow inaccurate?

Hi guys,

NetFlow on the Fortigate devices is a relatively new thing. I've been using 
sFlow on these devices for years, and it's been working very well.

We're planning to swap out a lot of the older Fortigate devices for new Cisco 
routers that can only do NetFlow, so I'd like to get NetFlow working on the 
remaining Fortigates as well, to have all flows handled by the same system.

I have sfacctd and nfacctd both setup and configured on the same server.
The configuration of the two are almost identical, yet the flow numbers I get 
are not even close. These are the entries in the database tables for netflow vs 
sflow of me downloading a 1054867456 byte .iso file.

Just to be clear, the fortigate is exporting both NetFlow and sFlow at the same 
time. I have tried to disable sFlow, but the NetFlow results are the same.

NetFlow:
| peer | src| dst| packets | bytes   |
stamp_inserted  |
+--+++-+-+-+
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |  112586 | 6181828 |
2015-11-01 11:34:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   93304 | 5117100 |
2015-11-01 11:33:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   90794 | 4988224 |
2015-11-01 11:32:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   94255 | 5162745 |
2015-11-01 11:31:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |4622 |  251893 |
2015-11-01 11:30:00 |
totalbytes accounted for: 21701790

sFlow:
| peer | src| dst| packets | bytes |
stamp_inserted  |
+--+++-+---+-+
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   78000 |   5724000 |
2015-11-01 11:34:00 |
| 10.112.166.1 | 194.177.224.50 | 10.112.166.241 |  162000 | 232956000 |
2015-11-01 11:34:00 |
| 10.112.166.1 | 194.177.224.50 | 10.112.166.241 |  19 | 27322 |
2015-11-01 11:33:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   96000 |   7024000 |
2015-11-01 11:33:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   84000 |   6128000 |
2015-11-01 11:32:00 |
| 10.112.166.1 | 194.177.224.50 | 10.112.166.241 |  168000 | 241584000 |
2015-11-01 11:32:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   92000 |   6744000 |
2015-11-01 11:31:00 |
| 10.112.166.1 | 194.177.224.50 | 10.112.166.241 |  178000 | 255964000 |
2015-11-01 11:31:00 |
| 10.112.166.1 | 10.112.166.241 | 194.177.224.50 |   46000 |   334 |
2015-11-01 11:30:00 |
| 10.112.166.1 | 194.177.224.50 | 10.112.166.241 |   84000 | 120792000 |
2015-11-01 11:30:00 |
total bytes accounted for: 1153476000

The NetFlow bytes values are less that 2% of the sflow bytes values.

Has anybody seen this before? Perhaps I'm missing some vital clue?

I have not dug very deep into the numbers Ireceive from the Cisco boxes, but 
those numbers seem to match the actual traffic way better.

nfacctd.conf:

aggregate[netflow1m]: peer_src_ip,src_host,dst_host
aggregate[netflow1h]: peer_src_ip,src_host,dst_host
aggregate_filter[netflow1m]: net 10.0.0.0/8 or net 172.16.0.0/12 or net
192.168.0.0/16
aggregate_filter[netflow1h]: net 10.0.0.0/8 or net 172.16.0.0/12 or net
192.168.0.0/16
interface: eth0
nfacctd_ip: x.x.x.x
nfacctd_port: 2055
nfacctd_time_new: true
nfacctd_renormalize: true
plugins: mysql[netflow1m], mysql[netflow1h]
sql_optimize_clauses: true
sql_num_hosts: true
sql_locking_style: row
sql_table[netflow1m]: netflow1m
sql_table[netflow1h]: netflow1h
sql_refresh_time[netflow1m]: 60
sql_refresh_time[netflow1h]: 300
sql_dont_try_update[netflow1m]: true
sql_dont_try_update[netflow1h]: false
sql_history[netflow1m]: 1m
sql_history[netflow1h]: 1h
sql_history_roundoff[netflow1m]: m
sql_history_roundoff[netflow1h]: h

sfacctd.conf:

aggregate[sflow1m]: peer_src_ip,src_host,dst_host
aggregate[sflow1h]: peer_src_ip,src_host,dst_host
aggregate_filter[sflow1m]: net 10.0.0.0/8 or net 172.16.0.0/12 or net
192.168.0.0/16
aggregate_filter[sflow1h]: net 10.0.0.0/8 or net 172.16.0.0/12 or net
192.168.0.0/16
interface: eth0
sfacctd_ip: x.x.x.x
sfacctd_port: 6343
sfacctd_renormalize: true
plugins: mysql[sflow1m], mysql[sflow1h]
sql_optimize_clauses: true
sql_num_hosts: true
sql_locking_style: row
sql_table[sflow1m]: sflow1m
sql_table[sflow1h]: sflow1h
sql_refresh_time[sflow1m]: 60

Re: [pmacct-discussion] multiple nfacctd files being written

[Jentsch, Mario]

Hey Edward,

each file contains the data of one timebin. Flows spread over a timeframe 
longer than one timebin cause pmacct to create/update multiple files. Depending 
on how long your Netflow exporter keeps the flow records before it flushes them 
 to the collector, the created/updated files will be more or less far in the 
past.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Edward Henigin
Sent: Wednesday, October 21, 2015 10:34 PM
To: pmacct-discuss.
Subject: Re: [pmacct-discussion] multiple nfacctd files being written

And sometimes the filenames look correct but again they all come out at the 
same time:

(root) packet1:/opt/pmacct/data# ls -lt | head
total 6670424
-rw---   1 root root 3005237 Oct 21 15:28 nfacct-20151021-1525.csv
-rw---   1 root root 1461133 Oct 21 15:28 nfacct-20151021-1527.csv
-rw---   1 root root 2292406 Oct 21 15:28 nfacct-20151021-1526.csv
-rw---   1 root root 3505033 Oct 21 15:27 nfacct-20151021-1524.csv
-rw---   1 root root 2178942 Oct 21 15:26 nfacct-20151021-1523.csv
-rw---   1 root root 2551924 Oct 21 15:24 nfacct-20151021-1522.csv
-rw---   1 root root 3633218 Oct 21 15:23 nfacct-20151021-1521.csv
-rw---   1 root root 2407956 Oct 21 15:22 nfacct-20151021-1520.csv
-rw---   1 root root 2669403 Oct 21 15:21 nfacct-20151021-1519.csv
(root) packet1:/opt/pmacct/data# perl -le 'foreach (qw{1519 1520 1521 1522 1523 
1524 1525 1526 1527}){print("$_: ". 
localtime((stat("nfacct-20151021-$_.csv"))[9]))}'
1519: Wed Oct 21 15:21:46 2015
1520: Wed Oct 21 15:22:50 2015
1521: Wed Oct 21 15:23:53 2015
1522: Wed Oct 21 15:24:58 2015
1523: Wed Oct 21 15:26:02 2015
1524: Wed Oct 21 15:27:06 2015
1525: Wed Oct 21 15:28:10 2015
1526: Wed Oct 21 15:28:10 2015
1527: Wed Oct 21 15:28:10 2015

Seems related?


On Wed, Oct 21, 2015 at 3:28 PM, Edward Henigin 
> wrote:
Hi Paolo,

Running pmacct 1.5.2, simply using the print plugin, I'm getting multiple files 
coming out at the same time with filenames suggesting they should be coming out 
at different times, and sometimes very strange filenames like the data is very 
old.

ls output:

(root) packet1:/opt/pmacct/data# ls -lt | head
total 6649800
-rw---   1 root root 1785873 Oct 21 15:20 nfacct-20151021-1450.csv
-rw---   1 root root 2740509 Oct 21 15:20 nfacct-20151021-1518.csv
-rw---   1 root root 2597403 Oct 21 15:20 nfacct-20151021-1519.csv
-rw---   1 root root 2778987 Oct 21 15:19 nfacct-20151021-1517.csv
-rw---   1 root root 3017902 Oct 21 15:18 nfacct-20151021-1516.csv
-rw---   1 root root 2860626 Oct 21 15:17 nfacct-20151021-1515.csv
-rw---   1 root root 3013418 Oct 21 15:16 nfacct-20151021-1514.csv
-rw---   1 root root 3433555 Oct 21 15:15 nfacct-20151021-1513.csv
-rw---   1 root root 2752513 Oct 21 15:14 nfacct-20151021-1512.csv

and timestamps to the second:

(root) packet1:/opt/pmacct/data# perl -le 'foreach (qw{1512 1513 1514 1515 1516 
1517 1518 1450}){print("$_: ". localtime((stat("nfacct-20151021-$_.csv"))[9]))}'
1512: Wed Oct 21 15:14:18 2015
1513: Wed Oct 21 15:15:22 2015
1514: Wed Oct 21 15:16:26 2015
1515: Wed Oct 21 15:17:30 2015
1516: Wed Oct 21 15:18:34 2015
1517: Wed Oct 21 15:19:38 2015
1518: Wed Oct 21 15:20:42 2015
1450: Wed Oct 21 15:20:42 2015

Where is filename "...-1450" coming from, and why is it coming out at the same 
time as -1518?

Configuration:

! nfacctd configuration file
aggregate: peer_src_ip,in_iface,dst_host,dst_mask
plugins: print
plugin_buffer_size: 10240
imt_buckets: 157
imt_mem_pools_number: 256
imt_mem_pools_size: 32768
syslog: daemon
daemonize: true
!
print_refresh_time: 64
print_history: 1m
print_output: csv
print_output_file: /opt/pmacct/data/nfacct-%Y%m%d-%H%M.csv
print_output_file_append: true
!
nfacctd_port: 2055

Syslog:

Oct 21 15:15:21 packet1 nfacctd[558]: INFO ( default/print ): *** Purging cache 
- START (PID: 558) ***
Oct 21 15:15:22 packet1 nfacctd[558]: INFO ( default/print ): *** Purging cache 
- END (PID: 558, QN: 80643/80847, ET: 1) ***
Oct 21 15:16:25 packet1 nfacctd[593]: INFO ( default/print ): *** Purging cache 
- START (PID: 593) ***
Oct 21 15:16:26 packet1 nfacctd[593]: INFO ( default/print ): *** Purging cache 
- END (PID: 593, QN: 74092/74503, ET: 1) ***
Oct 21 15:17:29 packet1 nfacctd[651]: INFO ( default/print ): *** Purging cache 
- START (PID: 651) ***
Oct 21 15:17:30 packet1 nfacctd[651]: INFO ( default/print ): *** Purging cache 
- END (PID: 651, QN: 74211/74621, ET: 1) ***
Oct 21 15:18:33 packet1 nfacctd[690]: INFO ( default/print ): *** Purging cache 
- START (PID: 690) ***
Oct 21 15:18:34 packet1 nfacctd[690]: INFO ( default/print ): *** Purging cache 
- END (PID: 690, QN: 71470/72233, ET: 1) ***
Oct 21 15:19:37 packet1 nfacctd[739]: INFO ( default/print ): *** Purging cache 
- START (PID: 739) ***
Oct 21 15:19:38 packet1 nfacctd[739]: INFO ( default/print ): *** Purging cache 
- END (PID: 739, QN: 

Re: [pmacct-discussion] Combining the BGP and tee plugins

[Jentsch, Mario]

Hi Paul,

this is based on my understanding of pmacct not possible out of the box. With 
the tee plugin you can forward/duplicate the raw streams but not “process” them 
and add data before forwarding.

Regards,
Mario

From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf 
Of Paul S.
Sent: Friday, June 19, 2015 9:39 AM
To: pmacct-discussion@pmacct.net
Subject: [pmacct-discussion] Combining the BGP and tee plugins

Hi guys,

I'm trying to receive datagrams from switches essentially, and infuse them with 
BGP information, then retransmit.

Is this something pmacct supports and may be used to do?

If yes, what might a sample configuration look like? Rather new to the tool, 
still trying to get a grasp on things.

I have read http://wiki.pmacct.net/OfficialExamples that describes the two 
functionalities separately.

Thanks!
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] Virtual interfaces setup problem

[Jentsch, Mario]

Hey Stefan,

I use nfacctd with a custom plugin, never used pmacctd nor one of the SQL 
plugins.
My guess is that you don't use aggregate[]: none and sql_history[]: 1d.
Whatever configuration proposal I give you is a shot in the dark.

To have the data of all interfaces in one table I would create a pretag map 
file for each of them, setting the tag to an interface index I choose (e.g. 
tag=1 for eth0, tag=2 for eth1, ...) and use the tag primitive in the 
aggregate directive.

MySQL table:

CREATE TABLE `if_daily` (
`agent_id` INT(10) UNSIGNED NOT NULL,
`stamp_inserted` DATETIME NOT NULL,
`packets` INT(10) UNSIGNED NOT NULL,
`bytes` BIGINT(20) UNSIGNED NOT NULL,
`stamp_updated` DATETIME NULL DEFAULT NULL,
PRIMARY KEY (`agent_id`, `stamp_inserted`)
)

File pretag-eth0.map file:

!
set_tag: 1
!

The configuration file:

!
plugins: mysql
!
sql_optimize_clauses: true
sql_history: 1d
sql_history_roundoff: d
sql_table: if_daily
sql_refresh_time: 300
!
aggregate: tag
interface: eth0
pre_tag_map: pretag-eth0.map
!
plugin_pipe_size: 1024
plugin_buffer_size: 102400
!
sql_host: mysqld-host
sql_db: db-name
sql_user: db-username
sql_passwd: db-password
!

agent_id in the MySQL table contains the interface id as of the pre-tagging.
Data is updated every 300 seconds.

Regards,
Mario

 -Original Message-
 From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
 On Behalf Of Whisky
 Sent: Dienstag, 1. April 2014 16:16
 To: pmacct-discussion@pmacct.net
 Subject: Re: [pmacct-discussion] Virtual interfaces setup problem
 
 Hi Mario.
 
 Thanks for your message. I think what would help me most would be a
 correct
 configuration. As I said, I only need the total in and out values for one
 specific interface without ports or protocol. Shouldn't be too difficult but
 I just can't figure out how a fitting config would look like.
 
 Regards,
 
 Stefan
 

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] print plugin and src_as_path

[Jentsch, Mario]

Hi Paolo,

thanks for taking care of this!

Regards,
Mario

 -Original Message-
 From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net]
 On Behalf Of Paolo Lucente
 Sent: Mittwoch, 26. Februar 2014 18:49
 To: pmacct-discussion@pmacct.net
 Subject: Re: [pmacct-discussion] print plugin and src_as_path
 
 Hi Mario, All,
 
 A quick note to say this has now been implemented:
 
 http://www.mail-archive.com/pmacct-commits@pmacct.net/msg01006.html
 
 Cheers,
 Paolo
 
 ___
 pmacct-discussion mailing list
 http://www.pmacct.net/#mailinglists

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] print plugin and src_as_path

[Jentsch, Mario]

Hi,

the print plugin doesn't output src_as_path (tested with 1.5.0rc2).
I verified a similar setup with the memory plugin successfully and checked the 
source code, src/print_plugin.c misses src_as_path. Duplicating the lines for 
as_path and updating them to reflect src_as_path worked for me.

Regards,
Mario
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists