Re: [pmacct-discussion] Collecting AS

[Scott Pettit - Vorco]

Hi Cameron,

Yes you can record AS details, but your Netflow exporter will need to send AS 
information with the flows.

Relevant bits from my config:

sql_table_version: 6
nfacctd_as_new: false
aggregate:src_host, dst_host, src_as, dst_as

You need to read into the various SQL table versions to ensure you have a 
format that suits what you want to collect. If you are using mySQL you should 
look at the sql_num_hosts feature and make use of INET_ATON/NTOA functions in 
mySQL to avoid performance issues with string values.

-Scott


From:  pmacct-discussion  on behalf of 
Cameron Murray 
Reply-To:  "pmacct-discussion@pmacct.net" 
Date:  Tuesday, 15 December 2015 at 21:47
To:  "pmacct-discussion@pmacct.net" 
Subject:  [pmacct-discussion] Collecting AS


Hi All,

New to pmacct so be kind please. Below is a copy of my config which I have only 
data for our IP's being filtered however it would be great also to record AS 
details so we can identify top traffic destinations so we can look at peering 
to save on transit
 costings.

Is there an easy way to get this data and store it?

 The content of this message and any attachments may be privileged, 
confidential or sensitive. Any unauthorised used is prohibited. Views expressed 
in this message are those of the individual sender, except where stated 
otherwise with appropriate authority. All pricing provided is valid at the time 
of writing only and due to factors such as the exchange rate, may change 
without notice. Sales are made subject to our Terms & Conditions, available on 
our website or on request.

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Re: [pmacct-discussion] mySQL ATON/NTOA and pmacct performance

[Scott Pettit - Vorco]

In MySQL (any SQL actually), string searches are computationally very
expensive when compared with integers. If you’re dealing with a relatively
small number of rows then the difference isn’t a problem.

It’s like asking “show me all the rows where the number is between 1 and
1000” versus “show me all the rows that look like something between one
and one thousand in words”.  The second statement is the same question but
requires a lot more processing because the database has to think about
every number as a word.  It might only take fractions of a second longer
for one row, but multiply that by 300 million rows (which is what I’m
trying to analyse) and the difference is huge. PostgreSQL handles this
really well as it has support for IP’s as a column type which presumably
converts to integers in the back end.

I will try pmacct with ipv4 on my test collector with sql_num_hosts and if
it works well then having the same for inet6_aton would be great.

-Scott


On 17/09/14 16:03, itria30...@itri.org.tw itria30...@itri.org.tw wrote:

Hi Scott and Paolo,

  May I ask a question per your good experience?

  What is the performance difference between using string ipv4 and
integer ipv4?  And potential impact?

  I have a PC running sfacctd that collects 2 giga-switches.  And it
collects around 5M records into mysql per day.

  I am satisfied with the default string ipv4 for its convenience of
being readable.  And I would like to know the performance drawback.


 The content of this message and any attachments may be privileged, 
confidential or sensitive. Any unauthorised used is prohibited. Views expressed 
in this message are those of the individual sender, except where stated 
otherwise with appropriate authority. All pricing provided is valid at the time 
of writing only and due to factors such as the exchange rate, may change 
without notice. Sales are made subject to our Terms  Conditions, available on 
our website or on request.

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

[pmacct-discussion] mySQL ATON/NTOA and pmacct performance

[Scott Pettit - Vorco]

Hi,

I've been running pmacct for a while with PostgreSQL and it works well with 
lots of netflow data presumably due to inbuilt handling of CIDR.

For various reasons we're migrating to a mySQL backend to integrate with our 
billing/customer portal environment better and mySQL does not appear to deal 
well with strings in complex reports.

We generate around 10GB of data per hour so when I want to do something like 
select all the flows over a week and then spit out the top 50 AS sources, show 
the total data from each of the top AS sources, and then the top 50 destination 
IP's in our network that those AS's are sending data to, mySQL chokes for an 
hour trying to work that out whereas PostgreSQL takes a few minutes but doesn't 
struggle. I've played with indexes and got some improvement but nothing 
significant.

I was looking into using ATON/NTOA functions in mySQL to store IP addresses as 
integers instead which I expect should dramatically increase performance 
especially when dealing with CIDR as I can query based on a range of integers.

As far as I can see pmacct doesn't seem to have an option to tell it to do its 
INSERTs of ip_src and ip_dst with INET_ATON and INET6_ATON. Is there such an 
option or a patch that exists? If not is there a reason why this would be a bad 
idea?

-Scott

The content of this message and any attachments may be privileged, confidential 
or sensitive. Any unauthorised used is prohibited. Views expressed in this 
message are those of the individual sender, except where stated otherwise with 
appropriate authority. All pricing provided is valid at the time of writing 
only and due to factors such as the exchange rate, may change without notice. 
Sales are made subject to our Terms  Conditions, available on our website or 
on request.

___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists