Re: [pmacct-discussion] Questions about IPFIX and pmacct
Hi Paolo, thank you very much for your prompt reply. My comments are also inline. On 28 May 2014, at 17:39, Paolo Lucente pa...@pmacct.net wrote: On Wed, May 28, 2014 at 07:47:45AM +, Thomas King wrote: - From the documentation I reasoned that pmacct/nfacct is able to handle IPFIX sampling. I use IPFIX sampling with a sampling rate of 1. From the results I see (pmacct or prng) the sampling rate is not recognised by pmacct/nfacct. I also tried to configure the sampling rate by using the configuration key nfacctd_ext_sampling_rate which did not resolve the issue. Is there a know issue with recognising the sampling rate from the IPFIX data? Or did I miss how to configure pmacct/nfacct correctly? * You are using a pre 1.5.0rc3 release. We are using the 1.5.0rc3 release. * Sampling information is not sent over by the router. This, in turn, can be because of a knob to enable on the router or due to a bug. Sniffing the raw IPFIX data and analizing with a tool like Wireshark can tell if it's the latter case. I'd be more than happy to help/support you with such analysis if we reckon all points in the direction of a bug. We double checked the IPFIX data coming from our router. The sampling rate is contained in the data. It comes via a data record (template id=256) and the relevant fields are named samplingPacketInterval and samplingPacketSpace. Do you know if pmacct is able to recognise this information? Is there is anything else (configuration file wise) what we can do? - The aggregate configuration directive comes with various values. However, I could not find a way to aggregate IPv4 and IPv6 traffic. Did I miss this in the documentation? Or is it not supported by pmacct/nfacct? I believe i should be correct decoding aggregate IPv4 and IPv6 traffic as: you want to collect traffic per source, destination and/or source-destination MAC address and distinguish v4 vs v6 traffic. If this is correct then you need the 'etype' primitive on your aggregation method. A value of 0x800 means v4, a value of 0x86dd means v6. If my understanding is not correct, please elaborate more. We tried “aggregate: etype” but we then see just 0x0800 (IPv4) traffic. We do not see any 0x86dd (IPv6) traffic. I assume the reason is that the template (L2-IP) we use does not provide any ethernet type field as I just learned. From my understanding the field IP Version (IANA element ID=60) would be the one that should be inspected. Does pmacct support the IP Version field? - I would like to generate rrd files for traffic going in and out of a MAC address. I also would like to generate rrd files for the communication between a MAC address and another MAC address (in and out). The configuration of pmacct/nfacct is actually quite easy. However, I had difficulties to generate the rrd files. I tried pnrg version 0.1 which is from 2006 and not updated ever since. It also has problems with creating rrd files and graphs based on MAC addresses. So I assume there should be a better solution than pnrg to generate rrd files. What is the default way of generating rrd files using pmacct/nfacct (I saw the section in the FAQ talking about rrd files, but this is nothing I can use as I would like to generate thousands rrd files :-))? Did you have difficulty injecting stats in RRD files or you had difficulty finding a tool that does it for you, ie. PNRG? I would like to have a tool that takes all the data available from pmacct via a memory socket and writes it periodically to rrd files. At a first glance PNRG did this. However, if a rrd filename is like a mac address PNRG stops working. Additionally, PNRG is not supported anymore. So I am looking for a similar tool. Are you aware of any tool that does this? Thanks again for your feedback! Best regards, Thomas signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Questions about IPFIX and pmacct
Hi Pierre-Yves, thanks for your suggestions. On 28 May 2014, at 18:24, Pierre-Yves Maunier pymaunier+li...@gmail.com wrote: 2014-05-28 9:47 GMT+02:00 Thomas King thomas.k...@de-cix.net: - From the documentation I reasoned that pmacct/nfacct is able to handle IPFIX sampling. I use IPFIX sampling with a sampling rate of 1. From the results I see (pmacct or prng) the sampling rate is not recognised by pmacct/nfacct. I also tried to configure the sampling rate by using the configuration key nfacctd_ext_sampling_rate which did not resolve the issue. Is there a know issue with recognising the sampling rate from the IPFIX data? Or did I miss how to configure pmacct/nfacct correctly? on my side I modify my sql queries to take our sampling rate into account : example : select round(SUM(bytes)*8000/300*8/100,2) AS mbps If I use a 8000 sampling rate, I just multiply the pmacct store value by 8000. I divide my 300 to have the number of bytes per seconds (My aggregation is done every 5 minutes) I multiply by 8 to have bits and not bytes And divide by 100 to have mbits instead of bits I can easily simplify the operations but in this example I kept it clear so it's easier to see. I would prefer something that is done by pmacct because otherwise if the sampling rates differs from port to port or router to router I have to cover each case in the database. As the sampling rate is available in the IPFIX data stream I would prefer if this data is recognised. - I would like to generate rrd files for traffic going in and out of a MAC address. I also would like to generate rrd files for the communication between a MAC address and another MAC address (in and out). The configuration of pmacct/nfacct is actually quite easy. However, I had difficulties to generate the rrd files. I tried pnrg version 0.1 which is from 2006 and not updated ever since. It also has problems with creating rrd files and graphs based on MAC addresses. So I assume there should be a better solution than pnrg to generate rrd files. What is the default way of generating rrd files using pmacct/nfacct (I saw the section in the FAQ talking about rrd files, but this is nothing I can use as I would like to generate thousands rrd files :-))? On our side we (one of my colleague did that actually) have collecd using the dbi plugin to generate RRD's using the results of SQL queries (https://collectd.org/wiki/index.php/Plugin:DBI) For instance we generate RRD's for all AS PATH which generate at least a certain amount of traffic (so I don't end up with a gazillion files). And when I want to draw the graph for a particular AS destination, I just stack all as-path rrd's ending by the AS in particular. Here is an example : https://www.dropbox.com/s/lqcp71epavci1iz/aspath_pmacct.png This graph clearly show when I setup a peering with AS XXX. (I have remove the name, scale and values of the AS Destination) Thanks for the idea. We plan to generate about 10 rrd files. So I am not sure if this solution scales with our needs. We already did some MySql-pmacct testing to test the scalability and it didn’t look too good. Currently our preferred solution is to go with rrd files as we know from other systems that working with 10 rrd files is possible. Do you have experience with such scalability demand? Best regards, Thomas signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Questions about IPFIX and pmacct
Hi Thomas, Comments in-line: On Tue, Jun 03, 2014 at 02:48:33PM +, Thomas King wrote: We double checked the IPFIX data coming from our router. The sampling rate is contained in the data. It comes via a data record (template id=256) and the relevant fields are named samplingPacketInterval and samplingPacketSpace. Do you know if pmacct is able to recognise this information? Is there is anything else (configuration file wise) what we can do? It definitely should. I recommend to send me over privately a brief trace of a few IPFIX packets (templates and data records) to look into the issue. If actual flow data records is a problem, you can send over only the sampling related part (option template plus option record), it might be sufficient to nail down the issue. We tried “aggregate: etype” but we then see just 0x0800 (IPv4) traffic. We do not see any 0x86dd (IPv6) traffic. I assume the reason is that the template (L2-IP) we use does not provide any ethernet type field as I just learned. From my understanding the field IP Version (IANA element ID=60) would be the one that should be inspected. Does pmacct support the IP Version field? Not natively. But you have a framework, aggregate_primitives config directive, and an example for its use in examples/primitives.lst in the standard pmacct code distribution, to define custom primitives. It should be as simple as you add the following to your config: aggregate_primitives: /path/to/primitives.lst aggregate: .. other primitives .. , ip_version Then define ip_version primitive in /path/to/primitives.lst as: name=ip_version field_type=60 len=1 semantics=u_int I would like to have a tool that takes all the data available from pmacct via a memory socket and writes it periodically to rrd files. At a first glance PNRG did this. However, if a rrd filename is like a mac address PNRG stops working. Additionally, PNRG is not supported anymore. So I am looking for a similar tool. Are you aware of any tool that does this? No. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Timestamps in RabbitMQ/JSON output
Hi Paolo,
On Tue, 3 Jun 2014, Paolo Lucente wrote:
What you describe for timestamps seems a good match for NetFlow, ie.
cast packets into flows and handle these via a flow-aware cache (so
active/passive expiration timers, max lifetime, etc.). All described is
already part of the nfprobe plugin. Collecting back such data via
nfacctd (on the same box where NetFlow is exported or ship it to some
central location) enables to use timestamp_start, timestamp_end
aggregation primitives - which should be precisely what you want to
achieve. The beauty is that you can have all time references possible at
once: timetamp_start, timestamp_end, stamp_inserted, stamp_updated.
Don't know how much you like/dislike the solution but i'd encourage to
run a proof-of-concept with these tools (which are all available
already) so to see we are in line with your requirements and hence take
it from there.
So at the moment I am developing this by running pmacctd (not nfacctd) on
my own laptop to collect and graph my own traffic. Thanks for the
suggestion of using timestamp_start and _end which I didn't know you could
aggregate on.
However when I added these to my aggregate line, I found that the
timestamp_start is in local time (not GMT) and a human-readable date
format, which is not great for processing in JavaScript, and timestamp_end
doesn't appear to work properly:
DEBUG ( default/amqp ): publishing [E=pmacct RK=acct DM=0]:
{timestamp_start: 2014-06-03 22:42:00.202820, ip_dst:
196.223.145.xxx, ip_proto: tcp, tos: 0, ip_src: 86.30.131.xxx,
bytes: 142, port_dst: 36363, packets: 1, port_src: 2201,
timestamp_end: 1970-01-01 03:00:00.0}
Is this a bug? Would it be easy to fix?
About sql_refresh_time less than one second. I've not considered it for
a simple reason: it seems to me like forcing an existing caching
mechanism towards a real-time use-case. Then better to disable it at all
and stream flows as they arrive onto the AMQP exchange. I have this on
my todo list - does it seem what you are looking for?
It might be. Because I'm mainly using pmacctd (not having any
netflow-capable hardware) I don't know how that would work in pmacctd.
Would you send every packet? That could be an awful lot of traffic, with
some flows having a thousand packets per second.
We could process and aggregate it all on the client side, and that has
uses (such as drilling down into individual packets), but it would be
great to have the option of aggregating them on the server as well, at a
resolution chosen by the user.
It's definitely not something that I need now, but would like you to have
it on your radar that this might be useful for some people.
Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Citylife House, Sturton Street, Cambridge, CB1 2QF, UK
Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
