Re: [pmacct-discussion] Request for Feedback: Additional Features for pmacct
Hi Andreas, thanks for your feedback. The thing we would mainly focus on our requirements and use cases. Would you be interested in adding your requirements and use cases to a piece of software we might open-source initialy? Best regards, Thomas On 18 Sep 2014, at 18:10, Andreas Larsen andreas.lar...@ip-only.se wrote: These addon looks great in my book. I really do like pmacct but have not had the chance to implement it other than in testing due the the lack of a good frontend ( and due to the fact that we lack time to develop one ourself) Med vänlig hälsning Andreas Larsen IP-Only Telecommunication AB| Postadress: 753 81 UPPSALA | Besöksadress: S:t Persgatan 6, Uppsala | Telefon: +46 (0)18 843 10 00 | Direkt: +46 (0)18 843 10 56 www.ip-only.se 18 sep 2014 kl. 15:32 skrev Karl O. Pinc k...@meme.com: On 09/16/2014 11:50:56 AM, Thomas King wrote: On 16 Sep 2014, at 17:37, Karl O. Pinc k...@meme.com wrote: On 09/16/2014 04:25:19 AM, Thomas King wrote: We are thinking about adding features in the following categories: - Reconfiguration via API: As we want to use pmacct in a dynamic environment we want to be able to change the configuration via an API without restarting pmacct. Just curious. Why is an API required. Isn't a signal to re-read configs enough? If this can be done without interrupting the pmacct operations (and losing any data) it is enough I like the idea of a signal, it being the simplest API there is. It'd also then be trivial to add support for a system pmacct startup script that supports a reload operation, without data loss. Just my 2 cents. Karl k...@meme.com Free Software: You don't pay back, you pay forward. -- Robert A. Heinlein ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Request for Feedback: Additional Features for pmacct
Dear all, we at DE-CIX think about adding new features to pmacct that are needed from our side. As pmacct is open-source we would like to release our additions as open-source as well. To assess if our features are of general interest I would like to discuss our features with you. The questions for us is here: Are the features we might add be supported by the community? We are thinking about adding features in the following categories: - High availability: We would like to enhance pmacct in such a way that if pmacct dies on one machine another will take over automatically. We assume that existing open-source tools (e.g., heart-beat) can be combined with pmacct in order to achieve this. - Reconfiguration via API: As we want to use pmacct in a dynamic environment we want to be able to change the configuration via an API without restarting pmacct. - Report and statistic engine: We would like to define a set of reports (e.g., throughput in bits per seconds pro MAC address or interface) which can be displayed on a website including fancy looking graphs. This will probably not be included into pmacct directly instead it will be a standalone tool that relies on pmacct. - Notification: If a threshold is met we want to be informed by mail (e.g., the throughput for an interface reaches a certain level we want to receive a mail). This will probably not be included into pmacct directly instead it will be a standalone tool that relies on pmacct. I am aware that these feature descriptions are somewhat high-level. We already created a more detailed technical description of what we need to add to pmacct so that it fits our requirements. We will share this list later but we do not want to steer the discussion to much in one direction. Any feedback to our ideas is highly appreciated! Best regards, Thomas -- Dr. Thomas King Manager Research Development DE-CIX Management GmbH | Lindleystraße 12 | 60314 Frankfurt am Main | Germany | www.de-cix.net Phone +49 69 1730902 87 | Mobile +49 175 1161428 | Fax +49 69 4056 2716 | thomas.k...@de-cix.net Geschaeftsfuehrer Harald A. Summa | Registergericht AG Koeln HRB 51135 signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Request for Feedback: Additional Features for pmacct
On 16 Sep 2014, at 17:37, Karl O. Pinc k...@meme.com wrote: On 09/16/2014 04:25:19 AM, Thomas King wrote: We are thinking about adding features in the following categories: - Reconfiguration via API: As we want to use pmacct in a dynamic environment we want to be able to change the configuration via an API without restarting pmacct. Just curious. Why is an API required. Isn't a signal to re-read configs enough? If this can be done without interrupting the pmacct operations (and losing any data) it is enough. Best regards, Thomas signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Questions about IPFIX and pmacct
Hi Paolo, thank you very much for your prompt reply. My comments are also inline. On 28 May 2014, at 17:39, Paolo Lucente pa...@pmacct.net wrote: On Wed, May 28, 2014 at 07:47:45AM +, Thomas King wrote: - From the documentation I reasoned that pmacct/nfacct is able to handle IPFIX sampling. I use IPFIX sampling with a sampling rate of 1. From the results I see (pmacct or prng) the sampling rate is not recognised by pmacct/nfacct. I also tried to configure the sampling rate by using the configuration key nfacctd_ext_sampling_rate which did not resolve the issue. Is there a know issue with recognising the sampling rate from the IPFIX data? Or did I miss how to configure pmacct/nfacct correctly? * You are using a pre 1.5.0rc3 release. We are using the 1.5.0rc3 release. * Sampling information is not sent over by the router. This, in turn, can be because of a knob to enable on the router or due to a bug. Sniffing the raw IPFIX data and analizing with a tool like Wireshark can tell if it's the latter case. I'd be more than happy to help/support you with such analysis if we reckon all points in the direction of a bug. We double checked the IPFIX data coming from our router. The sampling rate is contained in the data. It comes via a data record (template id=256) and the relevant fields are named samplingPacketInterval and samplingPacketSpace. Do you know if pmacct is able to recognise this information? Is there is anything else (configuration file wise) what we can do? - The aggregate configuration directive comes with various values. However, I could not find a way to aggregate IPv4 and IPv6 traffic. Did I miss this in the documentation? Or is it not supported by pmacct/nfacct? I believe i should be correct decoding aggregate IPv4 and IPv6 traffic as: you want to collect traffic per source, destination and/or source-destination MAC address and distinguish v4 vs v6 traffic. If this is correct then you need the 'etype' primitive on your aggregation method. A value of 0x800 means v4, a value of 0x86dd means v6. If my understanding is not correct, please elaborate more. We tried “aggregate: etype” but we then see just 0x0800 (IPv4) traffic. We do not see any 0x86dd (IPv6) traffic. I assume the reason is that the template (L2-IP) we use does not provide any ethernet type field as I just learned. From my understanding the field IP Version (IANA element ID=60) would be the one that should be inspected. Does pmacct support the IP Version field? - I would like to generate rrd files for traffic going in and out of a MAC address. I also would like to generate rrd files for the communication between a MAC address and another MAC address (in and out). The configuration of pmacct/nfacct is actually quite easy. However, I had difficulties to generate the rrd files. I tried pnrg version 0.1 which is from 2006 and not updated ever since. It also has problems with creating rrd files and graphs based on MAC addresses. So I assume there should be a better solution than pnrg to generate rrd files. What is the default way of generating rrd files using pmacct/nfacct (I saw the section in the FAQ talking about rrd files, but this is nothing I can use as I would like to generate thousands rrd files :-))? Did you have difficulty injecting stats in RRD files or you had difficulty finding a tool that does it for you, ie. PNRG? I would like to have a tool that takes all the data available from pmacct via a memory socket and writes it periodically to rrd files. At a first glance PNRG did this. However, if a rrd filename is like a mac address PNRG stops working. Additionally, PNRG is not supported anymore. So I am looking for a similar tool. Are you aware of any tool that does this? Thanks again for your feedback! Best regards, Thomas signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Questions about IPFIX and pmacct
Hi Pierre-Yves, thanks for your suggestions. On 28 May 2014, at 18:24, Pierre-Yves Maunier pymaunier+li...@gmail.com wrote: 2014-05-28 9:47 GMT+02:00 Thomas King thomas.k...@de-cix.net: - From the documentation I reasoned that pmacct/nfacct is able to handle IPFIX sampling. I use IPFIX sampling with a sampling rate of 1. From the results I see (pmacct or prng) the sampling rate is not recognised by pmacct/nfacct. I also tried to configure the sampling rate by using the configuration key nfacctd_ext_sampling_rate which did not resolve the issue. Is there a know issue with recognising the sampling rate from the IPFIX data? Or did I miss how to configure pmacct/nfacct correctly? on my side I modify my sql queries to take our sampling rate into account : example : select round(SUM(bytes)*8000/300*8/100,2) AS mbps If I use a 8000 sampling rate, I just multiply the pmacct store value by 8000. I divide my 300 to have the number of bytes per seconds (My aggregation is done every 5 minutes) I multiply by 8 to have bits and not bytes And divide by 100 to have mbits instead of bits I can easily simplify the operations but in this example I kept it clear so it's easier to see. I would prefer something that is done by pmacct because otherwise if the sampling rates differs from port to port or router to router I have to cover each case in the database. As the sampling rate is available in the IPFIX data stream I would prefer if this data is recognised. - I would like to generate rrd files for traffic going in and out of a MAC address. I also would like to generate rrd files for the communication between a MAC address and another MAC address (in and out). The configuration of pmacct/nfacct is actually quite easy. However, I had difficulties to generate the rrd files. I tried pnrg version 0.1 which is from 2006 and not updated ever since. It also has problems with creating rrd files and graphs based on MAC addresses. So I assume there should be a better solution than pnrg to generate rrd files. What is the default way of generating rrd files using pmacct/nfacct (I saw the section in the FAQ talking about rrd files, but this is nothing I can use as I would like to generate thousands rrd files :-))? On our side we (one of my colleague did that actually) have collecd using the dbi plugin to generate RRD's using the results of SQL queries (https://collectd.org/wiki/index.php/Plugin:DBI) For instance we generate RRD's for all AS PATH which generate at least a certain amount of traffic (so I don't end up with a gazillion files). And when I want to draw the graph for a particular AS destination, I just stack all as-path rrd's ending by the AS in particular. Here is an example : https://www.dropbox.com/s/lqcp71epavci1iz/aspath_pmacct.png This graph clearly show when I setup a peering with AS XXX. (I have remove the name, scale and values of the AS Destination) Thanks for the idea. We plan to generate about 10 rrd files. So I am not sure if this solution scales with our needs. We already did some MySql-pmacct testing to test the scalability and it didn’t look too good. Currently our preferred solution is to go with rrd files as we know from other systems that working with 10 rrd files is possible. Do you have experience with such scalability demand? Best regards, Thomas signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Questions about IPFIX and pmacct
Dear all, I am currently investigating how pmacct can be used together with IPFIX data coming from Alcatel Lucent 7750 routers. During the past couple of days I hit a few questions I could not answer by myself: - From the documentation I reasoned that pmacct/nfacct is able to handle IPFIX sampling. I use IPFIX sampling with a sampling rate of 1. From the results I see (pmacct or prng) the sampling rate is not recognised by pmacct/nfacct. I also tried to configure the sampling rate by using the configuration key nfacctd_ext_sampling_rate which did not resolve the issue. Is there a know issue with recognising the sampling rate from the IPFIX data? Or did I miss how to configure pmacct/nfacct correctly? - The aggregate configuration directive comes with various values. However, I could not find a way to aggregate IPv4 and IPv6 traffic. Did I miss this in the documentation? Or is it not supported by pmacct/nfacct? - I would like to generate rrd files for traffic going in and out of a MAC address. I also would like to generate rrd files for the communication between a MAC address and another MAC address (in and out). The configuration of pmacct/nfacct is actually quite easy. However, I had difficulties to generate the rrd files. I tried pnrg version 0.1 which is from 2006 and not updated ever since. It also has problems with creating rrd files and graphs based on MAC addresses. So I assume there should be a better solution than pnrg to generate rrd files. What is the default way of generating rrd files using pmacct/nfacct (I saw the section in the FAQ talking about rrd files, but this is nothing I can use as I would like to generate thousands rrd files :-))? Any help/hint/feedback is highly appreciated. Thanks in advance! Best regards, Thomas signature.asc Description: Message signed with OpenPGP using GPGMail ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
