Re: [pmacct-discussion] Redis support
On 14/11/2016 19:42, Rasto Rickardt wrote: Hello Paolo, +1 for me, main reasons are the usual Redis ones: Pmacct have already a lot of backend plugins, but if another is needed I vote for redis. Redis is blazing fast and can act both as a message queue and a an ephemeral storage. I think I can use it for live data/trends, and passing the history to influx ? With sql/amqp/kafka + redis pmacct will cover all the current technologies. (perhaps the mongodb plugins was not necessary ;) -- Raphael Mazelier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Graphing Options
On 10/11/2016 20:58, Catalin Petrescu wrote: Hi Robert, We had good results with grafana and influxdb using below : -src_as,dst_as,peer_src_as,peer_dst_as,peer_src_ip,as_path,dst_net,src_net,dst_mask,src_mask,in_iface,std_comm,med,proto,src_as_path We export the flow records from pmacctd to kafka then import into influxdb and from there is just a matter of creating the templates. The only custom thing is kafka to influx , and we use kafka-influxdb with a inhouse parser * (kafka-influxdb). <https://github.com/mre/kafka-influxdb>* take a look * here <http://cpmarvin.blogspot.co.uk/2016/08/open-source-netflow-analysis-tool.html>* Very interesting. I've made something similar using custom amqp consumer. Is influxdb support well as much data ? because you basically insert one ts point for one flow entry. -- Raphael Mazelier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] amqp user/pass
Le 01/03/2016 23:41, Paolo Lucente a écrit : Hi Raphael, Good point you raise: legacy is never a good reason but that's the reason; ideally amqp_user and amqp_passwd should be added to the configuration struct in cfg.h and all should be pointed to those. It's the same for a few other cases (not many fortunately), like the output filename of the print plugin that uses config.sql_table. It's a number of code cleanups that i have on my todo list. Their impact is minor: all works well unless, as part of the same configuration, one does not cast, say, co-existing sql_user and amqp_user directives, ie.: We all have some legacy things to fix :) What puzzled me is that amqp_* variable are mentionned is some docs/threads on the net. So the first thing to fix is perhaps the documentation :p .. plugins: amqp[a], mysql[b] sql_user: foo amqp_user: bar .. In this case 'bar' would be used also as sql_user for the MySQL plugin; if, instead, directives are casted properly, ie.: .. plugins: amqp[a], mysql[b] sql_user[a]: foo amqp_user[b]: bar .. Then all would work perfect. All in all, something to be fixed. Yep would be perfect. -- Raphael Mazelier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Some tools
Hi guys, I just want to share with you some stuff I wrote arround pmacct. This is very specific to my buiness, but can help other to construct gui or graph arround this fantastic tool :) Note: the design is very crappy, using mysql as a buffer is stupid, but the replacement code (via amqp) is not yet ready. The code is also ugly, but simple, so anyone should understand it. - https://github.com/ut0mt8/pmacct-example - https://github.com/ut0mt8/phpflow - https://github.com/ut0mt8/flow2influx Any feedback are welcome. Regards, -- Raphael Mazelier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Webinterface
Le 08/03/15 23:21, Daniel Eschner a écrit : HI, I dont mean phpmyadmin ;) I've made a little php gui on top of a mysql database filled by nfacctd. (it was a rewrite of the Pierre Yves one, thks for the proto :) It's very ISP centrics, and very naive, but it might be a staring point. I willl made some clean, and publish it on github. Regards, -- Raphael Mazelier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Native Elasticsearch backend development
Sorry for the last message, this was the classic fail to make an private message... Le 21/02/2015 21:52, Raphael Mazelier a écrit : Salut Jérôme, Oui c'est un peu ce que je penses faire. Btw Grafana doit gérer opentsdb nativement je crois, du coup c'est pratique pour faire un dashboard. Étonnement ES est de plus en plus utilisé comme stockage intermédiaire, ce qui est paradoxal, mais grace à la stack ELK c'est très utilisé dans le monitoring. Il y a moyen que tu partages tes scripts/conf pour me donner une idée/comparer ? Cdt, -- Raphael Mazelier ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Native Elasticsearch backend development
Salut Jérôme, Oui c'est un peu ce que je penses faire. Btw Grafana doit gérer opentsdb nativement je crois, du coup c'est pratique pour faire un dashboard. Étonnement ES est de plus en plus utilisé comme stockage intermédiaire, ce qui est paradoxal, mais grace à la stack ELK c'est très utilisé dans le monitoring. Il y a moyen que tu partages tes scripts/conf pour me donner une idée/comparer ? Cdt, Le 21/02/2015 09:37, Jérôme Fleury a écrit : Hello Raphael, This is how I'm doing it: - collect in memory (many different aggregations) - script to parse and insert in OpenTSDB - home made frontend to display graphs (AngularJS app + nvd3 graphs) I am definitely interested in the ES and grafana approach but haven't gone very far at the moment. On Thu, Feb 12, 2015 at 6:36 AM, Raphael Mazelier r...@futomaki.net wrote: Hello, I'm using pmacct with success for month now. Thanks again for such a great tool. I ve found this discussion very interresting, because for now I store all data in sql. I've made a php frontend to view results, it's ok but slow (too much rows) and not realtime. So I'm searching for another approach to view realtime metrics and to store it for graphing. Speaking with some network engineer friends, there are so many options: - collect in memory (or file), convert via a script (or logg, store in Elastic search, use kibana as frontend - collect directly via ES backend, use kibane as frontend - collect in db, use a script to compute metric and put in rrd/, use whatever frontend, - collect in memory, use a script to compute metric and put in influxd, use grafana frontend etc, The last have my preference in theory. I'm really wondering what was the best solutions, perhaps a mix ? best, -- Raphael Mazelier AS39605 Le 11/02/15 10:14, Xavier Romero a écrit : Hello, I would be very interested on native elasticsearch support. I've tried in the past pmacct - logstash - elasticsearch but I've found than logstash is not so much reliable when injecting high amounts of data so I discarded the whole thing. Best regards, Xavier Romero ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Native Elasticsearch backend development
Hello, I'm using pmacct with success for month now. Thanks again for such a great tool. I ve found this discussion very interresting, because for now I store all data in sql. I've made a php frontend to view results, it's ok but slow (too much rows) and not realtime. So I'm searching for another approach to view realtime metrics and to store it for graphing. Speaking with some network engineer friends, there are so many options: - collect in memory (or file), convert via a script (or logg, store in Elastic search, use kibana as frontend - collect directly via ES backend, use kibane as frontend - collect in db, use a script to compute metric and put in rrd/, use whatever frontend, - collect in memory, use a script to compute metric and put in influxd, use grafana frontend etc, The last have my preference in theory. I'm really wondering what was the best solutions, perhaps a mix ? best, -- Raphael Mazelier AS39605 Le 11/02/15 10:14, Xavier Romero a écrit : Hello, I would be very interested on native elasticsearch support. I've tried in the past pmacct - logstash - elasticsearch but I've found than logstash is not so much reliable when injecting high amounts of data so I discarded the whole thing. Best regards, Xavier Romero ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Error
Hello Paolo, I upgrade my system today to debian wheezy, percona-server-5.6. Since this upgrade I have error message like : Jul 01 18:18:01 ERROR ( out_hour/mysql ): Data too long for column 'ip_proto' at row 445 and no data are filled in the database. How I can debug this issue ? -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Paolo, Yes it work. Time to another query, in another thread for clarity :) To the other point I stuck with my pretag filter based on interface. Regards, Le 25/06/2014 09:26, Paolo Lucente a écrit : Hi Raphael, Addressing the last open point: On Tue, Jun 24, 2014 at 01:00:37PM +0200, Raphael Mazelier wrote: So I am supposed to use v9 sql schema ? (I think tag is far more clear than agent_id). Yes, agree and would recommend so. At least run sql_table_version: 9 and sql_table_type: bgp for the basic styling (ie. tags in the 'tag' field rather than 'agent_id', etc.) then you can customize your table (ie. which fields to include/exclude) with sql_optimize_clauses: true. Cheers, Paolo -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Mario, Well I'm using inline ipfix from Juniper. Reading the juniper doc it seems that the ipv4-template does not include 'direction' field. I will stay with my current solution, using interface in and out. Regards, Le 24/06/2014 13:42, Jentsch, Mario a écrit : Hi Raphael, looks like the field direction is not set in your netflow v? data. Depending on your devices that export the netflow data another way may be to export ingress and egress to different collector instances. I can't say anything to the sql_plugin setup... Regards, Mario -Original Message- From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Raphael Mazelier Sent: Dienstag, 24. Juni 2014 13:01 To: pmacct-discussion@pmacct.net Subject: Re: [pmacct-discussion] Splitting In and Out traffic, and others questions Hi Mario, I try a pretag.map like this : set_tag=100 ip=0.0.0.0/0 direction=0 set_tag=200 ip=0.0.0.0/0 direction=1 Unfortunately that did not work as expected :/ All my flow are tagged 100 (in) and so injected in my in table. It's strange because quoting Paolo from another thread You can use pre-tagging (pre_tag_map) to do it. How simple or how tricky this is depends on the NetFlow version and exporter: 1) NetFlow v9 and IPFIX have a direction field (0 = ingress, 1 = egress) This is exactly what I wanted. To my other point, adding tag field in aggregate directive solve my problem. This value is correctly reported to the agent_id sql column. btw, I ve read in the changelog that the agent_id column was renamed by tag in the last version. SQL plugins: agent_id, agent_id2 fields renamed to tag, tag2. Issued SQL table schema #9 for agent_id backward compatibility. Renaming agent_id2 to tag2 is going to be disruptive to existing deployments instead. So I am supposed to use v9 sql schema ? (I think tag is far more clear than agent_id). Thks. Le 24/06/2014 10:32, Jentsch, Mario a écrit : Hey Raphael, we use the 1st tag to distinguish ingress and egress of IPv4 and IPv6: ! tag=1 - inbound IPv4 traffic ! tag=2 - outbound IPv4 traffic ! tag=3 - inbound IPv6 traffic ! tag=4 - outbound IPv6 traffic ! set_tag=1 ip=0.0.0.0/0 direction=0 filter='ip' set_tag=2 ip=0.0.0.0/0 direction=1 filter='ip' set_tag=3 ip=0.0.0.0/0 direction=0 filter='ip6' set_tag=4 ip=0.0.0.0/0 direction=1 filter='ip6' set_tag=0 ip=0.0.0.0/0 ! This may also work for your setup... Regards, Mario -Original Message- From: pmacct-discussion [mailto:pmacct-discussion- boun...@pmacct.net] On Behalf Of Raphael Mazelier Sent: Montag, 23. Juni 2014 14:31 To: pmacct-discussion@pmacct.net Subject: [pmacct-discussion] Splitting In and Out traffic, and others questions Hi Paolo, All, First I would thank you Paolo for this great piece of software ! Thanks to my predecessor (hi Pym) I already have a working pmacctd installation which doing accounting on my network :) I have some questions tough : I have enabled inbound accounting in my network. I want to distinguish in and out traffic. For now I make something like this, using pre_tag filter : # more /etc/pmacct/pretag.map set_tag=100 ip=158.58.176.2 in=527 set_tag=100 ip=158.58.176.2 in=528 set_tag=100 ip=158.58.176.2 in=530 ... set_tag=200 ip=158.58.176.2 out=527 set_tag=200 ip=158.58.176.2 out=528 set_tag=200 ip=158.58.176.2 out=530 ... # more /etc/pmacct/nfacctd.conf ... pre_tag_filter[in_hour]: 100 pre_tag_filter[out_hour]: 200 ... ! sql outbound by hour sql_refresh_time[out_hour]: 300 sql_history[out_hour]: 5m sql_history_roundoff[out_hour]: m sql_table[out_hour]: netflow_out_hour_%Y%m%d_%H sql_table_schema[out_hour]: /etc/pmacct/netflow_out_hour.schema ! sql inbound by hour sql_refresh_time[in_hour]: 300 sql_history[in_hour]: 5m sql_history_roundoff[in_hour]: m sql_table[in_hour]: netflow_in_hour_%Y%m%d_%H sql_table_schema[in_hour]: /etc/pmacct/netflow_in_hour.schema It's working well, but I wonder if it exists another, more clear/simpler method ? because I have to maintain the pretag.map. Or perhaps I could mix In an Out flux in the sql table (but make the table much bigger). Side question about pretag filter ? the tag field in sql is always at '0' ? This is not blocking but I wonder why ? Another question about BGP src_as and dst_as fields : Depending on the direction the src_as or the dst_as are correclty filled, but not the other which is always '0' ? I would assume that it will be my As number ? Should I have to deal with network filter ? I have many other questions, but for now I think that is sufficient :) best, -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net
[pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Paolo, All, First I would thank you Paolo for this great piece of software ! Thanks to my predecessor (hi Pym) I already have a working pmacctd installation which doing accounting on my network :) I have some questions tough : I have enabled inbound accounting in my network. I want to distinguish in and out traffic. For now I make something like this, using pre_tag filter : # more /etc/pmacct/pretag.map set_tag=100 ip=158.58.176.2 in=527 set_tag=100 ip=158.58.176.2 in=528 set_tag=100 ip=158.58.176.2 in=530 ... set_tag=200 ip=158.58.176.2 out=527 set_tag=200 ip=158.58.176.2 out=528 set_tag=200 ip=158.58.176.2 out=530 ... # more /etc/pmacct/nfacctd.conf ... pre_tag_filter[in_hour]: 100 pre_tag_filter[out_hour]: 200 ... ! sql outbound by hour sql_refresh_time[out_hour]: 300 sql_history[out_hour]: 5m sql_history_roundoff[out_hour]: m sql_table[out_hour]: netflow_out_hour_%Y%m%d_%H sql_table_schema[out_hour]: /etc/pmacct/netflow_out_hour.schema ! sql inbound by hour sql_refresh_time[in_hour]: 300 sql_history[in_hour]: 5m sql_history_roundoff[in_hour]: m sql_table[in_hour]: netflow_in_hour_%Y%m%d_%H sql_table_schema[in_hour]: /etc/pmacct/netflow_in_hour.schema It's working well, but I wonder if it exists another, more clear/simpler method ? because I have to maintain the pretag.map. Or perhaps I could mix In an Out flux in the sql table (but make the table much bigger). Side question about pretag filter ? the tag field in sql is always at '0' ? This is not blocking but I wonder why ? Another question about BGP src_as and dst_as fields : Depending on the direction the src_as or the dst_as are correclty filled, but not the other which is always '0' ? I would assume that it will be my As number ? Should I have to deal with network filter ? I have many other questions, but for now I think that is sufficient :) best, -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
