Re: [pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Raphael, Addressing the last open point: On Tue, Jun 24, 2014 at 01:00:37PM +0200, Raphael Mazelier wrote: So I am supposed to use v9 sql schema ? (I think tag is far more clear than agent_id). Yes, agree and would recommend so. At least run sql_table_version: 9 and sql_table_type: bgp for the basic styling (ie. tags in the 'tag' field rather than 'agent_id', etc.) then you can customize your table (ie. which fields to include/exclude) with sql_optimize_clauses: true. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Paolo, Yes it work. Time to another query, in another thread for clarity :) To the other point I stuck with my pretag filter based on interface. Regards, Le 25/06/2014 09:26, Paolo Lucente a écrit : Hi Raphael, Addressing the last open point: On Tue, Jun 24, 2014 at 01:00:37PM +0200, Raphael Mazelier wrote: So I am supposed to use v9 sql schema ? (I think tag is far more clear than agent_id). Yes, agree and would recommend so. At least run sql_table_version: 9 and sql_table_type: bgp for the basic styling (ie. tags in the 'tag' field rather than 'agent_id', etc.) then you can customize your table (ie. which fields to include/exclude) with sql_optimize_clauses: true. Cheers, Paolo -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Mario, Well I'm using inline ipfix from Juniper. Reading the juniper doc it seems that the ipv4-template does not include 'direction' field. I will stay with my current solution, using interface in and out. Regards, Le 24/06/2014 13:42, Jentsch, Mario a écrit : Hi Raphael, looks like the field direction is not set in your netflow v? data. Depending on your devices that export the netflow data another way may be to export ingress and egress to different collector instances. I can't say anything to the sql_plugin setup... Regards, Mario -Original Message- From: pmacct-discussion [mailto:pmacct-discussion-boun...@pmacct.net] On Behalf Of Raphael Mazelier Sent: Dienstag, 24. Juni 2014 13:01 To: pmacct-discussion@pmacct.net Subject: Re: [pmacct-discussion] Splitting In and Out traffic, and others questions Hi Mario, I try a pretag.map like this : set_tag=100 ip=0.0.0.0/0 direction=0 set_tag=200 ip=0.0.0.0/0 direction=1 Unfortunately that did not work as expected :/ All my flow are tagged 100 (in) and so injected in my in table. It's strange because quoting Paolo from another thread You can use pre-tagging (pre_tag_map) to do it. How simple or how tricky this is depends on the NetFlow version and exporter: 1) NetFlow v9 and IPFIX have a direction field (0 = ingress, 1 = egress) This is exactly what I wanted. To my other point, adding tag field in aggregate directive solve my problem. This value is correctly reported to the agent_id sql column. btw, I ve read in the changelog that the agent_id column was renamed by tag in the last version. SQL plugins: agent_id, agent_id2 fields renamed to tag, tag2. Issued SQL table schema #9 for agent_id backward compatibility. Renaming agent_id2 to tag2 is going to be disruptive to existing deployments instead. So I am supposed to use v9 sql schema ? (I think tag is far more clear than agent_id). Thks. Le 24/06/2014 10:32, Jentsch, Mario a écrit : Hey Raphael, we use the 1st tag to distinguish ingress and egress of IPv4 and IPv6: ! tag=1 - inbound IPv4 traffic ! tag=2 - outbound IPv4 traffic ! tag=3 - inbound IPv6 traffic ! tag=4 - outbound IPv6 traffic ! set_tag=1 ip=0.0.0.0/0 direction=0 filter='ip' set_tag=2 ip=0.0.0.0/0 direction=1 filter='ip' set_tag=3 ip=0.0.0.0/0 direction=0 filter='ip6' set_tag=4 ip=0.0.0.0/0 direction=1 filter='ip6' set_tag=0 ip=0.0.0.0/0 ! This may also work for your setup... Regards, Mario -Original Message- From: pmacct-discussion [mailto:pmacct-discussion- boun...@pmacct.net] On Behalf Of Raphael Mazelier Sent: Montag, 23. Juni 2014 14:31 To: pmacct-discussion@pmacct.net Subject: [pmacct-discussion] Splitting In and Out traffic, and others questions Hi Paolo, All, First I would thank you Paolo for this great piece of software ! Thanks to my predecessor (hi Pym) I already have a working pmacctd installation which doing accounting on my network :) I have some questions tough : I have enabled inbound accounting in my network. I want to distinguish in and out traffic. For now I make something like this, using pre_tag filter : # more /etc/pmacct/pretag.map set_tag=100 ip=158.58.176.2 in=527 set_tag=100 ip=158.58.176.2 in=528 set_tag=100 ip=158.58.176.2 in=530 ... set_tag=200 ip=158.58.176.2 out=527 set_tag=200 ip=158.58.176.2 out=528 set_tag=200 ip=158.58.176.2 out=530 ... # more /etc/pmacct/nfacctd.conf ... pre_tag_filter[in_hour]: 100 pre_tag_filter[out_hour]: 200 ... ! sql outbound by hour sql_refresh_time[out_hour]: 300 sql_history[out_hour]: 5m sql_history_roundoff[out_hour]: m sql_table[out_hour]: netflow_out_hour_%Y%m%d_%H sql_table_schema[out_hour]: /etc/pmacct/netflow_out_hour.schema ! sql inbound by hour sql_refresh_time[in_hour]: 300 sql_history[in_hour]: 5m sql_history_roundoff[in_hour]: m sql_table[in_hour]: netflow_in_hour_%Y%m%d_%H sql_table_schema[in_hour]: /etc/pmacct/netflow_in_hour.schema It's working well, but I wonder if it exists another, more clear/simpler method ? because I have to maintain the pretag.map. Or perhaps I could mix In an Out flux in the sql table (but make the table much bigger). Side question about pretag filter ? the tag field in sql is always at '0' ? This is not blocking but I wonder why ? Another question about BGP src_as and dst_as fields : Depending on the direction the src_as or the dst_as are correclty filled, but not the other which is always '0' ? I would assume that it will be my As number ? Should I have to deal with network filter ? I have many other questions, but for now I think that is sufficient :) best, -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net
[pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Paolo, All, First I would thank you Paolo for this great piece of software ! Thanks to my predecessor (hi Pym) I already have a working pmacctd installation which doing accounting on my network :) I have some questions tough : I have enabled inbound accounting in my network. I want to distinguish in and out traffic. For now I make something like this, using pre_tag filter : # more /etc/pmacct/pretag.map set_tag=100 ip=158.58.176.2 in=527 set_tag=100 ip=158.58.176.2 in=528 set_tag=100 ip=158.58.176.2 in=530 ... set_tag=200 ip=158.58.176.2 out=527 set_tag=200 ip=158.58.176.2 out=528 set_tag=200 ip=158.58.176.2 out=530 ... # more /etc/pmacct/nfacctd.conf ... pre_tag_filter[in_hour]: 100 pre_tag_filter[out_hour]: 200 ... ! sql outbound by hour sql_refresh_time[out_hour]: 300 sql_history[out_hour]: 5m sql_history_roundoff[out_hour]: m sql_table[out_hour]: netflow_out_hour_%Y%m%d_%H sql_table_schema[out_hour]: /etc/pmacct/netflow_out_hour.schema ! sql inbound by hour sql_refresh_time[in_hour]: 300 sql_history[in_hour]: 5m sql_history_roundoff[in_hour]: m sql_table[in_hour]: netflow_in_hour_%Y%m%d_%H sql_table_schema[in_hour]: /etc/pmacct/netflow_in_hour.schema It's working well, but I wonder if it exists another, more clear/simpler method ? because I have to maintain the pretag.map. Or perhaps I could mix In an Out flux in the sql table (but make the table much bigger). Side question about pretag filter ? the tag field in sql is always at '0' ? This is not blocking but I wonder why ? Another question about BGP src_as and dst_as fields : Depending on the direction the src_as or the dst_as are correclty filled, but not the other which is always '0' ? I would assume that it will be my As number ? Should I have to deal with network filter ? I have many other questions, but for now I think that is sufficient :) best, -- Raphael Mazelier AS39605 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Splitting In and Out traffic, and others questions
Hi Raphael, Thanks for your kind words about the pmacct project. In-line: On Mon, Jun 23, 2014 at 02:30:35PM +0200, Raphael Mazelier wrote: It's working well, but I wonder if it exists another, more clear/simpler method ? because I have to maintain the pretag.map. Or perhaps I could mix In an Out flux in the sql table (but make the table much bigger). For sure you have to maintain a map to say what is input and what is output - would be great to find a way that is as most static as possible for you. What comes to mind for the purpose - all depends whether you have downstream ASNs, get at least a BGP feed or get src_as and dst_as populated from NetFlow, get MAC addresses from NetFlow, etc. - is you can use ASNs, IP prefixes, MAC addresses or interfaces (this last one is what you are doing at present). For example, should you not have downstream ASNs and get src_as and dst_as correctly populated by your router(s) via NetFlow you could simply match input traffic as dst_as=0 and output traffic as src_as=0 in your pre_tag_map. Side question about pretag filter ? the tag field in sql is always at '0' ? This is not blocking but I wonder why ? Is 'tag' part of your aggregation scheme, ie. 'aggregate' keyword in your config? If not, then that's the reason and zero is simply the default value imposed to the field in the SQL schema. Another question about BGP src_as and dst_as fields : Depending on the direction the src_as or the dst_as are correclty filled, but not the other which is always '0' ? I would assume that it will be my As number ? Should I have to deal with network filter ? Correct, when the ASN is zero then it's traffic delivered to/sourced by your own IP address space. You won't see your own ASN number being filled in - just like you don't see it in your own BGP routing table. But you can make some tricks, ie. use a networks_map, to do that. Let me know if interested. Cheers, Paolo ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists