Re: [pmacct-discussion] Netflow and 'ip' key in pretag.map
Hi Damian, Are you running BGP? Would it be feasible for you to past BGP feed(s) into pmacct (granted you run a recent 0.12 release or can upgrade to that)? Idea is you can attach BGP standard communities to IP prefixes as they are advertised or re-distributed into your network. Because comunities are supported as primitives in pmacct (and you can filter what you want to see and what should be thrashed instead), you might even avoid the whole 'id2' section. Just an idea. In case BGP is not an option: total number of mappings defined isn't an issue. In this sense, to store 2K entries you should make use of the pre_tag_map_entries directive (and set it above 2K). Not very nice to see but it works and should be straightforward to automate. It's important though to keep the walks through the maps as short as possible; so, id=1 ip=X jeq=user_X id=1 ip=Y jeq=user_Y ... id=1 ip=Z jeq=user_Z ! id=userid ip=X filter='...' label=user_X jeq=traffic_type_X id=userid ip=Y filter='...' label=user_Y jeq=traffic_type_Y ... id=userid ip=Z filter='...' label=user_Z jeq=traffic_type_Z ! id2=1 ip=X filter='...' label=traffic_type_X jeq=next ... id2=1 ip=Y filter='...' label=traffic_type_Y jeq=next ... id2=1 ip=Z filter='...' label=traffic_type_Z jeq=next ... Cheers, Paolo On Wed, Jul 21, 2010 at 11:33:38AM +1200, Damian Kissick wrote: [ ... ] So finally to the crux of the original question; on the single-router setup, netflow on the appropriate ingress and egress interfaces works and all the traffic is marked appropriately with a userid (tag) and traffic type (tag2). But I know that upon adding the additional routers for the other traffic users, I will currently have to duplicate the id2 mappings for each netflow agent's IP. I am trying to find a way around that to keep the pretag.map efficient (or maybe ~500 networks x 4 netflow agents for 2000 mappings is not actually too bad?) One solution that I am contemplating is to move away from netflow and enable sflow on our core switches which would keep the required duplication of mappings down. Or, in the same way our current system collects traffic data, use mirror ports (on the core switches) and then use the pmacctd daemon instead. I suspect I am overlooking some more obvious solutions so I really appreciate pointers on this and if you or anyone sees other issues or better design tips for this, then I welcome the feedback. Cheers, - Damian ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] How many ips can pmacctd monitorize
I was Reading CONFIG-KEYS and internals documents but im doing something wrong because the pmacct is stolen data. When I look into the table for the traffic of one ip I see that the accumulate is stolen. # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.1751277 14929314 # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.1753033 15176927 # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.172803477449 As you can see the bytes column is minor that before. The last time get 477449 when 10 seconds before has 15176927. My config is: ! ! pmacct: In + Out ! debug: false daemonize: true interface: eth1 promisc: true syslog: kern imt_mem_pools_size: 12000 imt_mem_pools_number: 20 plugin_buffer_size: 20480 networks_file: /etc/pmacct/networks2.def plugins: memory[net_in], memory[net_out], memory[host_in], memory[host_out] aggregate[net_in]: dst_net aggregate[net_out]: src_net aggregate[host_in]: dst_host aggregate[host_out]: src_host imt_path[net_in]: /tmp/net_in.pipe imt_path[net_out]: /tmp/net_out.pipe imt_path[host_in]: /tmp/host_in.pipe imt_path[host_out]: /tmp/host_out.pipe The traffic that I want to monitorize is 70 Mb out and 30 Mb In, is not to much but im getting stolen data. Thanks for the help and sorry for my bad English. Best regards. Jose Joaquín Antón Herrerías J jan...@abserver.es - Access Basic Server S.L. Elche Parque Industrial. C/Galileo Galilei, 12. 03203 Elche (Alicante). Telf. +34 96 568 29 04 / 902 380 380 Fax. +34 96 568 35 30 Cláusula de confidencialidad: Este mensaje se dirige exclusivamente al destinatario consignado. Puede contener información confidencial, de nuestra propiedad o legalmente protegida. Si usted no es el destinatario, le informamos que cualquier acceso, divulgación, copia o distribución de la información, así como cualquier acción u omisión realizada con base a la misma, queda prohibida y puede ser ilegal. En caso de haber recibido este mensaje por error, le rogamos nos lo reenvíe y notifique inmediatamente, borrando toda copia de su sistema. Gracias. _ Antes de imprimir este mensaje, asegúrese de que es necesario. Proteger el medio ambiente está en nuestra mano. Piensa en global, actúa en local. -Mensaje original- De: pmacct-discussion-boun...@pmacct.net [mailto:pmacct-discussion-boun...@pmacct.net] En nombre de Paolo Lucente Enviado el: lunes, 19 de julio de 2010 20:20 Para: pmacct-discussion@pmacct.net Asunto: Re: [pmacct-discussion] How many ips can pmacctd monitorize Hi, I'd say limit on IP addresses that can be accounted of is only imposed by resources available (memory) and, when using memory tables (which is your case), the imt_mem_pools_number setting (see CONFIG-KEYS for more details). I see a couple of issues with your config: * quite large plugin_buffer_size setting; i just wonder if you had to come up to such huge buffers because of packet loss. If this is the case then OK; otherwise, if traffic rate does not justify it i would advice to scale it down, say, to 20KB or 100KB. Buffering too much has also negative impact. * If i get correctly you want to have a two tiered approach: you want to account for /24 subnets (so that you have a more compact overview of what's going on) and for /32 IP addresses so that you can drill down who is actually generating traffic within a certin /24. If this is the case, i would recommend you to re-write your config as follows: ... networks_file: /etc/pmacct/networks.def plugins: memory[net_in], memory[net_out], memory[host_in], memory[host_out] aggregate[net_in]: dst_net aggregate[net_out]: src_net aggregate[host_in]: dst_host aggregate[host_out]: src_host imt_path[net_in]: /tmp/net_in.pipe imt_path[net_out]: /tmp/net_out.pipe imt_path[host_in]: /tmp/host_in.pipe imt_path[host_out]: /tmp/host_out.pipe ... And in the networks.def (which definition is shared among all the plugins) you put only the /24 ranges - scrap all the /32 definitions. I guess the rest should be pretty much intuitive. Cheers, Paolo On Mon, Jul 19, 2010 at 02:36:56PM +, Jose Joaquin Anton Herrerias wrote: Hi everyone, I'm new in pmacctd and i using pmacctd with cacti for drawing graph of the network, now I want to monitorize the traffic of all ips but are a lot, 2040 ips and the complete range for give a overall view of the range traffic. But I have a trouble, I configure the network.def with all ips (/32) and all ranges (/24) and I have my pmacctd.conf with: ! ! pmacct: In + Out ! debug: false daemonize: true interface: eth1 plugin_buffer_size: 2048000 plugin_pipe_size: 204800 networks_file: /etc/pmacct/networks.def
Re: [pmacct-discussion] How many ips can pmacctd monitorize
Hi, Try performing a locked access to the memory table. This can be done by appending a '-l' option to the command, ie. pmacct -s -l -p Let me know. Cheers, Paolo On Thu, Jul 22, 2010 at 10:35:18AM +, Jose Joaquin Anton Herrerias wrote: I was Reading CONFIG-KEYS and internals documents but im doing something wrong because the pmacct is stolen data. When I look into the table for the traffic of one ip I see that the accumulate is stolen. # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.1751277 14929314 # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.1753033 15176927 # pmacct -s -p /tmp/host_in.pipe |grep 184.17 XX.XXX.184.172803477449 As you can see the bytes column is minor that before. The last time get 477449 when 10 seconds before has 15176927. My config is: ! ! pmacct: In + Out ! debug: false daemonize: true interface: eth1 promisc: true syslog: kern imt_mem_pools_size: 12000 imt_mem_pools_number: 20 plugin_buffer_size: 20480 networks_file: /etc/pmacct/networks2.def plugins: memory[net_in], memory[net_out], memory[host_in], memory[host_out] aggregate[net_in]: dst_net aggregate[net_out]: src_net aggregate[host_in]: dst_host aggregate[host_out]: src_host imt_path[net_in]: /tmp/net_in.pipe imt_path[net_out]: /tmp/net_out.pipe imt_path[host_in]: /tmp/host_in.pipe imt_path[host_out]: /tmp/host_out.pipe The traffic that I want to monitorize is 70 Mb out and 30 Mb In, is not to much but im getting stolen data. Thanks for the help and sorry for my bad English. Best regards. Jose Joaqu??n Ant??n Herrer??as J jan...@abserver.es ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmpgplay - what does it do?
Hi, I'm using pmacct-0.12.0rc3 with postgresql. I've been trying to work out what pmpgplay does when it looks at a log file. I couldn't find much in my searches that go into the details of how it does things. It seems it generates a bunch of sql, but does it check to see if that record exists already before applying the change? I'm worried about replaying the log and it double-recording data (ie the data is already recorded in the db, and then replaying the log does it again), or will it only apply new records that it can't find in the db? I have a missing chunk of data from my database (about 1.5 hrs worth) that I can see in the logs, but the log is surrounded by data that could be in the database (I haven't checked each individual log entry). Relevant pmacct.conf info: plugin_pipe_size:1024000 plugin_buffer_size:8192 imt_buckets: 65537 imt_mem_pools_size: 1024000 sql_table_version:1 sql_refresh_time: 60 sql_history: 5m Any information is much appreciated, thanks! -- Postgresql php tutorials http://www.designmagick.com/ ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
