[pmacct-discussion] ODP: NFv9 Unknown Template
Hi Robert Could You show Your netflow/jflow configuration on Your SRX ? Best Adam Od: Robert Juric Wysłano: czwartek, 10 marca 2016 18:13 Do: pmacct-discussion@pmacct.net Temat: [pmacct-discussion] NFv9 Unknown Template I correct the mysql configuration and when I went to change and I restarted the service to change the table version I'm now seeing NFv9 packets received and discarded for Unknown Template. I've not been able to find much information regarding this. I'm using a Juniper SRX router with inline-jflow. root@debian-netflow:/etc/pmacct# nfacctd -l 2100 -P print -c none -d true DEBUG: [cmdline] plugin name/type: 'default'/'core'. DEBUG: [cmdline] plugin name/type: 'default'/'print'. DEBUG: [cmdline] nfacctd_port:2100 DEBUG: [cmdline] aggregate:none DEBUG: [cmdline] debug:true INFO ( default/core ): Reading configuration from cmdline. INFO ( default/print ): plugin_pipe_size=4096000 bytes plugin_buffer_size=228 bytes INFO ( default/print ): ctrl channel: obtained=212992 bytes target=143712 bytes INFO ( default/core ): waiting for NetFlow data on 0.0.0.0:2100 INFO ( default/print ): cache entries=16411 base cache memory=44769208 bytes PACKETS BYTES DEBUG ( default/core ): Received NetFlow/IPFIX packet from [192.168.1.1:55602] version [9] seqno [45617] DEBUG ( default/core ): Discarded NetFlow v9/IPFIX packet (R: unknown template 257 [192.168.1.1:142]) DEBUG ( default/core ): Received NetFlow/IPFIX packet from [192.168.1.1:55602] version [9] seqno [45618] DEBUG ( default/core ): Discarded NetFlow v9/IPFIX packet (R: unknown template 257 [192.168.1.1:142]) Could anyone point me in the right direction? ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Problem with flow replication
Hi, I have some odd problem This is what I have - Juniper router with logical-systems - I have some routes (BGP sessions) in primary routing table (non LS) and different routing table in one logical-system. I set up a BGP sessions from pmacct to both routers (non-LS and LS). Juniper export all flows with IP address from non-LS - because I need to resolve flows based on BGP in LS I'm doing this like this: flows are send to IP e.g. a.a.a.a on port 3000 from there I replicate flows to 127.0.0.1 to ports 5000 and 6000 Then I run two tee plugins with this configuration: nfacctd_port: 7000 nfacctd_ip: 127.0.0.1 plugins: tee[lo5] tee_receiver[lo5]: b.b.b.b:2001 tee_source_ip[lo5]: c.c.c.c tee_transparent[lo5]: false and second config: nfacctd_port: 6000 nfacctd_ip: 127.0.0.1 plugins: tee[lo6] tee_receiver[lo6]: b.b.b.b:2001 tee_transparent[lo6]: true Small explanation for above - flows from Juniper are replicated to ports above (5000 and 6000) and from them I send them to pmacct (b.b.b.b) - for lo5 I change IP address for IP from LS (c.c.c.c) and for lo6 I leave it unchanged (IP from non-LS) now on pmacct machine I get two exactly the same flows but visible as sent from two machines (to this point everything looks fine, I even checked packets send from Juniper to tee and then sent to pmacct (iptables counters) and it looks fine) The problem - when I enable only tee[lo5] I get proper traffic value on pmacct but when I enable tee[lo6] then traffic which I get in graphite instantly drop Here You can see how it looks like: http://postimg.org/image/zb9u1ywaj/ To 21:00 I get some traffic (enabled tee[lo5] and tee[lo6]) after 21:00 I disabled tee[lo6] and traffic instantly increased to the proper value - after 22:00 enabled tee[lo6] again The problem is exactly the same if I enable/disable tee[lo5] - then on tee[lo6] traffic value increase or deacrease I'm sitting on this second days and no idea where to search - nfacctd config file on pmacct machine should be ok - beacause it's working for others flows which I get from other routers This is screen from today: http://postimg.org/image/6qgd4sztj/ - after 16:00 I enabled one of tee In pmacct logs there are no errors - I even enabled debug for one of plugins: May 07 16:43:01 DEBUG ( DATA1/mysql ): 975 VALUES statements sent to the MySQL server. May 07 16:43:01 INFO ( DATA1/mysql ): *** Purging cache - END (PID: 2502, QN: 975/975, ET: 0) *** Only when I get data from sql - there is a big difference between 15:54 and 16:08 Anyone have any idea where to search ? Best regards Adam Bogdan ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Network file not properly load
Hi Joan, The problem is with this 2 lines: 123.123.123.123,55649,223.255.240.0/22 123.123.123.123,55649,223.255.240.0/24 Just delete the line with /24 and check then - I had similar problem with overlapping prefixes. Regards Adam 2014-02-13 15:36 GMT+01:00 Joan aseq...@gmail.com: While loading the attached network file, I get this strange errors on the logs (when debug is enabled), it seems that the networks are not properly imported (it seems related to the nested networks) but I couldn't simplify any more the test case. The problem is that when there are those errors the srcas and dstas never get populated on the flows. Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 17766 net: 223.255.235.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 45954 net: 223.255.244.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 45954 net: 223.255.245.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 45954 net: 223.255.246.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 45954 net: 223.255.247.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 55415 net: 223.255.254.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: 0.0.0.0 mask: 0 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] contains a default route Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: 0.0.0.0 mask: 0 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] contains a default route Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: 0.0.0.0 mask: 0 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] contains a default route Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: 0.0.0.0 mask: 0 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] contains a default route Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: peer asn: 0 asn: 0 net: 0.0.0.0 mask: 0 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] contains a default route Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 55649 net: 223.255.240.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 55649 net: 223.255.241.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 55649 net: 223.255.242.0 mask: 24 Feb 13 15:31:07 collector pmacctd[29186]: DEBUG ( /etc/pmacct/networks.lst ): [networks table IPv4] nh: 123.123.123.123 peer asn: 0 asn: 55649 net: 223.255.243.0 mask: 24 ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] Trafifc level from nfacctd and snmp
Hi,
I have a question - maybe somebody had a similar issue - I'm receiving
netflow from router (Juniper) - they are sampled 1:2000
After the traffic is recalculated by nfacctd - in comparision to statistics
received via snmp - I have strange values - in the lowest traffic level
snmp shows around 550Mbps - in the same time traffic calculated by nfacctd
is ~1.3Gbps - in max point - snmp is showing 6Gbps but nfacctd 3.9 Gbps
I understand that traffic is sampled so it won't be exactly at the same
level as counted by snmp - but isn't it too big difference ?
Instead of this - the characteristics of the traffic is correct - traffic
is growing in the same direction, traffic drops are present in the same
time etc. - only this traffic level..
This is conifguration from router - it's quite simple:
sampling {
input {
rate 2000;
max-packets-per-second 7000;
}
family inet {
output {
flow-server x.x.x.x {
port x;
autonomous-system-type origin;
no-local-dump;
source-address x.x.x.x;
version 5;
}
}
}
}
in nfacctd config file - I recalculate netflows like this:
sql_optimize_clauses: true
sql_dont_try_update: true
sql_multi_values: 1024000
sql_db: pmacct
sql_host: host
sql_passwd: pass
sql_table_version: 7
sql_table_type: bgp
sql_cache_entries: 256000
sql_preprocess: usrf=2000
From this what I checked - the problem - for sure - is not in nfacctd,
netflow data received and recalculated by nfdump was almost the same -
maybe there is something different what I should change/modify to get
the traffic level little more accurate.
Thanks for response
Regards
Adam
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] Problem with more than 2 mysql plugins
Hi Paolo, After changing sql_cache_entries from 256k to 64k - updates are made without problem :) Thank You for hint and help. BR Adam 2014-02-06 Adam Bogdan nelr...@gmail.com: Hi Paolo, Thanks for answer - I think the memory is not the problem in this case - I still have some of it available and even swap isn't used but I'll check it with smaller cache_entry value. I had problem with memory but then in logs I get this information: Unable to fork DB writer: Cannot allocate memory - for now I changed few options and the problem is gone. When I changed plugin from mysql to sqlite3 - updates are made. I'll check cache_entries and let u know BR Adam 2014-02-06 Paolo Lucente pa...@pmacct.net: Hi Adam, Is it possible you are running out of memory or so? And maybe as a side result of swapping also CPU is 100%? I see in your config you have 'sql_cache_entries: 256000' which should take quite some memory per each plugin defined. Cheers, Paolo On Thu, Feb 06, 2014 at 02:02:14PM +0100, Adam Bogdan wrote: Hi again, I have a problem with running nfacctd to serve 3 mysql plugins/tables - version nfacctd 1.5.0rc2 config: daemonize: true debug: true pidfile: /var/run/nfacctd_r7.pid syslog: daemon aggregate: tag, src_as, dst_as, peer_src_as, peer_dst_as nfacctd_ip: x.x.x.x nfacctd_port: x nfacctd_time_new: true nfacctd_as_new: fallback nfacctd_net: fallback nfacctd_disable_checks: true networks_file: /etc/pmacct/networks.lst pre_tag_map: /etc/pmacct/pretag.map pre_tag_filter[abc]: 11 pre_tag_filter[ddd]: 10 pre_tag_filter[vcc]: 20 plugins: mysql[abc], mysql[ddd], mysql[vcc] plugin_pipe_size: 6544 plugin_buffer_size: 3 bgp_daemon: true bgp_daemon_ip: x.x.x.x bgp_daemon_max_peers: 10 bgp_peer_src_as_type: bgp bgp_src_as_path_type: bgp bgp_daemon_msglog: false bgp_agent_map: /etc/pmacct/agent.map sql_optimize_clauses: true sql_dont_try_update: true sql_multi_values: 1024000 sql_db: pmacct sql_host: x sql_passwd: x sql_table_version: 7 sql_table_type: bgp sql_cache_entries: 256000 sql_preprocess: usrf=2000 sql_history_roundoff[abc]: m sql_history[abc]: 5m sql_refresh_time[abc]: 300 sql_table[abc]: acct_bgp_abc sql_history_roundoff[ddd]: m sql_history[ddd]: 5m sql_refresh_time[ddd]: 300 sql_table[ddd]: acct_bgp_ddd sql_history_roundoff[vcc]: m sql_history[vcc]: 5m sql_refresh_time[vcc]: 300 sql_table[vcc]: acct_bgp_r7_vcc And this configuration doesn't work :( - If I enable only mysql abc i ddd then it's ok - updates to DB are made and everything is working - when I added vcc - it's dead - nfacctd is running but no updates to DB - when I turn on debugging - I get only keepalives from BGP - neither one update. When I turn off abc and ddd - vcc is working fine (updates etc.) One more hint - when I run config above, I can't kill nfacctd normally - only with option -9 Thanks for help Regards Adam ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] pmacct and more than 1 AS
Hello Paolo, Thanks for answer - that's clarify a lot :) Regards 2013/11/18 Paolo Lucente pa...@pmacct.net Hi Adam, The scenario is supported by pmacct, there are two pieces to it: * pmacct BGP daemon acts as a passive BGP neighbor and replies to an incoming BGP OPEN message with the same AS number contained in the OPEN. This means a single collector can peer with different ASNs no problem. If your NetFlow export model is ingress at edge interfaces facing customers, peers and transits you should be mostly sorted. * If you want to get end to end visibility, ie. a flow from customer in AS111 to a transit in AS222: you collect ingress NetFlow at AS111 but want to see the exit point in AS222, you can use a mix of bgp_follow_nexthop and bgp_agent_map. bgp_follow_nexthop allows to define IP prefixes to be considered internal BGP next-hops so granted pmacct peers with all ASBRs, it can follow BGP tables until it hits a foreign BGP next-hop; bgp_agent_map is because routers would typically BGP peer with pmacct using one of their loopback interfaces; whereas it is very possible ASBRs of AS111 and AS222 are eBGP peering using their transfer network IP addresses (ie. /30): so bgp_agent_map is useful in this context to map these addresses back to the loopback interface used for the BGP peering. In case something of the above does not work, then it's most probably a bug (or we have to review assumptions) so feel free to follow with me privately for some troubleshooting. Cheers, Paolo On Mon, Nov 18, 2013 at 12:53:41PM +0100, Adam Bogdan wrote: Hi I have small problem with pmacct implementation I have network with 3 ASes - in each AS there is at least 2 routers, sometimes more Each of these ASes hold some part of the full BGP table It looks like this: AS 111 - R1,R2 AS 222 - R3,R4 AS 333 - R5,R6 R1 is connected to uplink1 and uplink2 R2 is connected to uplink3 and uplink4 R3 is connected to uplink5 and uplink6 and so on Routers in ASes are connected via ibgp and each router is connected with each other via ebgp If I connect customer to any of these ASes he's able to receive all bgp routes or some part of it - depends to which AS I'll connect him. And now I'd like to apply pmacct to see how traffic is flowing between uplinks and customers, which AS to which AS is generate what amount of traffic. Question is how to do it ? If I set a machine with quagga I can only configure 1 BGP session - should I set 3 collectors, one per AS. But then could a problem with peer_ip_src or peer_ip_dst. Any ideas ? Regards Adam ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] pmacct and more than 1 AS
Hi I have small problem with pmacct implementation I have network with 3 ASes - in each AS there is at least 2 routers, sometimes more Each of these ASes hold some part of the full BGP table It looks like this: AS 111 - R1,R2 AS 222 - R3,R4 AS 333 - R5,R6 R1 is connected to uplink1 and uplink2 R2 is connected to uplink3 and uplink4 R3 is connected to uplink5 and uplink6 and so on Routers in ASes are connected via ibgp and each router is connected with each other via ebgp If I connect customer to any of these ASes he's able to receive all bgp routes or some part of it - depends to which AS I'll connect him. And now I'd like to apply pmacct to see how traffic is flowing between uplinks and customers, which AS to which AS is generate what amount of traffic. Question is how to do it ? If I set a machine with quagga I can only configure 1 BGP session - should I set 3 collectors, one per AS. But then could a problem with peer_ip_src or peer_ip_dst. Any ideas ? Regards Adam ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
