Hi Mark,
Since 'nfprobe' plugin would generate sequence numbers on output of
IPFIX packets, that would rule out any buffering issue (not saying
buffers should not be looked at, just saying buffering considerations
are disjoint from the sequencing issue).
Do you have any multi-paths between your routers (where you generate
IPFIX packets) and your collector? Is it possible, in other words, we
may be looking at an out of order delivery issue? If you are not doing
that already, can you take a pcap sample on the router (that is, where
IPFIX data is generated)? Depending if the sequencing issue is notified
there, we can pin-point (or not!) to out of ordering. If it's not an out
of order issue it would help a lot if you could send me the pcap trace
privately for me to inspect it and get further clues.
Wrt buffers: if there is a buffering issue inside pmacct then you are
notified by the means of warning/error messages so it's easy to spot;
trickier could be to spot issues between the kernel and pmacct for the
case of libpcap: you should send a SIGUSR1 to pmacctd ('killall -USR1
pmacctd' would do) and read output back in the log file. Here it is
documented what kind of output you should expect (see 'dropped_packets'):
https://github.com/pmacct/pmacct/blob/master/docs/SIGNALS#L17-#L40
Paolo
On Fri, Apr 03, 2020 at 03:46:53PM +0200, Mark Schouten wrote:
>
> Hi,
>
> I'm from Tuxis, a small ISP from the Netherlands. We run a small network,
> with three transits and one peering network. Our routers are
> Debian/Bird-based, and perform pretty well, if I may say so myself.
>
> On each router, we run a pmacct per external interface. Primary goal is
> invoicing (through amqp), secondary goal is sFlow and Netflow for obvious
> reasons.
>
> I have sFlow sampling at 1:512 and all is working well there. Netflow
> however, is an issue.
>
> We export Netflow to nprobe, and nprobe complains about Flow Collection
> Drops. And I can confirm that with Wireshark, which reports:
> FlowSequence: 889946358 (expected 888912934)
> [Expert Info (Warning/Sequence): Unexpected flow sequence for domain
> ID 77 (expected 888912934, got 889946358)]
> [Unexpected flow sequence for domain ID 77 (expected 888912934,
> got 889946358)]
> [Severity level: Warning]
> [Group: Sequence]
>
>
> So it seems that pmacct is missing data on flows, probably due to my
> configuration. Two questions:
>
> 1: How can I see that pmacct buffers or buckets are overrunning?
> 2: Looking at my current configuration, which values should I alter to
> achieve both (mostly) complete flowdata and not too much CPU usage?
>
>
> Thanks in advance, also for the cool product that pmacctd is!
>
>
> daemonize: true
> pidfile: /var/run/pmacctd.pid
> syslog: daemon
>
> interface: v-amsix
>
> pmacctd_flow_buffer_size: 268435456 !256MB
> pmacctd_flow_buffer_buckets: 65536
> pmacctd_conntrack_buffer_size: 134217728 !128MB
> pmacctd_flow_tcp_lifetime: 3600
>
> plugins: nfprobe[in],nfprobe[out],sfprobe[fnm],amqp
> !
> plugin_buffer_size: 102400
> plugin_pipe_size: 10240
>
> sampling_rate: 1
> sampling_rate[fnm]: 512
> timestamps_since_epoch: true
> geoipv2_file: /var/lib/GeoIP/GeoLite2-Country.mmdb
> aggregate: src_host_country, dst_host_country
> aggregate:
> src_host,dst_host,src_port,dst_port,proto,src_mac,dst_mac,src_host_country,dst_host_country
> amqp_exchange: pmacct
> amqp_routing_key: acct
> amqp_refresh_time: 300
> amqp_history_roundoff: m
> amqp_host:
> amqp_user:
> amqp_passwd:
> amqp_exchange_type: direct
> amqp_persistent_msg: false
> amqp_cache_entries: 262147
> amqp_multi_values: 65536
> amqp_history: 5m
>
> sfprobe_agentsubid: 1402
> sfprobe_receiver:
>
> nfprobe_receiver:
> nfprobe_version: 10
> nfprobe_source_ip:
> nfprobe_direction[in]: in
> nfprobe_direction[out]: out
> nfprobe_ifindex[in]: 77
> nfprobe_ifindex[out]: 77
> nfprobe_engine: 77
>
>
> --
> Mark Schouten
>
> Tuxis, Ede, https://www.tuxis.nl
>
> T: +31 318 200208
>
>
> ___
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
