Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
Hi Brian, If i understand correctly the issue is with the fact that templates are sent out very frequently and hence the daemon keeps writing 'real time' to disk. You mention a configurable interval but, recalling the original purpose of the feature (if restarting a collector be able to immediately collect new data without waiting for templates to arrive), i wonder if it makes sense to write all to disk only upon clean exit of the daemon. By the way thanks very much for the feedback on nailing this down. On the SO_REUSEPORT point: sure and, please, if you could contribute some code that would be fantastic. For now i am advocating for a simpler architecture: 1) use a replicator (tee plugin) to balance flows onto multiple collectors (different UDP ports) and 2) use SO_REUSEPORT with an ACL, ie. bgp_daemon_allow_file, that should match tee config for BGP (i don't say also BMP or TCP-based Streaming Telemetry because for those, contrary to BGP, receiver port is typically configurable at the exporter). Paolo On Fri, Mar 08, 2019 at 09:28:31PM +, Brian Solar wrote: > > The culprit was actually the template file. It appears to block while > writing and it's really slow. When I remove the configuration option one > process could do what I could not accomplish using 80 processes with each > using a template file. > > Any consideration on a different implementation? > > Writing it out on a configurable interval would be a simple improvement. > > When load-balancing, particularly with SO_REUSEPORT, it would be nice to > allow them to communicate the template set to each other. Perhaps another > use for zeromq? > > Brian > > > > ‐‐‐ Original Message ‐‐‐ > On Sunday, February 24, 2019 5:02 PM, Paolo Lucente wrote: > > > > > > > Hi Brian, > > > > You are most probably looking for this: > > > > https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659 > > > > Should that not work, ie. too many input flows for the available > > resources, you have a couple load-balancing strategies possible: > > one is to configure a replicator (tee plugin, see in QUICKSTART). > > > > Paolo > > > > On Sun, Feb 24, 2019 at 05:31:55PM +, Brian Solar wrote: > > > > > Is there a way to adjust the UDP buffer receive size ? > > > Are there any other indications of nfacctd not keeping up? > > > cat /proc/net/udp |egrep drops\|0835 > > > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt > > > uid timeout inode ref pointer drops > > > 52366: :0835 : 07 :00034B80 00: > > > 0 0 20175528 2 89993febd940 7495601 > > > 7495601 drops w/ a buffer of 0x0034B80 or 214528 > > > sysctl -a |fgrep mem > > > net.core.optmem_max = 20480 > > > net.core.rmem_default = 212992 > > > net.core.rmem_max = 2147483647 > > > net.core.wmem_default = 212992 > > > net.core.wmem_max = 212992 > > > net.ipv4.igmp_max_memberships = 20 > > > net.ipv4.tcp_mem = 9249771 12333028 18499542 > > > net.ipv4.tcp_rmem = 4096 87380 6291456 > > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > > net.ipv4.udp_mem = 9252429 12336573 18504858 > > > net.ipv4.udp_rmem_min = 4096 > > > net.ipv4.udp_wmem_min = 4096 > > > vm.lowmem_reserve_ratio = 256 256 32 > > > vm.memory_failure_early_kill = 0 > > > vm.memory_failure_recovery = 1 > > > vm.nr_hugepages_mempolicy = 0 > > > vm.overcommit_memory = 0 > > > > > pmacct-discussion mailing list > > > http://www.pmacct.net/#mailinglists > > ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
The culprit was actually the template file. It appears to block while writing and it's really slow. When I remove the configuration option one process could do what I could not accomplish using 80 processes with each using a template file. Any consideration on a different implementation? Writing it out on a configurable interval would be a simple improvement. When load-balancing, particularly with SO_REUSEPORT, it would be nice to allow them to communicate the template set to each other. Perhaps another use for zeromq? Brian ‐‐‐ Original Message ‐‐‐ On Sunday, February 24, 2019 5:02 PM, Paolo Lucente wrote: > > > Hi Brian, > > You are most probably looking for this: > > https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659 > > Should that not work, ie. too many input flows for the available > resources, you have a couple load-balancing strategies possible: > one is to configure a replicator (tee plugin, see in QUICKSTART). > > Paolo > > On Sun, Feb 24, 2019 at 05:31:55PM +, Brian Solar wrote: > > > Is there a way to adjust the UDP buffer receive size ? > > Are there any other indications of nfacctd not keeping up? > > cat /proc/net/udp |egrep drops\|0835 > > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid > > timeout inode ref pointer drops > > 52366: :0835 : 07 :00034B80 00: > > 0 0 20175528 2 89993febd940 7495601 > > 7495601 drops w/ a buffer of 0x0034B80 or 214528 > > sysctl -a |fgrep mem > > net.core.optmem_max = 20480 > > net.core.rmem_default = 212992 > > net.core.rmem_max = 2147483647 > > net.core.wmem_default = 212992 > > net.core.wmem_max = 212992 > > net.ipv4.igmp_max_memberships = 20 > > net.ipv4.tcp_mem = 9249771 12333028 18499542 > > net.ipv4.tcp_rmem = 4096 87380 6291456 > > net.ipv4.tcp_wmem = 4096 16384 4194304 > > net.ipv4.udp_mem = 9252429 12336573 18504858 > > net.ipv4.udp_rmem_min = 4096 > > net.ipv4.udp_wmem_min = 4096 > > vm.lowmem_reserve_ratio = 256 256 32 > > vm.memory_failure_early_kill = 0 > > vm.memory_failure_recovery = 1 > > vm.nr_hugepages_mempolicy = 0 > > vm.overcommit_memory = 0 > > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
Hi Brian,
No, it's not currently possible to send exporter system time / uptime to
Kafka (also because doing it per-flow would be lots of wasted space).
Also, there are minimal protections, yes, for example for the case of
flows stopping before they start. But not for time format errors, ie.
not making a call for how old is old rather fix (or have fixed) your
exporting device.
Wrt Fortigate: "this way" you mean very old timestamps or stop time is
earlier than start time? If the former case, any chance they export with
timestamps in secs instead of msecs (in which case there would be a knob
for this 'timestamps_secs: true')?
Paolo
On Tue, Mar 05, 2019 at 12:46:35AM +, Brian Solar wrote:
> Is there a way to send the devices decoded system time and uptime to Kafka?
>
> Are there protections for flows stopping before they start? Or other time
> format errors?
>
> I have yet to track down actual packets, but Ive seen usual timestamps and
> very old time stamps in Kafka. This makes nfacctd generate millions of
> entries at times.
>
> Fortigate devices seem to report this way at times. I haven't noticed it on
> other devices as of yet.
>
> Original Message
> On Feb 25, 2019, 9:28 AM, Paolo Lucente wrote:
>
> Hi Brian,
>
> Thanks very much for the nginx config, definitely something to add to
> docs as a possible option. QN reads 'Queries Number' (inherited from the
> SQL plugins, hence the queries wording); the first number is now many
> are sent to the backend, the second is how many should be sent as part
> of the purge event.
>
> They should normally be aligned. In case of NetFlow/IPFIX, among the
> different possibilities, it may reveal time sync issues among exporters
> and the collector; easiest to resolve / experiment is to consider as
> timestamp in pmacct the arrival time at the collector (versus the start
> time of flows) by setting nfacctd_time_new to true.
>
> Paolo
>
> On Mon, Feb 25, 2019 at 03:23:42AM +, Brian Solar wrote:
> >
> > Thanks for the response Paolo. I am using nginx to stream load balance (see
> > config below).
> >
> > Another quick question on the Kafka plugin. What Does the QN portion of the
> > purging cache end line indicate/mean?
> >
> >
> > 2019-02-25T03:05:04Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
> > 387033, QN: 12786/13291, ET: 1) ***
> >
> > 2019-02-25T03:16:22Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
> > 150221, QN: 426663/426663, ET: 19) ***
> >
> > # Load balance UDP-based FLOW traffic across two servers
> > stream {
> >
> > log_format combined '$remote_addr - - [$time_local] $protocol $status
> > $bytes_sent $bytes_received $session_time "$upstream_addr"';
> >
> > access_log /var/log/nginx/stream-access.log combined;
> >
> > upstream flow_upstreams {
> > #hash $remote_addr consistent;
> > server 10.20.25.11:2100;
> > #
> > server 10.20.25.12:2100;
> >
> > }
> >
> > server {
> > listen 2201 udp;
> > proxy_pass flow_upstreams;
> > #proxy_timeout 1s;
> > proxy_responses 0;
> > # must have user: root in main config
> > proxy_bind $remote_addr transparent;
> > error_log /var/log/nginx/stream-flow-err.log;
> > }
> > }
> >
> >
> >
> >
> > ‐‐‐ Original Message ‐‐‐
> > On Sunday, February 24, 2019 5:02 PM, Paolo Lucente wrote:
> >
> > >
> > >
> > > Hi Brian,
> > >
> > > You are most probably looking for this:
> > >
> > > https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659
> > >
> > > Should that not work, ie. too many input flows for the available
> > > resources, you have a couple load-balancing strategies possible:
> > > one is to configure a replicator (tee plugin, see in QUICKSTART).
> > >
> > > Paolo
> > >
> > > On Sun, Feb 24, 2019 at 05:31:55PM +, Brian Solar wrote:
> > >
> > > > Is there a way to adjust the UDP buffer receive size ?
> > > > Are there any other indications of nfacctd not keeping up?
> > > > cat /proc/net/udp |egrep drops\|0835
> > > > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt
> > > > uid timeout inode ref pointer drops
> > > > 52366: :0835 : 07 :00034B80 00:
> > > > 0 0 20175528 2 89993febd940 7495601
> > > > 7495601 drops w/ a buffer of 0x0034B80 or 214528
> > > > sysctl -a |fgrep mem
> > > > net.core.optmem_max = 20480
> > > > net.core.rmem_default = 212992
> > > > net.core.rmem_max = 2147483647
> > > > net.core.wmem_default = 212992
> > > > net.core.wmem_max = 212992
> > > > net.ipv4.igmp_max_memberships = 20
> > > > net.ipv4.tcp_mem = 9249771 12333028 18499542
> > > > net.ipv4.tcp_rmem = 4096 87380 6291456
> > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > net.ipv4.udp_mem = 9252429 12336573 18504858
> > > > net.ipv4.udp_rmem_min = 4096
> > > > net.ipv4.udp_wmem_min = 4096
> > > > vm.lowmem_reserve_ratio = 256 256 32
> > > > vm.memory_failure_early_kill = 0
> > > > vm.memory_failure_recovery = 1
> > > > vm.nr_hugepages_mempolicy = 0
> > > >
Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
Hi Brian,
Thanks very much for the nginx config, definitely something to add to
docs as a possible option. QN reads 'Queries Number' (inherited from the
SQL plugins, hence the queries wording); the first number is now many
are sent to the backend, the second is how many should be sent as part
of the purge event.
They should normally be aligned. In case of NetFlow/IPFIX, among the
different possibilities, it may reveal time sync issues among exporters
and the collector; easiest to resolve / experiment is to consider as
timestamp in pmacct the arrival time at the collector (versus the start
time of flows) by setting nfacctd_time_new to true.
Paolo
On Mon, Feb 25, 2019 at 03:23:42AM +, Brian Solar wrote:
>
> Thanks for the response Paolo. I am using nginx to stream load balance (see
> config below).
>
> Another quick question on the Kafka plugin. What Does the QN portion of the
> purging cache end line indicate/mean?
>
>
> 2019-02-25T03:05:04Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
> 387033, QN: 12786/13291, ET: 1) ***
>
> 2019-02-25T03:16:22Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
> 150221, QN: 426663/426663, ET: 19) ***
>
> # Load balance UDP-based FLOW traffic across two servers
> stream {
>
>log_format combined '$remote_addr - - [$time_local] $protocol $status
> $bytes_sent $bytes_received $session_time "$upstream_addr"';
>
>access_log /var/log/nginx/stream-access.log combined;
>
> upstream flow_upstreams {
> #hash $remote_addr consistent;
> server 10.20.25.11:2100;
> #
> server 10.20.25.12:2100;
>
> }
>
> server {
> listen 2201 udp;
> proxy_pass flow_upstreams;
> #proxy_timeout 1s;
> proxy_responses 0;
> # must have user: root in main config
> proxy_bind $remote_addr transparent;
> error_log /var/log/nginx/stream-flow-err.log;
> }
> }
>
>
>
>
> ‐‐‐ Original Message ‐‐‐
> On Sunday, February 24, 2019 5:02 PM, Paolo Lucente wrote:
>
> >
> >
> > Hi Brian,
> >
> > You are most probably looking for this:
> >
> > https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659
> >
> > Should that not work, ie. too many input flows for the available
> > resources, you have a couple load-balancing strategies possible:
> > one is to configure a replicator (tee plugin, see in QUICKSTART).
> >
> > Paolo
> >
> > On Sun, Feb 24, 2019 at 05:31:55PM +, Brian Solar wrote:
> >
> > > Is there a way to adjust the UDP buffer receive size ?
> > > Are there any other indications of nfacctd not keeping up?
> > > cat /proc/net/udp |egrep drops\|0835
> > > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt
> > > uid timeout inode ref pointer drops
> > > 52366: :0835 : 07 :00034B80 00:
> > > 0 0 20175528 2 89993febd940 7495601
> > > 7495601 drops w/ a buffer of 0x0034B80 or 214528
> > > sysctl -a |fgrep mem
> > > net.core.optmem_max = 20480
> > > net.core.rmem_default = 212992
> > > net.core.rmem_max = 2147483647
> > > net.core.wmem_default = 212992
> > > net.core.wmem_max = 212992
> > > net.ipv4.igmp_max_memberships = 20
> > > net.ipv4.tcp_mem = 9249771 12333028 18499542
> > > net.ipv4.tcp_rmem = 4096 87380 6291456
> > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > net.ipv4.udp_mem = 9252429 12336573 18504858
> > > net.ipv4.udp_rmem_min = 4096
> > > net.ipv4.udp_wmem_min = 4096
> > > vm.lowmem_reserve_ratio = 256 256 32
> > > vm.memory_failure_early_kill = 0
> > > vm.memory_failure_recovery = 1
> > > vm.nr_hugepages_mempolicy = 0
> > > vm.overcommit_memory = 0
> >
> > > pmacct-discussion mailing list
> > > http://www.pmacct.net/#mailinglists
>
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
Thanks for the response Paolo. I am using nginx to stream load balance (see
config below).
Another quick question on the Kafka plugin. What Does the QN portion of the
purging cache end line indicate/mean?
2019-02-25T03:05:04Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
387033, QN: 12786/13291, ET: 1) ***
2019-02-25T03:16:22Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
150221, QN: 426663/426663, ET: 19) ***
# Load balance UDP-based FLOW traffic across two servers
stream {
log_format combined '$remote_addr - - [$time_local] $protocol $status
$bytes_sent $bytes_received $session_time "$upstream_addr"';
access_log /var/log/nginx/stream-access.log combined;
upstream flow_upstreams {
#hash $remote_addr consistent;
server 10.20.25.11:2100;
#
server 10.20.25.12:2100;
}
server {
listen 2201 udp;
proxy_pass flow_upstreams;
#proxy_timeout 1s;
proxy_responses 0;
# must have user: root in main config
proxy_bind $remote_addr transparent;
error_log /var/log/nginx/stream-flow-err.log;
}
}
‐‐‐ Original Message ‐‐‐
On Sunday, February 24, 2019 5:02 PM, Paolo Lucente wrote:
>
>
> Hi Brian,
>
> You are most probably looking for this:
>
> https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659
>
> Should that not work, ie. too many input flows for the available
> resources, you have a couple load-balancing strategies possible:
> one is to configure a replicator (tee plugin, see in QUICKSTART).
>
> Paolo
>
> On Sun, Feb 24, 2019 at 05:31:55PM +, Brian Solar wrote:
>
> > Is there a way to adjust the UDP buffer receive size ?
> > Are there any other indications of nfacctd not keeping up?
> > cat /proc/net/udp |egrep drops\|0835
> > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid
> > timeout inode ref pointer drops
> > 52366: :0835 : 07 :00034B80 00:
> > 0 0 20175528 2 89993febd940 7495601
> > 7495601 drops w/ a buffer of 0x0034B80 or 214528
> > sysctl -a |fgrep mem
> > net.core.optmem_max = 20480
> > net.core.rmem_default = 212992
> > net.core.rmem_max = 2147483647
> > net.core.wmem_default = 212992
> > net.core.wmem_max = 212992
> > net.ipv4.igmp_max_memberships = 20
> > net.ipv4.tcp_mem = 9249771 12333028 18499542
> > net.ipv4.tcp_rmem = 4096 87380 6291456
> > net.ipv4.tcp_wmem = 4096 16384 4194304
> > net.ipv4.udp_mem = 9252429 12336573 18504858
> > net.ipv4.udp_rmem_min = 4096
> > net.ipv4.udp_wmem_min = 4096
> > vm.lowmem_reserve_ratio = 256 256 32
> > vm.memory_failure_early_kill = 0
> > vm.memory_failure_recovery = 1
> > vm.nr_hugepages_mempolicy = 0
> > vm.overcommit_memory = 0
>
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
___
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
Re: [pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
Hi Brian, You are most probably looking for this: https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659 Should that not work, ie. too many input flows for the available resources, you have a couple load-balancing strategies possible: one is to configure a replicator (tee plugin, see in QUICKSTART). Paolo On Sun, Feb 24, 2019 at 05:31:55PM +, Brian Solar wrote: > Is there a way to adjust the UDP buffer receive size ? > > Are there any other indications of nfacctd not keeping up? > > cat /proc/net/udp |egrep drops\|0835 > > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt > uid timeout inode ref pointer drops > > 52366: :0835 : 07 :00034B80 00: >00 20175528 2 89993febd940 7495601 > > 7495601 drops w/ a buffer of 0x0034B80 or 214528 > > sysctl -a |fgrep mem > > net.core.optmem_max = 20480 > > net.core.rmem_default = 212992 > > net.core.rmem_max = 2147483647 > > net.core.wmem_default = 212992 > > net.core.wmem_max = 212992 > > net.ipv4.igmp_max_memberships = 20 > > net.ipv4.tcp_mem = 9249771 1233302818499542 > > net.ipv4.tcp_rmem = 409687380 6291456 > > net.ipv4.tcp_wmem = 409616384 4194304 > > net.ipv4.udp_mem = 9252429 1233657318504858 > > net.ipv4.udp_rmem_min = 4096 > > net.ipv4.udp_wmem_min = 4096 > > vm.lowmem_reserve_ratio = 256 256 32 > > vm.memory_failure_early_kill = 0 > > vm.memory_failure_recovery = 1 > > vm.nr_hugepages_mempolicy = 0 > > vm.overcommit_memory = 0 > ___ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists ___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
[pmacct-discussion] nfacctd Dropping UDP. Buffer Receive Size?
Is there a way to adjust the UDP buffer receive size ? Are there any other indications of nfacctd not keeping up? cat /proc/net/udp |egrep drops\|0835 sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops 52366: :0835 : 07 :00034B80 00: 00 20175528 2 89993febd940 7495601 7495601 drops w/ a buffer of 0x0034B80 or 214528 sysctl -a |fgrep mem net.core.optmem_max = 20480 net.core.rmem_default = 212992 net.core.rmem_max = 2147483647 net.core.wmem_default = 212992 net.core.wmem_max = 212992 net.ipv4.igmp_max_memberships = 20 net.ipv4.tcp_mem = 9249771 1233302818499542 net.ipv4.tcp_rmem = 409687380 6291456 net.ipv4.tcp_wmem = 409616384 4194304 net.ipv4.udp_mem = 9252429 1233657318504858 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096 vm.lowmem_reserve_ratio = 256 256 32 vm.memory_failure_early_kill = 0 vm.memory_failure_recovery = 1 vm.nr_hugepages_mempolicy = 0 vm.overcommit_memory = 0___ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
