Hi Brian,
No, it's not currently possible to send exporter system time / uptime to
Kafka (also because doing it per-flow would be lots of wasted space).
Also, there are minimal protections, yes, for example for the case of
flows stopping before they start. But not for time format errors, ie.
not making a call for how old is old rather fix (or have fixed) your
exporting device.
Wrt Fortigate: "this way" you mean very old timestamps or stop time is
earlier than start time? If the former case, any chance they export with
timestamps in secs instead of msecs (in which case there would be a knob
for this 'timestamps_secs: true')?
Paolo
On Tue, Mar 05, 2019 at 12:46:35AM +0000, Brian Solar wrote:
> Is there a way to send the devices decoded system time and uptime to Kafka?
>
> Are there protections for flows stopping before they start? Or other time
> format errors?
>
> I have yet to track down actual packets, but Ive seen usual timestamps and
> very old time stamps in Kafka. This makes nfacctd generate millions of
> entries at times.
>
> Fortigate devices seem to report this way at times. I haven't noticed it on
> other devices as of yet.
>
> -------- Original Message --------
> On Feb 25, 2019, 9:28 AM, Paolo Lucente wrote:
>
> Hi Brian,
>
> Thanks very much for the nginx config, definitely something to add to
> docs as a possible option. QN reads 'Queries Number' (inherited from the
> SQL plugins, hence the queries wording); the first number is now many
> are sent to the backend, the second is how many should be sent as part
> of the purge event.
>
> They should normally be aligned. In case of NetFlow/IPFIX, among the
> different possibilities, it may reveal time sync issues among exporters
> and the collector; easiest to resolve / experiment is to consider as
> timestamp in pmacct the arrival time at the collector (versus the start
> time of flows) by setting nfacctd_time_new to true.
>
> Paolo
>
> On Mon, Feb 25, 2019 at 03:23:42AM +0000, Brian Solar wrote:
> >
> > Thanks for the response Paolo. I am using nginx to stream load balance (see
> > config below).
> >
> > Another quick question on the Kafka plugin. What Does the QN portion of the
> > purging cache end line indicate/mean?
> >
> >
> > 2019-02-25T03:05:04Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
> > 387033, QN: 12786/13291, ET: 1) ***
> >
> > 2019-02-25T03:16:22Z INFO ( kafka2/kafka ): *** Purging cache - END (PID:
> > 150221, QN: 426663/426663, ET: 19) ***
> >
> > # Load balance UDP-based FLOW traffic across two servers
> > stream {
> >
> > log_format combined '$remote_addr - - [$time_local] $protocol $status
> > $bytes_sent $bytes_received $session_time "$upstream_addr"';
> >
> > access_log /var/log/nginx/stream-access.log combined;
> >
> > upstream flow_upstreams {
> > #hash $remote_addr consistent;
> > server 10.20.25.11:2100;
> > #
> > server 10.20.25.12:2100;
> >
> > }
> >
> > server {
> > listen 2201 udp;
> > proxy_pass flow_upstreams;
> > #proxy_timeout 1s;
> > proxy_responses 0;
> > # must have user: root in main config
> > proxy_bind $remote_addr transparent;
> > error_log /var/log/nginx/stream-flow-err.log;
> > }
> > }
> >
> >
> >
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Sunday, February 24, 2019 5:02 PM, Paolo Lucente wrote:
> >
> > >
> > >
> > > Hi Brian,
> > >
> > > You are most probably looking for this:
> > >
> > > https://github.com/pmacct/pmacct/blob/master/QUICKSTART#L2644-#L2659
> > >
> > > Should that not work, ie. too many input flows for the available
> > > resources, you have a couple load-balancing strategies possible:
> > > one is to configure a replicator (tee plugin, see in QUICKSTART).
> > >
> > > Paolo
> > >
> > > On Sun, Feb 24, 2019 at 05:31:55PM +0000, Brian Solar wrote:
> > >
> > > > Is there a way to adjust the UDP buffer receive size ?
> > > > Are there any other indications of nfacctd not keeping up?
> > > > cat /proc/net/udp |egrep drops\|0835
> > > > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt
> > > > uid timeout inode ref pointer drops
> > > > 52366: 00000000:0835 00000000:0000 07 00000000:00034B80 00:00000000
> > > > 00000000 0 0 20175528 2 ffff89993febd940 7495601
> > > > 7495601 drops w/ a buffer of 0x0034B80 or 214528
> > > > sysctl -a |fgrep mem
> > > > net.core.optmem_max = 20480
> > > > net.core.rmem_default = 212992
> > > > net.core.rmem_max = 2147483647
> > > > net.core.wmem_default = 212992
> > > > net.core.wmem_max = 212992
> > > > net.ipv4.igmp_max_memberships = 20
> > > > net.ipv4.tcp_mem = 9249771 12333028 18499542
> > > > net.ipv4.tcp_rmem = 4096 87380 6291456
> > > > net.ipv4.tcp_wmem = 4096 16384 4194304
> > > > net.ipv4.udp_mem = 9252429 12336573 18504858
> > > > net.ipv4.udp_rmem_min = 4096
> > > > net.ipv4.udp_wmem_min = 4096
> > > > vm.lowmem_reserve_ratio = 256 256 32
> > > > vm.memory_failure_early_kill = 0
> > > > vm.memory_failure_recovery = 1
> > > > vm.nr_hugepages_mempolicy = 0
> > > > vm.overcommit_memory = 0
> > >
> > > > pmacct-discussion mailing list
> > > > http://www.pmacct.net/#mailinglists
> >
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
