Thank you that sorted the problem out!
On Fri, Apr 16, 2021 at 02:25:08PM +0000, Paolo Lucente wrote:
>
> Hi Hendrik,
>
> You may see these messages appearing in your log (i can spot one in your
> excerpt in your previous email): "Finished cache entries (ie.
> print_cache_entries). Purging.". This is the reason for the intermediate
> purges. You have more entries to store for the 300 seconds interval than
> available cache entries; for the Kafka plugin the default is 16411. Set
> kafka_cache_entries to something (ideally a prime number) greater than
> that. Dunno, go for 99991. See if it works and take it from there (to
> further increase it or, if unhappy with memory consumed, reduce it).
>
> Paolo
>
> On Fri, Apr 16, 2021 at 07:36:38AM +0200, Hendrik Meyburgh wrote:
> > Hi.
> >
> > I have a problem where the data is being purged in multiple intervals and
> > this is causing a calculation issue, as it doesn't seem to be resetting the
> > values back to zero after each purge. My understanding is
> > that kafka_refresh_time should set it to fixed values, in this case 300
> > seconds, but from the logs it is clear that it is not happening like that.
> > Am I missing a config option or am I misunderstanding the configuration?
> >
> > Some sample data for the IP x.x.x.x, it is the same IP for both events:
> > {"event_type": "purge", "ip_src": "x.x.x.x", "stamp_inserted": "2021-04-16
> > 07:10:00", "stamp_updated": "2021-04-16 07:14:00", "packets": 12288,
> > "bytes": 9193472, "writer_id": "subscriber_usage/20793"}
> >
> > {"event_type": "purge", "ip_src": "x.x.x.x", "stamp_inserted": "2021-04-16
> > 07:10:00", "stamp_updated": "2021-04-16 07:15:03", "packets": 12288,
> > "bytes": 13235200, "writer_id": "subscriber_usage/20945"}
> >
> > Some of the logs
> >
> > 2021-04-16T07:20:01+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> > cache - START (PID: 21144) ***
> >
> > 2021-04-16T07:20:05+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> > cache - END (PID: 21144, QN: 106679/106679, ET: 2) ***
> >
> > 2021-04-16T07:23:51+02:00 INFO ( subscriber_usage/kafka ): Finished cache
> > entries (ie. print_cache_entries). Purging.
> >
> > 2021-04-16T07:23:51+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> > cache - START (PID: 21194) ***
> >
> > 2021-04-16T07:23:56+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> > cache - END (PID: 21194, QN: 180344/180344, ET: 3) ***
> >
> > 2021-04-16T07:25:01+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> > cache - START (PID: 21412) ***
> >
> > 2021-04-16T07:25:05+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> > cache - END (PID: 21412, QN: 109942/109942, ET: 2) ***
> >
> > Current running configuration.
> >
> > daemonize: true
> >
> > logfile: /var/log/nfacctd.log
> >
> > !debug: true
> >
> > plugins: kafka[subscriber_usage]
> >
> > aggregate[subscriber_usage]: sum_host
> >
> > nfacctd_renormalize: true
> >
> > !nfacctd_ext_sampling_rate: 1024
> >
> > nfacctd_pro_rating: true
> >
> > !
> >
> > nfacctd_time_new: true
> >
> > !
> >
> > kafka_output: json
> >
> > kafka_topic: pmacct.test
> >
> > kafka_refresh_time: 300
> >
> > kafka_history: 5m
> >
> > kafka_history_roundoff: m
> >
> > kafka_broker_host: a.a.a.a
> >
> >
> > Thank you.
>
> > _______________________________________________
> > pmacct-discussion mailing list
> > http://www.pmacct.net/#mailinglists
>
>
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
