Hi Hendrik,
You may see these messages appearing in your log (i can spot one in your
excerpt in your previous email): "Finished cache entries (ie.
print_cache_entries). Purging.". This is the reason for the intermediate
purges. You have more entries to store for the 300 seconds interval than
available cache entries; for the Kafka plugin the default is 16411. Set
kafka_cache_entries to something (ideally a prime number) greater than
that. Dunno, go for 99991. See if it works and take it from there (to
further increase it or, if unhappy with memory consumed, reduce it).
Paolo
On Fri, Apr 16, 2021 at 07:36:38AM +0200, Hendrik Meyburgh wrote:
> Hi.
>
> I have a problem where the data is being purged in multiple intervals and
> this is causing a calculation issue, as it doesn't seem to be resetting the
> values back to zero after each purge. My understanding is
> that kafka_refresh_time should set it to fixed values, in this case 300
> seconds, but from the logs it is clear that it is not happening like that.
> Am I missing a config option or am I misunderstanding the configuration?
>
> Some sample data for the IP x.x.x.x, it is the same IP for both events:
> {"event_type": "purge", "ip_src": "x.x.x.x", "stamp_inserted": "2021-04-16
> 07:10:00", "stamp_updated": "2021-04-16 07:14:00", "packets": 12288,
> "bytes": 9193472, "writer_id": "subscriber_usage/20793"}
>
> {"event_type": "purge", "ip_src": "x.x.x.x", "stamp_inserted": "2021-04-16
> 07:10:00", "stamp_updated": "2021-04-16 07:15:03", "packets": 12288,
> "bytes": 13235200, "writer_id": "subscriber_usage/20945"}
>
> Some of the logs
>
> 2021-04-16T07:20:01+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> cache - START (PID: 21144) ***
>
> 2021-04-16T07:20:05+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> cache - END (PID: 21144, QN: 106679/106679, ET: 2) ***
>
> 2021-04-16T07:23:51+02:00 INFO ( subscriber_usage/kafka ): Finished cache
> entries (ie. print_cache_entries). Purging.
>
> 2021-04-16T07:23:51+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> cache - START (PID: 21194) ***
>
> 2021-04-16T07:23:56+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> cache - END (PID: 21194, QN: 180344/180344, ET: 3) ***
>
> 2021-04-16T07:25:01+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> cache - START (PID: 21412) ***
>
> 2021-04-16T07:25:05+02:00 INFO ( subscriber_usage/kafka ): *** Purging
> cache - END (PID: 21412, QN: 109942/109942, ET: 2) ***
>
> Current running configuration.
>
> daemonize: true
>
> logfile: /var/log/nfacctd.log
>
> !debug: true
>
> plugins: kafka[subscriber_usage]
>
> aggregate[subscriber_usage]: sum_host
>
> nfacctd_renormalize: true
>
> !nfacctd_ext_sampling_rate: 1024
>
> nfacctd_pro_rating: true
>
> !
>
> nfacctd_time_new: true
>
> !
>
> kafka_output: json
>
> kafka_topic: pmacct.test
>
> kafka_refresh_time: 300
>
> kafka_history: 5m
>
> kafka_history_roundoff: m
>
> kafka_broker_host: a.a.a.a
>
>
> Thank you.
> _______________________________________________
> pmacct-discussion mailing list
> http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
