Dear Paolo,

I'm using nfacctd, with Kafka, and the data are written in InfluxDB, and
I'd like some help to clarify an issue that I encountered.

As far as I've understood nfacctd's way of operation, when using
historical accounting, it is using the timestamps of the flow generation
to do the aggregates.

So, if I have:

kafka_refresh_time[x]: 300
kafka_history[x]: 5m
kafka_history_roundoff[x]: m

Then, it will aggregate all flows with timestamp_start say 18:00 to
18:05, and then at 18:05, have an "event_type": "purge", with
{"stamp_inserted" : 18:00}.

However, if an out-of-order record X arrives (say timestamp_arrival:
18:07 but timestamp_start: 18:04), it will emit again another message with:

{ "event_type": "purge", "stamp_inserted" : 18:00}.


And this would contain only the payload of record X, not X + the
aggregate of 18:00-18:05, of that entry, right?


Since InfluxDB doesn't contain any read-add-replace, and I'm using the
stamp_inserted as the time value, the old entries are replaced as soon
as X type of records arrive.


To my understanding, I could use the kafka_startup_delay parameter, to
solve this, but would it be possible to have a value here that is larger
than the kafka_history / kafka_refresh_time (Assuming the active timeout
is also large)?

[A potential solution might be to use nfacctd_time_new, however I'd
prefer to keep the original times.]


Alternatively, would it be possible (if it also makes sense in other
usecases) for you to add two (optional) parameters for the kafka plugin,
that do something like:

- kafka_update_event_type: if the event is an update, then the
event_type, field of the message is set to "update", and not to purge

- kafka_update_event_topic: specify a different topic for the events
that are updates.


Thank you for your time and your help.

Best Regards,
Georgios


-- 
------------------------------------------------------------------
Georgios Kaklamanos
Research Assistant, e-Science Group, GWDG
mailto: georgios.kaklama...@gwdg.de
Telefon: 0551 201-26803
------------------------------------------------------------------
GWDG - Gesellschaft für wissenschaftliche
Datenverarbeitung mbH Göttingen
Am Faßberg 11, 37077 Göttingen, Germany

WWW: www.gwdg.de    mailto: g...@gwdg.de
Phone: +49 (0) 551 201-1510
Fax:   +49 (0) 551 201-2150
------------------------------------------------------------------
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen
Handelsregister-Nr. B 598
------------------------------------------------------------------
Zertifiziert nach ISO 9001
------------------------------------------------------------------

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists

Reply via email to