Dear Paolo,

I'm using nfacctd, with Kafka, and the data are written in InfluxDB, and
I'd like some help to clarify an issue that I encountered.

As far as I've understood nfacctd's way of operation, when using
historical accounting, it is using the timestamps of the flow generation
to do the aggregates.

So, if I have:

kafka_refresh_time[x]: 300
kafka_history[x]: 5m
kafka_history_roundoff[x]: m

Then, it will aggregate all flows with timestamp_start say 18:00 to
18:05, and then at 18:05, have an "event_type": "purge", with
{"stamp_inserted" : 18:00}.

However, if an out-of-order record X arrives (say timestamp_arrival:
18:07 but timestamp_start: 18:04), it will emit again another message with:

{ "event_type": "purge", "stamp_inserted" : 18:00}.

And this would contain only the payload of record X, not X + the
aggregate of 18:00-18:05, of that entry, right?

Since InfluxDB doesn't contain any read-add-replace, and I'm using the
stamp_inserted as the time value, the old entries are replaced as soon
as X type of records arrive.

To my understanding, I could use the kafka_startup_delay parameter, to
solve this, but would it be possible to have a value here that is larger
than the kafka_history / kafka_refresh_time (Assuming the active timeout
is also large)?

[A potential solution might be to use nfacctd_time_new, however I'd
prefer to keep the original times.]

Alternatively, would it be possible (if it also makes sense in other
usecases) for you to add two (optional) parameters for the kafka plugin,
that do something like:

- kafka_update_event_type: if the event is an update, then the
event_type, field of the message is set to "update", and not to purge

- kafka_update_event_topic: specify a different topic for the events
that are updates.

Thank you for your time and your help.

Best Regards,

Georgios Kaklamanos
Research Assistant, e-Science Group, GWDG
Telefon: 0551 201-26803
GWDG - Gesellschaft für wissenschaftliche
Datenverarbeitung mbH Göttingen
Am Faßberg 11, 37077 Göttingen, Germany

WWW:    mailto:
Phone: +49 (0) 551 201-1510
Fax:   +49 (0) 551 201-2150
Geschäftsführer: Prof. Dr. Ramin Yahyapour
Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen
Handelsregister-Nr. B 598
Zertifiziert nach ISO 9001

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature

pmacct-discussion mailing list

Reply via email to