you should be able to configure kafka to not throw away logs in it's queue.
In your example below, you have configured rsyslog to throw away messages when
the queue fills up to 60000 messages.
can you show the pstats data that shows that rsyslog is dropping messages?
David Lang
On Tue, 6 Feb 2024, Adriaan de Waal via rsyslog wrote:
Good day
I've configured the omkafka module to push logs to Kafka and added a
disk-assisted queue configuration to this action. The idea being, that if the
Kafka cluster is temporarily unavailable, logs will be queued and prevent
message loss. It should also be noted that this whole Kafka configuration is in
its own ruleset and called from other rulesets in order to output data to Kafka.
The issue I'm having is that the queue doesn't hold onto messages and simply
passes it on (I can see in the relevant statistics that the queue processed the
messages) even when the Kafka cluster can't be reached and the action suspends.
The librdkafka library - used by omkafka - also has its own queue, but this
fills up and message loss occurs. I've also tried to rather set the queue on
the ruleset itself, but that too makes no difference.
Below is the queue configuration within the action(type="omkafka" ...)
definition.
# Queue Settings
queue.type="LinkedList" # In-memory queue with dynamic memory allocation
queue.spoolDirectory="/var/spool/rsyslog" # Path for disk-assisted queue
queue.size="65000" # Total queue size
queue.highwatermark="60000" # Write messages to disk if highwatermark
breached
queue.lowwatermark="50000" # Use in-memory queue again below this mark
queue.filename="kafka_output" # Enables disk-assisted queue and names it
queue.maxdiskspace="18g" # Maximum disk space disk-queue can use
queue.saveonshutdown="on" # Save queue contents to disk on shutdown
Any assistance in this regard would be greatly appreciated.
Kind Regards
---
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
