I'm sure I'm missing something.
I'm trying to send messages to kafka reliably but when I test
blocking/dropping packets with the firewall to simulate network problems,
log messages are lost.
I've tried several values for failedMsgFile with and without paths and
checked file permissions. Selinux is disabled for the testing.
It is a Centos 7 system with rsyslog-kafka-8.38.0-1.el7.x86_64 installed.
Also, is it possible to have the kafka module make back pressure to use the
native rsyslog queues instead of using the failedMsgFile or in addition to
it?
Here is the ruleset sending to kafka:
ruleset(name="Send_Event_obj_to_Kafka_as_json"){
set $.evnt!tags = "preped_by_rsyslog";
# action(type="omfwd" Target="192.168.2.11" Port="10514" Protocol="tcp" )
# for testing back pressure to earlier disk queue.
action(
type="omkafka"
topic="log.json"
template="json_from_event_obj"
queue.filename="action_send_json_to_kafka"
queue.spoolDirectory="/var/lib/rsyslog"
queue.type="linkedlist"
queue.size="200" #RAM used depends on actual messages size. Guestimate
is 512b per message 20,000 * 512B = 10MB of RAM
queue.highwatermark="-1" # -1 so the disk cache only gets used when
rsyslog is stopped or restarted.
# queue.lowwatermark="50" # should be irrelevant for this use case since
high watermark can not be reached.
queue.maxdiskspace="10737418" # 20MB
queue.discardmark="-1" # discard nothing in this queue.
queue.discardseverity="8" # discard nothing in this queue. 7 would
discard debug.
# queue.timeoutenqueue="10"
queue.dequeuebatchsize="40" # max number of messages to send at once to
kafka. Will send less if less are available when it is ready to send.
queue.saveonshutdown="on"
broker="192.168.14.26"
closeTimeout="2000" # ms to drain msg to kafka when shutting down
resubmitOnFailure="on"
KeepFailedMessages="on"
failedMsgFile="/tmp/kafka_faild_msgs_cache"
partitions.auto="on"
errorFile="/var/log/rsyslog_kafka_err.log"
confParam=[
"compression.codec=snappy",
"socket.timeout.ms=1000",
"socket.keepalive.enable=true"
]
)
}
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
