I don't know kafka, but when dealing with issues like this, I like to write the
data to a local file using the template that you use to send it to the remote
system, that way you can check to see if there is something odd in the log
messages that trigger issues.
David Lang
On Thu, 10 Dec 2015, Sury Bu wrote:
Date: Thu, 10 Dec 2015 17:52:11 +0800
From: Sury Bu <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: [email protected]
Subject: [rsyslog] rsyslog omkafka define partition problem
Hi All,
I use rsyslog as log collect tool in the ELK stacks, logs will send to
kafka, rsyslog.conf configured as below:
module(load="imfile" PollingInterval="5")
module(load="omkafka")
$WorkDirectory /var/lib/rsyslog
input(type="imfile" File="/usr/local/tomcat6/logs/xx/log4j.log" Tag="xx:"
Severity="info" Facility="local0")
local0.* action(type="omkafka" topic="xx" broker="elkkafka1.example.com:9092
,elkkafka2.example.com:9092,elkkafka3.example.com.p.xyidc:9092"
partitions.number="24"
confParam=["compression.codec=snappy","socket.keepalive.enable=true"]
queue.spoolDirectory="/usr/local/xx" queue.filename="rsyslogq"
queue.saveonshutdown="on" queue.size="1000000" queue.type="LinkedList"
queue.highwatermark="600000" queue.lowwatermark="20000"
queue.discardmark="800000" queue.maxfilesize="1g" queue.maxdiskspace="10g"
action.ResumeInterval="10" action.ResumeRetryCount="-1"
action.reportSuspension="on" action.reportSuspensionContinuation="on")
Sometimes I can find these ERROR messages in the /var/log/messages,
partitions becomes from -1 to -23, while it should from 0 to 23, can anyone
meat the same problem?
Dec 10 10:56:01 xx1 rsyslogd-2359: action 'action 7' resumed (module
'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ]
Dec 10 10:56:01 xx1 rsyslogd-2424: omkafka: Failed to produce to topic 'xx'
partition -4: Local: Unknown partition [v8.14.0 try
http://www.rsyslog.com/e/2424 ]
Dec 10 10:56:11 xx1 rsyslogd-2359: action 'action 7' resumed (module
'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ]
Dec 10 10:56:11 xx1 rsyslogd-2424: omkafka: Failed to produce to topic 'xx'
partition -3: Local: Unknown partition [v8.14.0 try
http://www.rsyslog.com/e/2424 ]
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
