This might be a case of integer overflow with the partition counter if MAX_INT is reached.
@Sury - what OS/arch are you using? On Thu, Dec 10, 2015 at 1:35 PM, David Lang <[email protected]> wrote: > I don't know kafka, but when dealing with issues like this, I like to write > the data to a local file using the template that you use to send it to the > remote system, that way you can check to see if there is something odd in > the log messages that trigger issues. > > David Lang > > On Thu, 10 Dec 2015, Sury Bu wrote: > >> Date: Thu, 10 Dec 2015 17:52:11 +0800 >> From: Sury Bu <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: [email protected] >> Subject: [rsyslog] rsyslog omkafka define partition problem >> >> >> Hi All, >> >> I use rsyslog as log collect tool in the ELK stacks, logs will send to >> kafka, rsyslog.conf configured as below: >> >> module(load="imfile" PollingInterval="5") >> module(load="omkafka") >> $WorkDirectory /var/lib/rsyslog >> input(type="imfile" File="/usr/local/tomcat6/logs/xx/log4j.log" Tag="xx:" >> Severity="info" Facility="local0") >> >> local0.* action(type="omkafka" topic="xx" >> broker="elkkafka1.example.com:9092 >> ,elkkafka2.example.com:9092,elkkafka3.example.com.p.xyidc:9092" >> partitions.number="24" >> confParam=["compression.codec=snappy","socket.keepalive.enable=true"] >> queue.spoolDirectory="/usr/local/xx" queue.filename="rsyslogq" >> queue.saveonshutdown="on" queue.size="1000000" queue.type="LinkedList" >> queue.highwatermark="600000" queue.lowwatermark="20000" >> queue.discardmark="800000" queue.maxfilesize="1g" queue.maxdiskspace="10g" >> action.ResumeInterval="10" action.ResumeRetryCount="-1" >> action.reportSuspension="on" action.reportSuspensionContinuation="on") >> >> Sometimes I can find these ERROR messages in the /var/log/messages, >> partitions becomes from -1 to -23, while it should from 0 to 23, can >> anyone >> meat the same problem? >> >> Dec 10 10:56:01 xx1 rsyslogd-2359: action 'action 7' resumed (module >> 'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ] >> Dec 10 10:56:01 xx1 rsyslogd-2424: omkafka: Failed to produce to topic >> 'xx' >> partition -4: Local: Unknown partition [v8.14.0 try >> http://www.rsyslog.com/e/2424 ] >> Dec 10 10:56:11 xx1 rsyslogd-2359: action 'action 7' resumed (module >> 'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ] >> Dec 10 10:56:11 xx1 rsyslogd-2424: omkafka: Failed to produce to topic >> 'xx' >> partition -3: Local: Unknown partition [v8.14.0 try >> http://www.rsyslog.com/e/2424 ] >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
