CentOS release 6.3 (Final) 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
The -23 to -1 partitions only appears at some unknown special situation, and at most time the rsyslog kafka module can work fine in my environment. On Fri, Dec 11, 2015 at 11:04 AM, Tait Clarridge <[email protected]> wrote: > This might be a case of integer overflow with the partition counter if > MAX_INT is reached. > > @Sury - what OS/arch are you using? > > On Thu, Dec 10, 2015 at 1:35 PM, David Lang <[email protected]> wrote: > > I don't know kafka, but when dealing with issues like this, I like to > write > > the data to a local file using the template that you use to send it to > the > > remote system, that way you can check to see if there is something odd in > > the log messages that trigger issues. > > > > David Lang > > > > On Thu, 10 Dec 2015, Sury Bu wrote: > > > >> Date: Thu, 10 Dec 2015 17:52:11 +0800 > >> From: Sury Bu <[email protected]> > >> Reply-To: rsyslog-users <[email protected]> > >> To: [email protected] > >> Subject: [rsyslog] rsyslog omkafka define partition problem > >> > >> > >> Hi All, > >> > >> I use rsyslog as log collect tool in the ELK stacks, logs will send to > >> kafka, rsyslog.conf configured as below: > >> > >> module(load="imfile" PollingInterval="5") > >> module(load="omkafka") > >> $WorkDirectory /var/lib/rsyslog > >> input(type="imfile" File="/usr/local/tomcat6/logs/xx/log4j.log" > Tag="xx:" > >> Severity="info" Facility="local0") > >> > >> local0.* action(type="omkafka" topic="xx" > >> broker="elkkafka1.example.com:9092 > >> ,elkkafka2.example.com:9092,elkkafka3.example.com.p.xyidc:9092" > >> partitions.number="24" > >> confParam=["compression.codec=snappy","socket.keepalive.enable=true"] > >> queue.spoolDirectory="/usr/local/xx" queue.filename="rsyslogq" > >> queue.saveonshutdown="on" queue.size="1000000" queue.type="LinkedList" > >> queue.highwatermark="600000" queue.lowwatermark="20000" > >> queue.discardmark="800000" queue.maxfilesize="1g" > queue.maxdiskspace="10g" > >> action.ResumeInterval="10" action.ResumeRetryCount="-1" > >> action.reportSuspension="on" action.reportSuspensionContinuation="on") > >> > >> Sometimes I can find these ERROR messages in the /var/log/messages, > >> partitions becomes from -1 to -23, while it should from 0 to 23, can > >> anyone > >> meat the same problem? > >> > >> Dec 10 10:56:01 xx1 rsyslogd-2359: action 'action 7' resumed (module > >> 'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ] > >> Dec 10 10:56:01 xx1 rsyslogd-2424: omkafka: Failed to produce to topic > >> 'xx' > >> partition -4: Local: Unknown partition [v8.14.0 try > >> http://www.rsyslog.com/e/2424 ] > >> Dec 10 10:56:11 xx1 rsyslogd-2359: action 'action 7' resumed (module > >> 'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ] > >> Dec 10 10:56:11 xx1 rsyslogd-2424: omkafka: Failed to produce to topic > >> 'xx' > >> partition -3: Local: Unknown partition [v8.14.0 try > >> http://www.rsyslog.com/e/2424 ] > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
