Does the issue occur after it rsyslog been running for a long time?
On Thu, Dec 10, 2015 at 10:33 PM, Sury Bu <[email protected]> wrote: > CentOS release 6.3 (Final) 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 > UTC 2012 x86_64 x86_64 x86_64 GNU/Linux > > The -23 to -1 partitions only appears at some unknown special situation, > and at most time the rsyslog kafka module can work fine in my environment. > > On Fri, Dec 11, 2015 at 11:04 AM, Tait Clarridge <[email protected]> wrote: > >> This might be a case of integer overflow with the partition counter if >> MAX_INT is reached. >> >> @Sury - what OS/arch are you using? >> >> On Thu, Dec 10, 2015 at 1:35 PM, David Lang <[email protected]> wrote: >> > I don't know kafka, but when dealing with issues like this, I like to >> write >> > the data to a local file using the template that you use to send it to >> the >> > remote system, that way you can check to see if there is something odd in >> > the log messages that trigger issues. >> > >> > David Lang >> > >> > On Thu, 10 Dec 2015, Sury Bu wrote: >> > >> >> Date: Thu, 10 Dec 2015 17:52:11 +0800 >> >> From: Sury Bu <[email protected]> >> >> Reply-To: rsyslog-users <[email protected]> >> >> To: [email protected] >> >> Subject: [rsyslog] rsyslog omkafka define partition problem >> >> >> >> >> >> Hi All, >> >> >> >> I use rsyslog as log collect tool in the ELK stacks, logs will send to >> >> kafka, rsyslog.conf configured as below: >> >> >> >> module(load="imfile" PollingInterval="5") >> >> module(load="omkafka") >> >> $WorkDirectory /var/lib/rsyslog >> >> input(type="imfile" File="/usr/local/tomcat6/logs/xx/log4j.log" >> Tag="xx:" >> >> Severity="info" Facility="local0") >> >> >> >> local0.* action(type="omkafka" topic="xx" >> >> broker="elkkafka1.example.com:9092 >> >> ,elkkafka2.example.com:9092,elkkafka3.example.com.p.xyidc:9092" >> >> partitions.number="24" >> >> confParam=["compression.codec=snappy","socket.keepalive.enable=true"] >> >> queue.spoolDirectory="/usr/local/xx" queue.filename="rsyslogq" >> >> queue.saveonshutdown="on" queue.size="1000000" queue.type="LinkedList" >> >> queue.highwatermark="600000" queue.lowwatermark="20000" >> >> queue.discardmark="800000" queue.maxfilesize="1g" >> queue.maxdiskspace="10g" >> >> action.ResumeInterval="10" action.ResumeRetryCount="-1" >> >> action.reportSuspension="on" action.reportSuspensionContinuation="on") >> >> >> >> Sometimes I can find these ERROR messages in the /var/log/messages, >> >> partitions becomes from -1 to -23, while it should from 0 to 23, can >> >> anyone >> >> meat the same problem? >> >> >> >> Dec 10 10:56:01 xx1 rsyslogd-2359: action 'action 7' resumed (module >> >> 'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ] >> >> Dec 10 10:56:01 xx1 rsyslogd-2424: omkafka: Failed to produce to topic >> >> 'xx' >> >> partition -4: Local: Unknown partition [v8.14.0 try >> >> http://www.rsyslog.com/e/2424 ] >> >> Dec 10 10:56:11 xx1 rsyslogd-2359: action 'action 7' resumed (module >> >> 'omkafka') [v8.14.0 try http://www.rsyslog.com/e/2359 ] >> >> Dec 10 10:56:11 xx1 rsyslogd-2424: omkafka: Failed to produce to topic >> >> 'xx' >> >> partition -3: Local: Unknown partition [v8.14.0 try >> >> http://www.rsyslog.com/e/2424 ] >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T >> >> LIKE THAT. >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of >> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> > LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
