Wrote too soon, not enough RTFM :) omkafka already has "confParam" and "topicConfParam", sorry, my bad Rainer :)
But "request.required.acks=1" is redundant since that is the default for librdkafka and omkafka does not change that. I hereby increase the rating of this module to 15/10 from 10/10 :P On Sun, Dec 21, 2014 at 7:11 AM, Tim Smith <[email protected]> wrote: > Looking at librdkafka documentation, I understand why the extra kafka > parameters did not get honoured: > https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md > > "request.required.acks=1" -> This is a per topic configuration so might > require code change in omkafka to support "confGlobalParam" and > "confTopicParam" instead of the generic "confParam". #FeatureRequest > "producer.type=async" -> librdkafka is async only so this parameter makes > no sense and isn't supported by the API > "batch.num.messages=10000" -> I withdrew it before testing to keep things > simple. Tried it again and it worked :) > > > > > On Sun, Dec 21, 2014 at 6:58 AM, Tim Smith <[email protected]> wrote: > >> Finally, I had a chance to test the new -kafka packages on RHEL6.5. >> >> First the quirks/errors. I tried this config but didn't work: >> action(type="omkafka" topic="rawunstruct" >> broker="test-kafkanode1-1-node.ops.abc.net:9092, >> test-kafkanode1-2-node.ops.abc.net:9092, >> test-kafkanode1-3-node.ops.abc.net:9092, >> test-kafkanode1-4-node.ops.abc.net:9092, >> test-kafkanode1-5-node.ops.abc.net:9092" >> partitions.number="5" >> errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" >> confParam=["compression.codec=snappy", >> "request.required.acks=1", >> "producer.type=async", >> "batch.num.messages=10000" >> ] >> queue.filename="toKafka" >> queue.size="360000000" >> queue.maxdiskspace="360G" >> queue.highwatermark="216000000" >> queue.discardmark="288000000" >> queue.type="LinkedList" >> queue.dequeuebatchsize="4096" >> queue.timeoutenqueue="0" >> queue.maxfilesize="4G" >> queue.saveonshutdown="on" >> queue.workerThreads="4" >> template="rawfwd" >> ) stop >> >> rsyslogd complained: >> 2014-12-21T00:31:58.980146+00:00 test-kafkanode1-24-node rsyslogd-1000: >> error in kafka parameter 'request.required.acks=1': No such configuration >> property: "request.required.acks" [try http://www.rsyslog.com/e/1000 ] >> 2014-12-21T00:33:23.507670+00:00 test-kafkanode1-24-node rsyslogd: >> [origin software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="8265" >> x-info="http://www.rsyslog.com";] exiting on signal 15. >> 2014-12-21T00:33:26.624810+00:00 test-kafkanode1-24-node rsyslogd: >> [origin software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="19759" >> x-info="http://www.rsyslog.com";] start >> 2014-12-21T00:33:26.624811+00:00 test-kafkanode1-24-node rsyslogd-1000: >> error in kafka parameter 'producer.type=async': No such configuration >> property: "producer.type" [try http://www.rsyslog.com/e/1000 ] >> 2014-12-21T00:33:26.616090+00:00 test-kafkanode1-24-node rsyslogd0: debug >> log file is '/data/data01/rsyslog/debug/debug.log', fd 3 [try >> http://www.rsyslog.com/e/0 ] >> 2014-12-21T00:33:53.092950+00:00 test-kafkanode1-24-node rsyslogd-1000: >> error in kafka parameter 'producer.type=async': No such configuration >> property: "producer.type" [try http://www.rsyslog.com/e/1000 ] >> >> So I trimmed down the config to: >> action(type="omkafka" topic="rawunstruct" >> broker="test-kafkanode1-1-node.ops.abc.net:9092, >> test-kafkanode1-2-node.ops.abc.net:9092, >> test-kafkanode1-3-node.ops.abc.net:9092, >> test-kafkanode1-4-node.ops.abc.net:9092, >> test-kafkanode1-5-node.ops.abc.net:9092" >> partitions.number="5" >> errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" >> confParam=["compression.codec=snappy"] >> queue.filename="toKafka" >> queue.size="360000000" >> queue.maxdiskspace="360G" >> queue.highwatermark="216000000" >> queue.discardmark="288000000" >> queue.type="LinkedList" >> queue.dequeuebatchsize="4096" >> queue.timeoutenqueue="0" >> queue.maxfilesize="4G" >> queue.saveonshutdown="on" >> queue.workerThreads="4" >> template="rawfwd" >> ) stop >> >> >> At restart, rsyslogd failed to stop cleanly, throwing these errors in >> /var/log/messages: >> 2014-12-21T00:35:33.314674+00:00 test-kafkanode1-24-node kernel: >> rs:action 1 que[19764]: segfault at 21 ip 00007f9c329f4689 sp >> 00007f9c31a78b18 error 4 in libc-2.12.so[7f9c328c9000+18b000] >> >> But start was ok. >> >> Then, there was a crash but abrtd did not keep the crash dump :( >> >> So, I set rsyslogd to run in debug mode but after several tens of Gigs >> worth of debug logs, I couldn't re-produce the crash so went back to debug >> set to '0'. >> >> Now, so far, haven't had a crash in about half an hour and I would say >> about ~40 million messages must have passed through in this time :) >> >> For a first cut, I give this release a 10/10 :D >> >> Hoping to test more and smoke out any bugs. As a current user of >> rsyslog->flume->kafka, rsyslog->kafka simply kicks butt!!! I can't wait to >> say goodbye to resource hungry and sloooowww flume/java :) >> >> Thanks, >> >> Tim >> >> >> >> >> On Tue, Dec 16, 2014 at 2:11 PM, Rainer Gerhards < >> [email protected]> wrote: >> >>> Andre has now also created some test packages for Centos 6 & 7. If you >>> would like to use them, add this package source: >>> >>> http://rpms.adiscon.com/testing/ >>> >>> Note that this source is obviously not stable, but we will keep it >>> available until the official release happens. >>> >>> Rainer >>> >>> 2014-12-15 19:06 GMT+01:00 Tait Clarridge <[email protected]>: >>> > >>> > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards >>> > <[email protected]> wrote: >>> > > So here it is: >>> > > >>> > > https://github.com/rsyslog/rsyslog/tree/master-omkafka >>> > > >>> > > I have deliberately left out the impstats calls as I saw you included >>> > them >>> > > in your source, so feel free to add them again ;) >>> > > >>> > > Note that there is a different approach to parameters: my version has >>> > only >>> > > very few hardcoded ones, but instead permits the user to set actual >>> kafka >>> > > parameters via the confparam and topicparam parameters. In my >>> experience, >>> > > this is far better with a rapidly evolving backend, because now any >>> new >>> > or >>> > > changed parameter is immediately available. >>> > > >>> > > I'll write up some bare doc later today. So far this is tested on >>> Ubuntu >>> > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone >>> else) >>> > > could test it - especially as I have only very limited Kafka >>> know-how and >>> > > no real use case myself. >>> > > >>> > > Thanks, >>> > > Rainer >>> > > >>> > >>> > Great! Looks much better than my original version, the "dynamic" >>> > configuration for conf and topicconf are definitely a win there >>> > (hardcoding is the worst). >>> > I had submitted my pull request with the template enabled topics, I >>> > had to shuffle some of the items around to make the topics work on >>> > each call of writeKafka. >>> > >>> > Found here: >>> > https://github.com/rsyslog/rsyslog/pull/186 >>> > >>> > Tait >>> > _______________________________________________ >>> > rsyslog mailing list >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >>> > http://www.rsyslog.com/professional-services/ >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>> myriad >>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> > DON'T LIKE THAT. >>> > >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
