Looking at librdkafka documentation, I understand why the extra kafka parameters did not get honoured: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
"request.required.acks=1" -> This is a per topic configuration so might require code change in omkafka to support "confGlobalParam" and "confTopicParam" instead of the generic "confParam". #FeatureRequest "producer.type=async" -> librdkafka is async only so this parameter makes no sense and isn't supported by the API "batch.num.messages=10000" -> I withdrew it before testing to keep things simple. Tried it again and it worked :) On Sun, Dec 21, 2014 at 6:58 AM, Tim Smith <[email protected]> wrote: > Finally, I had a chance to test the new -kafka packages on RHEL6.5. > > First the quirks/errors. I tried this config but didn't work: > action(type="omkafka" topic="rawunstruct" > broker="test-kafkanode1-1-node.ops.abc.net:9092, > test-kafkanode1-2-node.ops.abc.net:9092, > test-kafkanode1-3-node.ops.abc.net:9092, > test-kafkanode1-4-node.ops.abc.net:9092, > test-kafkanode1-5-node.ops.abc.net:9092" > partitions.number="5" > errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" > confParam=["compression.codec=snappy", > "request.required.acks=1", > "producer.type=async", > "batch.num.messages=10000" > ] > queue.filename="toKafka" > queue.size="360000000" > queue.maxdiskspace="360G" > queue.highwatermark="216000000" > queue.discardmark="288000000" > queue.type="LinkedList" > queue.dequeuebatchsize="4096" > queue.timeoutenqueue="0" > queue.maxfilesize="4G" > queue.saveonshutdown="on" > queue.workerThreads="4" > template="rawfwd" > ) stop > > rsyslogd complained: > 2014-12-21T00:31:58.980146+00:00 test-kafkanode1-24-node rsyslogd-1000: > error in kafka parameter 'request.required.acks=1': No such configuration > property: "request.required.acks" [try http://www.rsyslog.com/e/1000 ] > 2014-12-21T00:33:23.507670+00:00 test-kafkanode1-24-node rsyslogd: [origin > software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="8265" x-info=" > http://www.rsyslog.com";] exiting on signal 15. > 2014-12-21T00:33:26.624810+00:00 test-kafkanode1-24-node rsyslogd: [origin > software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="19759" x-info=" > http://www.rsyslog.com";] start > 2014-12-21T00:33:26.624811+00:00 test-kafkanode1-24-node rsyslogd-1000: > error in kafka parameter 'producer.type=async': No such configuration > property: "producer.type" [try http://www.rsyslog.com/e/1000 ] > 2014-12-21T00:33:26.616090+00:00 test-kafkanode1-24-node rsyslogd0: debug > log file is '/data/data01/rsyslog/debug/debug.log', fd 3 [try > http://www.rsyslog.com/e/0 ] > 2014-12-21T00:33:53.092950+00:00 test-kafkanode1-24-node rsyslogd-1000: > error in kafka parameter 'producer.type=async': No such configuration > property: "producer.type" [try http://www.rsyslog.com/e/1000 ] > > So I trimmed down the config to: > action(type="omkafka" topic="rawunstruct" > broker="test-kafkanode1-1-node.ops.abc.net:9092, > test-kafkanode1-2-node.ops.abc.net:9092, > test-kafkanode1-3-node.ops.abc.net:9092, > test-kafkanode1-4-node.ops.abc.net:9092, > test-kafkanode1-5-node.ops.abc.net:9092" > partitions.number="5" > errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" > confParam=["compression.codec=snappy"] > queue.filename="toKafka" > queue.size="360000000" > queue.maxdiskspace="360G" > queue.highwatermark="216000000" > queue.discardmark="288000000" > queue.type="LinkedList" > queue.dequeuebatchsize="4096" > queue.timeoutenqueue="0" > queue.maxfilesize="4G" > queue.saveonshutdown="on" > queue.workerThreads="4" > template="rawfwd" > ) stop > > > At restart, rsyslogd failed to stop cleanly, throwing these errors in > /var/log/messages: > 2014-12-21T00:35:33.314674+00:00 test-kafkanode1-24-node kernel: rs:action > 1 que[19764]: segfault at 21 ip 00007f9c329f4689 sp 00007f9c31a78b18 error > 4 in libc-2.12.so[7f9c328c9000+18b000] > > But start was ok. > > Then, there was a crash but abrtd did not keep the crash dump :( > > So, I set rsyslogd to run in debug mode but after several tens of Gigs > worth of debug logs, I couldn't re-produce the crash so went back to debug > set to '0'. > > Now, so far, haven't had a crash in about half an hour and I would say > about ~40 million messages must have passed through in this time :) > > For a first cut, I give this release a 10/10 :D > > Hoping to test more and smoke out any bugs. As a current user of > rsyslog->flume->kafka, rsyslog->kafka simply kicks butt!!! I can't wait to > say goodbye to resource hungry and sloooowww flume/java :) > > Thanks, > > Tim > > > > > On Tue, Dec 16, 2014 at 2:11 PM, Rainer Gerhards <[email protected] > > wrote: > >> Andre has now also created some test packages for Centos 6 & 7. If you >> would like to use them, add this package source: >> >> http://rpms.adiscon.com/testing/ >> >> Note that this source is obviously not stable, but we will keep it >> available until the official release happens. >> >> Rainer >> >> 2014-12-15 19:06 GMT+01:00 Tait Clarridge <[email protected]>: >> > >> > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards >> > <[email protected]> wrote: >> > > So here it is: >> > > >> > > https://github.com/rsyslog/rsyslog/tree/master-omkafka >> > > >> > > I have deliberately left out the impstats calls as I saw you included >> > them >> > > in your source, so feel free to add them again ;) >> > > >> > > Note that there is a different approach to parameters: my version has >> > only >> > > very few hardcoded ones, but instead permits the user to set actual >> kafka >> > > parameters via the confparam and topicparam parameters. In my >> experience, >> > > this is far better with a rapidly evolving backend, because now any >> new >> > or >> > > changed parameter is immediately available. >> > > >> > > I'll write up some bare doc later today. So far this is tested on >> Ubuntu >> > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone >> else) >> > > could test it - especially as I have only very limited Kafka know-how >> and >> > > no real use case myself. >> > > >> > > Thanks, >> > > Rainer >> > > >> > >> > Great! Looks much better than my original version, the "dynamic" >> > configuration for conf and topicconf are definitely a win there >> > (hardcoding is the worst). >> > I had submitted my pull request with the template enabled topics, I >> > had to shuffle some of the items around to make the topics work on >> > each call of writeKafka. >> > >> > Found here: >> > https://github.com/rsyslog/rsyslog/pull/186 >> > >> > Tait >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
