Broker is an array parameter just like confirmation parameters. So you need the braces if you have multiple values.
Sent from phone, thus brief. Am 21.12.2014 01:28 schrieb "Tim Smith" <[email protected]>: > Finally, I had a chance to test the new -kafka packages on RHEL6.5. > > First the quirks/errors. I tried this config but didn't work: > action(type="omkafka" topic="rawunstruct" > broker="test-kafkanode1-1-node.ops.abc.net:9092, > test-kafkanode1-2-node.ops.abc.net:9092, > test-kafkanode1-3-node.ops.abc.net:9092, > test-kafkanode1-4-node.ops.abc.net:9092, > test-kafkanode1-5-node.ops.abc.net:9092" > partitions.number="5" > errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" > confParam=["compression.codec=snappy", > "request.required.acks=1", > "producer.type=async", > "batch.num.messages=10000" > ] > queue.filename="toKafka" > queue.size="360000000" > queue.maxdiskspace="360G" > queue.highwatermark="216000000" > queue.discardmark="288000000" > queue.type="LinkedList" > queue.dequeuebatchsize="4096" > queue.timeoutenqueue="0" > queue.maxfilesize="4G" > queue.saveonshutdown="on" > queue.workerThreads="4" > template="rawfwd" > ) stop > > rsyslogd complained: > 2014-12-21T00:31:58.980146+00:00 test-kafkanode1-24-node rsyslogd-1000: > error in kafka parameter 'request.required.acks=1': No such configuration > property: "request.required.acks" [try http://www.rsyslog.com/e/1000 ] > 2014-12-21T00:33:23.507670+00:00 test-kafkanode1-24-node rsyslogd: [origin > software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="8265" x-info=" > http://www.rsyslog.com";] exiting on signal 15. > 2014-12-21T00:33:26.624810+00:00 test-kafkanode1-24-node rsyslogd: [origin > software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="19759" x-info=" > http://www.rsyslog.com";] start > 2014-12-21T00:33:26.624811+00:00 test-kafkanode1-24-node rsyslogd-1000: > error in kafka parameter 'producer.type=async': No such configuration > property: "producer.type" [try http://www.rsyslog.com/e/1000 ] > 2014-12-21T00:33:26.616090+00:00 test-kafkanode1-24-node rsyslogd0: debug > log file is '/data/data01/rsyslog/debug/debug.log', fd 3 [try > http://www.rsyslog.com/e/0 ] > 2014-12-21T00:33:53.092950+00:00 test-kafkanode1-24-node rsyslogd-1000: > error in kafka parameter 'producer.type=async': No such configuration > property: "producer.type" [try http://www.rsyslog.com/e/1000 ] > > So I trimmed down the config to: > action(type="omkafka" topic="rawunstruct" > broker="test-kafkanode1-1-node.ops.abc.net:9092, > test-kafkanode1-2-node.ops.abc.net:9092, > test-kafkanode1-3-node.ops.abc.net:9092, > test-kafkanode1-4-node.ops.abc.net:9092, > test-kafkanode1-5-node.ops.abc.net:9092" > partitions.number="5" > errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" > confParam=["compression.codec=snappy"] > queue.filename="toKafka" > queue.size="360000000" > queue.maxdiskspace="360G" > queue.highwatermark="216000000" > queue.discardmark="288000000" > queue.type="LinkedList" > queue.dequeuebatchsize="4096" > queue.timeoutenqueue="0" > queue.maxfilesize="4G" > queue.saveonshutdown="on" > queue.workerThreads="4" > template="rawfwd" > ) stop > > > At restart, rsyslogd failed to stop cleanly, throwing these errors in > /var/log/messages: > 2014-12-21T00:35:33.314674+00:00 test-kafkanode1-24-node kernel: rs:action > 1 que[19764]: segfault at 21 ip 00007f9c329f4689 sp 00007f9c31a78b18 error > 4 in libc-2.12.so[7f9c328c9000+18b000] > > But start was ok. > > Then, there was a crash but abrtd did not keep the crash dump :( > > So, I set rsyslogd to run in debug mode but after several tens of Gigs > worth of debug logs, I couldn't re-produce the crash so went back to debug > set to '0'. > > Now, so far, haven't had a crash in about half an hour and I would say > about ~40 million messages must have passed through in this time :) > > For a first cut, I give this release a 10/10 :D > > Hoping to test more and smoke out any bugs. As a current user of > rsyslog->flume->kafka, rsyslog->kafka simply kicks butt!!! I can't wait to > say goodbye to resource hungry and sloooowww flume/java :) > > Thanks, > > Tim > > > > > On Tue, Dec 16, 2014 at 2:11 PM, Rainer Gerhards <[email protected] > > > wrote: > > > Andre has now also created some test packages for Centos 6 & 7. If you > > would like to use them, add this package source: > > > > http://rpms.adiscon.com/testing/ > > > > Note that this source is obviously not stable, but we will keep it > > available until the official release happens. > > > > Rainer > > > > 2014-12-15 19:06 GMT+01:00 Tait Clarridge <[email protected]>: > > > > > > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards > > > <[email protected]> wrote: > > > > So here it is: > > > > > > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > > > > > > > I have deliberately left out the impstats calls as I saw you included > > > them > > > > in your source, so feel free to add them again ;) > > > > > > > > Note that there is a different approach to parameters: my version has > > > only > > > > very few hardcoded ones, but instead permits the user to set actual > > kafka > > > > parameters via the confparam and topicparam parameters. In my > > experience, > > > > this is far better with a rapidly evolving backend, because now any > new > > > or > > > > changed parameter is immediately available. > > > > > > > > I'll write up some bare doc later today. So far this is tested on > > Ubuntu > > > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone > else) > > > > could test it - especially as I have only very limited Kafka know-how > > and > > > > no real use case myself. > > > > > > > > Thanks, > > > > Rainer > > > > > > > > > > Great! Looks much better than my original version, the "dynamic" > > > configuration for conf and topicconf are definitely a win there > > > (hardcoding is the worst). > > > I had submitted my pull request with the template enabled topics, I > > > had to shuffle some of the items around to make the topics work on > > > each call of writeKafka. > > > > > > Found here: > > > https://github.com/rsyslog/rsyslog/pull/186 > > > > > > Tait > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
