Thanks, good to know. Without the braces, it didn't throw an error and connects to all the partitions right but maybe it just connected to the first broker in the list and got other brokers from the leader. Would be good to have some "informational" logging about what output services/hosts it connects and disconnects from, maybe?
On Sun, Dec 21, 2014 at 2:03 PM, Rainer Gerhards <[email protected]> wrote: > Broker is an array parameter just like confirmation parameters. So you > need the braces if you have multiple values. > > Sent from phone, thus brief. > Am 21.12.2014 01:28 schrieb "Tim Smith" <[email protected]>: > > > Finally, I had a chance to test the new -kafka packages on RHEL6.5. > > > > First the quirks/errors. I tried this config but didn't work: > > action(type="omkafka" topic="rawunstruct" > > broker="test-kafkanode1-1-node.ops.abc.net:9092, > > test-kafkanode1-2-node.ops.abc.net:9092, > > test-kafkanode1-3-node.ops.abc.net:9092, > > test-kafkanode1-4-node.ops.abc.net:9092, > > test-kafkanode1-5-node.ops.abc.net:9092" > > partitions.number="5" > > errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" > > confParam=["compression.codec=snappy", > > "request.required.acks=1", > > "producer.type=async", > > "batch.num.messages=10000" > > ] > > queue.filename="toKafka" > > queue.size="360000000" > > queue.maxdiskspace="360G" > > queue.highwatermark="216000000" > > queue.discardmark="288000000" > > queue.type="LinkedList" > > queue.dequeuebatchsize="4096" > > queue.timeoutenqueue="0" > > queue.maxfilesize="4G" > > queue.saveonshutdown="on" > > queue.workerThreads="4" > > template="rawfwd" > > ) stop > > > > rsyslogd complained: > > 2014-12-21T00:31:58.980146+00:00 test-kafkanode1-24-node rsyslogd-1000: > > error in kafka parameter 'request.required.acks=1': No such configuration > > property: "request.required.acks" [try http://www.rsyslog.com/e/1000 ] > > 2014-12-21T00:33:23.507670+00:00 test-kafkanode1-24-node rsyslogd: > [origin > > software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="8265" x-info=" > > http://www.rsyslog.com";] exiting on signal 15. > > 2014-12-21T00:33:26.624810+00:00 test-kafkanode1-24-node rsyslogd: > [origin > > software="rsyslogd" swVersion="8.7.0.master-kafka1" x-pid="19759" > x-info=" > > http://www.rsyslog.com";] start > > 2014-12-21T00:33:26.624811+00:00 test-kafkanode1-24-node rsyslogd-1000: > > error in kafka parameter 'producer.type=async': No such configuration > > property: "producer.type" [try http://www.rsyslog.com/e/1000 ] > > 2014-12-21T00:33:26.616090+00:00 test-kafkanode1-24-node rsyslogd0: debug > > log file is '/data/data01/rsyslog/debug/debug.log', fd 3 [try > > http://www.rsyslog.com/e/0 ] > > 2014-12-21T00:33:53.092950+00:00 test-kafkanode1-24-node rsyslogd-1000: > > error in kafka parameter 'producer.type=async': No such configuration > > property: "producer.type" [try http://www.rsyslog.com/e/1000 ] > > > > So I trimmed down the config to: > > action(type="omkafka" topic="rawunstruct" > > broker="test-kafkanode1-1-node.ops.abc.net:9092, > > test-kafkanode1-2-node.ops.abc.net:9092, > > test-kafkanode1-3-node.ops.abc.net:9092, > > test-kafkanode1-4-node.ops.abc.net:9092, > > test-kafkanode1-5-node.ops.abc.net:9092" > > partitions.number="5" > > errorFile="/data/data01/rsyslog/errors/kafka-output-err.log" > > confParam=["compression.codec=snappy"] > > queue.filename="toKafka" > > queue.size="360000000" > > queue.maxdiskspace="360G" > > queue.highwatermark="216000000" > > queue.discardmark="288000000" > > queue.type="LinkedList" > > queue.dequeuebatchsize="4096" > > queue.timeoutenqueue="0" > > queue.maxfilesize="4G" > > queue.saveonshutdown="on" > > queue.workerThreads="4" > > template="rawfwd" > > ) stop > > > > > > At restart, rsyslogd failed to stop cleanly, throwing these errors in > > /var/log/messages: > > 2014-12-21T00:35:33.314674+00:00 test-kafkanode1-24-node kernel: > rs:action > > 1 que[19764]: segfault at 21 ip 00007f9c329f4689 sp 00007f9c31a78b18 > error > > 4 in libc-2.12.so[7f9c328c9000+18b000] > > > > But start was ok. > > > > Then, there was a crash but abrtd did not keep the crash dump :( > > > > So, I set rsyslogd to run in debug mode but after several tens of Gigs > > worth of debug logs, I couldn't re-produce the crash so went back to > debug > > set to '0'. > > > > Now, so far, haven't had a crash in about half an hour and I would say > > about ~40 million messages must have passed through in this time :) > > > > For a first cut, I give this release a 10/10 :D > > > > Hoping to test more and smoke out any bugs. As a current user of > > rsyslog->flume->kafka, rsyslog->kafka simply kicks butt!!! I can't wait > to > > say goodbye to resource hungry and sloooowww flume/java :) > > > > Thanks, > > > > Tim > > > > > > > > > > On Tue, Dec 16, 2014 at 2:11 PM, Rainer Gerhards < > [email protected] > > > > > wrote: > > > > > Andre has now also created some test packages for Centos 6 & 7. If you > > > would like to use them, add this package source: > > > > > > http://rpms.adiscon.com/testing/ > > > > > > Note that this source is obviously not stable, but we will keep it > > > available until the official release happens. > > > > > > Rainer > > > > > > 2014-12-15 19:06 GMT+01:00 Tait Clarridge <[email protected]>: > > > > > > > > On Mon, Dec 15, 2014 at 5:04 AM, Rainer Gerhards > > > > <[email protected]> wrote: > > > > > So here it is: > > > > > > > > > > https://github.com/rsyslog/rsyslog/tree/master-omkafka > > > > > > > > > > I have deliberately left out the impstats calls as I saw you > included > > > > them > > > > > in your source, so feel free to add them again ;) > > > > > > > > > > Note that there is a different approach to parameters: my version > has > > > > only > > > > > very few hardcoded ones, but instead permits the user to set actual > > > kafka > > > > > parameters via the confparam and topicparam parameters. In my > > > experience, > > > > > this is far better with a rapidly evolving backend, because now any > > new > > > > or > > > > > changed parameter is immediately available. > > > > > > > > > > I'll write up some bare doc later today. So far this is tested on > > > Ubuntu > > > > > 14.04LTS only, and I would *deeply* appreciate if you (and anyone > > else) > > > > > could test it - especially as I have only very limited Kafka > know-how > > > and > > > > > no real use case myself. > > > > > > > > > > Thanks, > > > > > Rainer > > > > > > > > > > > > > Great! Looks much better than my original version, the "dynamic" > > > > configuration for conf and topicconf are definitely a win there > > > > (hardcoding is the worst). > > > > I had submitted my pull request with the template enabled topics, I > > > > had to shuffle some of the items around to make the topics work on > > > > each call of writeKafka. > > > > > > > > Found here: > > > > https://github.com/rsyslog/rsyslog/pull/186 > > > > > > > > Tait > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > > > DON'T LIKE THAT. > > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
