El 30/01/17 a las 19:25, Andrew Griffin escribió:
I have a rsyslog -> kafka -> splunk stack working pretty well, I could
probably answer a few of your questions -
You can list topics (and a lot of other stuff) on the kafka brokers
themselves using the kafka-topics.sh script included with kafka. e.g.:
bin/kafka-topics.sh —zookeeper=localhost:2181 —list
only __consumer_offsets is shown, so probably I'm not adding topics
correctly
bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
Topic: __consumer_offsets Partition: 0 Leader: 9 Replicas:
9,10,15 Isr: 10,9,15
Topic: __consumer_offsets Partition: 1 Leader: 10 Replicas:
10,15,9 Isr: 10,9,15
Topic: __consumer_offsets Partition: 2 Leader: 15 Replicas:
15,9,10 Isr: 10,9,15
...
Topic: __consumer_offsets Partition: 47 Leader: 15 Replicas:
15,10,9 Isr: 10,9,15
Topic: __consumer_offsets Partition: 48 Leader: 9 Replicas:
9,10,15 Isr: 10,9,15
Topic: __consumer_offsets Partition: 49 Leader: 10 Replicas:
10,15,9 Isr: 10,9,15
does this mean the cluster is properly formed?
I’d recommend using kafka-manager to manage your cluster. It’ll give
you a much quicker look in to your topics, your brokers, consumers,
and throughput. It also makes creating and deleting topics easy.
It isn't able to show cluster list, so perhaps problems connecting to zk?
If you’re not seeing your topics get created the first place I’d look
is in the kafka broker logs themselves - server.log and
kafkaServer.out - then work your way back from there. As you’ve
found, omkafka isn’t terribly verbose when it comes to error reporting.
plenty of:
[2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
For your timeout issues, the first place I’d look to is the local
firewall configuration. Also in your "broker=["cluster_kafka”]”
portion are you including the port number for the broker (I’m assuming
9092)?
I thought port was added by default. I'll try again.
Thanks!
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
