Hi,
I’ve deployed a topology consisting of a KafkaSpout using Kafka 0.10.0.1 and
Zookeeper 3.4.6. All of the services, including the Nimbus and Supervisor, run
on the same instance.
However, by examining the worker.log file, I’ve found that the KafkaSpout is
unable to find partitions information of the given topic. The following log
messages for each of the 12 partitions appear:
2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] No partition
information found, using configuration to determine offset
2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Last commit offset from
zookeeper: 0
2016-09-17 17:37:33.333 o.a.s.k.PartitionManager [INFO] Commit offset 0 is more
than 9223372036854775807 behind latest offset 0, resetting to startOffsetTime=-2
Whereas, the SpoutConf is instantiated as follows:
SpoutConfig spoutConfig = new SpoutConfig(hosts, "bytes", new String(),
UUID.randomUUID().toString());
spoutConfig.scheme = new RawMultiScheme();
spoutConfig.zkServers = Arrays.asList("localhost");
spoutConfig.zkPort = 2181;
spoutConfig.zkRoot = "/brokers”;
By examining Zookeeper’s zNodes using the zkCli script, I’ve found that the
partitions information is stored properly as prescribed by the default
configuration: /brokers/topics/<topic_name>/partitions/…
What exactly might be the reason behind KafkaSpout not being able to consume
messages from Kafka, i.e. find partition information? Is there a configuration
mistake I’ve made?
Thanks in advance!
Dominik
