Yes. Storm and Kafka both share the same zookeeper ensemble.
I found the following error in my worker.log file,
2016-10-19 09:21:04.326 o.a.s.k.DynamicBrokersReader [INFO] Read partition info
from zookeeper: GlobalPartitionInformation{topic=mytopic,
partitionMap={0=kafka-broker01:9092, 1=kafka-broker02:9092,
2=kafka-broker03:9092}}
2016-10-19 09:21:04.330 o.a.s.k.KafkaUtils [INFO] Task [2/3] assigned
[Partition{host=kafka-broker01:9092, topic=mytopic, partition=1}]
2016-10-19 09:21:04.330 o.a.s.k.ZkCoordinator [INFO] Task [2/3] Deleted
partition managers: []
2016-10-19 09:21:04.330 o.a.s.k.ZkCoordinator [INFO] Task [2/3] New partition
managers: [Partition{host=kafka-broker01:9092, topic=mytopic, partition=1}]
2016-10-19 09:21:04.872 o.a.s.k.PartitionManager [INFO] Read partition
information from: /mytopic/myconsumergroup/partition_1 --> null
2016-10-19 09:21:05.006 o.a.s.k.PartitionManager [INFO] No partition
information found, using configuration to determine offset
2016-10-19 09:21:05.007 o.a.s.k.PartitionManager [INFO] Last commit offset from
zookeeper: 0
And In my Zookeeper Client, I could find only the partition_2 file in the
zookeeper path,
[zk: localhost:2181(CONNECTED) 6] ls /mytopic/myconsumergroup
[partition_2]
Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : [email protected]
________________________________
From: [email protected] [[email protected]] on behalf of Junfeng Chen
[[email protected]]
Sent: Wednesday, October 19, 2016 3:11 PM
To: user@storm apache. org
Subject: Re: KafkaSpout offset stored under wrong partition in zookeeper
Does the storm share the same zookeeper with Kafka? And where did you find the
error?
Regard,
Junfeng Chen
On Wed, Oct 19, 2016 at 5:30 PM, Daniccan VP
<[email protected]<mailto:[email protected]>> wrote:
String zookeeperConnectionString =
"zookeeper-server01:2181,zookeeper-server02:2181,zookeeper-server03:2181,zookeeper-server04:2181";
BrokerHosts hosts = new ZkHosts(zookeeperConnectionString);
SpoutConfig spoutConfig = new SpoutConfig(hosts, kafkaTopicName, "/" +
kafkaTopicName, kafkaSpoutConsumerGroupId);
spoutConfig.bufferSizeBytes = 1024 * 1024 * 5;
spoutConfig.fetchSizeBytes = 1024 * 1024 * 5;
spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime();
spoutConfig.useStartOffsetTimeIfOffsetOutOfRange = true;
spoutConfig.scheme = new KeyValueSchemeAsMultiScheme(new
CustomizedKeyValueScheme());
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : [email protected]<http://iqsystech.com>
________________________________
From: [email protected]<mailto:[email protected]>
[[email protected]<mailto:[email protected]>] on behalf of Junfeng Chen
[[email protected]<mailto:[email protected]>]
Sent: Wednesday, October 19, 2016 2:43 PM
To: user@storm apache. org
Subject: Re: KafkaSpout offset stored under wrong partition in zookeeper
Can you provide your KafkaSpout configuration?
Regard,
Junfeng Chen
On Wed, Oct 19, 2016 at 5:07 PM, Daniccan VP
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I am running a Storm Topology with a KafkaSpout which consumes messages from
Kafka Brokers. The issue I am facing is that the assigned partition for a
worker process and the partition and offset information of the consumer stored
in zookeeper are different.
For example, One of the worker processes is assigned with a partition id of 1.
But, it keeps searching for the last offset under partition 2 in Zookeeper
path. Has anyone else faced a similar issue ?
Thanks and Regards,
Daniccan VP | Junior Software Engineer
Email : [email protected]<http://iqsystech.com>
*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they have been
addressed. If you are not the intended recipient, you are notified that
disclosing, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete
this e-mail from your system.
*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they have been
addressed. If you are not the intended recipient, you are notified that
disclosing, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete
this e-mail from your system.
*******************************************************************************************************************************************************************
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they have been
addressed. If you are not the intended recipient, you are notified that
disclosing, copying, distributing or taking any action in reliance on the
contents of this information is strictly prohibited. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete
this e-mail from your system.
