String zookeeperConnectionString = "zookeeper-server01:2181,zookeeper-server02:2181,zookeeper-server03:2181,zookeeper-server04:2181";
BrokerHosts hosts = new ZkHosts(zookeeperConnectionString); SpoutConfig spoutConfig = new SpoutConfig(hosts, kafkaTopicName, "/" + kafkaTopicName, kafkaSpoutConsumerGroupId); spoutConfig.bufferSizeBytes = 1024 * 1024 * 5; spoutConfig.fetchSizeBytes = 1024 * 1024 * 5; spoutConfig.startOffsetTime = kafka.api.OffsetRequest.EarliestTime(); spoutConfig.useStartOffsetTimeIfOffsetOutOfRange = true; spoutConfig.scheme = new KeyValueSchemeAsMultiScheme(new CustomizedKeyValueScheme()); KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig); Thanks and Regards, Daniccan VP | Junior Software Engineer Email : [email protected] ________________________________ From: [email protected] [[email protected]] on behalf of Junfeng Chen [[email protected]] Sent: Wednesday, October 19, 2016 2:43 PM To: user@storm apache. org Subject: Re: KafkaSpout offset stored under wrong partition in zookeeper Can you provide your KafkaSpout configuration? Regard, Junfeng Chen On Wed, Oct 19, 2016 at 5:07 PM, Daniccan VP <[email protected]<mailto:[email protected]>> wrote: Hi, I am running a Storm Topology with a KafkaSpout which consumes messages from Kafka Brokers. The issue I am facing is that the assigned partition for a worker process and the partition and offset information of the consumer stored in zookeeper are different. For example, One of the worker processes is assigned with a partition id of 1. But, it keeps searching for the last offset under partition 2 in Zookeeper path. Has anyone else faced a similar issue ? Thanks and Regards, Daniccan VP | Junior Software Engineer Email : [email protected]<http://iqsystech.com> ******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. ******************************************************************************************************************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they have been addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
