Francois, I've used the same pattern you've described and didn't have any problems. Your approach is valid. There must be something else going on.
On Mon, Mar 23, 2015 at 2:28 PM Harsha <[email protected]> wrote: > It looks like your approach is right. Once you turn off forceFromStart and > set the offset time to earliestTime only new events from kafka topic will > be read. Are you sure that your kafka topic has new data coming in? > > -- > Harsha > > > On March 23, 2015 at 12:48:04 PM, François Méthot ([email protected]) > wrote: > > Hi, > > We have a storm topology that uses Kafka to read a topic with 6 > partitions. ( Kafka 0.8.2, Storm 0.9.3 ) > > Recently, we had to set the KafkaSpout to read from the beginning, so we > temporary configured our KafkaConfig this way: > > kafkaConfig.forceFromStart=true > kafkaConfig.startOffsetTime = OffsetRequest.EarliestTime() > > It worked well, but afterward, setting those parameters back to false and > to LatestTime respectively had no effect. In fact the topology won't read > from our topic anymore. > > When the topology starts, The spout successully logs the offset and > consumer group's cursor position for each partition in the worker log. But > nothing is read. > The only way we can read back from our Topic is to give our SpoutConfig a > new Kafka ConsumerGroup Id in the SpoutConfig: > ex: new SpoutConfig(zk, topic, zkStormRoot, newConsumerGroupID) > > Now, the only way I can see to read from the beginning would be to write > the position we want to read from in Zookeeper where Consumer Group offset > are stored and to restart our topology. I haven't tried it yet. > Bottom line, it looks like as you as you use this forceFromStart flag, > the consumer group id become unusable. > Would anyone know if this is a bug in the KafkaSpout or an issue inherited > from bug in Kafka? > > Thanks > Francois > > >
