hi Sachin, As Guy said, your topology is too slow. The rate you are processing tuples is slower than the rate of them being deleted from the kafka servers.
There can be innumerable reasons for why this is, but basically you need to increase the throughput of your topology's processing of the data read from kafka. More workers, executors, etc. might help, but don't just do increase the values thoughtlessly, try doing it with some planning and care. - Erik On Sun, Nov 22, 2015 at 10:14 PM, Sachin Pasalkar < [email protected]> wrote: > Yes its running for a week now. I was also guessing the same on spout > lagging just wanted to double check. We have retention hours as 24 hrs. But > couldn’t make out why topology halts after these kinds of warnings :( > > > -----Original Message----- > From: Guy Spielberg [mailto:[email protected]] > Sent: Monday, November 23, 2015 11:18 AM > To: [email protected] > Subject: Re: Why am I getting OffsetOutOfRange: Updating offset from > offset? > > Hi Sachin, > > I guess these "few days of topology run" will be a week? > Your Kafka spout is probably lagging behind and messages located at the > offset requested were deleted from the Kafka topic (log.retention.hours), > causing the following fetch request to throw `OffsetOutOfRange`. > > Thanks, > Guy > > > On Mon, Nov 23, 2015 at 7:25 AM, Sachin Pasalkar < > [email protected]> wrote: > > > Can someone help us on this? > > > > From: Sachin Pasalkar <[email protected]<mailto: > > [email protected]>> > > Reply-To: "[email protected]<mailto:[email protected]>" < > > [email protected]<mailto:[email protected]>> > > Date: Friday, 20 November 2015 11:53 am > > To: "[email protected]<mailto:[email protected]>" < > > [email protected]<mailto:[email protected]>> > > Subject: Why am I getting OffsetOutOfRange: Updating offset from offset? > > > > Hi, > > > > We are developing application where after some days of topology run, > > we get continuous warning messages > > > > > > 2015-11-20 05:05:42.226 s.k.KafkaUtils [WARN] Got fetch request with > > offset out of range: [7238824446] > > > > 2015-11-20 05:05:42.229 s.k.t.TridentKafkaEmitter [WARN] > OffsetOutOfRange: > > Updating offset from offset = 7238824446 to offset = 7241183683 > > > > 2015-11-20 05:05:43.207 s.k.KafkaUtils [WARN] Got fetch request with > > offset out of range: [7022945051] > > > > 2015-11-20 05:05:43.208 s.k.t.TridentKafkaEmitter [WARN] > OffsetOutOfRange: > > Updating offset from offset = 7022945051 to offset = 7025309343 > > > > 2015-11-20 05:05:44.260 s.k.KafkaUtils [WARN] Got fetch request with > > offset out of range: [7170559432] > > > > 2015-11-20 05:05:44.264 s.k.t.TridentKafkaEmitter [WARN] > OffsetOutOfRange: > > Updating offset from offset = 7170559432 to offset = 7172920769 > > > > 2015-11-20 05:05:45.332 s.k.KafkaUtils [WARN] Got fetch request with > > offset out of range: [7132495867]…… > > > > > > After some point topology stop processing messages, I need to > > rebalance it to start it again. > > > > > > My spout config is > > > > > > BrokerHosts brokers = new ZkHosts((String) > > stormConfiguration.get(ZOOKEEPER_HOSTS)); > > > > TridentKafkaConfig spoutConfig = new TridentKafkaConfig(brokers, > > (String) stormConfiguration.get(KAFKA_INPUT_TOPIC)); > > > > > > spoutConfig.scheme = getSpoutScheme(stormConfiguration); > > > > Boolean forceFromStart = (Boolean) > > stormConfiguration.get(FORCE_FROM_START); > > > > > > spoutConfig.ignoreZkOffsets = false; > > > > spoutConfig.fetchSizeBytes = > > stormConfiguration.getIntProperty(KAFKA_CONSUMER_FETCH_SIZE_BYTE, > > KAFKA_CONSUMER_DEFAULT_FETCH_SIZE_BYTE); > > > > spoutConfig.bufferSizeBytes = > > stormConfiguration.getIntProperty(KAFKA_CONSUMER_BUFFER_SIZE_BYTE, > > KAFKA_CONSUMER_DEFAULT_BUFFER_SIZE_BYTE); > > > > As per my knowledge, only thing we are doing wrong is topic has 12 > > partitions but we are reading using only 1 spout, but that’s > > limitation on our side. I am not sure why its getting halted? It just > > keep printing below lines and does nothing > > > > > > 2015-11-20 05:44:41.574 b.s.m.n.Server [INFO] Getting metrics for > > server on port 6700 > > > > 2015-11-20 05:44:41.574 b.s.m.n.Client [INFO] Getting metrics for > > client connection to Netty-Client-b-bdata-xx.net/xxx.xx.xxx.xxx:6700 > > > > 2015-11-20 05:44:41.574 b.s.m.n.Client [INFO] Getting metrics for > > client connection to Netty-Client-b-bdata-xx.net/xxx.xx.xxx.xxx:6709 > > > > 2015-11-20 05:44:41.574 b.s.m.n.Client [INFO] Getting metrics for > > client connection to Netty-Client-b-bdata-xx.net/xxx.xx.xxx.xxx:6707 > > > > > > Thanks, > > > > Sachin > > > > >
