Please grab the settings/values for the spout as detailed in the
HortonWorks Kafka Spout paper. Send them to this list.
According to the Storm UI, the entire topology:
* what is the complete latency(ms) for the topology @ 1 day?, @ 3 hours?
According to the Storm UI, the spout:
* how many records Ackd
* how many records Failed?
You're the guy who thought he was having problems when the Kafka topic
partition rolled over right? What ended up happening there?
run this on one of your kafka nodes:
/opt/kafka_2.9.2-0.8.1.1/bin/kafka-topics.sh --zookeeper {kafkas_zk}
--describe
Please send the results for the topic in question
Thank you for your time!
+++++++++++++++++++++
Jeff Maass <[email protected]>
linkedin.com/in/jeffmaass
stackoverflow.com/users/373418/maassql
+++++++++++++++++++++
On Mon, May 18, 2015 at 4:13 PM, Benjamin Cuthbert <[email protected]>
wrote:
> Thanks Jeff,
>
> So I looked over the docs but I don’t understand is it runs for 2+ hours
> then just starts going:
>
> 2015-05-18T22:12:53.673+0100 s.k.PartitionManager [WARN] Using new offset:
> 64429892
> 2015-05-18T22:12:53.705+0100 s.k.KafkaUtils [WARN] Got fetch request with
> offset out of range: [63610973]; retrying with default start offset time
> from configuration. configured start offset time: [-2]
> 2015-05-18T22:12:53.743+0100 s.k.PartitionManager [WARN] Using new offset:
> 64429934
> 2015-05-18T22:12:53.773+0100 s.k.KafkaUtils [WARN] Got fetch request with
> offset out of range: [63610973]; retrying with default start offset time
> from configuration. configured start offset time: [-2]
>
> So why does the offset get reset?
>
>
> On 18 May 2015, at 20:37, Jeffery Maass <[email protected]> wrote:
>
> The answer will be in how you configured the kafka spout. If after
> reading the below, you still need help, please grab the values for all of
> the settings mentioned in the doc and send them on.
>
> See this document about the Kafka Spout:
>
> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/Storm_UG_v22/Storm_UG_v22.pdf
>
> See this archive message:
>
> http://mail-archives.apache.org/mod_mbox/storm-user/201503.mbox/%3ccahzwdygra33uiv+po01mvthvrjsxvqlvx6babymzye8zu_d...@mail.gmail.com%3E
>
> "
> Not exactly.. forceFromStart=true will tell the spout to start reading from
> whatever is set in startOffsetTime (available options are the earliest
> offset or the latest offset). If forceFromStart=false then startOffsetTime
> is not used at all and the offset is just retrieved from zookeeper, if it's
> available.
>
> The "Start" in "forceFromStart" has nothing to do with consuming from the
> beginning of the topic. I interpret it as referring to whether you are
> going to force starting consumption from a different offset.
> "
>
> Thank you for your time!
>
> +++++++++++++++++++++
> Jeff Maass <[email protected]>
> linkedin.com/in/jeffmaass
> stackoverflow.com/users/373418/maassql
> +++++++++++++++++++++
>
>
> On Mon, May 18, 2015 at 1:53 PM, Benjamin Cuthbert <[email protected]
> > wrote:
>
>> All,
>>
>> We are getting loads of these errors
>>
>> 2015-05-18T19:52:44.038+0100 s.k.KafkaUtils [WARN] Got fetch request with
>> offset out of range: [62379213]; retrying with default start offset time
>> from configuration. configured start offset time: [-2]
>> 2015-05-18T19:52:44.066+0100 s.k.PartitionManager [WARN] Using new
>> offset: 63495047
>>
>> And it is stopping messages being pulled from Kafka into the spout and
>> onto the bolts. Any ideas on how to fix this?
>
>
>
>
