I am running 4 topologies on a storm cluster each with one bolt and one
kafka spout. Of these, 3 are showing a high number of failures (in UI) in
the spout itself. I looked at the logs and found that the offset isn't just
moving (in fact, at times it is reduced by one). Sample log for one of the
partitions below:
anuj.agrawal@server-ingestion1:/var/log/storm$ grep "2014-07-26 17:5"
worker-6704.log | grep "partition=24"
2014-07-26 17:50:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:50:27 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:50:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:50:57 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:51:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:51:27 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:51:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:51:57 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:52:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:52:27 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:52:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:52:57 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:53:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:53:27 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:53:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:53:57 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:54:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:54:27 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:54:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:54:57 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:55:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:55:27 s.k.PartitionManager [INFO] Committed offset 96435 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:55:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:55:57 s.k.ZkState [INFO] Writing
/server/cp-kafka/AndroidAppEventIngestion/partition_24 the data
{topology={id=8fba6b24-e1cd-4476-91a6-bb493a0e7c87,
name=AndriodAppEventIngestion}, offset=96434, partition=24,
broker={host=server-kafka5.local, port=9092}, topic=AndroidApp}
2014-07-26 17:55:57 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:56:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:56:27 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:56:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:56:57 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:57:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:57:27 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:57:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:57:57 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:58:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:58:27 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:58:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:58:57 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:59:27 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:59:27 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
2014-07-26 17:59:57 s.k.PartitionManager [INFO] Committing offset for
Partition{host=server-kafka5.local:9092, partition=24}
2014-07-26 17:59:57 s.k.PartitionManager [INFO] Committed offset 96434 for
Partition{host=server-kafka5.local:9092, partition=24} for topology:
8fba6b24-e1cd-4476-91a6-bb493a0e7c87
anuj.agrawal@server-ingestion1:/var/log/storm$
anuj.agrawal@server-ingestion1:/var/log/storm$
anuj.agrawal@server-ingestion1:/var/log/storm$
anuj.agrawal@server-ingestion1:/var/log/storm$
anuj.agrawal@server-ingestion1:/var/log/storm$ grep "2014-07-26 17:5"
worker-6704.log | grep ":24"
*2014-07-26 17:55:27 s.k.PartitionManager [INFO] Fetched 1526 messages from
Kafka: server-kafka5.local:24*
*2014-07-26 17:55:27 s.k.PartitionManager [INFO] Added 1526 messages from
Kafka: server-kafka5.local:24 to internal buffers*
anuj.agrawal@server-ingestion1:/var/log/storm$ grep "2014-07-26 17:5"
worker-6704.log | grep "partition_24"
*2014-07-26 17:55:57 s.k.ZkState [INFO] Writing
/server/cp-kafka/AndroidAppEventIngestion/partition_24 the data
{topology={id=8fba6b24-e1cd-4476-91a6-bb493a0e7c87,
name=AndriodAppEventIngestion}, offset=96434, partition=24,
broker={host=server-kafka5.local, port=9092}, topic=AndroidApp}*
anuj.agrawal@server-ingestion1:/var/log/storm$
anuj.agrawal@server-ingestion1:/var/log/storm$
I am unable to figure out what is going on here. Can anyone please help?
