Hello,
I'm having problems with my transactional Trident topology. It has been
running fine for about 20 days, and suddenly is stuck processing a single
batch, with no tuples being emitted nor tuples being persisted by the
TridentState (IBackingMap).
It's a simple topology which consumes messages off a Kafka queue. The spout
is an instance of storm-kafka-0.8-plus TransactionalTridentKafkaSpout and I
use the trident-mssql transactional TridentState implementation to
persistentAggregate() data into a SQL database.
In Zookeeper I can see Storm is re-trying a batch, i.e.
"/transactional/<myTopologyName>/coordinator/currattempts" is
"{"29698959":6487}"
... and the attempt count keeps increasing. It seems the batch with txid
29698959 is stuck, as the attempt count in Zookeeper keeps increasing --
seems like the batch isn't being acked by Trident and I have no idea why,
especially since the topology has been running successfully the last 20
days.
I did rebalance the topology on one occasion, after which it continued
running normally. Other than that, no other modifications were done. Storm
is at version 0.9.0.1.
Any hints on how to debug the stuck topology? Any other useful info I might
provide?
Thanks,
--
Danijel Schiavuzzi
E: [email protected]
W: www.schiavuzzi.com
T: +385989035562
Skype: danijel.schiavuzzi
