Looks like this was some kind of network issue... I ran the topology on another box and don't have the problem any more.
I have a related question about the TransactionalTridentKafkaSpout and OpaqueTridentKafkaSpout though: does anyone know where these spouts store their partition offsets to use for recovery in the case of failure? I know the normal (non-trident) Kafka spout stores its offsets in Zookeeper, and it has a spout config setting ZkHosts which determines where they are stored. But the trident Kafka spouts don't have this setting. Looking at the source code I can't find any place where it is persisting offsets. Plus it's using the SimpleConsumer so it's not relying on Kafka persisting its offsets. On Wed, Apr 29, 2015 at 4:29 PM, Josh <[email protected]> wrote: > Hi all, > > I'm using the OpaqueTridentKafkaSpout to read batches of tuples from > Kafka, I'm doing some filtering and aggregation and then calling > persistAggregate to maintain a map state: > > stream > .each(new Fields("msg"), new MyFilter()) > .each(new Fields("msg"), new ExtractFieldsFunction(), new > Fields("id")) > .groupBy(new Fields("id")) > .persistentAggregate(MyMapState.getNonTransactional, new > CountAggregator(), new Fields("count")) > .parallelismHint(1) > > It works fine, for the first batch, but then I am having a very strange > problem where after the first batch my map state is no longer called. (i.e. > there a call to multiGet and multiPut for the first batch only). > > The spout is still providing tuples, as when I debug I can see that the > filter and function both continue to process input tuples (indefinitely). > But the map state never gets called again! > > Why would this happen? > > I found a couple of very similar threads to this, but they have gone > unanswered: > > https://groups.google.com/forum/#!searchin/storm-user/trident$20state/storm-user/wJbKbHDxRqI/KCiBE5lTdGEJ > > https://groups.google.com/forum/#!searchin/storm-user/trident$20state/storm-user/OlYUoCjx-uU/99XdNkdaSnIJ > > > Thanks for any help on this, > > Josh >
