Hi Gergely, Thanks for using storm-redis, and report issue.
The thing Storm was trying to serialize is BinaryJedis$SetFromList, not BinaryJedis. https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java#L74 Jedis.smembers() returns SetFromList, which wraps List to provide Set interface. (It is just one of the optimizations in Jedis.) Unfortunately SetFromList doesn't implement Serializable for now, so you can encounter NotSerializableException when trying to emit the result of smembers directly. Two workarounds are here, 1. Implement custom Kyro serializer for SetFromList and register to topology. http://storm.apache.org/documentation/Serialization.html 2. Copy the result into HashSet and emit instead. I'll try to craft the patch to Jedis which makes SetFromList to implement Serializable. Before applying and releasing, I'll also try to make storm-redis to provide easy workaround. (Considering 1. for now.) Thanks! Jungtaek Lim (HeartSaVioR) 2015-12-14 21:54 GMT+09:00 Gergely Nagy <[email protected]>: > Hello fellow storm troopers. > > I am trying to build a storm topology in Storm 0.10.0. I am also trying to > use storm-redis to save some interim results during operation. > > Now my problem is that when I run my topology locally everything works > fine, but when I run it remotely, storm sends me this nice stacktrace: > https://gist.github.com/Fogetti/23304178a534fa1f8f63 > > I don't get it. What's going on? Why is storm serializing BinaryJedis? > Shouldn't BinaryJedis be transient? And anyway, what is this cryptic thing > called 'consumeBatchToCursor()' doing? > > The "offending" class can be seen below. It's just a simple bolt extending > AbstractRedisBolt. > > https://github.com/Fogetti/fogetti-phish-storm/blob/master/src/main/java/fogetti/phish/storm/relatedness/GoogleSemBolt.java > > Does anyone know what might be the root cause for this? > > Regards, > Gergely > -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
