I think fixed it. Realised i missed "jedis.close();" from my storm bolt. I think that if too many redis connections are created without being closed, it might be sending info to a TCP port that happens to be scanned by my TCPflow. If the problem reappears, ill update the thread.
thanks again On Wed, Mar 11, 2015 at 3:45 PM, 임정택 <[email protected]> wrote: > As I understood correctly, you're turning on tcpflow and send a command to > Redis, right? > Then maybe Redis protocol could be captured by tcpflow and published to > Kafka. > Please check your tcpflow setting. > > 2015-03-11 21:10 GMT+09:00 Mithun Kalan <[email protected]>: > >> Hi >> I've got something weird going on. >> >> [my setup is: I get TCP data using tcpflow, publish to kafka using >> kafkacat, storm picks it up with kafkaspout to process and publishes to >> redis using jedis. They all run off a single machine with a single >> zookeeper] >> >> Storm breaks a few minutes into the run because the messages coming from >> kafka do not conform to the format that is expected. >> Digging deeper, i found that the culprit messages in kafka look like the >> ones in redis. >> >> How is this possible? i never told storm to publish into kafka. Kafka >> should never have those messages. I cant find a pattern either. its not a >> specific message or a timing issue. >> >> *The lines in kafka should have been:* >> 2015-02-02 14:03 catergoy: A price: 123 >> 2015-02-02 14:04 catergoy: A price: 124 >> 2015-02-02 14:05 catergoy: A price: 122 >> >> *the lines in redis are * >> 14:04 change +1 >> 14:05 change -2 >> >> *but i get this * >> 2015-02-02 14:03 catergoy: A price: 123 >> 2015-02-02 14:04 catergoy: A price: 124 >> *3 >> $7 >> message >> $8 >> $76 >> 14:04 change +1 >> 2015-02-02 14:05 catergoy: A price: 122 >> >> >> i suspect its a because kafka and storm share the same zookeeper. I will >> try to separate them to see if it works. >> but has anyone had a similar issue? how did you solve it? >> >> Thank you >> >> Mithun Kalan >> >> > > > -- > Name : 임 정택 > Blog : http://www.heartsavior.net / http://dev.heartsavior.net > Twitter : http://twitter.com/heartsavior > LinkedIn : http://www.linkedin.com/in/heartsavior > -- Mithun Kalan [email protected]
