Thanks Onur. the problem is resolved. What I noticed is that the supervisor ids are not synced between the zookeeper cluster nodes. Some of the supervisors were registered and some were not. Basically, i stopped the storm cluster, delete the storm-data, cleaned up zookeeper zpath and restarted and it works as of now.
Thanks Kashyap On Fri, Sep 25, 2015 at 9:56 AM, Onur Yalazı <[email protected]> wrote: > I mean in size. I had to introduce a compressing serializer to lower our > bw usage a few days ago. > > SDN is software defined network, like the networking used via cloud > providers or openstack like tools. > > I would start by looking into dns issues first. Try resolving each nodes > hostname on each and every node of the system. Use hostnames as it's > reported on the stormmui. > > Does the cluster start and then stop or never fire any tuple? Also the > issue may be that the spout having problems generating tuples. > > If the cluster starts and stops a while later, start using some sort of > network monitoring and watch for bw usage between supervisor nodes. > On Sep 25, 2015 5:47 PM, "Kashyap Mhaisekar" <[email protected]> wrote: > >> Onur, >> By huge tuples - do you mean in number or size? And what is SDN? >> >> There are no firewalls for this and all zookeepers are correctly listed >> on storm.yaml. Any advices on how I should go about looking for the root >> cause? >> >> Thanks >> kashyap >> >> On Fri, Sep 25, 2015 at 9:00 AM, Onur Yalazı <[email protected]> >> wrote: >> >>> That situation happens even if a single node's hostname can not be >>> resolved. >>> >>> Also if you are running with huge tuples like we do (big mistake I >>> know), and if the network has low capacity compared to what you need, like >>> a soft limit on sdn kicking in, cluster stops to communicate and run. >>> On Sep 25, 2015 4:45 PM, "Srividhya Shanmugam" < >>> [email protected]> wrote: >>> >>>> Are the zookeeper servers listed in storm.yaml file running properly? >>>> >>>> -----Original Message----- >>>> From: Derek Dagit [mailto:[email protected]] >>>> Sent: Friday, September 25, 2015 9:39 AM >>>> To: [email protected] >>>> Subject: Re: Nodes dont talk to each other >>>> >>>> I saw it happen before when the firewalls were not configured correctly. >>>> >>>> -- >>>> Derek >>>> >>>> >>>> ________________________________ >>>> From: Kashyap Mhaisekar <[email protected]> >>>> To: [email protected] >>>> Sent: Thursday, September 24, 2015 6:58 PM >>>> Subject: Nodes dont talk to each other >>>> >>>> >>>> >>>> HI, >>>> I am now facing a unique issue where the nodes of a cluster do not talk >>>> to each other. The spout does not accept messages and even if there are >>>> some messages, they are not sent to other workers on the same node or >>>> different nodes. >>>> >>>> Logs also dont indicate much. Has anyone faced this issue before? >>>> >>>> I am on Storm 0.9.5 and have 8 machines with 8 cores and 32 GB each. >>>> >>>> Thanks for the time. >>>> >>>> Kashyap >>>> >>>> This email and any files transmitted with it are confidential, >>>> proprietary and intended solely for the individual or entity to whom they >>>> are addressed. If you have received this email in error please delete it >>>> immediately. >>>> >>> >>
