Hi, It seems like this is a bug in the code. When SlotCoordinator is changed submitCurrentSafeZone gets connectionException. When this exception is thrown, SlotMessageCounter should update its local thrift coordinator details from hazelcast map. That part is not implemented in the code. Will fix the issue in the Alpha release.
Thank you, Sajini On Wed, Mar 25, 2015 at 9:33 AM, Isuru Haththotuwa <[email protected]> wrote: > Hi devs, > > I experienced this error as well in a two node cluster in ec2 (two VMs). > Here is what I observed: > > I have configured each nodes IP as the thrift host name in the broker.xml > file. When the first node is started, its selected as the Slot Coordinator. > And if the current Slot Coordinator goes down, the second node is elected > as the Slot Coordinator. > > However, when the first node goes down, the second node consistently > prints this error. I assume its trying to connect and publish some > information to the previous Slot Coordinator that results in this broken > pipe, since the first node is no more. Should the second node continue to > try to publish to the first node, even when it becomes the Slot Coordinator > itself (assuming the non coordinators publish this information to the > coordinator) ? > > On Thu, Mar 19, 2015 at 8:55 AM, Hasitha Amal De Silva <[email protected]> > wrote: > >> Its possible since the first node will try to communicate the safe zone >> to the second node once its up, through the offset thrift port. >> >> On Thu, Mar 19, 2015 at 6:11 PM, Hemika Kodikara <[email protected]> wrote: >> >>> The first node was having an offset which was set by mistake. But would >>> that have caused in error ? When I tried it for the first time, this >>> started coming up when the second node got connected. >>> >>> Regards, >>> Hemika >>> >>> Hemika Kodikara >>> Software Engineer >>> WSO2 Inc. >>> lean . enterprise . middleware >>> http://wso2.com >>> >>> Mobile : +94777688882 >>> >>> On Thu, Mar 19, 2015 at 5:44 PM, Asitha Nanayakkara <[email protected]> >>> wrote: >>> >>>> When the safe zone communication event is triggered, this exception is >>>> thrown. This happens all the time not an intermittent issue. It seems like >>>> ThriftClient couldn't read/write using the port. >>>> >>>> Can this be a port issue, where we haven't opened the ports for Thrift >>>> client in this EC2 instance? >>>> >>>> >>>> On Thu, Mar 19, 2015 at 5:26 PM, Hasitha Amal De Silva < >>>> [email protected]> wrote: >>>> >>>>> Hi Hemika, >>>>> >>>>> Does this happen at a specific moment when new nodes are joining etc. >>>>> ? or purely random ? This could as well be a thrift connection breakup >>>>> issue too. >>>>> >>>>> Thanks >>>>> >>>>> On Thu, Mar 19, 2015 at 5:16 PM, Hemika Kodikara <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> I have setup a 3 MB node cluster with 3 datastax cassandra message >>>>>> stores each on EC2 instances. >>>>>> >>>>>> When starting up the 2nd node of MB, I am getting the following error >>>>>> (The error is coming from time to time, I believe its coming from a >>>>>> timer.) >>>>>> : >>>>>> >>>>>> TID: [] [] [2015-03-19 11:26:09,746] ERROR >>>>>> {org.wso2.andes.thrift.MBThriftClient} - Error while sending slot >>>>>> deletion >>>>>> safe zone update {org.wso2.andes.thrift.MBThriftClient} >>>>>> org.apache.thrift.transport.TTransportException >>>>>> at >>>>>> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) >>>>>> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) >>>>>> at >>>>>> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) >>>>>> at >>>>>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Client.recv_updateCurrentMessageIdForSafeZone(SlotManagementService.java:183) >>>>>> at >>>>>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Client.updateCurrentMessageIdForSafeZone(SlotManagementService.java:169) >>>>>> at >>>>>> org.wso2.andes.thrift.MBThriftClient.updateSlotDeletionSafeZone(MBThriftClient.java:323) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotCoordinatorCluster.updateSlotDeletionSafeZone(SlotCoordinatorCluster.java:61) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter.submitCurrentSafeZone(SlotMessageCounter.java:134) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter.access$500(SlotMessageCounter.java:36) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter$1.run(SlotMessageCounter.java:99) >>>>>> at java.util.TimerThread.mainLoop(Timer.java:555) >>>>>> at java.util.TimerThread.run(Timer.java:505) >>>>>> TID: [] [] [2015-03-19 11:26:22,357] INFO >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> - >>>>>> Member joined [8c25be13-8649-4826-a54a-cd881aaf07d8]: / >>>>>> 10.23.156.141:4000 >>>>>> {org.wso2.carbon.core.clustering.hazelcast.wka.WKABasedMembershipScheme} >>>>>> TID: [] [] [2015-03-19 11:26:22,357] INFO >>>>>> {org.wso2.andes.server.cluster.coordination.hazelcast.AndesMembershipListener} >>>>>> - Handling cluster gossip: New member joined to the cluster. Member >>>>>> Socket >>>>>> Address:/10.23.156.141:4000 >>>>>> UUID:8c25be13-8649-4826-a54a-cd881aaf07d8 >>>>>> {org.wso2.andes.server.cluster.coordination.hazelcast.AndesMembershipListener} >>>>>> TID: [] [] [2015-03-19 11:26:22,357] INFO >>>>>> {org.wso2.andes.server.cluster.ClusterManager} - This node is elected as >>>>>> the Slot Coordinator. Registering 10.23.10.50:7612 >>>>>> {org.wso2.andes.server.cluster.ClusterManager} >>>>>> TID: [] [] [2015-03-19 11:26:39,745] ERROR >>>>>> {org.wso2.andes.thrift.MBThriftClient} - Error while sending slot >>>>>> deletion >>>>>> safe zone update {org.wso2.andes.thrift.MBThriftClient} >>>>>> org.apache.thrift.transport.TTransportException: >>>>>> java.net.SocketException: Broken pipe >>>>>> at >>>>>> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) >>>>>> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) >>>>>> at >>>>>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Client.send_updateCurrentMessageIdForSafeZone(SlotManagementService.java:177) >>>>>> at >>>>>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Client.updateCurrentMessageIdForSafeZone(SlotManagementService.java:168) >>>>>> at >>>>>> org.wso2.andes.thrift.MBThriftClient.updateSlotDeletionSafeZone(MBThriftClient.java:323) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotCoordinatorCluster.updateSlotDeletionSafeZone(SlotCoordinatorCluster.java:61) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter.submitCurrentSafeZone(SlotMessageCounter.java:134) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter.access$500(SlotMessageCounter.java:36) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter$1.run(SlotMessageCounter.java:99) >>>>>> at java.util.TimerThread.mainLoop(Timer.java:555) >>>>>> at java.util.TimerThread.run(Timer.java:505) >>>>>> Caused by: java.net.SocketException: Broken pipe >>>>>> at java.net.SocketOutputStream.socketWrite0(Native Method) >>>>>> at >>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) >>>>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159) >>>>>> at >>>>>> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) >>>>>> ... 12 more >>>>>> TID: [] [] [2015-03-19 11:27:09,745] ERROR >>>>>> {org.wso2.andes.thrift.MBThriftClient} - Error while sending slot >>>>>> deletion >>>>>> safe zone update {org.wso2.andes.thrift.MBThriftClient} >>>>>> org.apache.thrift.transport.TTransportException: >>>>>> java.net.SocketException: Broken pipe >>>>>> at >>>>>> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) >>>>>> at >>>>>> org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) >>>>>> at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) >>>>>> at >>>>>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Client.send_updateCurrentMessageIdForSafeZone(SlotManagementService.java:177) >>>>>> at >>>>>> org.wso2.andes.thrift.slot.gen.SlotManagementService$Client.updateCurrentMessageIdForSafeZone(SlotManagementService.java:168) >>>>>> at >>>>>> org.wso2.andes.thrift.MBThriftClient.updateSlotDeletionSafeZone(MBThriftClient.java:323) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotCoordinatorCluster.updateSlotDeletionSafeZone(SlotCoordinatorCluster.java:61) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter.submitCurrentSafeZone(SlotMessageCounter.java:134) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter.access$500(SlotMessageCounter.java:36) >>>>>> at >>>>>> org.wso2.andes.kernel.slot.SlotMessageCounter$1.run(SlotMessageCounter.java:99) >>>>>> at java.util.TimerThread.mainLoop(Timer.java:555) >>>>>> at java.util.TimerThread.run(Timer.java:505) >>>>>> Caused by: java.net.SocketException: Broken pipe >>>>>> at java.net.SocketOutputStream.socketWrite0(Native Method) >>>>>> at >>>>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) >>>>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:159) >>>>>> at >>>>>> org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145) >>>>>> ... 12 more >>>>>> >>>>>> Any idea what can be done ? The pack was taken from jenkins. >>>>>> >>>>>> Regards, >>>>>> Hemika >>>>>> >>>>>> >>>>>> Hemika Kodikara >>>>>> Software Engineer >>>>>> WSO2 Inc. >>>>>> lean . enterprise . middleware >>>>>> http://wso2.com >>>>>> >>>>>> Mobile : +94777688882 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> >>>>> Hasitha Amal De Silva >>>>> Software Engineer >>>>> Mobile : 0772037426 >>>>> Blog : http://devnutshell.tumblr.com/ >>>>> WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> [email protected] >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Asitha Nanayakkara* >>>> Software Engineer >>>> WSO2, Inc. http://wso2.com/ >>>> Mob: + 94 77 85 30 682 >>>> >>>> >>> >> >> >> -- >> Cheers, >> >> Hasitha Amal De Silva >> Software Engineer >> Mobile : 0772037426 >> Blog : http://devnutshell.tumblr.com/ >> WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. ) >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Thanks and Regards, > > Isuru H. > +94 716 358 048* <http://wso2.com/>* > > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: [email protected] Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva Phone: +94 712797729
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
