Re: "create region" cmd stuck on wan setup
Please file a bug for this. Even if that ordering doesn't work, nothing should hang because of it. Thanks, Kirk On Wed, Jul 28, 2021 at 5:27 PM Barrett Oglesby wrote: > I reproduced your issue with your scripts. > > They do: > > create gateway-receiver > create disk-store > create gateway-sender > create region > > With that order, I see the hang you mentioned. I'm not 100% sure why that > is happening but you can prevent it by reordering these elements. > > As Anil said, you should start your GatewayReceiver last like: > > create disk-store > create gateway-sender > create region > create gateway-receiver > > With that order, cluster1 restarts fine. > > btw 1 - with the order you had regardless of the hang, you'll see lots of > dropped WAN events since the region doesn't exist yet when the receiver is > started: > > [info 2021/07/28 17:02:39.795 PDT server1_1 Processor2> tid=0x3c] The GatewayReceiver started on port : 5411 > > [warn 2021/07/28 17:02:39.883 PDT server1_1 Thread 1> tid=0x4a] Server connection from > [identity(192.168.1.7(server2_2:25891):41005,connection=1; port=52554]: > Caught exception processing batch create request 0 for 100 events > org.apache.geode.cache.RegionDestroyedException: Region /testregion was > not found during batch create request 0 > > btw 2 - I use CacheCreation.create to see the order that elements should > be started. Thats the object that the old GemFire cache xml uses to start > things in the right order. > > Barry > ________ > From: Anilkumar Gingade > Sent: Wednesday, July 28, 2021 3:45 PM > To: dev@geode.apache.org > Subject: Re: "create region" cmd stuck on wan setup > > The recommendation with WAN setup is: > - Create/start WAN Senders first > - Create Regions > - Create/Start WAN receivers last > > That way when wan receiver is started; the regions are created on all the > sites. Sorry, I have not looked at your scripts... > > -Anil. > > > > On 7/28/21, 3:31 AM, "Alberto Bustamante Reyes" > wrote: > > Hi Geode devs, > > I have been analyzing an issue that occurs in the following scenario: > > 1) I start two Geode clusters (cluster1 & cluster2) with one locator > and two servers each. > Both clusters host a partitioned region called "testregion", which is > replicated using a parallel gateway sender and a gateway receiver. > These are the gfsh files I have been using for creating the clusters: > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Falb3rtobr%2Fe230623255632937fa68265f31e97f3adata=04%7C01%7Cboglesby%40vmware.com%7C6e6bff680f5d46c6bbcc08d952195ff7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637631091210347322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=V%2Fnsqn8wiEnEpjf9GQZ4Ta38rPk5ha79RYqlZWZIXzY%3Dreserved=0 > > 2) I run a client connected to cluster2 performing operations on > testregion. > > 3) cluster1 is stopped and all persistent data is deleted. And then, I > create cluster1 again. > > 4) At this point, the command to create "testregion" get stuck. > > > After checking the thread stack and the code, I found that the problem > is the following. > > This thread is trapped on an infinite loop waiting for a bucket > primary election at "PartitionedRegion.waitForNoStorageOrPrimary": > > > "Function Execution Processor4" tid=0x55 > java.lang.Thread.State: TIMED_WAITING > at java.base@11.0.11/java.lang.Object.wait(Native Method) > - waiting on > org.apache.geode.internal.cache.BucketAdvisor@28be7ae0 > at > app//org.apache.geode.internal.cache.BucketAdvisor.waitForPrimaryMember(BucketAdvisor.java:1433) > at > app//org.apache.geode.internal.cache.BucketAdvisor.waitForNewPrimary(BucketAdvisor.java:825) > at > app//org.apache.geode.internal.cache.BucketAdvisor.getPrimary(BucketAdvisor.java:794) > at > app//org.apache.geode.internal.cache.partitioned.RegionAdvisor.getPrimaryMemberForBucket(RegionAdvisor.java:1032) > at > app//org.apache.geode.internal.cache.PartitionedRegion.getBucketPrimary(PartitionedRegion.java:9081) > at > app//org.apache.geode.internal.cache.PartitionedRegion.waitForNoStorageOrPrimary(PartitionedRegion.java:3249) > at > app//org.apache.geode.internal.cache.PartitionedRegion.getNodeForBucketWrite(PartitionedRegion.java:3234) > at > app//org.apache.geode.internal.cache.PartitionedRegion.shadowPRWaitForBucketRecovery
Re: "create region" cmd stuck on wan setup
I reproduced your issue with your scripts. They do: create gateway-receiver create disk-store create gateway-sender create region With that order, I see the hang you mentioned. I'm not 100% sure why that is happening but you can prevent it by reordering these elements. As Anil said, you should start your GatewayReceiver last like: create disk-store create gateway-sender create region create gateway-receiver With that order, cluster1 restarts fine. btw 1 - with the order you had regardless of the hang, you'll see lots of dropped WAN events since the region doesn't exist yet when the receiver is started: [info 2021/07/28 17:02:39.795 PDT server1_1 tid=0x3c] The GatewayReceiver started on port : 5411 [warn 2021/07/28 17:02:39.883 PDT server1_1 tid=0x4a] Server connection from [identity(192.168.1.7(server2_2:25891):41005,connection=1; port=52554]: Caught exception processing batch create request 0 for 100 events org.apache.geode.cache.RegionDestroyedException: Region /testregion was not found during batch create request 0 btw 2 - I use CacheCreation.create to see the order that elements should be started. Thats the object that the old GemFire cache xml uses to start things in the right order. Barry From: Anilkumar Gingade Sent: Wednesday, July 28, 2021 3:45 PM To: dev@geode.apache.org Subject: Re: "create region" cmd stuck on wan setup The recommendation with WAN setup is: - Create/start WAN Senders first - Create Regions - Create/Start WAN receivers last That way when wan receiver is started; the regions are created on all the sites. Sorry, I have not looked at your scripts... -Anil. On 7/28/21, 3:31 AM, "Alberto Bustamante Reyes" wrote: Hi Geode devs, I have been analyzing an issue that occurs in the following scenario: 1) I start two Geode clusters (cluster1 & cluster2) with one locator and two servers each. Both clusters host a partitioned region called "testregion", which is replicated using a parallel gateway sender and a gateway receiver. These are the gfsh files I have been using for creating the clusters: https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Falb3rtobr%2Fe230623255632937fa68265f31e97f3adata=04%7C01%7Cboglesby%40vmware.com%7C6e6bff680f5d46c6bbcc08d952195ff7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637631091210347322%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=V%2Fnsqn8wiEnEpjf9GQZ4Ta38rPk5ha79RYqlZWZIXzY%3Dreserved=0 2) I run a client connected to cluster2 performing operations on testregion. 3) cluster1 is stopped and all persistent data is deleted. And then, I create cluster1 again. 4) At this point, the command to create "testregion" get stuck. After checking the thread stack and the code, I found that the problem is the following. This thread is trapped on an infinite loop waiting for a bucket primary election at "PartitionedRegion.waitForNoStorageOrPrimary": "Function Execution Processor4" tid=0x55 java.lang.Thread.State: TIMED_WAITING at java.base@11.0.11/java.lang.Object.wait(Native Method) - waiting on org.apache.geode.internal.cache.BucketAdvisor@28be7ae0 at app//org.apache.geode.internal.cache.BucketAdvisor.waitForPrimaryMember(BucketAdvisor.java:1433) at app//org.apache.geode.internal.cache.BucketAdvisor.waitForNewPrimary(BucketAdvisor.java:825) at app//org.apache.geode.internal.cache.BucketAdvisor.getPrimary(BucketAdvisor.java:794) at app//org.apache.geode.internal.cache.partitioned.RegionAdvisor.getPrimaryMemberForBucket(RegionAdvisor.java:1032) at app//org.apache.geode.internal.cache.PartitionedRegion.getBucketPrimary(PartitionedRegion.java:9081) at app//org.apache.geode.internal.cache.PartitionedRegion.waitForNoStorageOrPrimary(PartitionedRegion.java:3249) at app//org.apache.geode.internal.cache.PartitionedRegion.getNodeForBucketWrite(PartitionedRegion.java:3234) at app//org.apache.geode.internal.cache.PartitionedRegion.shadowPRWaitForBucketRecovery(PartitionedRegion.java:10110) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:564) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:443) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderEventProcessor.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderEventProcessor.java:195) at app//org.apache.geode.internal.cache.wan.parallel.ConcurrentParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(Co
Re: "create region" cmd stuck on wan setup
The recommendation with WAN setup is: - Create/start WAN Senders first - Create Regions - Create/Start WAN receivers last That way when wan receiver is started; the regions are created on all the sites. Sorry, I have not looked at your scripts... -Anil. On 7/28/21, 3:31 AM, "Alberto Bustamante Reyes" wrote: Hi Geode devs, I have been analyzing an issue that occurs in the following scenario: 1) I start two Geode clusters (cluster1 & cluster2) with one locator and two servers each. Both clusters host a partitioned region called "testregion", which is replicated using a parallel gateway sender and a gateway receiver. These are the gfsh files I have been using for creating the clusters: https://gist.github.com/alb3rtobr/e230623255632937fa68265f31e97f3a 2) I run a client connected to cluster2 performing operations on testregion. 3) cluster1 is stopped and all persistent data is deleted. And then, I create cluster1 again. 4) At this point, the command to create "testregion" get stuck. After checking the thread stack and the code, I found that the problem is the following. This thread is trapped on an infinite loop waiting for a bucket primary election at "PartitionedRegion.waitForNoStorageOrPrimary": "Function Execution Processor4" tid=0x55 java.lang.Thread.State: TIMED_WAITING at java.base@11.0.11/java.lang.Object.wait(Native Method) - waiting on org.apache.geode.internal.cache.BucketAdvisor@28be7ae0 at app//org.apache.geode.internal.cache.BucketAdvisor.waitForPrimaryMember(BucketAdvisor.java:1433) at app//org.apache.geode.internal.cache.BucketAdvisor.waitForNewPrimary(BucketAdvisor.java:825) at app//org.apache.geode.internal.cache.BucketAdvisor.getPrimary(BucketAdvisor.java:794) at app//org.apache.geode.internal.cache.partitioned.RegionAdvisor.getPrimaryMemberForBucket(RegionAdvisor.java:1032) at app//org.apache.geode.internal.cache.PartitionedRegion.getBucketPrimary(PartitionedRegion.java:9081) at app//org.apache.geode.internal.cache.PartitionedRegion.waitForNoStorageOrPrimary(PartitionedRegion.java:3249) at app//org.apache.geode.internal.cache.PartitionedRegion.getNodeForBucketWrite(PartitionedRegion.java:3234) at app//org.apache.geode.internal.cache.PartitionedRegion.shadowPRWaitForBucketRecovery(PartitionedRegion.java:10110) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:564) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:443) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderEventProcessor.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderEventProcessor.java:195) at app//org.apache.geode.internal.cache.wan.parallel.ConcurrentParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ConcurrentParallelGatewaySenderQueue.java:183) at app//org.apache.geode.internal.cache.PartitionedRegion.postCreateRegion(PartitionedRegion.java:1177) at app//org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3050) at app//org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2910) at app//org.apache.geode.internal.cache.GemFireCacheImpl.createRegion(GemFireCacheImpl.java:2894) at app//org.apache.geode.cache.RegionFactory.create(RegionFactory.java:773) After creating testregion, the sender queue partitioned region is created. While that region buckets are recovered the command is trapped on an infinite loop waiting for a primary bucket election at PartitionedRegion.waitForNoStorageOrPrimary. This seems to be a known issue because in PartitionedRegion.getNodeForBucketWrite, there is the following command before calling waitForNoStorageOrPrimary (and the command has been there since Geode's first commit!) : // Possible race with loss of redundancy at this point. // This loop can possibly create a soft hang if no primary is ever selected. // This is preferable to returning null since it will prevent obtaining the // bucket lock for bucket creation. return waitForNoStorageOrPrimary(bucketId, "write"); Any idea about why the primary bucket is not elected? It seems the failure is related with the fact that "testregion" is receiving updates from the receiver before the "create region" command has finished. If the test is repeated without traffic on cluster2 or if I create the cluster1's receiver after creating "testregion", this problem is not happening. Is there any recommendation on the startup order of regions, senders and
"create region" cmd stuck on wan setup
Hi Geode devs, I have been analyzing an issue that occurs in the following scenario: 1) I start two Geode clusters (cluster1 & cluster2) with one locator and two servers each. Both clusters host a partitioned region called "testregion", which is replicated using a parallel gateway sender and a gateway receiver. These are the gfsh files I have been using for creating the clusters: https://gist.github.com/alb3rtobr/e230623255632937fa68265f31e97f3a 2) I run a client connected to cluster2 performing operations on testregion. 3) cluster1 is stopped and all persistent data is deleted. And then, I create cluster1 again. 4) At this point, the command to create "testregion" get stuck. After checking the thread stack and the code, I found that the problem is the following. This thread is trapped on an infinite loop waiting for a bucket primary election at "PartitionedRegion.waitForNoStorageOrPrimary": "Function Execution Processor4" tid=0x55 java.lang.Thread.State: TIMED_WAITING at java.base@11.0.11/java.lang.Object.wait(Native Method) - waiting on org.apache.geode.internal.cache.BucketAdvisor@28be7ae0 at app//org.apache.geode.internal.cache.BucketAdvisor.waitForPrimaryMember(BucketAdvisor.java:1433) at app//org.apache.geode.internal.cache.BucketAdvisor.waitForNewPrimary(BucketAdvisor.java:825) at app//org.apache.geode.internal.cache.BucketAdvisor.getPrimary(BucketAdvisor.java:794) at app//org.apache.geode.internal.cache.partitioned.RegionAdvisor.getPrimaryMemberForBucket(RegionAdvisor.java:1032) at app//org.apache.geode.internal.cache.PartitionedRegion.getBucketPrimary(PartitionedRegion.java:9081) at app//org.apache.geode.internal.cache.PartitionedRegion.waitForNoStorageOrPrimary(PartitionedRegion.java:3249) at app//org.apache.geode.internal.cache.PartitionedRegion.getNodeForBucketWrite(PartitionedRegion.java:3234) at app//org.apache.geode.internal.cache.PartitionedRegion.shadowPRWaitForBucketRecovery(PartitionedRegion.java:10110) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:564) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderQueue.java:443) at app//org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderEventProcessor.addShadowPartitionedRegionForUserPR(ParallelGatewaySenderEventProcessor.java:195) at app//org.apache.geode.internal.cache.wan.parallel.ConcurrentParallelGatewaySenderQueue.addShadowPartitionedRegionForUserPR(ConcurrentParallelGatewaySenderQueue.java:183) at app//org.apache.geode.internal.cache.PartitionedRegion.postCreateRegion(PartitionedRegion.java:1177) at app//org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3050) at app//org.apache.geode.internal.cache.GemFireCacheImpl.basicCreateRegion(GemFireCacheImpl.java:2910) at app//org.apache.geode.internal.cache.GemFireCacheImpl.createRegion(GemFireCacheImpl.java:2894) at app//org.apache.geode.cache.RegionFactory.create(RegionFactory.java:773) After creating testregion, the sender queue partitioned region is created. While that region buckets are recovered the command is trapped on an infinite loop waiting for a primary bucket election at PartitionedRegion.waitForNoStorageOrPrimary. This seems to be a known issue because in PartitionedRegion.getNodeForBucketWrite, there is the following command before calling waitForNoStorageOrPrimary (and the command has been there since Geode's first commit!) : // Possible race with loss of redundancy at this point. // This loop can possibly create a soft hang if no primary is ever selected. // This is preferable to returning null since it will prevent obtaining the // bucket lock for bucket creation. return waitForNoStorageOrPrimary(bucketId, "write"); Any idea about why the primary bucket is not elected? It seems the failure is related with the fact that "testregion" is receiving updates from the receiver before the "create region" command has finished. If the test is repeated without traffic on cluster2 or if I create the cluster1's receiver after creating "testregion", this problem is not happening. Is there any recommendation on the startup order of regions, senders and receivers for an scenario like the one described? Thanks in advance, Alberto B.
