locator is not able to join the distributed system.
Hi All. we are using Geode 1.8.0 in peer to peer configuration. I am facing a problem where a locator is not able to join the distributed system. >From the logs I can see that the locator is attempting to join through the coordinator but it is not able to join. It keeps retrying with the following message "waiting for a join-response" Also I have checked the coordinator logs and I see no join request in its logs. I have checked the ports 41000 and 41001, both of them are open. I suspect network latency might be the reason but how can I make it resistant to network latency. following is the meesage in logs where it tries to join the distributed system [info 2020/03/19 16:06:09.312 UTC iHubCacheLocatorProcess_10.248.72.107 tid=0x1] Attempting to join the distributed system through coordinator 10.248.72.106(iHubCacheLocatorProcess_10.248.72.106:14712:locator):41000 using address 10.248.72.107(iHubCacheLocatorProcess_10.248.72.107:36893:locator):41000 After sometime i see the following in logs where its still waiting for response [info 2020/03/19 16:06:19.408 UTC iHubCacheLocatorProcess_10.248.72.107 tid=0x1] Probable coordinator is still 10.248.72.106(iHubCacheLocatorProcess_10.248.72.106:14712:locator):41000 - waiting for a join-response Thanks, Rajesh
Re: region name is incorrect while trying to get using region.get
Hi All, Thanks for your inputs, region not getting created is the root cause of the issue. The error message was misleading and we focused our efforts on client side. Thanks, Rajesh On Wed, Dec 4, 2019 at 4:14 PM Ju@N wrote: > Hello Rajesh, > > *>> Do you think this might have caused the problem? if yes was it due to > the client trying to connect to a node where the region is not present?* > Yes, that's certainly the cause of the problem: you will hit the exception > every time the client tries to execute the operation on one of the servers > that doesn't have the region created. > > The region is *REPLICATE* so the client pool "assumes" that it will be > created in every single server, if you want a *REPLICATE* region to exist > only in some servers you need to user server groups. The exception is > certainly misleading, though, it shouldn't contain the name twice and it > would probably be better to throw something like *RegionNotFoundException* > instead > of *RegionDestroyedException*, but that's another discussion. > > Cheers. > > On Wed, Dec 4, 2019 at 10:32 AM Rajesh Kumar > wrote: > >> Hi Ju, >> >> I have checked the logs and I can see that out of the 5 nodes in the >> cluster, the region is created only in 2 nodes and in rest of the nodes >> region creation failed due to various reasons. Do you think this might >> have caused the problem? if yes was it due to the client trying to connect >> to a node where the region is not present? >> >> Thanks, >> Rajesh >> >> On Wed, Dec 4, 2019 at 3:41 PM Ju@N wrote: >> >>> Hello Rajesh, >>> >>> I've seen this issue 4 or 5 times in the past and, even though the >>> Exception is misleading, I certainly remember that the root cause of the >>> issue was always the same: the client application was trying to execute the >>> operation on a server that didn't have the region created. As Dan pointed >>> out, I'd suggest to double check whether you have the region >>> *AuthIdRegion* configured in all your servers. >>> Best regards. >>> >>> >>> On Wed, Dec 4, 2019 at 9:40 AM Rajesh Kumar >>> wrote: >>> >>>> Hi Udo, >>>> >>>> Thanks for the reply. One of our clients is facing this issue in >>>> older version of our product where we use apache geode 1.1.0. We have >>>> upgraded to 1.8.0 in our latest release. >>>> The region /AuthIdRegion is created by using java code, verified it >>>> by querying it using gfsh. >>>> we will take a look at the spring boot for out further releases but >>>> can you suggest me a fix/solution for the problem. >>>> >>>> Thanks, >>>> Rajesh >>>> >>>> >>>> On Wed, Dec 4, 2019 at 12:00 AM Udo Kohlmeyer wrote: >>>> >>>>> Reporting this, since it seems it did not make it through last night. >>>>> On 12/3/19 12:04 AM, Udo Kohlmeyer wrote: >>>>> >>>>> Hi there Rajesh, >>>>> >>>>> Thank you for raising this. I have seen this before and I think it has >>>>> been tracked to it actually being a toString() issue when logging the >>>>> exception and not that it is not finding the region. >>>>> >>>>> Just for interest sake, why are you using Geode 1.1.0? Did you know >>>>> Geode 1.10 is already out and we are in the process of cutting 1.11. >>>>> >>>>> In addition, can you confirm that you have created a region of the >>>>> same name on the server side? >>>>> >>>>> If not, please create it either in Java code (pretty much the same as >>>>> what you are doing with the client region) OR using GFSH, connected to the >>>>> cluster to create it. >>>>> >>>>> If you'd like, have a look at Spring Boot for Geode >>>>> <https://github.com/spring-projects/spring-boot-data-geode>. I'd >>>>> recommend it.. you won't be disappointed. It will simplify the way you >>>>> code >>>>> and interact with the Geode system. >>>>> >>>>> --Udo >>>>> On 12/2/19 8:04 PM, Rajesh Kumar wrote: >>>>> >>>>> we are using apache geode 1.1.0 version. sometimes randomly while >>>>> trying to get details from a region we are getting exceptions. On looking >>>>> at the logs it seems t
Re: region name is incorrect while trying to get using region.get
Hi Ju,
I have checked the logs and I can see that out of the 5 nodes in the
cluster, the region is created only in 2 nodes and in rest of the nodes
region creation failed due to various reasons. Do you think this might
have caused the problem? if yes was it due to the client trying to connect
to a node where the region is not present?
Thanks,
Rajesh
On Wed, Dec 4, 2019 at 3:41 PM Ju@N wrote:
> Hello Rajesh,
>
> I've seen this issue 4 or 5 times in the past and, even though the
> Exception is misleading, I certainly remember that the root cause of the
> issue was always the same: the client application was trying to execute the
> operation on a server that didn't have the region created. As Dan pointed
> out, I'd suggest to double check whether you have the region
> *AuthIdRegion* configured in all your servers.
> Best regards.
>
>
> On Wed, Dec 4, 2019 at 9:40 AM Rajesh Kumar
> wrote:
>
>> Hi Udo,
>>
>> Thanks for the reply. One of our clients is facing this issue in
>> older version of our product where we use apache geode 1.1.0. We have
>> upgraded to 1.8.0 in our latest release.
>> The region /AuthIdRegion is created by using java code, verified it
>> by querying it using gfsh.
>> we will take a look at the spring boot for out further releases but
>> can you suggest me a fix/solution for the problem.
>>
>> Thanks,
>> Rajesh
>>
>>
>> On Wed, Dec 4, 2019 at 12:00 AM Udo Kohlmeyer wrote:
>>
>>> Reporting this, since it seems it did not make it through last night.
>>> On 12/3/19 12:04 AM, Udo Kohlmeyer wrote:
>>>
>>> Hi there Rajesh,
>>>
>>> Thank you for raising this. I have seen this before and I think it has
>>> been tracked to it actually being a toString() issue when logging the
>>> exception and not that it is not finding the region.
>>>
>>> Just for interest sake, why are you using Geode 1.1.0? Did you know
>>> Geode 1.10 is already out and we are in the process of cutting 1.11.
>>>
>>> In addition, can you confirm that you have created a region of the same
>>> name on the server side?
>>>
>>> If not, please create it either in Java code (pretty much the same as
>>> what you are doing with the client region) OR using GFSH, connected to the
>>> cluster to create it.
>>>
>>> If you'd like, have a look at Spring Boot for Geode
>>> <https://github.com/spring-projects/spring-boot-data-geode>. I'd
>>> recommend it.. you won't be disappointed. It will simplify the way you code
>>> and interact with the Geode system.
>>>
>>> --Udo
>>> On 12/2/19 8:04 PM, Rajesh Kumar wrote:
>>>
>>> we are using apache geode 1.1.0 version. sometimes randomly while trying
>>> to get details from a region we are getting exceptions. On looking at the
>>> logs it seems the region is appended to itself and a region not found
>>> exception is thrown.
>>>
>>>
>>>
>>> Caused by: org.apache.geode.cache.RegionDestroyedException: Server
>>> connection from
>>> [identity(10.43.164.241(10772:loner):62563:aebd35c5,connection=1;
>>> port=62563]: Region named /AuthIdRegion/AuthIdRegion was not found during
>>> get request
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.writeRegionDestroyedEx(BaseCommand.java:633)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.command.Get70.cmdExecute(Get70.java:126)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1171)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
>>>
>>> ... 1 more
>>>
>>>
>>>
>>>
>>>
>>> The name of my region is /AuthIdRegion but it tries to query using
>>> "/AuthIdRegion/AuthIdRegion"
>>>
>>>
>>>
>>> we use client region factory to create a region instance
>>>
>>> "authIdCacheRegionInstance = GeodeCache.getCacheInstance(
>>> ).createClientRegionFactory(ClientRegionShortcut.PROXY).create("AuthIdRegion");"
>>>
>>>
>>>
>>> It will be really helpful if someone can point what exactly is the issue.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Rajesh
>>>
>>>
>
> --
> Ju@N
>
Re: region name is incorrect while trying to get using region.get
have an error/configuration
> problem, but at least the error message should be more insightful. Please
> include the full stack trace and error message along with how you
> configured the Region on the server.
>
> -j
>
>
>
> On Tue, Dec 3, 2019 at 9:53 AM Dan Smith wrote:
>
>> Huh, that's weird. Is it possible that somewhere in your code you are
>> calling createSubregion on the returned authIdCacheRegionInstance? The
>> syntax /AuthIdRegion/AuthIdRegion indicates a region named AuthIdRegion
>> with a subregion also named AuthIdRegion.
>>
>> -Dan
>>
>> On Mon, Dec 2, 2019 at 8:04 PM Rajesh Kumar
>> wrote:
>>
>>> we are using apache geode 1.1.0 version. sometimes randomly while trying
>>> to get details from a region we are getting exceptions. On looking at the
>>> logs it seems the region is appended to itself and a region not found
>>> exception is thrown.
>>>
>>>
>>>
>>> Caused by: org.apache.geode.cache.RegionDestroyedException: Server
>>> connection from
>>> [identity(10.43.164.241(10772:loner):62563:aebd35c5,connection=1;
>>> port=62563]: Region named /AuthIdRegion/AuthIdRegion was not found during
>>> get request
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.writeRegionDestroyedEx(BaseCommand.java:633)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.command.Get70.cmdExecute(Get70.java:126)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1171)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
>>>
>>> ... 1 more
>>>
>>>
>>>
>>>
>>>
>>> The name of my region is /AuthIdRegion but it tries to query using
>>> "/AuthIdRegion/AuthIdRegion"
>>>
>>>
>>>
>>> we use client region factory to create a region instance
>>>
>>> "authIdCacheRegionInstance = GeodeCache.getCacheInstance(
>>> ).createClientRegionFactory(ClientRegionShortcut.PROXY).create("AuthIdRegion");"
>>>
>>>
>>>
>>> It will be really helpful if someone can point what exactly is the issue.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Rajesh
>>>
>>
>
> --
> -John
> john.blum10101 (skype)
>
Re: region name is incorrect while trying to get using region.get
Hi Udo,
Thanks for the reply. One of our clients is facing this issue in
older version of our product where we use apache geode 1.1.0. We have
upgraded to 1.8.0 in our latest release.
The region /AuthIdRegion is created by using java code, verified it by
querying it using gfsh.
we will take a look at the spring boot for out further releases but can
you suggest me a fix/solution for the problem.
Thanks,
Rajesh
On Wed, Dec 4, 2019 at 12:00 AM Udo Kohlmeyer wrote:
> Reporting this, since it seems it did not make it through last night.
> On 12/3/19 12:04 AM, Udo Kohlmeyer wrote:
>
> Hi there Rajesh,
>
> Thank you for raising this. I have seen this before and I think it has
> been tracked to it actually being a toString() issue when logging the
> exception and not that it is not finding the region.
>
> Just for interest sake, why are you using Geode 1.1.0? Did you know Geode
> 1.10 is already out and we are in the process of cutting 1.11.
>
> In addition, can you confirm that you have created a region of the same
> name on the server side?
>
> If not, please create it either in Java code (pretty much the same as what
> you are doing with the client region) OR using GFSH, connected to the
> cluster to create it.
>
> If you'd like, have a look at Spring Boot for Geode
> <https://github.com/spring-projects/spring-boot-data-geode>. I'd
> recommend it.. you won't be disappointed. It will simplify the way you code
> and interact with the Geode system.
>
> --Udo
> On 12/2/19 8:04 PM, Rajesh Kumar wrote:
>
> we are using apache geode 1.1.0 version. sometimes randomly while trying
> to get details from a region we are getting exceptions. On looking at the
> logs it seems the region is appended to itself and a region not found
> exception is thrown.
>
>
>
> Caused by: org.apache.geode.cache.RegionDestroyedException: Server
> connection from
> [identity(10.43.164.241(10772:loner):62563:aebd35c5,connection=1;
> port=62563]: Region named /AuthIdRegion/AuthIdRegion was not found during
> get request
>
> at
> org.apache.geode.internal.cache.tier.sockets.BaseCommand.writeRegionDestroyedEx(BaseCommand.java:633)
>
> at
> org.apache.geode.internal.cache.tier.sockets.command.Get70.cmdExecute(Get70.java:126)
>
> at
> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
>
> at
> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1171)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> at
> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
>
> ... 1 more
>
>
>
>
>
> The name of my region is /AuthIdRegion but it tries to query using
> "/AuthIdRegion/AuthIdRegion"
>
>
>
> we use client region factory to create a region instance
>
> "authIdCacheRegionInstance = GeodeCache.getCacheInstance(
> ).createClientRegionFactory(ClientRegionShortcut.PROXY).create("AuthIdRegion");"
>
>
>
> It will be really helpful if someone can point what exactly is the issue.
>
>
>
> Thanks,
>
> Rajesh
>
>
Re: region name is incorrect while trying to get using region.get
Hi John,
Thanks for the reply. We suspected the same. One of our clients is
facing this issue in older version of our product where we use apache geode
1.1.0. We have upgraded to 1.8.0 in our latest release. But is there any
fix/solution for this as we cannot upgrade the client environment. at this
point.
Thanks,
Rajesh
On Tue, Dec 3, 2019 at 11:40 PM John Blum wrote:
> This is an old bug (which was most certainly present in Apache Geode 1.1
> since I think I remember seeing this bug all the way up through 1.3).
> Technically, IIRC, there is no Region on the server by the name indicated
> in the misleading error message, /AuthIdRegion/AuthIdRegion, it was
> simply a problem with the error message. Off the top of my head, I cannot
> remember what ultimately caused the RegionDestroyedException in this
> case, though.
>
> Rajesh - is it possible for you to upgrade to Apache Geode 1.10, which is
> now the latest version? You might still have an error/configuration
> problem, but at least the error message should be more insightful. Please
> include the full stack trace and error message along with how you
> configured the Region on the server.
>
> -j
>
>
>
> On Tue, Dec 3, 2019 at 9:53 AM Dan Smith wrote:
>
>> Huh, that's weird. Is it possible that somewhere in your code you are
>> calling createSubregion on the returned authIdCacheRegionInstance? The
>> syntax /AuthIdRegion/AuthIdRegion indicates a region named AuthIdRegion
>> with a subregion also named AuthIdRegion.
>>
>> -Dan
>>
>> On Mon, Dec 2, 2019 at 8:04 PM Rajesh Kumar
>> wrote:
>>
>>> we are using apache geode 1.1.0 version. sometimes randomly while trying
>>> to get details from a region we are getting exceptions. On looking at the
>>> logs it seems the region is appended to itself and a region not found
>>> exception is thrown.
>>>
>>>
>>>
>>> Caused by: org.apache.geode.cache.RegionDestroyedException: Server
>>> connection from
>>> [identity(10.43.164.241(10772:loner):62563:aebd35c5,connection=1;
>>> port=62563]: Region named /AuthIdRegion/AuthIdRegion was not found during
>>> get request
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.writeRegionDestroyedEx(BaseCommand.java:633)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.command.Get70.cmdExecute(Get70.java:126)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1171)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>
>>> at
>>> org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
>>>
>>> ... 1 more
>>>
>>>
>>>
>>>
>>>
>>> The name of my region is /AuthIdRegion but it tries to query using
>>> "/AuthIdRegion/AuthIdRegion"
>>>
>>>
>>>
>>> we use client region factory to create a region instance
>>>
>>> "authIdCacheRegionInstance = GeodeCache.getCacheInstance(
>>> ).createClientRegionFactory(ClientRegionShortcut.PROXY).create("AuthIdRegion");"
>>>
>>>
>>>
>>> It will be really helpful if someone can point what exactly is the issue.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Rajesh
>>>
>>
>
> --
> -John
> john.blum10101 (skype)
>
