Re: AuthenticationRequiredException on force disconnection

[Bruce Schuchardt]

This might work for a server.  The locator sets an override 
configuration on the DistributedSystem that it creates, so files won't 
be accessed and any system properties will be ignored.



On 4/19/18 12:07 PM, John Blum wrote:

HI Dharam-

Alternatively, (*@Bruce* - please confirm), that this problem does not 
exist when a user uses /Gfsh/ to launch the server.  Let's explore why 
that is.


Well, this is only the case because when using the /Gfsh/, `start 
server` command, the properties (GemFire and Security properties) are 
typically "externalized", as in, put into separate (properties) files, 
e.g. gemfire.properties and gfSecurity.properties, respectively.


Therefore, it would seem that Geode is able to reconnect when the 
(GemFire/Security) properties are actually kept in files since, 
non-apparently, Geode will *re-read* these files when it reconstructs 
the /Cache/ (and to be more precise, the /DistributedSystem/, which is 
responsible for reading the configuration, which it delegates to an 
instance of /DistributedConfig/).


Some people are quick to point out that this is only a problem when 
using SDG.  However, that is simply not true and those people are not 
comparing apples to apples when using SDG to configure/bootstrap your 
Geode server vs /Gfsh/.  SDG would be more equivalent to the following 
(i.e. using the public API)...


Cache cache = *new* CacheFactory()
    .set("locators", "...")
    .set("security-manager", "...")
    .set("security-username", "...")
    .set("security-password", "...")
    .set("disable-auto-reconnect", "false")
    ...
    .create();
I guarantee you, before the fix for GEODE-5085 [1], using the Geode 
public API would not work either, because this is exactly what SDG 
does (*new* CacheFactory(gemfireProperties)), where 
"gemfireProperties" are your "geodeProperties" properties specified in 
*geode-server-context.xml*...



  



/So, what does this mean for your Spring configure/bootstrapped Geode 
server; How do you work around this problem?/


Well, all the properties you specified in ""geodeProperties" simply 
need to be externalized so that Geode re-reads in the properties on a 
reconnect.

Re: AuthenticationRequiredException on force disconnection

[John Blum]

HI Dharam-

Alternatively, (*@Bruce* - please confirm), that this problem does not
exist when a user uses *Gfsh* to launch the server.  Let's explore why that
is.

Well, this is only the case because when using the *Gfsh*, `start server`
command, the properties (GemFire and Security properties) are typically
"externalized", as in, put into separate (properties) files, e.g.
gemfire.properties and gfSecurity.properties, respectively.

Therefore, it would seem that Geode is able to reconnect when the
(GemFire/Security) properties are actually kept in files since,
non-apparently, Geode will *re-read* these files when it reconstructs the
*Cache* (and to be more precise, the *DistributedSystem*, which is
responsible for reading the configuration, which it delegates to an
instance of *DistributedConfig*).

Some people are quick to point out that this is only a problem when using
SDG.  However, that is simply not true and those people are not comparing
apples to apples when using SDG to configure/bootstrap your Geode server vs
*Gfsh*.  SDG would be more equivalent to the following (i.e. using the
public API)...

Cache cache = *new* CacheFactory()
.set("locators", "...")
.set("security-manager", "...")
.set("security-username", "...")
.set("security-password", "...")
.set("disable-auto-reconnect", "false")
...
.create();

I guarantee you, before the fix for GEODE-5085 [1], using the Geode public
API would not work either, because this is exactly what SDG does (*new*
CacheFactory(gemfireProperties)), where "gemfireProperties" are your "
geodeProperties" properties specified in *geode-server-context.xml*...






*So, what does this mean for your Spring configure/bootstrapped Geode
server; How do you work around this problem?*

Well, all the properties you specified in ""geodeProperties" simply need to
be externalized so that Geode re-reads in the properties on a reconnect.



Fortunately, with *Spring*, these properties can come from any "location"
as supported by *Spring's* Resource abstraction [2] by using the resource
location specifiers (e.g. 'classpath:', 'file:', etc as seen in *Table 10 -
Resource strings* of [3]).

Simply put all the Geode properties you specified in the
geode-server-context.xml into a properties file, specify the location of
this properties file in  using the location attribute, and
you should be in business.

Hope this helps util the fix for GEODE-5085 [1] is rolled out in Geode 1.6.

Cheers,
John


[1] https://issues.apache.org/jira/browse/GEODE-5085
[2]
https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources
[3]
https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources-resourceloader


On Thu, Apr 19, 2018 at 10:10 AM, Bruce Schuchardt <bschucha...@pivotal.io>
wrote:

> Hi Dharam
>
> This problem is fixed in the upcoming 1.6 release, which should be out
> later this month.
>
> As a workaround you could disable auto-reconnect (property
> disable-auto-reconnect) so that the process will just exit. Or you could
> get a build of the "develop" branch of the git repo and use that.
>
> The problem was that Geode was not retaining security settings along with
> other properties when it went into an auto-reconnect state.  It had all of
> the other settings but security settings were lost.
>
> On 4/16/18 1:57 AM, Thacker, Dharam wrote:
>
> Hi John,
>
>
>
> *1.  **What type of application are you building to configure your
> (embedded) peer cache?  E.g. is this a Web application with an embedded
> peer cache instance?  Or, are you simply using Spring to configure a peer
> cache, Geode server node, that is itself not an actual application?*
>
>
>
> It is client/server topology where both client & server nodes are
> configured using spring data geode.
>
> I am using spring data geode to simply configure geode server node
> [Attached actual config]
>
>
>
> *2.   *I have enabled enable-auto-reconnect="true" as well.
>
>
>
> *3.   *As per pulse and gfsh shell, node was thrown out of the
> distributed system as it does not show there.
>
>
>
> *4.   *security-udp-dhalgo is coming as “**” on node restart
> where I am assuming that it might be setting some value by default. I am
> not sure as it’s masked.
>
>
>
>
>
> *Ok. I would request geode team to validate on issue part.*
>
>
>
> But the point for *spring data geode* is,
>
> >> *How to deal with such scenario where node is no more a part of
> distributed system but app is running?*
>
>
>
> Usually one would like to take some automated corrective action o

Re: AuthenticationRequiredException on force disconnection

[John Blum]

te on issue part.*
>
>
>
> But the point for *spring data geode* is,
>
> >> *How to deal with such scenario where node is no more a part of
> distributed system but app is running?*
>
>
>
> Usually one would like to take some automated corrective action on such
> alerts.
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* John Blum [mailtojb...@pivotal.io]
> *Sent:* Monday, April 16, 2018 1:03 PM
> *To:* Thacker, Dharam
> *Cc:* user@geode.apache.org; Bruce Schuchardt
>
> *Subject:* Re: AuthenticationRequiredException on force disconnection
>
>
>
> Dharam-
>
>
>
> There is nothing *Spring* can do if Apache Geode is not respecting the
> setting for the Apache Geode property, "*security-udp-dhalgo*".  That is
> all Apache Geode.
>
>
>
> Also, I suspect that the AuthenticationRequiredException is not bubbling
> out because somewhere Geode is eating the Exception in the chain of calls
> and simply logging it (at log level, "warning", no less, as in it is
> considered "fatal").  Though, I would point out that even if the Exception
> was propagated out that it is not necessarily going to cause the JVM to
> exit either.  It all depends on the existing Thread types (e.g. non/daemon)
> and their state (e.g. BLOCKED).
>
>
>
> *What type of application are you building to configure your (embedded)
> peer cache?  E.g. is this a Web application with an embedded peer cache
> instance?  Or, are you simply using Spring to configure a peer cache, Geode
> server node, that is itself not an actual application?*
>
>
>
> A quick Thread dump on the JVM when this Exception occurs will quickly
> reveal what "non-daemon" Threads are blocking the JVM from shutting down,
> whether that is perhaps a Web application Thread (e.g. embedded Web
> server/container request processing Thread, etc) or a Geode Thread, holding
> up the JVM.
>
>
>
> Also keep in mind, that I deliberately disabled [1] the reconnecting state
> when using SDG, since I would assume that a user is using SDG to build an
> application using Apache Geode (whether as a client cache app, the common
> scenario, or even when building embedded peer cache applications).  As
> such, a user would need to explicitly enable this Geode property, for
> instance, when the user is only using Spring to configure a peer cache,
> Geode server.  To enable this "reconnectable" state in an application is
> futile since, if a Geode node gets "disconnected", then all Geode object
> references (from the Cache to Regions, to anything else) are not "stale".
> Therefore, if you have injected references to Geode objects into your
> application components (e.g. like a Region into a DAO/Repo), then you are
> going to have problems.
>
>
>
> As far as reporting the state, you can determine whether your peer node
> (application) is still connected by querying... cache.
> getDistributedSystem()
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem-->
> [2].isConnected()
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isConnected-->
> [3], or even isReconnecting()
> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isReconnecting-->
> [4].  As far as trying to determining whether an
> AuthenticationRequiredException has happened, great question!
>
>
>
> If I think of anything, I will let you know.
>
>
>
> Regards,
>
> John
>
>
>
>
>
> [1] https://github.com/spring-projects/spring-data-geode/
> blob/2.0.6.RELEASE/src/main/java/org/springframework/data/
> gemfire/CacheFactoryBean.java#L204-L205
>
> [2] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/cache/GemFireCache.html#getDistributedSystem--
>
> [3] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/distributed/DistributedSystem.html#isConnected--
>
> [4] http://geode.apache.org/releases/latest/javadoc/org/
> apache/geode/distributed/DistributedSystem.html#isReconnecting--
>
>
>
>
>
> On Sun, Apr 15, 2018 at 11:08 PM, Thacker, Dharam <
> dharam.thac...@jpmorgan.com> wrote:
>
> Thanks John/Bruce!
>
>
>
> But it does not work as expected.
>
>
>
> I tried setting (*security-udp-dhalgo=*) in properties file in both
> locators and servers.
>
> I also confirmed the same by verifying config level logs and using locator
> with following command which explicitly mentions that *“security-udp-dhalgo”
> is empty (gemfire.sys.security-udp-dhalgo =)*
&g

Re: AuthenticationRequiredException on force disconnection

[Bruce Schuchardt]

I've reproduced your problem and am looking into it.  It turns out that 
it doesn't have to do with security-udp-dhalgo but with security 
credentials.



On 4/15/18 11:08 PM, Thacker, Dharam wrote:


Thanks John/Bruce!

But it does not work as expected.

I tried setting (*security-udp-dhalgo=*) in properties file in both 
locators and servers.


I also confirmed the same by verifying config level logs and using 
locator with following command which explicitly mentions that 
*“security-udp-dhalgo” is empty (gemfire.sys.security-udp-dhalgo =)*


>> describe config –member=locator1

>> describe config –member=server1

But even after that, I see following exception which is same as before.

More, it looks like that once GEODE server member reboot itself after 
force disconnection, it does not respect *security-udp-dhalgo* 
override in properties file (/My assumption based on logs/)


I see *security-udp-dhalgo=* in startup configuration *after 
member’s attempt to connect to distributed system*.


[warning 2018/04/15 21:23:57.095 EDT event-server-1 
tid=0x215] Exception occurred while trying to connect the system 
during reconnect


org.apache.geode.security.AuthenticationRequiredException: Failed to 
find credentials from [host-001(event-server-1:32054):1025]


at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)


at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)


at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)


at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)


at 
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)


at 
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)


at 
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)


at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1112)


at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1160)


at 
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)


at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)


at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)


at 
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)


at 
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)


   at 
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)


at 
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)


at 
org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)


John,

This does not kill GEODE application. It still runs as it is. This 
makes APM tool to assume that application is healthy and is not facing 
any issues.


What do you suggest to rectify this?

Is there any example if I can report state of GEODE server as 
“UNHEALTHY”/”DISCONNECTED”?


Is there any example if I can listen to these notifications and come 
up with some health check service?


Thanks & Regards,

Dharam

*From:*John Blum [mailtojb...@pivotal.io <mailto:jb...@pivotal.io>]
*Sent:* Friday, April 13, 2018 2:59 AM
*To:* user@geode.apache.org <mailto:user@geode.apache.org>
*Subject:* Re: AuthenticationRequiredException on force disconnection

Regarding /Spring/, not really too differently actually, see here 
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33> [1] 
(XML) and here 
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84> [2] 
(/JavaConfig/) (followed by this 
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91> [3] 
and this 
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96> [4]).


There is even an Annotation-based approach 
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java> [5] 
for the curious onlooker.

RE: AuthenticationRequiredException on force disconnection

[Thacker, Dharam]

Hi John,


1.  What type of application are you building to configure your (embedded) 
peer cache?  E.g. is this a Web application with an embedded peer cache 
instance?  Or, are you simply using Spring to configure a peer cache, Geode 
server node, that is itself not an actual application?

It is client/server topology where both client & server nodes are configured 
using spring data geode.
I am using spring data geode to simply configure geode server node [Attached 
actual config]


2.   I have enabled enable-auto-reconnect="true" as well.


3.   As per pulse and gfsh shell, node was thrown out of the distributed 
system as it does not show there.



4.   security-udp-dhalgo is coming as “**” on node restart where I am 
assuming that it might be setting some value by default. I am not sure as it’s 
masked.


Ok. I would request geode team to validate on issue part.

But the point for spring data geode is,
>> How to deal with such scenario where node is no more a part of distributed 
>> system but app is running?

Usually one would like to take some automated corrective action on such alerts.

Thanks & Regards,
Dharam

From: John Blum [mailto:jb...@pivotal.io]
Sent: Monday, April 16, 2018 1:03 PM
To: Thacker, Dharam
Cc: user@geode.apache.org; Bruce Schuchardt
Subject: Re: AuthenticationRequiredException on force disconnection

Dharam-

There is nothing Spring can do if Apache Geode is not respecting the setting 
for the Apache Geode property, "security-udp-dhalgo".  That is all Apache Geode.

Also, I suspect that the AuthenticationRequiredException is not bubbling out 
because somewhere Geode is eating the Exception in the chain of calls and 
simply logging it (at log level, "warning", no less, as in it is considered 
"fatal").  Though, I would point out that even if the Exception was propagated 
out that it is not necessarily going to cause the JVM to exit either.  It all 
depends on the existing Thread types (e.g. non/daemon) and their state (e.g. 
BLOCKED).

What type of application are you building to configure your (embedded) peer 
cache?  E.g. is this a Web application with an embedded peer cache instance?  
Or, are you simply using Spring to configure a peer cache, Geode server node, 
that is itself not an actual application?

A quick Thread dump on the JVM when this Exception occurs will quickly reveal 
what "non-daemon" Threads are blocking the JVM from shutting down, whether that 
is perhaps a Web application Thread (e.g. embedded Web server/container request 
processing Thread, etc) or a Geode Thread, holding up the JVM.

Also keep in mind, that I deliberately disabled [1] the reconnecting state when 
using SDG, since I would assume that a user is using SDG to build an 
application using Apache Geode (whether as a client cache app, the common 
scenario, or even when building embedded peer cache applications).  As such, a 
user would need to explicitly enable this Geode property, for instance, when 
the user is only using Spring to configure a peer cache, Geode server.  To 
enable this "reconnectable" state in an application is futile since, if a Geode 
node gets "disconnected", then all Geode object references (from the Cache to 
Regions, to anything else) are not "stale".  Therefore, if you have injected 
references to Geode objects into your application components (e.g. like a 
Region into a DAO/Repo), then you are going to have problems.

As far as reporting the state, you can determine whether your peer node 
(application) is still connected by querying... 
cache.getDistributedSystem()<https://secureweb.jpmchase.net/readonly/http:/geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem-->[2].isConnected()<https://secureweb.jpmchase.net/readonly/http:/geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isConnected-->[3],
 or even 
isReconnecting()<https://secureweb.jpmchase.net/readonly/http:/geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isReconnecting-->[4].
  As far as trying to determining whether an AuthenticationRequiredException 
has happened, great question!

If I think of anything, I will let you know.

Regards,
John


[1] 
https://github.com/spring-projects/spring-data-geode/blob/2.0.6.RELEASE/src/main/java/org/springframework/data/gemfire/CacheFactoryBean.java#L204-L205<https://secureweb.jpmchase.net/readonly/https:/github.com/spring-projects/spring-data-geode/blob/2.0.6.RELEASE/src/main/java/org/springframework/data/gemfire/CacheFactoryBean.java#L204-L205>
[2] 
http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem--<https://secureweb.jpmchase.net/readonly/http:/geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#g

Re: AuthenticationRequiredException on force disconnection

[John Blum]

rship.GMSJoinLeave.join(GMSJoinLeave.java:318)
>
> at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.join(GMSMembershipManager.java:656)
>
> at org.apache.geode.distributed.internal.membership.gms.mgr.
> GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
>
> at org.apache.geode.distributed.internal.membership.gms.
> Services.start(Services.java:181)
>
> at org.apache.geode.distributed.internal.membership.gms.
> GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
>
> at org.apache.geode.distributed.internal.membership.MemberFactory.
> newMembershipManager(MemberFactory.java:89)
>
> at org.apache.geode.distributed.internal.DistributionManager.<
> init>(DistributionManager.java:1112)
>
> at org.apache.geode.distributed.internal.DistributionManager.<
> init>(DistributionManager.java:1160)
>
> at org.apache.geode.distributed.internal.DistributionManager.
> create(DistributionManager.java:531)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
>
> at org.apache.geode.distributed.DistributedSystem.connect(
> DistributedSystem.java:202)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.tryReconnect(InternalDistributedSystem.
> java:2508)
>
> at org.apache.geode.distributed.internal.
> InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
>
> at org.apache.geode.distributed.internal.DistributionManager$
> MyListener.membershipFailure(DistributionManager.java:4307)
>
>
>
> John,
>
>
>
> This does not kill GEODE application. It still runs as it is. This makes
> APM tool to assume that application is healthy and is not facing any issues.
>
> What do you suggest to rectify this?
>
>
>
> Is there any example if I can report state of GEODE server as
> “UNHEALTHY”/”DISCONNECTED”?
>
> Is there any example if I can listen to these notifications and come up
> with some health check service?
>
>
>
> Thanks & Regards,
>
> Dharam
>
>
>
> *From:* John Blum [mailtojb...@pivotal.io]
> *Sent:* Friday, April 13, 2018 2:59 AM
> *To:* user@geode.apache.org
> *Subject:* Re: AuthenticationRequiredException on force disconnection
>
>
>
> Regarding *Spring*, not really too differently actually, see here
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33>
>  [1]
> (XML) and here
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84>
>  [2]
> (*JavaConfig*) (followed by this
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91>
>  [3]
> and this
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96>
>  [4]).
>
>
>
> There is even an Annotation-based approach
> <https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java>
>  [5]
> for the curious onlooker.
>
>
>
>
>
> [1] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/resources/spring-
> server-cache.xml#L24-L33
>
> [2] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/spring/java/server/
> JavaConfiguredGeodeServerApplication.java#L66-L84
>
> [3] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/spring/java/server/
> JavaConfiguredGeodeServerApplication.java#L91
>
> [4] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/spring/java/server/
> JavaConfiguredGeodeServerApplication.java#L96
>
> [5] https://github.com/jxblum/contacts-application/blob/
> master/configuration-example/src/main/java/example/app/
> spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java
>
>
&g

RE: AuthenticationRequiredException on force disconnection

[Thacker, Dharam]

Thanks John/Bruce!

But it does not work as expected.

I tried setting (security-udp-dhalgo=) in properties file in both locators and 
servers.
I also confirmed the same by verifying config level logs and using locator with 
following command which explicitly mentions that “security-udp-dhalgo” is empty 
(gemfire.sys.security-udp-dhalgo =)

>> describe config –member=locator1
>> describe config –member=server1

But even after that, I see following exception which is same as before.

More, it looks like that once GEODE server member reboot itself after force 
disconnection, it does not respect security-udp-dhalgo override in properties 
file (My assumption based on logs)
I see security-udp-dhalgo= in startup configuration after member’s 
attempt to connect to distributed system.

[warning 2018/04/15 21:23:57.095 EDT event-server-1 tid=0x215] 
Exception occurred while trying to connect the system during reconnect
org.apache.geode.security.AuthenticationRequiredException: Failed to find 
credentials from [host-001(event-server-1:32054):1025]
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)
at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
at 
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
at 
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
at 
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1112)
at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1160)
at 
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
at 
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)
at 
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
at 
org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)

John,

This does not kill GEODE application. It still runs as it is. This makes APM 
tool to assume that application is healthy and is not facing any issues.
What do you suggest to rectify this?

Is there any example if I can report state of GEODE server as 
“UNHEALTHY”/”DISCONNECTED”?
Is there any example if I can listen to these notifications and come up with 
some health check service?

Thanks & Regards,
Dharam

From: John Blum [mailto:jb...@pivotal.io]
Sent: Friday, April 13, 2018 2:59 AM
To: user@geode.apache.org
Subject: Re: AuthenticationRequiredException on force disconnection

Regarding Spring, not really too differently actually, see 
here<https://secureweb.jpmchase.net/readonly/https:/github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33>
 [1] (XML) and 
here<https://secureweb.jpmchase.net/readonly/https:/github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84>
 [2] (JavaConfig) (followed by 
this<https://secureweb.jpmchase.net/readonly/https:/github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91>
 [3] and 
this<https://secureweb.jpmchase.net/readonly/https:/github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96>
 [4]).

There is even an Annotation-based 
approach<https://secureweb.jpmchase.net/readonly/https:/github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java>
 [5] for the curious onlooker.


[1] 
https://gith

Re: AuthenticationRequiredException on force disconnection

[John Blum]

Regarding *Spring*, not really too differently actually, see here

[1]
(XML) and here

[2]
(*JavaConfig*) (followed by this

[3]
and this

 [4]).

There is even an Annotation-based approach

[5]
for the curious onlooker.


[1]
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33
[2]
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84
[3]
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91
[4]
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96
[5]
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java


On Thu, Apr 12, 2018 at 2:17 PM, Bruce Schuchardt 
wrote:

> The setting merely causes Geode to encrypt packets sent over UDP.
>
>
> On 4/11/18 10:29 AM, Thacker, Dharam wrote:
>
>>
>> Would there be any negative impact on disabling 'security-udp-dhalgo' on
>> peer to peer members or pulse or jmx notifications ?
>>
>> Thanks,
>> Dharam
>>
>>
>


-- 
-John
john.blum10101 (skype)

Re: AuthenticationRequiredException on force disconnection

[Bruce Schuchardt]

The setting merely causes Geode to encrypt packets sent over UDP.

On 4/11/18 10:29 AM, Thacker, Dharam wrote:


Would there be any negative impact on disabling 'security-udp-dhalgo' 
on peer to peer members or pulse or jmx notifications ?


Thanks,
Dharam

Re: AuthenticationRequiredException on force disconnection

[Bruce Schuchardt]

I'm not sure what your development context is so it's hard to answer 
that question.  If you're programatically creating a cache then set the 
cache property ConfigurationProperties.SECURITY_UDP_DHALGO to an empty 
string. If you're using a properties file set it to blank.


security-udp-dhalgo=

-or-

cachefactory.set(SECURITY_UDP_DHALGO, "")

I don't recall how you set properties for the cache under Spring.


On 4/11/18 11:44 PM, Thacker, Dharam wrote:

Hello Bruce,

I have not manually specified this property to enable udp encryption 
using "security-udp-dhalgo" anywhere. I am using TCP mode only.


Is it by default enabled? If yes, how can I disable it?

I could not find any documentation on it.

Thanks,
Dharam

Sent with BlackBerry Work (www.blackberry.com)

*From: *"Thacker, Dharam" <dharam.thac...@jpmorgan.com 
<mailto:dharam.thac...@jpmorgan.com>>

*Sent: *Apr 11, 2018 10:59 PM
*To: *user@geode.apache.org <mailto:user@geode.apache.org>
*Subject: *RE: AuthenticationRequiredException on force disconnection

Thank you Bruce!

I will surely open a JIRA soon.

"Geode sends membership information, alerts and on rare occasions a PDX
registration message over UDP"

Would there be any negative impact on disabling 'security-udp-dhalgo' 
on peer to peer members or pulse or jmx notifications ?


Thanks,
Dharam

Sent with BlackBerry Work (www.blackberry.com <http://www.blackberry.com>)

*From: *Bruce Schuchardt <bschucha...@pivotal.io 
<mailto:bschucha...@pivotal.io>>

*Sent: *Apr 11, 2018 8:45 PM
*To: *user@geode.apache.org <mailto:user@geode.apache.org>
*Subject: *Re: AuthenticationRequiredException on force disconnection

That looks like a bug in UDP encryption.  Can you open a JIRA ticket to
track this?  Set the component to "membership".  Looking at the unit
test suite I don't think there is any coverage for auto-reconnect with
security-udp-dhalgo enabled.

As a workaround you could, if you're comfortable doing so, disable
security-udp-dhalgo until this is fixed.  There are other known issues
with this fairly new setting that people have been working on recently.

Geode sends membership information, alerts and on rare occasions a PDX
registration message over UDP.  No client/server messages are sent over
UDP so its use is confined to your server cluster. No messages
containing application objects (keys, values, callback args etc) are
sent over UDP unless you set disable-tcp=true to disable use of tcp/ip
stream sockets.


On 4/11/18 4:38 AM, Thacker, Dharam wrote:
> warning 2018/04/10 02:40:59.541 EDT event-server-1  
tid=0x217] Exception occurred while trying to connect the system 
during reconnect
> org.apache.geode.security.AuthenticationRequiredException: Failed to 
find credentials from [host001(event-server-1:3525):1026]
>  at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
>  at 
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
>  at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)
>  at 
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
>  at 
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
>  at 
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
>  at 
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
>  at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1112)
>  at 
org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1160)
>  at 
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
>  at 
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
>  at 
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
>  at 
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
>  at 
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
>  at 
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)
>  at 
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSy

RE: AuthenticationRequiredException on force disconnection

[Thacker, Dharam]

Hello Bruce,

I have not manually specified this property to enable udp encryption using 
"security-udp-dhalgo" anywhere. I am using TCP mode only.

Is it by default enabled? If yes, how can I disable it?

I could not find any documentation on it.

Thanks,
Dharam

Sent with BlackBerry Work (www.blackberry.com)

From: "Thacker, Dharam" <dharam.thac...@jpmorgan.com>
Sent: Apr 11, 2018 10:59 PM
To: user@geode.apache.org
Subject: RE: AuthenticationRequiredException on force disconnection

Thank you Bruce!

I will surely open a JIRA soon.

"Geode sends membership information, alerts and on rare occasions a PDX
registration message over UDP"

Would there be any negative impact on disabling 'security-udp-dhalgo' on peer 
to peer members or pulse or jmx notifications ?

Thanks,
Dharam

Sent with BlackBerry Work 
(www.blackberry.com<https://secureweb.jpmchase.net/readonly/http://www.blackberry.com>)

From: Bruce Schuchardt <bschucha...@pivotal.io<mailto:bschucha...@pivotal.io>>
Sent: Apr 11, 2018 8:45 PM
To: user@geode.apache.org<mailto:user@geode.apache.org>
Subject: Re: AuthenticationRequiredException on force disconnection

That looks like a bug in UDP encryption.  Can you open a JIRA ticket to
track this?  Set the component to "membership".  Looking at the unit
test suite I don't think there is any coverage for auto-reconnect with
security-udp-dhalgo enabled.

As a workaround you could, if you're comfortable doing so, disable
security-udp-dhalgo until this is fixed.  There are other known issues
with this fairly new setting that people have been working on recently.

Geode sends membership information, alerts and on rare occasions a PDX
registration message over UDP.  No client/server messages are sent over
UDP so its use is confined to your server cluster. No messages
containing application objects (keys, values, callback args etc) are
sent over UDP unless you set disable-tcp=true to disable use of tcp/ip
stream sockets.


On 4/11/18 4:38 AM, Thacker, Dharam wrote:
> warning 2018/04/10 02:40:59.541 EDT event-server-1  
> tid=0x217] Exception occurred while trying to connect the system during 
> reconnect
> org.apache.geode.security.AuthenticationRequiredException: Failed to find 
> credentials from [host001(event-server-1:3525):1026]
>  at 
> org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
>  at 
> org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
>  at 
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)
>  at 
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
>  at 
> org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
>  at 
> org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
>  at 
> org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
>  at 
> org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1112)
>  at 
> org.apache.geode.distributed.internal.DistributionManager.(DistributionManager.java:1160)
>  at 
> org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
>  at 
> org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
>  at 
> org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
>  at 
> org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
>  at 
> org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
>  at 
> org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)
>  at 
> org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
>  at 
> org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)
>  at 
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1530)
>  at 
> org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$0(GMSMembershipManager.java:2550)
>  at java.lang.Thread.run(Thread.java:745)



This