Re: AuthenticationRequiredException on force disconnection

[Bruce Schuchardt]

This might work for a server.  The locator sets an override 
configuration on the DistributedSystem that it creates, so files won't 
be accessed and any system properties will be ignored.

On 4/19/18 12:07 PM, John Blum wrote:
HI Dharam-

Alternatively, (*@Bruce* - please confirm), that this problem does not 
exist when a user uses /Gfsh/ to launch the server.  Let's explore why 
that is.

Well, this is only the case because when using the /Gfsh/, `start 
server` command, the properties (GemFire and Security properties) are 
typically "externalized", as in, put into separate (properties) files, 
e.g. gemfire.properties and gfSecurity.properties, respectively.

Therefore, it would seem that Geode is able to reconnect when the 
(GemFire/Security) properties are actually kept in files since, 
non-apparently, Geode will *re-read* these files when it reconstructs 
the /Cache/ (and to be more precise, the /DistributedSystem/, which is 
responsible for reading the configuration, which it delegates to an 
instance of /DistributedConfig/).

Some people are quick to point out that this is only a problem when 
using SDG.  However, that is simply not true and those people are not 
comparing apples to apples when using SDG to configure/bootstrap your 
Geode server vs /Gfsh/.  SDG would be more equivalent to the following 
(i.e. using the public API)...

Cache cache = *new* CacheFactory()
    .set("locators", "...")
    .set("security-manager", "...")
    .set("security-username", "...")
    .set("security-password", "...")
    .set("disable-auto-reconnect", "false")
    ...
    .create();
I guarantee you, before the fix for GEODE-5085 [1], using the Geode 
public API would not work either, because this is exactly what SDG 
does (*new* CacheFactory(gemfireProperties)), where 
"gemfireProperties" are your "geodeProperties" properties specified in 
*geode-server-context.xml*...

<util:properties id="geodeProperties">
  ....
</util:properties>


/So, what does this mean for your Spring configure/bootstrapped Geode 
server; How do you work around this problem?/

Well, all the properties you specified in ""geodeProperties" simply 
need to be externalized so that Geode re-reads in the properties on a 
reconnect.

<util:propertiesid="geodeProperties" 
location="file:/resource/path/to/geode.properties">

Fortunately, with /Spring/, these properties can come from any 
"location" as supported by /Spring's/ Resource abstraction [2] by 
using the resource location specifiers (e.g. 'classpath:', 'file:', 
etc as seen in *Table 10 - /Resource strings/* of [3]).

Simply put all the Geode properties you specified in the 
geode-server-context.xml into a properties file, specify the location 
of this properties file in <util:properties> using the location 
attribute, and you should be in business.

Hope this helps util the fix for GEODE-5085 [1] is rolled out in Geode 
1.6.

Cheers,
John


[1] https://issues.apache.org/jira/browse/GEODE-5085
[2] 
https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources
[3] 
https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#resources-resourceloader


On Thu, Apr 19, 2018 at 10:10 AM, Bruce Schuchardt 
<bschucha...@pivotal.io <mailto:bschucha...@pivotal.io>> wrote:

    Hi Dharam

    This problem is fixed in the upcoming 1.6 release, which should be
    out later this month.

    As a workaround you could disable auto-reconnect (property
    disable-auto-reconnect) so that the process will just exit. Or you
    could get a build of the "develop" branch of the git repo and use
    that.

    The problem was that Geode was not retaining security settings
    along with other properties when it went into an auto-reconnect
    state.  It had all of the other settings but security settings
    were lost.


    On 4/16/18 1:57 AM, Thacker, Dharam wrote:

    Hi John,

    /1.//What type of application are you building to configure your
    (embedded) peer cache?  E.g. is this a Web application with an
    embedded peer cache instance?  Or, are you simply using Spring to
    configure a peer cache, Geode server node, that is itself not an
    actual application?/

    It is client/server topology where both client & server nodes are
    configured using spring data geode.

    I am using spring data geode to simply configure geode server
    node [Attached actual config]

    /2./I have enabled enable-auto-reconnect="true" as well.

    /3./As per pulse and gfsh shell, node was thrown out of the
    distributed system as it does not show there.

    /4./security-udp-dhalgo is coming as “******” on node restart
    where I am assuming that it might be setting some value by
    default. I am not sure as it’s masked.

    *Ok. I would request geode team to validate on issue part.*

    But the point for *spring data geode* is,

    >> *How to deal with such scenario where node is no more a part of
    distributed system but app is running?*

    Usually one would like to take some automated corrective action
    on such alerts.

    Thanks & Regards,

    Dharam

    *From:*John Blum [mailtojb...@pivotal.io <mailto:jb...@pivotal.io>]
    *Sent:* Monday, April 16, 2018 1:03 PM
    *To:* Thacker, Dharam
    *Cc:* user@geode.apache.org <mailto:user@geode.apache.org>; Bruce
    Schuchardt
    *Subject:* Re: AuthenticationRequiredException on force disconnection

    Dharam-

    There is nothing /Spring/ can do if Apache Geode is not
    respecting the setting for the Apache Geode property,
    "*security-udp-dhalgo*". That is all Apache Geode.

    Also, I suspect that the AuthenticationRequiredException is not
    bubbling out because somewhere Geode is eating the Exception in
    the chain of calls and simply logging it (at log level,
    "warning", no less, as in it is considered "fatal").  Though, I
    would point out that even if the Exception was propagated out
    that it is not necessarily going to cause the JVM to exit
    either.  It all depends on the existing Thread types (e.g.
    non/daemon) and their state (e.g. BLOCKED).

    /What type of application are you building to configure your
    (embedded) peer cache?  E.g. is this a Web application with an
    embedded peer cache instance?  Or, are you simply using Spring to
    configure a peer cache, Geode server node, that is itself not an
    actual application?/

    A quick Thread dump on the JVM when this Exception occurs will
    quickly reveal what "non-daemon" Threads are blocking the JVM
    from shutting down, whether that is perhaps a Web application
    Thread (e.g. embedded Web server/container request processing
    Thread, etc) or a Geode Thread, holding up the JVM.

    Also keep in mind, that I deliberately disabled [1] the
    reconnecting state when using SDG, since I would assume that a
    user is using SDG to build an application using Apache Geode
    (whether as a client cache app, the common scenario, or even when
    building embedded peer cache applications).  As such, a user
    would need to explicitly enable this Geode property, for
    instance, when the user is only using Spring to configure a peer
    cache, Geode server.  To enable this "reconnectable" state in an
    application is futile since, if a Geode node gets "disconnected",
    then all Geode object references (from the Cache to Regions, to
    anything else) are not "stale". Therefore, if you have injected
    references to Geode objects into your application components
    (e.g. like a Region into a DAO/Repo), then you are going to have
    problems.

    As far as reporting the state, you can determine whether your
    peer node (application) is still connected by querying...
    cache.getDistributedSystem()
    
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem-->[2].isConnected()
    
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isConnected-->[3],
    or even isReconnecting()
    
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isReconnecting-->[4].
    As far as trying to determining whether an
    AuthenticationRequiredException has happened, great question!

    If I think of anything, I will let you know.

    Regards,

    John

    [1]
    
https://github.com/spring-projects/spring-data-geode/blob/2.0.6.RELEASE/src/main/java/org/springframework/data/gemfire/CacheFactoryBean.java#L204-L205
    
<https://github.com/spring-projects/spring-data-geode/blob/2.0.6.RELEASE/src/main/java/org/springframework/data/gemfire/CacheFactoryBean.java#L204-L205>

    [2]
    
http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem--
    
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/GemFireCache.html#getDistributedSystem-->

    [3]
    
http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isConnected--
    
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isConnected-->

    [4]
    
http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isReconnecting--
    
<http://geode.apache.org/releases/latest/javadoc/org/apache/geode/distributed/DistributedSystem.html#isReconnecting-->

    On Sun, Apr 15, 2018 at 11:08 PM, Thacker, Dharam
    <dharam.thac...@jpmorgan.com
    <mailto:dharam.thac...@jpmorgan.com>> wrote:

    Thanks John/Bruce!

    But it does not work as expected.

    I tried setting (*security-udp-dhalgo=*) in properties file in
    both locators and servers.

    I also confirmed the same by verifying config level logs and
    using locator with following command which explicitly mentions
    that *“security-udp-dhalgo” is empty
    (gemfire.sys.security-udp-dhalgo =)*

    >> describe config –member=locator1

    >> describe config –member=server1

    But even after that, I see following exception which is same as
    before.

    More, it looks like that once GEODE server member reboot itself
    after force disconnection, it does not respect
    *security-udp-dhalgo* override in properties file (/My assumption
    based on logs/)

    I see *security-udp-dhalgo=********* in startup configuration
    *after member’s attempt to connect to distributed system*.

    [warning 2018/04/15 21:23:57.095 EDT
    event-server-1<ReconnectThread> tid=0x215] Exception occurred
    while trying to connect the system during reconnect

    org.apache.geode.security.AuthenticationRequiredException: Failed
    to find credentials from [host-001(event-server-1:32054)<ec>:1025]

    at
    
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)

    at
    
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)

    at
    
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)

    at
    
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)

    at
    
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)

    at
    
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)

    at
    
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)

    at
    
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1112)

    at
    
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1160)

    at
    
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)

    at
    
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)

    at
    
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)

    at
    
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)

    at
    
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)

       at
    
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)

    at
    
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)

    at
    
org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)

    John,

    This does not kill GEODE application. It still runs as it is.
    This makes APM tool to assume that application is healthy and is
    not facing any issues.

    What do you suggest to rectify this?

    Is there any example if I can report state of GEODE server as
    “UNHEALTHY”/”DISCONNECTED”?

    Is there any example if I can listen to these notifications and
    come up with some health check service?

    Thanks & Regards,

    Dharam

    *From:*John Blum [mailtojb...@pivotal.io <mailto:jb...@pivotal.io>]
    *Sent:* Friday, April 13, 2018 2:59 AM
    *To:* user@geode.apache.org <mailto:user@geode.apache.org>
    *Subject:* Re: AuthenticationRequiredException on force disconnection

    Regarding /Spring/, not really too differently actually, see here
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33>
 [1]
    (XML) and here
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84>
 [2]
    (/JavaConfig/) (followed by this
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91>
 [3]
    and this
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96>
 [4]).

    There is even an Annotation-based approach
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java>
 [5]
    for the curious onlooker.

    [1]
    
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/resources/spring-server-cache.xml#L24-L33>

    [2]
    
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L66-L84>

    [3]
    
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L91>

    [4]
    
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/java/server/JavaConfiguredGeodeServerApplication.java#L96>

    [5]
    
https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java
    
<https://github.com/jxblum/contacts-application/blob/master/configuration-example/src/main/java/example/app/spring/annotation/server/AnnotationConfiguredGeodeServerApplication.java>

    On Thu, Apr 12, 2018 at 2:17 PM, Bruce Schuchardt
    <bschucha...@pivotal.io <mailto:bschucha...@pivotal.io>> wrote:

    The setting merely causes Geode to encrypt packets sent over UDP.



    On 4/11/18 10:29 AM, Thacker, Dharam wrote:


    Would there be any negative impact on disabling
    'security-udp-dhalgo' on peer to peer members or pulse or jmx
    notifications ?

    Thanks,
    Dharam



    -- 

    -John

    john.blum10101 (skype)

    This message is confidential and subject to terms at:
    http://www.jpmorgan.com/emaildisclaimer
    <http://www.jpmorgan.com/emaildisclaimer> including on
    confidentiality, legal privilege, viruses and monitoring of
    electronic messages. If you are not the intended recipient,
    please delete this message and notify the sender immediately. Any
    unauthorized use is strictly prohibited.



    ---------- Forwarded message ----------
    From: Bruce Schuchardt <bschucha...@pivotal.io
    <mailto:bschucha...@pivotal.io>>
    To: "user@geode.apache.org <mailto:user@geode.apache.org>"
    <user@geode.apache.org <mailto:user@geode.apache.org>>
    Cc:
    Bcc:
    Date: Thu, 12 Apr 2018 21:15:21 +0000
    Subject: Re: AuthenticationRequiredException on force disconnection

    I'm not sure what your development context is so it's hard to
    answer that question.  If you're programatically creating a cache
    then set the cache property
    ConfigurationProperties.SECURITY_UDP_DHALGO to an empty string. 
    If you're using a properties file set it to blank.

    security-udp-dhalgo=

    -or-

    cachefactory.set(SECURITY_UDP_DHALGO, "")

    I don't recall how you set properties for the cache under Spring.

    On 4/11/18 11:44 PM, Thacker, Dharam wrote:

        Hello Bruce,

        I have not manually specified this property to enable udp
        encryption using "security-udp-dhalgo" anywhere. I am using
        TCP mode only.

        Is it by default enabled? If yes, how can I disable it?

        I could not find any documentation on it.

        Thanks,

        Dharam

        Sent with BlackBerry Work (www.blackberry.com
        <http://www.blackberry.com>)

        ------------------------------------------------------------------------

        *From: *"Thacker, Dharam" <dharam.thac...@jpmorgan.com
        <mailto:dharam.thac...@jpmorgan.com>>
        *Sent: *Apr 11, 2018 10:59 PM
        *To: *user@geode.apache.org <mailto:user@geode.apache.org>
        *Subject: *RE: AuthenticationRequiredException on force
        disconnection

        Thank you Bruce!

        I will surely open a JIRA soon.

        "Geode sends membership information, alerts and on rare
        occasions a PDX

        registration message over UDP"

        Would there be any negative impact on disabling
        'security-udp-dhalgo' on peer to peer members or pulse or jmx
        notifications ?

        Thanks,

        Dharam

        Sent with BlackBerry Work (www.blackberry.com
        <http://www.blackberry.com>)

        ------------------------------------------------------------------------

        *From: *Bruce Schuchardt <bschucha...@pivotal.io
        <mailto:bschucha...@pivotal.io>>
        *Sent: *Apr 11, 2018 8:45 PM
        *To: *user@geode.apache.org <mailto:user@geode.apache.org>
        *Subject: *Re: AuthenticationRequiredException on force
        disconnection

        That looks like a bug in UDP encryption.  Can you open a JIRA
        ticket to
        track this?  Set the component to "membership".  Looking at
        the unit
        test suite I don't think there is any coverage for
        auto-reconnect with
        security-udp-dhalgo enabled.

        As a workaround you could, if you're comfortable doing so,
        disable
        security-udp-dhalgo until this is fixed.  There are other
        known issues
        with this fairly new setting that people have been working on
        recently.

        Geode sends membership information, alerts and on rare
        occasions a PDX
        registration message over UDP.  No client/server messages are
        sent over
        UDP so its use is confined to your server cluster. No messages
        containing application objects (keys, values, callback args
        etc) are
        sent over UDP unless you set disable-tcp=true to disable use
        of tcp/ip
        stream sockets.


        On 4/11/18 4:38 AM, Thacker, Dharam wrote:
        > warning 2018/04/10 02:40:59.541 EDT event-server-1
        <ReconnectThread> tid=0x217] Exception occurred while trying
        to connect the system during reconnect
        > org.apache.geode.security.AuthenticationRequiredException:
        Failed to find credentials from
        [host001(event-server-1:3525)<ec>:1026]
        >          at
        
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.attemptToJoin(GMSJoinLeave.java:424)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:318)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:656)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:745)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:181)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:102)
        >          at
        
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:89)
        >          at
        
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1112)
        >          at
        
org.apache.geode.distributed.internal.DistributionManager.<init>(DistributionManager.java:1160)
        >          at
        
org.apache.geode.distributed.internal.DistributionManager.create(DistributionManager.java:531)
        >          at
        
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:687)
        >          at
        
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:299)
        >          at
        
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:202)
        >          at
        
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2675)
        >          at
        
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2508)
        >          at
        
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:983)
        >          at
        
org.apache.geode.distributed.internal.DistributionManager$MyListener.membershipFailure(DistributionManager.java:4307)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1530)
        >          at
        
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$0(GMSMembershipManager.java:2550)
        >          at java.lang.Thread.run(Thread.java:745)

        This message is confidential and subject to terms at:
        http://www.jpmorgan.com/emaildisclaimer
        <http://www.jpmorgan.com/emaildisclaimer> including on
        confidentiality, legal privilege, viruses and monitoring of
        electronic messages. If you are not the intended recipient,
        please delete this message and notify the sender immediately.
        Any unauthorized use is strictly prohibited.

        This message is confidential and subject to terms at:
        http://www.jpmorgan.com/emaildisclaimer
        <http://www.jpmorgan.com/emaildisclaimer> including on
        confidentiality, legal privilege, viruses and monitoring of
        electronic messages. If you are not the intended recipient,
        please delete this message and notify the sender immediately.
        Any unauthorized use is strictly prohibited.



    -- 

    -John

    john.blum10101 (skype)

    This message is confidential and subject to terms at:
    http://www.jpmorgan.com/emaildisclaimer
    <http://www.jpmorgan.com/emaildisclaimer> including on
    confidentiality, legal privilege, viruses and monitoring of
    electronic messages. If you are not the intended recipient,
    please delete this message and notify the sender immediately. Any
    unauthorized use is strictly prohibited.





--
-John
john.blum10101 (skype)