[
https://issues.apache.org/jira/browse/GEODE-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated GEODE-4451:
----------------------------------
Labels: pull-request-available (was: )
> When a gateway sender is created using gfsh and fails to connect due to
> AuthenticationFailedException, it is left in an inconsistent state
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: GEODE-4451
> URL: https://issues.apache.org/jira/browse/GEODE-4451
> Project: Geode
> Issue Type: Bug
> Components: wan
> Reporter: Barry Oglesby
> Priority: Major
> Labels: pull-request-available
>
> If I attempt to create a gateway sender with invalid credentials using gfsh,
> it fails like:
> {noformat}
> (2) Executing - create gateway-sender --remote-distributed-system-id=1
> --id=ny --dispatcher-threads=1 --parallel=true
> Member | Status
> ------ |
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> ln-1 | ERROR: Could not start a gateway sender ny because of exception Could
> not start a gateway sender 84 because of exception
> org.apache.geode.security.AuthenticationFailedException: Authentication
> error. Please check your credentials.
> {noformat}
> The 84 should be the gateway sender id; instead its the thread id (from
> \{{ConcurrentParallelGatewaySenderEventProcessor.waitForRunningStatus}}.
> In the server log, I see 5 AuthenticationFailedException stacks like (one per
> dispatcher-thread):
> {noformat}
> [severe 2018/01/31 16:13:12.638 PST ln-1 <Event Processor for
> GatewaySender_ny_0> tid=0x55] Message dispatch failed due to unexpected
> exception..
> org.apache.geode.internal.cache.wan.GatewaySenderException:
> org.apache.geode.security.AuthenticationFailedException: Authentication
> error. Please check your credentials., caused by
> org.apache.geode.security.AuthenticationFailedException: Authentication
> error. Please check your credentials.
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:432)
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.<init>(GatewaySenderEventRemoteDispatcher.java:75)
> at
> org.apache.geode.internal.cache.wan.parallel.RemoteParallelGatewaySenderEventProcessor.initializeEventDispatcher(RemoteParallelGatewaySenderEventProcessor.java:74)
> at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.setRunningStatus(AbstractGatewaySenderEventProcessor.java:1078)
> at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1050)
> Caused by: org.apache.geode.security.AuthenticationFailedException:
> Authentication error. Please check your credentials.
> at
> org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397)
> at
> org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253)
> at
> org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118)
> at
> org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136)
> at
> org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259)
> at
> org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206)
> at
> org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902)
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:388)
> ... 4 more
> {noformat}
> A thread dump shows the Event Processor threads are gone, but 5
> BatchRemovalThreads like this exist:
> {noformat}
> "BatchRemovalThread" #96 daemon prio=10 os_prio=31 tid=0x00007fc1dd009000
> nid=0xcb03 waiting on condition [0x000070000e245000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000007a04043c8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
> at
> org.apache.geode.internal.util.concurrent.StoppableCondition.await(StoppableCondition.java:61)
> at
> org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue$BatchRemovalThread.run(ParallelGatewaySenderQueue.java:1613)
> {noformat}
> list gateways shows the sender in the \{{Not Running}} state:
> {noformat}
> GatewaySender Id | Member | Remote Cluster Id | Type | Status | Queued Events
> | Receiver Location
> ---------------- | -------------------------------- | ----------------- |
> -------- | ----------- | ------------- | -----------------
> ny | 192.168.2.4(ln-1:13883)<v3>:1027 | 1 | Parallel | Not Running | 0 | null
> {noformat}
> The sender is left in an inconsistent state. It should either be completely
> gone or retrying to connect.
> {\{GatewaySenderEventRemoteDispatcher.<init>}} catches the
> \{{GatewaySenderException}} attempting to \{{initializeConnection}} and
> rethrows it if is caused by \{{GemFireSecurityException}}:
> {noformat}
> public GatewaySenderEventRemoteDispatcher(AbstractGatewaySenderEventProcessor
> eventProcessor) {
> this.processor = eventProcessor;
> this.sender = eventProcessor.getSender();
> // this.ackReaderThread = new AckReaderThread(sender);
> try {
> initializeConnection();
> } catch (GatewaySenderException e) {
> if (e.getCause() instanceof GemFireSecurityException) {
> throw e;
> }
> }
> }
> {noformat}
> {\{AbstractGatewaySenderEventProcessor.run}} catches that exception and logs
> severe message and stack above.
> If I change this area of code to log the warning instead of stopping the
> Event Processor threads, I see this exception after puts are done:
> {noformat}
> [severe 2018/01/31 17:37:16.324 PST ln-1 <Event Processor for
> GatewaySender_ny_0> tid=0x49] Stopping the processor because the following
> exception occurred while processing a batch:
> org.apache.geode.internal.cache.wan.GatewaySenderException:
> org.apache.geode.security.AuthenticationFailedException: Authentication
> error. Please check your credentials., caused by
> org.apache.geode.security.AuthenticationFailedException: Authentication
> error. Please check your credentials.
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:431)
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.getConnection(GatewaySenderEventRemoteDispatcher.java:320)
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher._dispatchBatch(GatewaySenderEventRemoteDispatcher.java:198)
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.dispatchBatch(GatewaySenderEventRemoteDispatcher.java:147)
> at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:609)
> at
> org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1051)
> Caused by: org.apache.geode.security.AuthenticationFailedException:
> Authentication error. Please check your credentials.
> at
> org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397)
> at
> org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253)
> at
> org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118)
> at
> org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136)
> at
> org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259)
> at
> org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206)
> at
> org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902)
> at
> org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:387)
> ... 5 more
> {noformat}
> That is coming from \{{GatewaySenderEventRemoteDispatcher.dispatchBatch}}:
> {noformat}
> } catch (GatewaySenderException ge) {
> if (...) {
> ...
> } else {
> logger.fatal(
> LocalizedMessage.create(
>
> LocalizedStrings.GatewayEventRemoteDispatcher_STOPPING_THE_PROCESSOR_BECAUSE_THE_FOLLOWING_EXCEPTION_OCCURRED_WHILE_PROCESSING_A_BATCH),
> ge);
> this.processor.setIsStopped(true);
> }
> {noformat}
> So, \{{AuthenticationFailedExceptions}} are definitely meant to stop the
> sender.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
