[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[
https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379003#comment-16379003
]
ASF subversion and git services commented on GEODE-4675:
Commit 889da898b5adf1864dd61a05ab01549ae6edcaf5 in geode's branch
refs/heads/feature/GEODE-4685 from [~dschneider]
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=889da89 ]
GEODE-4675: remove checkConnected calls while notifying listeners (#1495)
A call of checkConnected ended up being made from notifyListenersMemberRemoved
which could cause it to throw DistributedSystemDisconnectedException if
the distributed system was being shutdown.
So now the distribution manager is made without this check being done.
Also change to log level from fatal to warn and improved the log message.
> CI failure (suspect strings): DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected reported as fatal
> log message during shutdown
> -
>
> Key: GEODE-4675
> URL: https://issues.apache.org/jira/browse/GEODE-4675
> Project: Geode
> Issue Type: Bug
> Components: regions
>Affects Versions: 1.5.0
>Reporter: Shelley Lynn Hughes-Godfrey
>Assignee: Darrel Schneider
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This failure occurred during CI on geode:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140
> {noformat}
> org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> > testPartitionedParallelPropagationHA FAILED
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> ---
> Found suspect string in log4j at line 9339
> [fatal 2018/02/13 21:12:48.099 UTC tid=891]
> Unexpected exception:
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected.
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499)
> at
> org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
> at
> org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
> at
> org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
> at java.lang.Thread.run(Thread.java:748)
> ---
> {noformat}
> According to the logs, this looks like it occurs during shutdown ...
> {noformat}
> [vm1] [info 2018/02/13 21:12:48.075 UTC
> tid=30] Stopping membership services
> [vm0] [info
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[
https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377589#comment-16377589
]
ASF subversion and git services commented on GEODE-4675:
Commit 889da898b5adf1864dd61a05ab01549ae6edcaf5 in geode's branch
refs/heads/develop from [~dschneider]
[ https://gitbox.apache.org/repos/asf?p=geode.git;h=889da89 ]
GEODE-4675: remove checkConnected calls while notifying listeners (#1495)
A call of checkConnected ended up being made from notifyListenersMemberRemoved
which could cause it to throw DistributedSystemDisconnectedException if
the distributed system was being shutdown.
So now the distribution manager is made without this check being done.
Also change to log level from fatal to warn and improved the log message.
> CI failure (suspect strings): DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected reported as fatal
> log message during shutdown
> -
>
> Key: GEODE-4675
> URL: https://issues.apache.org/jira/browse/GEODE-4675
> Project: Geode
> Issue Type: Bug
> Components: regions
>Affects Versions: 1.5.0
>Reporter: Shelley Lynn Hughes-Godfrey
>Assignee: Darrel Schneider
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This failure occurred during CI on geode:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140
> {noformat}
> org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> > testPartitionedParallelPropagationHA FAILED
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> ---
> Found suspect string in log4j at line 9339
> [fatal 2018/02/13 21:12:48.099 UTC tid=891]
> Unexpected exception:
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected.
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499)
> at
> org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
> at
> org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
> at
> org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
> at java.lang.Thread.run(Thread.java:748)
> ---
> {noformat}
> According to the logs, this looks like it occurs during shutdown ...
> {noformat}
> [vm1] [info 2018/02/13 21:12:48.075 UTC
> tid=30] Stopping membership services
> [vm0] [info 2018/02/13 21:12:48.077 UTC thread 0>
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[
https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377475#comment-16377475
]
xiaojian zhou commented on GEODE-4675:
--
{noformat}
Found in CI develop DistributedTest #162
org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> testParallelPropagationWithUnEqualBucketDivision FAILED
java.lang.AssertionError: Suspicious strings were written to the log during
this run.
Fix the strings or use IgnoredException.addIgnoredException to ignore
.
---
Found suspect string in log4j at line 9083
[fatal 2018/02/24 06:27:04.231 UTC tid=1086] Unexpected
exception:
org.apache.geode.distributed.DistributedSystemDisconnectedException: This
connection to a distributed system has been disconnected.
at
org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1493)
at
org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
at
org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
at
org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
at
org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
at
org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
at
org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
at
org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
at
org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
at
org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
at java.lang.Thread.run(Thread.java:748{noformat}
[
|https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/162#L5a6ba88b:628]
> CI failure (suspect strings): DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected reported as fatal
> log message during shutdown
> -
>
> Key: GEODE-4675
> URL: https://issues.apache.org/jira/browse/GEODE-4675
> Project: Geode
> Issue Type: Bug
> Components: regions
>Affects Versions: 1.5.0
>Reporter: Shelley Lynn Hughes-Godfrey
>Assignee: Darrel Schneider
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This failure occurred during CI on geode:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140
> {noformat}
> org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> > testPartitionedParallelPropagationHA FAILED
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> ---
> Found suspect string in log4j at line 9339
> [fatal 2018/02/13 21:12:48.099 UTC tid=891]
> Unexpected exception:
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected.
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> at
>
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[
https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365966#comment-16365966
]
Shelley Lynn Hughes-Godfrey commented on GEODE-4675:
Seen again in another test in CI:
https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/144
```
org.apache.geode.internal.cache.wan.offheap.ParallelWANPersistenceEnabledGatewaySenderOffHeapDUnitTest
> testPersistentPRWithGatewaySenderPersistenceEnabled_Restart2 FAILED
java.lang.AssertionError: Suspicious strings were written to the log during
this run.
Fix the strings or use IgnoredException.addIgnoredException to ignore.
---
Found suspect string in log4j at line 16228
[fatal 2018/02/15 03:31:53.578 UTC tid=447]
Unexpected exception:
org.apache.geode.distributed.DistributedSystemDisconnectedException: This
connection to a distributed system has been disconnected.
at
org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499)
at
org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
at
org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
at
org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
at
org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
at
org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
at
org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
at
org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
at
org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
at
org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
at java.lang.Thread.run(Thread.java:748)
---
```
> CI failure (suspect strings): DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected reported as fatal
> log message during shutdown
> -
>
> Key: GEODE-4675
> URL: https://issues.apache.org/jira/browse/GEODE-4675
> Project: Geode
> Issue Type: Bug
> Components: regions
>Affects Versions: 1.5.0
>Reporter: Shelley Lynn Hughes-Godfrey
>Assignee: Darrel Schneider
>Priority: Major
>
> This failure occurred during CI on geode:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140
> {noformat}
> org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> > testPartitionedParallelPropagationHA FAILED
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> ---
> Found suspect string in log4j at line 9339
> [fatal 2018/02/13 21:12:48.099 UTC tid=891]
> Unexpected exception:
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected.
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> at
>
[jira] [Commented] (GEODE-4675) CI failure (suspect strings): DistributedSystemDisconnectedException: This connection to a distributed system has been disconnected reported as fatal log message during
[
https://issues.apache.org/jira/browse/GEODE-4675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364668#comment-16364668
]
Bruce Schuchardt commented on GEODE-4675:
-
This was caused by Darrel's recent commit,
fc3e14440afbb66f19b3e6dc79ded168adb8dca3, that added a getDistributionManager()
call to DistributionAdvisor.notifyListenersMemberRemoved(). The method used to
get the distribution manager in this case should avoid checking to see if the
DistributedSystem is closed.
> CI failure (suspect strings): DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected reported as fatal
> log message during shutdown
> -
>
> Key: GEODE-4675
> URL: https://issues.apache.org/jira/browse/GEODE-4675
> Project: Geode
> Issue Type: Bug
> Components: regions
>Affects Versions: 1.5.0
>Reporter: Shelley Lynn Hughes-Godfrey
>Assignee: Darrel Schneider
>Priority: Major
>
> This failure occurred during CI on geode:
> https://concourse.apachegeode-ci.info/teams/main/pipelines/develop/jobs/DistributedTest/builds/140
> {noformat}
> org.apache.geode.internal.cache.wan.concurrent.ConcurrentParallelGatewaySenderOffHeapDUnitTest
> > testPartitionedParallelPropagationHA FAILED
> java.lang.AssertionError: Suspicious strings were written to the log
> during this run.
> Fix the strings or use IgnoredException.addIgnoredException to ignore.
> ---
> Found suspect string in log4j at line 9339
> [fatal 2018/02/13 21:12:48.099 UTC tid=891]
> Unexpected exception:
> org.apache.geode.distributed.DistributedSystemDisconnectedException: This
> connection to a distributed system has been disconnected.
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.checkConnected(InternalDistributedSystem.java:911)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.getDistributionManager(InternalDistributedSystem.java:1499)
> at
> org.apache.geode.internal.cache.AbstractRegion.getDistributionManager(AbstractRegion.java:1757)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.getDistributionManager(DistributionAdvisor.java:380)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.notifyListenersMemberRemoved(DistributionAdvisor.java:1225)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.basicRemoveId(DistributionAdvisor.java:897)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.doRemoveId(DistributionAdvisor.java:964)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor.removeId(DistributionAdvisor.java:926)
> at
> org.apache.geode.internal.cache.CacheDistributionAdvisor.removeId(CacheDistributionAdvisor.java:1183)
> at
> org.apache.geode.internal.cache.partitioned.RegionAdvisor.removeId(RegionAdvisor.java:391)
> at
> org.apache.geode.distributed.internal.DistributionAdvisor$1.memberDeparted(DistributionAdvisor.java:232)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberDepartedEvent.handleEvent(ClusterDistributionManager.java:4198)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4127)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEvent.handleEvent(ClusterDistributionManager.java:4116)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.handleMemberEvent(ClusterDistributionManager.java:2218)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager.access$900(ClusterDistributionManager.java:109)
> at
> org.apache.geode.distributed.internal.ClusterDistributionManager$MemberEventInvoker.run(ClusterDistributionManager.java:2250)
> at java.lang.Thread.run(Thread.java:748)
> ---
> {noformat}
> According to the logs, this looks like it occurs during shutdown ...
> {noformat}
> [vm1] [info 2018/02/13 21:12:48.075 UTC
> tid=30] Stopping membership services
> [vm0] [info 2018/02/13 21:12:48.077 UTC thread 0> tid=398] GMSHealthMonitor server thread exiting
> [vm0] [info 2018/02/13 21:12:48.078 UTC
> tid=30] GMSHealthMonitor serverSocketExecutor is terminated
> [vm3] [info 2018/02/13 21:12:48.079 UTC
> tid=896] received leave request from 172.17.0.2:32771 for
> 172.17.0.2(180):32771
> [vm0] [info 2018/02/13 21:12:48.084 UTC
> tid=30] DistributionManager stopped in 121ms.
> [vm0] [info 2018/02/13
