[
https://issues.apache.org/jira/browse/GEODE-514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007176#comment-15007176
]
Barry Oglesby edited comment on GEODE-514 at 11/16/15 7:47 PM:
---------------------------------------------------------------
This failure reproduced in:
Geode_develop_DistributedTests Build #594 (Nov 16, 2015 5:09:12 AM).
Revision: 88da702593157d8a0c014295cab16149fc088dfc
https://brazil.gemstone.com:8080/job/Geode_develop_DistributedTests/594/testReport/junit/com.gemstone.gemfire.distributed.internal/DistributionManagerDUnitTest/testKickOutSickMember/
{noformat}
Suspicious strings were written to the log during this run.
Fix the strings or use DistributedTestCase.addExpectedException to ignore.
-----------------------------------------------------------------------
Found suspect string in log4j at line 981
[fatal 2015/11/16 08:30:55.682 PST <Quorum Lost Notification> tid=0x7b0]
Possible loss of quorum due to the loss of 2 cache processes:
[timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
-----------------------------------------------------------------------
Found suspect string in log4j at line 985
[fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification> tid=0x308]
Possible loss of quorum due to the loss of 2 cache processes:
[timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
{noformat}
Here is output from the log file:
{noformat}
[setup] START TEST DistributionManagerDUnitTest.testKickOutSickMember
[locator][warn 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (2) for view preparation [timor(5835:locator)<v0>:26739|173]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528] after
12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.210 PST <Failed to ACK member verify
Thread-1> tid=0x2f5] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|174]
[timor(5835:locator)<v0>:26739/53740] crashed mbrs:
[timor(5801)<v163>:50799/3528]] (1 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|174]
[timor(5835:locator)<v0>:26739/53740] crashed mbrs:
[timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:10.365 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|175]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528,
timor(putter:5801)<v175>:55014/18309]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:11.812 PST <P2P message reader@1411ccf1>
tid=0x2f9] Admitting member <timor(putter:5801)<v175>:55014>. Now there are 2
non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|175]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528,
timor(putter:5801)<v175>:55014/18309] after 12,437ms, missing ACKs from
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <Failed to ACK member verify
Thread-1> tid=0x2fc] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|176]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|176]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=3, loss threshold=2 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:22.807 PST <UDP Loopback Message Handler>
tid=0x24] Membership: lead member is now timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:22.808 PST <FD_SOCK Ping thread> tid=0x2fe]
GemFire failure detection is now monitoring timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:23.827 PST <Pooled High Priority Message
Processor 3> tid=0x102] Member timor(putter:5801)<v175>:55014 is equivalent or
in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:24.017 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|177]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069]] (4 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:24.023 PST <P2P message reader@1ddc7ffb>
tid=0x300] Admitting member <timor(sleeper:5856)<v177>:10709>. Now there are 3
non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (4) for view preparation [timor(5835:locator)<v0>:26739|177]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069] after
12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <Failed to ACK member verify
Thread-1> tid=0x302] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:36.459 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|178]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(sleeper:5856)<v177>:10709/55069] crashed mbrs:
[timor(5801)<v163>:50799/3528]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|178]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(sleeper:5856)<v177>:10709/55069] crashed mbrs:
[timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:38.044 PST <Pooled High Priority Message
Processor 3> tid=0x102] Member timor(sleeper:5856)<v177>:10709 is equivalent or
in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:38.084 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(sleeper:5856)<v177>:10709.
[locator]
[locator][info 2015/11/16 08:30:43.080 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(putter:5801)<v175>:55014. Reason=Failed to respond within
ack-wait-threshold
[locator]
[locator][info 2015/11/16 08:30:43.085 PST <VERIFY_SUSPECT.TimerThread>
tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response
received from timor(sleeper:5856)<v177>:10709 in 5001 milliseconds: I believe
it is gone.
[locator]
[locator][info 2015/11/16 08:30:43.238 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|179]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528] crashed mbrs:
[timor(sleeper:5856)<v177>:10709/55069]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:48.082 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <VERIFY_SUSPECT.TimerThread>
tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response
received from timor(sleeper:5856)<v177>:10709 in 5002 milliseconds: I believe
it is gone.
[locator]
[locator][warn 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|179]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528] crashed mbrs:
[timor(sleeper:5856)<v177>:10709/55069] after 12,437ms, missing ACKs from
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <Failed to ACK member verify
Thread-1> tid=0x307] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|180]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069,
timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39]
[locator] member timor(5835:locator)<v0>:26739 has a weight of 3
[locator] member timor(putter:5801)<v175>:55014 has a weight of 15
[locator] member timor(sleeper:5856)<v177>:10709 has a weight of 10
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] quorum
weight calculation: oldWeight=28 failureWeight=10 threshold=51% (failure
weight must be < 14)
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|180]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069,
timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=28, loss threshold=14 and failed weight=20
[locator]
[locator][info 2015/11/16 08:30:55.682 PST <View Message Processor> tid=0x3c]
Member at timor(sleeper:5856)<v177>:10709 unexpectedly left the distributed
cache: departed JGroups view
[locator]
[locator][fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification>
tid=0x308] Possible loss of quorum due to the loss of 2 cache processes:
[timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.691 PST <Pooled High Priority Message
Processor 3> tid=0x102] Member at timor(putter:5801)<v175>:55014 gracefully
left the distributed cache: shutdown message received
[locator]
[locator][info 2015/11/16 08:30:55.835 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|181]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.836 PST <FD_SOCK Ping thread> tid=0x2fe]
Member timor(putter:5801)<v175>:55014 shut down with normal termination.
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP
Connection(25)-10.118.32.89> tid=0x12] Received method:
dunit.DistributedTestCase$5.run with 0 args on object:
dunit.DistributedTestCase$5@7a954221
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP
Connection(25)-10.118.32.89> tid=0x12] Got result: null
[locator] from dunit.DistributedTestCase$5.run with 0 args on object:
dunit.DistributedTestCase$5@7a954221 (took 0 ms)
[locator]
{noformat}
was (Author: barry.oglesby):
This failure reproduced in Geode_develop_DistributedTests Build #594 (Nov 16,
2015 5:09:12 AM).
https://brazil.gemstone.com:8080/job/Geode_develop_DistributedTests/594/testReport/junit/com.gemstone.gemfire.distributed.internal/DistributionManagerDUnitTest/testKickOutSickMember/
{noformat}
Suspicious strings were written to the log during this run.
Fix the strings or use DistributedTestCase.addExpectedException to ignore.
-----------------------------------------------------------------------
Found suspect string in log4j at line 981
[fatal 2015/11/16 08:30:55.682 PST <Quorum Lost Notification> tid=0x7b0]
Possible loss of quorum due to the loss of 2 cache processes:
[timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
-----------------------------------------------------------------------
Found suspect string in log4j at line 985
[fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification> tid=0x308]
Possible loss of quorum due to the loss of 2 cache processes:
[timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
{noformat}
Here is output from the log file:
{noformat}
[setup] START TEST DistributionManagerDUnitTest.testKickOutSickMember
[locator][warn 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (2) for view preparation [timor(5835:locator)<v0>:26739|173]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528] after
12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.209 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:10.210 PST <Failed to ACK member verify
Thread-1> tid=0x2f5] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|174]
[timor(5835:locator)<v0>:26739/53740] crashed mbrs:
[timor(5801)<v163>:50799/3528]] (1 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|174]
[timor(5835:locator)<v0>:26739/53740] crashed mbrs:
[timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:10.211 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:10.365 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|175]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528,
timor(putter:5801)<v175>:55014/18309]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:11.812 PST <P2P message reader@1411ccf1>
tid=0x2f9] Admitting member <timor(putter:5801)<v175>:55014>. Now there are 2
non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|175]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528,
timor(putter:5801)<v175>:55014/18309] after 12,437ms, missing ACKs from
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:22.805 PST <Failed to ACK member verify
Thread-1> tid=0x2fc] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|176]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|176]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:22.806 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=3, loss threshold=2 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:22.807 PST <UDP Loopback Message Handler>
tid=0x24] Membership: lead member is now timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:22.808 PST <FD_SOCK Ping thread> tid=0x2fe]
GemFire failure detection is now monitoring timor(putter:5801)<v175>:55014
[locator]
[locator][info 2015/11/16 08:30:23.827 PST <Pooled High Priority Message
Processor 3> tid=0x102] Member timor(putter:5801)<v175>:55014 is equivalent or
in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:24.017 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|177]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069]] (4 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:24.023 PST <P2P message reader@1ddc7ffb>
tid=0x300] Admitting member <timor(sleeper:5856)<v177>:10709>. Now there are 3
non-admin member(s).
[locator]
[locator][warn 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (4) for view preparation [timor(5835:locator)<v0>:26739|177]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528, timor(sleeper:5856)<v177>:10709/55069] after
12,437ms, missing ACKs from [timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:36.458 PST <Failed to ACK member verify
Thread-1> tid=0x302] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:36.459 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|178]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(sleeper:5856)<v177>:10709/55069] crashed mbrs:
[timor(5801)<v163>:50799/3528]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|178]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(sleeper:5856)<v177>:10709/55069] crashed mbrs:
[timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:36.460 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=18, loss threshold=9 and failed weight=10
[locator]
[locator][info 2015/11/16 08:30:38.044 PST <Pooled High Priority Message
Processor 3> tid=0x102] Member timor(sleeper:5856)<v177>:10709 is equivalent or
in the same redundancy zone.
[locator]
[locator][info 2015/11/16 08:30:38.084 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(sleeper:5856)<v177>:10709.
[locator]
[locator][info 2015/11/16 08:30:43.080 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(putter:5801)<v175>:55014. Reason=Failed to respond within
ack-wait-threshold
[locator]
[locator][info 2015/11/16 08:30:43.085 PST <VERIFY_SUSPECT.TimerThread>
tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response
received from timor(sleeper:5856)<v177>:10709 in 5001 milliseconds: I believe
it is gone.
[locator]
[locator][info 2015/11/16 08:30:43.238 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|179]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528] crashed mbrs:
[timor(sleeper:5856)<v177>:10709/55069]] (3 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:48.082 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <UDP ucast receiver> tid=0x25]
Received Suspect notification for member(s) [timor(sleeper:5856)<v177>:10709]
from timor(putter:5801)<v175>:55014. Reason=still being suspected
[locator]
[locator][info 2015/11/16 08:30:53.085 PST <VERIFY_SUSPECT.TimerThread>
tid=0x305] timor(5835:locator)<v0>:26739: No suspect verification response
received from timor(sleeper:5856)<v177>:10709 in 5002 milliseconds: I believe
it is gone.
[locator]
[locator][warn 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] failed to
collect all ACKs (3) for view preparation [timor(5835:locator)<v0>:26739|179]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309,
timor(5801)<v163>:50799/3528] crashed mbrs:
[timor(sleeper:5856)<v177>:10709/55069] after 12,437ms, missing ACKs from
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <ViewHandler> tid=0x39] attempting
to contact members that did not respond to view preparation:
[timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.679 PST <Failed to ACK member verify
Thread-1> tid=0x307] Checking member timor(5801)<v163>:50799
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|180]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069,
timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39]
[locator] member timor(5835:locator)<v0>:26739 has a weight of 3
[locator] member timor(putter:5801)<v175>:55014 has a weight of 15
[locator] member timor(sleeper:5856)<v177>:10709 has a weight of 10
[locator]
[locator][info 2015/11/16 08:30:55.680 PST <ViewHandler> tid=0x39] quorum
weight calculation: oldWeight=28 failureWeight=10 threshold=51% (failure
weight must be < 14)
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler>
tid=0x24] Membership: received new view [timor(5835:locator)<v0>:26739|180]
[timor(5835:locator)<v0>:26739/53740, timor(putter:5801)<v175>:55014/18309]
crashed mbrs: [timor(sleeper:5856)<v177>:10709/55069,
timor(5801)<v163>:50799/3528]
[locator]
[locator][info 2015/11/16 08:30:55.681 PST <UDP Loopback Message Handler>
tid=0x24] old membership weight=28, loss threshold=14 and failed weight=20
[locator]
[locator][info 2015/11/16 08:30:55.682 PST <View Message Processor> tid=0x3c]
Member at timor(sleeper:5856)<v177>:10709 unexpectedly left the distributed
cache: departed JGroups view
[locator]
[locator][fatal 2015/11/16 08:30:55.683 PST <Quorum Lost Notification>
tid=0x308] Possible loss of quorum due to the loss of 2 cache processes:
[timor(sleeper:5856)<v177>:10709, timor(5801)<v163>:50799]
[locator]
[locator][info 2015/11/16 08:30:55.691 PST <Pooled High Priority Message
Processor 3> tid=0x102] Member at timor(putter:5801)<v175>:55014 gracefully
left the distributed cache: shutdown message received
[locator]
[locator][info 2015/11/16 08:30:55.835 PST <ViewHandler> tid=0x39] Membership:
sending new view [[timor(5835:locator)<v0>:26739|181]
[timor(5835:locator)<v0>:26739/53740, timor(5801)<v163>:50799/3528]] (2 mbrs)
[locator]
[locator]
[locator][info 2015/11/16 08:30:55.836 PST <FD_SOCK Ping thread> tid=0x2fe]
Member timor(putter:5801)<v175>:55014 shut down with normal termination.
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP
Connection(25)-10.118.32.89> tid=0x12] Received method:
dunit.DistributedTestCase$5.run with 0 args on object:
dunit.DistributedTestCase$5@7a954221
[locator]
[locator][info 2015/11/16 08:30:55.853 PST <RMI TCP
Connection(25)-10.118.32.89> tid=0x12] Got result: null
[locator] from dunit.DistributedTestCase$5.run with 0 args on object:
dunit.DistributedTestCase$5@7a954221 (took 0 ms)
[locator]
{noformat}
> DistributionManagerDUnitTest.testKickOutSickMember has suspect string
> ---------------------------------------------------------------------
>
> Key: GEODE-514
> URL: https://issues.apache.org/jira/browse/GEODE-514
> Project: Geode
> Issue Type: Bug
> Reporter: xiaojian zhou
> Labels: CI
>
> revision 0cc9d895b9f4465138d0fa223b0a0cadc1107893
> {noformat}
> java.lang.AssertionError: Suspicious strings were written to the log during
> this run.
> Fix the strings or use DistributedTestCase.addExpectedException to ignore.
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 990
> [fatal 2015/10/29 06:28:31.322 PDT <Quorum Lost Notification> tid=0xada]
> Possible loss of quorum due to the loss of 2 cache processes:
> [cc8-rh64(16364)<v658>:1121, cc8-rh64(sleeper:16419)<v680>:30890]
> -----------------------------------------------------------------------
> Found suspect string in log4j at line 992
> [fatal 2015/10/29 06:28:31.323 PDT <Quorum Lost Notification> tid=0xe04]
> Possible loss of quorum due to the loss of 2 cache processes:
> [cc8-rh64(16364)<v658>:1121, cc8-rh64(sleeper:16419)<v680>:30890]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
