[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273930#comment-14273930
]
Richard Low commented on CASSANDRA-5913:
I have seen this on 1.2.19 and 2.0.9. I suspect the root cause is
CASSANDRA-6125.
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141276#comment-14141276
]
Philip Thompson commented on CASSANDRA-5913:
Can you reproduce this issue on a newer 1.2 release, or on either 2.0 or 2.1?
This hasn't been touched in a year.
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789631#comment-13789631
]
Matt Kapilevich commented on CASSANDRA-5913:
We're having the same issue. One of our nodes crashed due to bad disk, but it
was still marked UP in nodetool ring.
{code}
54.244.26.215 2b Up Normal 453.39 GB 6.25%
138239711561631250781995934269155835904
{code}
When we run gossipinfo, here's what we see:
{code}
/54.244.26.215
RELEASE_VERSION:1.2.6.6
RACK:2b
SCHEMA:1f295f0d-e536-324f-8e54-10acb1844f90
DC:us-west-2
INTERNAL_IP:10.89.8.232
SEVERITY:0.0
NET_VERSION:6
HOST_ID:89c15817-de39-4fe5-86f8-5f1cf56e2b17
RPC_ADDRESS:0.0.0.0
{code}
STATUS is not set at all, and SEVERITY is 0.0. We're running 1.2.6
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789654#comment-13789654
]
Chris Burroughs commented on CASSANDRA-5913:
I looked into this a bit, U is basically reaching all the way into
Gossiper.instance.getLiveMembers(). N is not joining, leaving or moving (and
is also reaching deep in). Changing what getLiveMembers() does vis a vis no
STATUS seems risky (even if it was right, which I'm not sure it is). I also
don't see anything in StorageServiceMBean that maps to what we need for a quick
cosmetic fix.
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789782#comment-13789782
]
Brandon Williams commented on CASSANDRA-5913:
-
What especially doesn't make sense here is that, as you say, everything is
reaching deep into the gossiper, and getLiveMembers is essentially controlled
by notifications from the FailureDetector. Nothing has changed in the FD in a
long time.
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776384#comment-13776384
]
Jeremy Hanna commented on CASSANDRA-5913:
-
Another case where there was no gossip STATUS for node X across the cluster
except for on node X. In that case there was a recently bootstrapped node and
it couldn't see node X in either nodetool describering nor in nodetool status
(1.2.6). It could be seen in nodetool status in the rest of the cluster
however. Anyway, so restarting node X refreshed the gossip information and
restored the STATUS part - as well as put it in the describering and nodetool
status output.
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13774548#comment-13774548
]
Chris Burroughs commented on CASSANDRA-5913:
A status like this also results in UN from CASSANDRA-6082
{noformat}
[ldc01] out: /10.20.2.34
[ldc01] out: SCHEMA:6fe64f2c-c74f-37a5-a40c-5be27078ade5
[ldc01] out: RPC_ADDRESS:0.0.0.0
[ldc01] out: NET_VERSION:6
[ldc01] out: RACK:rack1
[ldc01] out: LOAD:5.2676994631E10
[ldc01] out: SEVERITY:0.0
[ldc01] out: HOST_ID:da96337b-f4ac-47d6-9642-ab37feea81f4
[ldc01] out: DC:datacenter1
[ldc01] out: RELEASE_VERSION:1.2.9-atm2
{noformat}
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746518#comment-13746518
]
Brandon Williams commented on CASSANDRA-5913:
-
I'm going to need steps to reproduce, since not having a STATUS state should be
impossible, especially since STATUS and TOKENS are added in the same place
(tokens is hidden from gossipinfo on purpose, however.) LOAD and SEVERITY
appear to be missing as well.
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5913) Nodes with no gossip STATUS shown as UN by nodetool:status
[
https://issues.apache.org/jira/browse/CASSANDRA-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746520#comment-13746520
]
Chris Burroughs commented on CASSANDRA-5913:
Sorry, trying to write up separate discrete tickets instead of one giant
CASSANDRA AND I HAD A BAD MORNING. It was from a replace_node on a down node
that failed as in CASSANDRA-5914
Nodes with no gossip STATUS shown as UN by nodetool:status
Key: CASSANDRA-5913
URL: https://issues.apache.org/jira/browse/CASSANDRA-5913
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 1.2.8
Reporter: Chris Burroughs
Priority: Minor
I have no idea if this is a valid situation or a larger problem, but either
way nodetool status should not make it look like everything is a-okay.
From nt:gossipinfo:
{noformat}
/64.215.255.182
RACK:NOP
NET_VERSION:6
HOST_ID:4f3b214b-b03e-46eb-8214-5fab2662a06b
RELEASE_VERSION:1.2.8
DC:IAD
INTERNAL_IP:10.15.2.182
SCHEMA:59adb24e-f3cd-3e02-97f0-5b395827453f
RPC_ADDRESS:0.0.0.0
{noformat}
{noformat}
$ ./bin/nt.sh status | grep -i 4055109d-800d-4743-8efa-4ecfff883463
UN 64.215.255.182 63.84 GB 256 2.5%
4055109d-800d-4743-8efa-4ecfff883463 NOP
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
