[jira] [Updated] (CASSANDRA-9129) HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11 and 2.0.12
[
https://issues.apache.org/jira/browse/CASSANDRA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-9129:
---
Component/s: Observability
> HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11
> and 2.0.12
> ---
>
> Key: CASSANDRA-9129
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9129
> Project: Cassandra
> Issue Type: Bug
> Components: Observability
> Environment: Ubuntu 12.04.5 LTS
> AWS (m3.xlarge)
> 15G RAM
> 4 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
> Cassandra 2.0.14
>Reporter: Russ Lavoie
>Assignee: Sam Tunnicliffe
> Fix For: 2.0.17, 2.1.9
>
> Attachments: 9129-2.0.txt
>
>
> Upgrading from Cassandra 2.0.11 or 2.0.12 to 2.0.14 I am seeing a pending
> hinted hand off that never clears. New hinted hand offs that go into pending
> waiting for a node to come up clear as expected. But 1 always remains.
> I through the following steps.
> 1) stop cassandra
> 2) Upgrade cassandra to 2.0.14
> 3) Start cassandra
> 4) nodetool tpstats
> There are no errors in the logs, to help with this issue. I ran a few
> nodetool commands to get some data and pasted them below:
> Below is what is shown after running nodetool status on each node in the ring
> {code}Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> UN279.8 MB 256 34.9% rack1
> UN279.79 MB 256 33.0% rack1
> UN279.87 MB 256 32.1% rack1
> {code}
> Below is what is shown after running nodetool tpstats on each node in the
> ring showing a single HintedHandoff in pending status that never clears
> {code}
> Pool NameActive Pending Completed Blocked All
> time blocked
> ReadStage 0 0 14550 0
> 0
> RequestResponseStage 0 0 113040 0
> 0
> MutationStage 0 0 168873 0
> 0
> ReadRepairStage 0 0 1147 0
> 0
> ReplicateOnWriteStage 0 0 0 0
> 0
> GossipStage 0 0 232112 0
> 0
> CacheCleanupExecutor 0 0 0 0
> 0
> MigrationStage0 0 0 0
> 0
> MemoryMeter 0 0 6 0
> 0
> FlushWriter 0 0 38 0
> 0
> ValidationExecutor0 0 0 0
> 0
> InternalResponseStage 0 0 0 0
> 0
> AntiEntropyStage 0 0 0 0
> 0
> MemtablePostFlusher 0 0 1333 0
> 0
> MiscStage 0 0 0 0
> 0
> PendingRangeCalculator0 0 6 0
> 0
> CompactionExecutor0 0178 0
> 0
> commitlog_archiver0 0 0 0
> 0
> HintedHandoff 0 1133 0
> 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> PAGED_RANGE 0
> BINARY 0
> READ 0
> MUTATION 0
> _TRACE 0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
> {code}
> Below is what is shown after running nodetool cfstats system.hints on all 3
> nodes.
> {code}
> Keyspace: system
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Table: hints
> SSTable count: 0
> Space used (live), bytes: 0
> Space used (total), bytes: 0
> Off heap memory used (total), bytes: 0
> SSTable Compression Ratio: 0.0
> Number of keys (estimate): 0
> Memtable cell count: 0
> Memtable data size, bytes: 0
> Memtable switch count: 0
> Local read count: 0
> Local read latency: 0.000 ms
> Local write count: 0
> Local write latency: 0.000 ms
>
[jira] [Updated] (CASSANDRA-9129) HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11 and 2.0.12
[
https://issues.apache.org/jira/browse/CASSANDRA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-9129:
---
Fix Version/s: 2.1.9
HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11
and 2.0.12
---
Key: CASSANDRA-9129
URL: https://issues.apache.org/jira/browse/CASSANDRA-9129
Project: Cassandra
Issue Type: Bug
Environment: Ubuntu 12.04.5 LTS
AWS (m3.xlarge)
15G RAM
4 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Cassandra 2.0.14
Reporter: Russ Lavoie
Assignee: Sam Tunnicliffe
Fix For: 2.1.9, 2.0.17
Attachments: 9129-2.0.txt
Upgrading from Cassandra 2.0.11 or 2.0.12 to 2.0.14 I am seeing a pending
hinted hand off that never clears. New hinted hand offs that go into pending
waiting for a node to come up clear as expected. But 1 always remains.
I through the following steps.
1) stop cassandra
2) Upgrade cassandra to 2.0.14
3) Start cassandra
4) nodetool tpstats
There are no errors in the logs, to help with this issue. I ran a few
nodetool commands to get some data and pasted them below:
Below is what is shown after running nodetool status on each node in the ring
{code}Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN NODE1 279.8 MB 256 34.9% HOSTID rack1
UN NODE2 279.79 MB 256 33.0% HOSTID rack1
UN NODE3 279.87 MB 256 32.1% HOSTID rack1
{code}
Below is what is shown after running nodetool tpstats on each node in the
ring showing a single HintedHandoff in pending status that never clears
{code}
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 0 0 14550 0
0
RequestResponseStage 0 0 113040 0
0
MutationStage 0 0 168873 0
0
ReadRepairStage 0 0 1147 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 232112 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage0 0 0 0
0
MemoryMeter 0 0 6 0
0
FlushWriter 0 0 38 0
0
ValidationExecutor0 0 0 0
0
InternalResponseStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
MemtablePostFlusher 0 0 1333 0
0
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 6 0
0
CompactionExecutor0 0178 0
0
commitlog_archiver0 0 0 0
0
HintedHandoff 0 1133 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{code}
Below is what is shown after running nodetool cfstats system.hints on all 3
nodes.
{code}
Keyspace: system
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Table: hints
SSTable count: 0
Space used (live), bytes: 0
Space used (total), bytes: 0
Off heap memory used (total), bytes: 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size, bytes: 0
Memtable switch count: 0
Local read count: 0
Local read latency: 0.000 ms
Local write count: 0
Local write latency: 0.000 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
[jira] [Updated] (CASSANDRA-9129) HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11 and 2.0.12
[
https://issues.apache.org/jira/browse/CASSANDRA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-9129:
--
Reviewer: Aleksey Yeschenko
HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11
and 2.0.12
---
Key: CASSANDRA-9129
URL: https://issues.apache.org/jira/browse/CASSANDRA-9129
Project: Cassandra
Issue Type: Bug
Environment: Ubuntu 12.04.5 LTS
AWS (m3.xlarge)
15G RAM
4 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Cassandra 2.0.14
Reporter: Russ Lavoie
Assignee: Sam Tunnicliffe
Fix For: 2.0.x
Attachments: 9129-2.0.txt
Upgrading from Cassandra 2.0.11 or 2.0.12 to 2.0.14 I am seeing a pending
hinted hand off that never clears. New hinted hand offs that go into pending
waiting for a node to come up clear as expected. But 1 always remains.
I through the following steps.
1) stop cassandra
2) Upgrade cassandra to 2.0.14
3) Start cassandra
4) nodetool tpstats
There are no errors in the logs, to help with this issue. I ran a few
nodetool commands to get some data and pasted them below:
Below is what is shown after running nodetool status on each node in the ring
{code}Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN NODE1 279.8 MB 256 34.9% HOSTID rack1
UN NODE2 279.79 MB 256 33.0% HOSTID rack1
UN NODE3 279.87 MB 256 32.1% HOSTID rack1
{code}
Below is what is shown after running nodetool tpstats on each node in the
ring showing a single HintedHandoff in pending status that never clears
{code}
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 0 0 14550 0
0
RequestResponseStage 0 0 113040 0
0
MutationStage 0 0 168873 0
0
ReadRepairStage 0 0 1147 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 232112 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage0 0 0 0
0
MemoryMeter 0 0 6 0
0
FlushWriter 0 0 38 0
0
ValidationExecutor0 0 0 0
0
InternalResponseStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
MemtablePostFlusher 0 0 1333 0
0
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 6 0
0
CompactionExecutor0 0178 0
0
commitlog_archiver0 0 0 0
0
HintedHandoff 0 1133 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{code}
Below is what is shown after running nodetool cfstats system.hints on all 3
nodes.
{code}
Keyspace: system
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Table: hints
SSTable count: 0
Space used (live), bytes: 0
Space used (total), bytes: 0
Off heap memory used (total), bytes: 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size, bytes: 0
Memtable switch count: 0
Local read count: 0
Local read latency: 0.000 ms
Local write count: 0
Local write latency: 0.000 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
[jira] [Updated] (CASSANDRA-9129) HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11 and 2.0.12
[
https://issues.apache.org/jira/browse/CASSANDRA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-9129:
---
Attachment: 9129-2.0.txt
The attached patch adds a second ScheduledExecutor to HintedHandoffManager.
This reverts back to something more like pre-8285 behaviour where only the
actual hint delivery tasks are run on the executor exposed through JMX
tpstats.
The periodic scheduling task, plus deletions from truncations of the hints
table are run on this new executor, so the stats will go back to just reporting
the number of hints delivered/pending etc.
Cassci:
[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-9219-testall/]
[dtests|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-9219-dtest/]
HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11
and 2.0.12
---
Key: CASSANDRA-9129
URL: https://issues.apache.org/jira/browse/CASSANDRA-9129
Project: Cassandra
Issue Type: Bug
Environment: Ubuntu 12.04.5 LTS
AWS (m3.xlarge)
15G RAM
4 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Cassandra 2.0.14
Reporter: Russ Lavoie
Assignee: Sam Tunnicliffe
Fix For: 2.0.x
Attachments: 9129-2.0.txt
Upgrading from Cassandra 2.0.11 or 2.0.12 to 2.0.14 I am seeing a pending
hinted hand off that never clears. New hinted hand offs that go into pending
waiting for a node to come up clear as expected. But 1 always remains.
I through the following steps.
1) stop cassandra
2) Upgrade cassandra to 2.0.14
3) Start cassandra
4) nodetool tpstats
There are no errors in the logs, to help with this issue. I ran a few
nodetool commands to get some data and pasted them below:
Below is what is shown after running nodetool status on each node in the ring
{code}Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN NODE1 279.8 MB 256 34.9% HOSTID rack1
UN NODE2 279.79 MB 256 33.0% HOSTID rack1
UN NODE3 279.87 MB 256 32.1% HOSTID rack1
{code}
Below is what is shown after running nodetool tpstats on each node in the
ring showing a single HintedHandoff in pending status that never clears
{code}
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 0 0 14550 0
0
RequestResponseStage 0 0 113040 0
0
MutationStage 0 0 168873 0
0
ReadRepairStage 0 0 1147 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 232112 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage0 0 0 0
0
MemoryMeter 0 0 6 0
0
FlushWriter 0 0 38 0
0
ValidationExecutor0 0 0 0
0
InternalResponseStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
MemtablePostFlusher 0 0 1333 0
0
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 6 0
0
CompactionExecutor0 0178 0
0
commitlog_archiver0 0 0 0
0
HintedHandoff 0 1133 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{code}
Below is what is shown after running nodetool cfstats system.hints on all 3
nodes.
{code}
Keyspace: system
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Table: hints
SSTable count: 0
Space used
[jira] [Updated] (CASSANDRA-9129) HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11 and 2.0.12
[
https://issues.apache.org/jira/browse/CASSANDRA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9129:
---
Assignee: Aleksey Yeschenko
HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11
and 2.0.12
---
Key: CASSANDRA-9129
URL: https://issues.apache.org/jira/browse/CASSANDRA-9129
Project: Cassandra
Issue Type: Bug
Environment: Ubuntu 12.04.5 LTS
AWS (m3.xlarge)
15G RAM
4 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Cassandra 2.0.14
Reporter: Russ Lavoie
Assignee: Aleksey Yeschenko
Fix For: 2.0.15
Upgrading from Cassandra 2.0.11 or 2.0.12 to 2.0.14 I am seeing a pending
hinted hand off that never clears. New hinted hand offs that go into pending
waiting for a node to come up clear as expected. But 1 always remains.
I through the following steps.
1) stop cassandra
2) Upgrade cassandra to 2.0.14
3) Start cassandra
4) nodetool tpstats
There are no errors in the logs, to help with this issue. I ran a few
nodetool commands to get some data and pasted them below:
Below is what is shown after running nodetool status on each node in the ring
{code}Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN NODE1 279.8 MB 256 34.9% HOSTID rack1
UN NODE2 279.79 MB 256 33.0% HOSTID rack1
UN NODE3 279.87 MB 256 32.1% HOSTID rack1
{code}
Below is what is shown after running nodetool tpstats on each node in the
ring showing a single HintedHandoff in pending status that never clears
{code}
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 0 0 14550 0
0
RequestResponseStage 0 0 113040 0
0
MutationStage 0 0 168873 0
0
ReadRepairStage 0 0 1147 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 232112 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage0 0 0 0
0
MemoryMeter 0 0 6 0
0
FlushWriter 0 0 38 0
0
ValidationExecutor0 0 0 0
0
InternalResponseStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
MemtablePostFlusher 0 0 1333 0
0
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 6 0
0
CompactionExecutor0 0178 0
0
commitlog_archiver0 0 0 0
0
HintedHandoff 0 1133 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{code}
Below is what is shown after running nodetool cfstats system.hints on all 3
nodes.
{code}
Keyspace: system
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Table: hints
SSTable count: 0
Space used (live), bytes: 0
Space used (total), bytes: 0
Off heap memory used (total), bytes: 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size, bytes: 0
Memtable switch count: 0
Local read count: 0
Local read latency: 0.000 ms
Local write count: 0
Local write latency: 0.000 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space
[jira] [Updated] (CASSANDRA-9129) HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11 and 2.0.12
[
https://issues.apache.org/jira/browse/CASSANDRA-9129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9129:
---
Fix Version/s: (was: 2.0.14)
2.0.15
HintedHandoff in pending state forever after upgrading to 2.0.14 from 2.0.11
and 2.0.12
---
Key: CASSANDRA-9129
URL: https://issues.apache.org/jira/browse/CASSANDRA-9129
Project: Cassandra
Issue Type: Bug
Environment: Ubuntu 12.04.5 LTS
AWS (m3.xlarge)
15G RAM
4 core Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Cassandra 2.0.14
Reporter: Russ Lavoie
Fix For: 2.0.15
Upgrading from Cassandra 2.0.11 or 2.0.12 to 2.0.14 I am seeing a pending
hinted hand off that never clears. New hinted hand offs that go into pending
waiting for a node to come up clear as expected. But 1 always remains.
I through the following steps.
1) stop cassandra
2) Upgrade cassandra to 2.0.14
3) Start cassandra
4) nodetool tpstats
There are no errors in the logs, to help with this issue. I ran a few
nodetool commands to get some data and pasted them below:
Below is what is shown after running nodetool status on each node in the ring
{code}Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN NODE1 279.8 MB 256 34.9% HOSTID rack1
UN NODE2 279.79 MB 256 33.0% HOSTID rack1
UN NODE3 279.87 MB 256 32.1% HOSTID rack1
{code}
Below is what is shown after running nodetool tpstats on each node in the
ring showing a single HintedHandoff in pending status that never clears
{code}
Pool NameActive Pending Completed Blocked All
time blocked
ReadStage 0 0 14550 0
0
RequestResponseStage 0 0 113040 0
0
MutationStage 0 0 168873 0
0
ReadRepairStage 0 0 1147 0
0
ReplicateOnWriteStage 0 0 0 0
0
GossipStage 0 0 232112 0
0
CacheCleanupExecutor 0 0 0 0
0
MigrationStage0 0 0 0
0
MemoryMeter 0 0 6 0
0
FlushWriter 0 0 38 0
0
ValidationExecutor0 0 0 0
0
InternalResponseStage 0 0 0 0
0
AntiEntropyStage 0 0 0 0
0
MemtablePostFlusher 0 0 1333 0
0
MiscStage 0 0 0 0
0
PendingRangeCalculator0 0 6 0
0
CompactionExecutor0 0178 0
0
commitlog_archiver0 0 0 0
0
HintedHandoff 0 1133 0
0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{code}
Below is what is shown after running nodetool cfstats system.hints on all 3
nodes.
{code}
Keyspace: system
Read Count: 0
Read Latency: NaN ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Table: hints
SSTable count: 0
Space used (live), bytes: 0
Space used (total), bytes: 0
Off heap memory used (total), bytes: 0
SSTable Compression Ratio: 0.0
Number of keys (estimate): 0
Memtable cell count: 0
Memtable data size, bytes: 0
Memtable switch count: 0
Local read count: 0
Local read latency: 0.000 ms
Local write count: 0
Local write latency: 0.000 ms
Pending tasks: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used,
