[
https://issues.apache.org/jira/browse/CASSANDRA-13020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Ivanov updated CASSANDRA-13020:
-----------------------------------------
Summary: Stuck in LEAVING state (Transferring all hints to null) (was:
Transferring all hints to null)
> Stuck in LEAVING state (Transferring all hints to null)
> -------------------------------------------------------
>
> Key: CASSANDRA-13020
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13020
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: v3.0.9
> Reporter: Aleksandr Ivanov
> Labels: decommission, hints
>
> I tried to decommission one node.
> Node sent all data to another node and got stuck in LEAVING state.
> Log message shows Exception in HintsDispatcher thread.
> Could it be reason of stuck in LEAVING state?
> command output:
> {noformat}
> root@cas-node6:~# time nodetool decommission
> error: null
> -- StackTrace --
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:203)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:168)
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> real 147m7.483s
> user 0m17.388s
> sys 0m1.968s
> {noformat}
> nodetool netstats:
> {noformat}
> root@cas-node6:~# nodetool netstats
> Mode: LEAVING
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 35082
> Mismatch (Blocking): 18
> Mismatch (Background): 0
> Pool Name Active Pending Completed Dropped
> Large messages n/a 1 0 0
> Small messages n/a 0 16109860 112
> Gossip messages n/a 0 287074 0
> {noformat}
> Log:
> {noformat}
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:52:59,467
> StorageService.java:1170 - LEAVING: sleeping 30000 ms for batch processing
> and pending range setup
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,455
> StorageService.java:1170 - LEAVING: replaying batch log and streaming data to
> other nodes
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,910
> StreamResultFuture.java:87 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Executing streaming plan for Unbootstrap
> INFO [StreamConnectionEstablisher:1] 2016-12-07 12:53:39,911
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.17
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,911
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,912
> StorageService.java:1170 - LEAVING: streaming hints to other nodes
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,912
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.10 is complete
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,912
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.11 is complete
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,912
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.12 is complete
> INFO [StreamConnectionEstablisher:5] 2016-12-07 12:53:39,912
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.13
> INFO [StreamConnectionEstablisher:6] 2016-12-07 12:53:39,912
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:8] 2016-12-07 12:53:39,912
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.14
> INFO [RMI TCP Connection(58)-127.0.0.1] 2016-12-07 12:53:39,913
> HintsService.java:218 - Resumed hints dispatch
> INFO [StreamConnectionEstablisher:7] 2016-12-07 12:53:39,913
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.15
> INFO [StreamConnectionEstablisher:6] 2016-12-07 12:53:39,914
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.16 is complete
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,914
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.10
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,914
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.11
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,914
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.12
> INFO [StreamConnectionEstablisher:6] 2016-12-07 12:53:39,917
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.16
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,917
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,917
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:6] 2016-12-07 12:53:39,917
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.18
> INFO [StreamConnectionEstablisher:3] 2016-12-07 12:53:39,917
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.19
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,917
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.20 is complete
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,917
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.21 is complete
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,920
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.20
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,920
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.21
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,920
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:4] 2016-12-07 12:53:39,920
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.22
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,921
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.23 is complete
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,921
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.23
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,921
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,921
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.24 is complete
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,921
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.24
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,921
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,922
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.25 is complete
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,922
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.25
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:39,922
> StreamSession.java:239 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Starting streaming to /10.10.10.26 through /192.168.101.70
> INFO [HintsDispatcher:1] 2016-12-07 12:53:39,926
> HintsDispatchExecutor.java:140 - Transferring all hints to null
> ERROR [HintsDispatcher:1] 2016-12-07 12:53:39,928 CassandraDaemon.java:205 -
> Exception in thread Thread[HintsDispatcher:1,1,main]
> java.lang.NullPointerException: null
> at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> ~[na:1.8.0_92]
> at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> ~[na:1.8.0_92]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:203)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> ~[na:1.8.0_92]
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> ~[na:1.8.0_92]
> at
> java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
> ~[na:1.8.0_92]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> ~[na:1.8.0_92]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> ~[na:1.8.0_92]
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> ~[na:1.8.0_92]
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> ~[na:1.8.0_92]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> ~[na:1.8.0_92]
> at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> ~[na:1.8.0_92]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:168)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:141)
> ~[apache-cassandra-3.0.9.jar:3.0.9]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_92]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_92]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_92]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_92]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:40,093
> StreamResultFuture.java:169 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb
> ID#0] Prepare completed. Receiving 0 files(0 bytes), sending 2 files(
> 133006 bytes)
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:40,097
> StreamCoordinator.java:213 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb,
> ID#0] Beginning stream session with /10.10.10.26
> INFO [StreamConnectionEstablisher:2] 2016-12-07 12:53:40,097
> StreamSession.java:232 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session does not have any tasks.
> ...
> INFO [IndexSummaryManager:1] 2016-12-07 14:51:44,761
> IndexSummaryRedistribution.java:74 - Redistributing index summaries
> INFO [STREAM-IN-/10.10.10.27] 2016-12-07 15:20:06,006
> StreamResultFuture.java:183 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> Session with /10.10.10.27 is complete
> INFO [STREAM-IN-/10.10.10.27] 2016-12-07 15:20:06,007
> StreamResultFuture.java:215 - [Stream #2cc874c0-bc7c-11e6-b0df-e7f1ecd3dcfb]
> All sessions completed
> {noformat}
> all real IPs replaced with IPs from 10.10.10.x network in log
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
