Hi Jeff,
Today another node was shutdown,I have attached the exception log
file,could you please help to analyze?Thanks.
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
发件人: Jeff Jirsa <jji...@gmail.com>
发送时间: 2018年3月27日 11:50
收件人: Xiangfei Ni <xiangfei...@cm-dt.com>
抄送: user@cassandra.apache.org
主题: Re: 答复: A node down every day in a 6 nodes cluster
Only one node having the problem is suspicious. May be that your application is
improperly pooling connections, or you have a hardware problem.
I dont see anything in nodetool that explains it, though you certainly have a
data model likely to cause problems over time (the cardinality of
rt_ac_stat.idx_rt_ac_stat_prot_verrt_ac_stat.idx_rt_ac_stat_prot_ver is such
that you have very wide partitions and it'll be difficult to read).
On Mon, Mar 26, 2018 at 8:26 PM, Xiangfei Ni
<xiangfei...@cm-dt.com<mailto:xiangfei...@cm-dt.com>> wrote:
Hi Jeff,
I need to restart the node manually every time,only one node has this
problem.
I have attached the nodetool output,thanks.
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811<tel:+86%20137%209700%207811>|Tel: + 86 27 5024
2516<tel:+86%2027%205024%202516>
发件人: Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>>
发送时间: 2018年3月27日 11:03
收件人: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
主题: Re: A node down every day in a 6 nodes cluster
That warning isn’t sufficient to understand why the node is going down
Cassandra 3.9 has some pretty serious known issues - upgrading to 3.11.3 is
likely a good idea
Are the nodes coming up on their own? Or are you restarting them?
Paste the output of nodetool tpstats and nodetool cfstats
--
Jeff Jirsa
On Mar 26, 2018, at 7:56 PM, Xiangfei Ni
<xiangfei...@cm-dt.com<mailto:xiangfei...@cm-dt.com>> wrote:
Hi Cassandra experts,
I am facing an issue,a node downs every day in a 6 nodes cluster,the cluster
is just in one DC,
Every node has 4C 16G,and the heap configuration is MAX_HEAP_SIZE=8192m
HEAP_NEWSIZE=512m,every node load about 200G data,the RF for the business CF is
3,a node downs one time every day,the system.log shows below info:
WARN [Native-Transport-Requests-19] 2018-03-26 18:53:17,128
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize #<User
nev_tsp_sa> for <table nev_prod_tsp.latest_rt_alarm>
ERROR [Native-Transport-Requests-19] 2018-03-26 18:53:17,129
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
~[guava-18.0.jar:na]
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.ClientState.authorize(ClientState.java:419)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:185)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
[apache-cassandra-3.9.jar:3.9]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:102)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183)
~[apache-cassandra-3.9.jar:3.9]
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
~[guava-18.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
~[guava-18.0.jar:na]
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
~[guava-18.0.jar:na]
... 26 common frames omitted
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1718)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1667)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1608)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1527)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:975)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:232)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraAuthorizer.addPermissionsForRole(CassandraAuthorizer.java:227)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:93)
~[apache-cassandra-3.9.jar:3.9]
... 32 common frames omitted
WARN [Native-Transport-Requests-23] 2018-03-26 18:53:17,131
CassandraAuthorizer.java:101 - CassandraAuthorizer failed to authorize #<User
nev_tsp_sa> for <table nev_prod_tsp.rt_alarm_unite>
ERROR [Native-Transport-Requests-64] 2018-03-26 18:53:17,135
QueryMessage.java:128 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
~[guava-18.0.jar:na]
I have confirmed that nev_tsp_sa has all rights on nev_prod_tsp keyspace:
cassandra@cqlsh:system_auth> select * from role_permissions where role =
'nev_tsp_sa';
role | resource | permissions
------------+-------------------+--------------------------------------------------------------
nev_tsp_sa | data/nev_prod_tsp | {'ALTER', 'AUTHORIZE', 'CREATE', 'DROP',
'MODIFY', 'SELECT'}
the cache disk can be read/write as normal.
Highly appreciated if anyone can help,thanks very much !
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811<tel:+86%20137%209700%207811>|Tel: + 86 27 5024
2516<tel:+86%2027%205024%202516>
ERROR [Native-Transport-Requests-34] 2018-03-28 00:39:25,049 Message.java:617 -
Unexpected exception during request; channel = [id: 0xcb106799,
L:/10.21.20.24:9042 ! R:/10.21.20.72:60796]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:298)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:79)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
[apache-cassandra-3.9.jar:3.9]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1718)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1667)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1608)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1527)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:975)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:232)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:497)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485)
~[apache-cassandra-3.9.jar:3.9]
... 13 common frames omitted
INFO [Service Thread] 2018-03-28 00:39:25,049 StatusLogger.java:106 -
system_traces.events 0,0
WARN [GossipTasks:1] 2018-03-28 00:39:25,051 Gossiper.java:771 - Gossip stage
has 2392 pending tasks; skipping status check (no nodes will be marked down)
INFO [Service Thread] 2018-03-28 00:39:25,058 GCInspector.java:284 -
ConcurrentMarkSweep GC in 310079ms. Par Survivor Space: 53673520 -> 49496248
INFO [Service Thread] 2018-03-28 00:40:01,918 StatusLogger.java:52 - Pool Name
Active Pending Completed Blocked All Time Blocked
WARN [GossipTasks:1] 2018-03-28 00:40:25,397 Gossiper.java:771 - Gossip stage
has 2474 pending tasks; skipping status check (no nodes will be marked down)
WARN [GossipTasks:1] 2018-03-28 00:48:00,332 Gossiper.java:771 - Gossip stage
has 3028 pending tasks; skipping status check (no nodes will be marked down)
INFO [Service Thread] 2018-03-28 00:48:00,337 StatusLogger.java:106 -
system_traces.events 0,0
INFO [Service Thread] 2018-03-28 00:48:26,414 GCInspector.java:284 -
ConcurrentMarkSweep GC in 541035ms. CMS Old Gen: 8053063648 -> 8053063680; Par
Eden Space: 429522944 -> 429522936; Par Survivor Space: 53673912 -> 49507016
INFO [Service Thread] 2018-03-28 00:48:26,414 StatusLogger.java:52 - Pool Name
Active Pending Completed Blocked All Time Blocked
ERROR [STREAM-OUT-/10.21.20.28:36986] 2018-03-28 01:28:08,895
CassandraDaemon.java:226 - Exception in thread
Thread[STREAM-OUT-/10.21.20.28:36986,5,main]
java.lang.OutOfMemoryError: Java heap space
at
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
~[logback-classic-1.1.3.jar:na]
at
ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.error(Logger.java:555)
~[logback-classic-1.1.3.jar:na]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:533)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:377)
~[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
INFO [ScheduledTasks:1] 2018-03-28 02:14:20,234 MessagingService.java:1048 -
READ messages were dropped in last 5000 ms: 1 for internal timeout and 0 for
cross node timeout. Mean internal dropped latency: 0 ms and Mean cross-node
dropped latency: 0 ms
ERROR [STREAM-OUT-/10.21.20.28:4282] 2018-03-28 02:15:32,398
StreamSession.java:533 - [Stream #4dfbbd80-31b7-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.26] 2018-03-28 02:40:14,326
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.26,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:12062] 2018-03-28 02:40:57,686
StreamSession.java:533 - [Stream #4e7caa20-31b3-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
INFO [Service Thread] 2018-03-28 02:41:25,516 StatusLogger.java:56 -
MutationStage 32 719673 18032055 0
0
ERROR [STREAM-OUT-/10.21.20.28:7878] 2018-03-28 02:51:08,862
StreamSession.java:533 - [Stream #04f62810-31b6-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:7066] 2018-03-28 02:42:04,812
StreamSession.java:533 - [Stream #364b7090-31b7-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.26] 2018-03-28 03:25:51,435
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.26,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.25] 2018-03-28 03:34:26,579
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.25,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:8733] 2018-03-28 03:36:32,059
StreamSession.java:533 - [Stream #d32ebcd0-31b4-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.25:30439] 2018-03-28 02:45:02,591
StreamSession.java:533 - [Stream #e3d69890-31ba-11e8-b768-f997b824a5a9]
Streaming error occurred on session with peer 10.21.20.25
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:48176] 2018-03-28 03:50:45,386
StreamSession.java:533 - [Stream #f1fe0c00-31b5-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:40626] 2018-03-28 03:55:54,316
StreamSession.java:533 - [Stream #e2a12a00-31b3-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.25:28491] 2018-03-28 03:56:22,664
StreamSession.java:533 - [Stream #cbb931f0-31ba-11e8-b768-f997b824a5a9]
Streaming error occurred on session with peer 10.21.20.25
java.lang.OutOfMemoryError: Java heap space
ERROR [InternalResponseStage:226] 2018-03-28 03:56:22,664
CassandraDaemon.java:226 - Exception in thread
Thread[InternalResponseStage:226,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.27] 2018-03-28 03:56:46,768
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.27,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.27] 2018-03-28 03:59:52,351
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.27,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.28] 2018-03-28 04:03:25,703
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.28,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2018-03-28 04:14:29,566
CassandraDaemon.java:226 - Exception in thread
Thread[PERIODIC-COMMIT-LOG-SYNCER,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipTasks:1] 2018-03-28 04:14:53,963 CassandraDaemon.java:226 -
Exception in thread Thread[GossipTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [Native-Transport-Requests-72] 2018-03-28 04:15:56,836 Message.java:617 -
Unexpected exception during request; channel = [id: 0x59942749,
L:/10.21.20.24:9042 - R:/10.21.20.45:31617]
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-1] 2018-03-28 04:19:36,951 Slf4JLogger.java:146 -
An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level
for full stacktrace] was thrown by a user handler's exceptionCaught() method
while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
ERROR [Native-Transport-Requests-39] 2018-03-28 04:38:30,582 Message.java:617 -
Unexpected exception during request; channel = [id: 0x7050a76f,
L:/10.21.20.24:9042 ! R:/10.21.20.71:42213]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:298)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:79)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
[apache-cassandra-3.9.jar:3.9]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
ERROR [STREAM-OUT-/10.21.20.28:48176] 2018-03-28 03:50:45,386
StreamSession.java:533 - [Stream #f1fe0c00-31b5-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:40626] 2018-03-28 03:55:54,316
StreamSession.java:533 - [Stream #e2a12a00-31b3-11e8-8fc5-39c988b353c2]
Streaming error occurred on session with peer 10.21.20.28
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.25:28491] 2018-03-28 03:56:22,664
StreamSession.java:533 - [Stream #cbb931f0-31ba-11e8-b768-f997b824a5a9]
Streaming error occurred on session with peer 10.21.20.25
java.lang.OutOfMemoryError: Java heap space
ERROR [InternalResponseStage:226] 2018-03-28 03:56:22,664
CassandraDaemon.java:226 - Exception in thread
Thread[InternalResponseStage:226,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.27] 2018-03-28 03:56:46,768
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.27,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.27] 2018-03-28 03:59:52,351
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.27,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.28] 2018-03-28 04:03:25,703
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.28,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2018-03-28 04:14:29,566
CassandraDaemon.java:226 - Exception in thread
Thread[PERIODIC-COMMIT-LOG-SYNCER,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipTasks:1] 2018-03-28 04:14:53,963 CassandraDaemon.java:226 -
Exception in thread Thread[GossipTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [Native-Transport-Requests-72] 2018-03-28 04:15:56,836 Message.java:617 -
Unexpected exception during request; channel = [id: 0x59942749,
L:/10.21.20.24:9042 - R:/10.21.20.45:31617]
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-1] 2018-03-28 04:19:36,951 Slf4JLogger.java:146 -
An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level
for full stacktrace] was thrown by
a user handler's exceptionCaught() method while handling the following
exception:
java.lang.OutOfMemoryError: Java heap space
ERROR [Native-Transport-Requests-39] 2018-03-28 04:38:30,582 Message.java:617 -
Unexpected exception during request; channel = [id: 0x7050a76f,
L:/10.21.20.24:9042 ! R:/10.21.20.71:42213]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:298)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:79)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
[apache-cassandra-3.9.jar:3.9]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1718)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1667)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1608)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1527)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:975)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:232)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:497)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485)
~[apache-cassandra-3.9.jar:3.9]
... 13 common frames omitted
ERROR [Native-Transport-Requests-1] 2018-03-28 04:46:18,782 Message.java:617 -
Unexpected exception during request; channel = [id: 0xe29dd66d,
L:/10.21.20.24:9042 ! R:/10.21.20.71:42398]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
received only 0 responses.
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.canLogin(CassandraRoleManager.java:298)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ClientState.login(ClientState.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:79)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
[apache-cassandra-3.9.jar:3.9]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.39.Final.jar:4.0.39.Final]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_91]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
timed out - received only 0 responses.
at
org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132)
~[apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1718)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1667)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1608)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1527)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:975)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:271)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:232)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:497)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485)
~[apache-cassandra-3.9.jar:3.9]
... 13 common frames omitted
INFO [ScheduledTasks:1] 2018-03-28 04:48:13,521 StatusLogger.java:52 - Pool
Name Active Pending Completed Blocked All Time
Blocked
ERROR [Native-Transport-Requests-41] 2018-03-28 04:55:11,513
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [Native-Transport-Requests-35] 2018-03-28 04:55:59,873
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.28:36986] 2018-03-28 04:55:59,874
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
~[logback-classic-1.1.3.jar:na]
at
ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
~[logback-classic-1.1.3.jar:na]
at ch.qos.logback.classic.Logger.error(Logger.java:555)
~[logback-classic-1.1.3.jar:na]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:533)
~[apache-cassandra-3.9.jar:3.9]
at
org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:377)
~[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
ERROR [Native-Transport-Requests-38] 2018-03-28 04:55:59,874
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
INFO [STREAM-OUT-/10.21.20.28:4282] 2018-03-28 05:00:46,843
StreamResultFuture.java:188 - [Stream #4dfbbd80-31b7-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
ERROR [IndexSummaryManager:1] 2018-03-28 05:02:15,535
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.26] 2018-03-28 05:09:27,777
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
INFO [STREAM-OUT-/10.21.20.28:7066] 2018-03-28 05:20:58,308
StreamResultFuture.java:188 - [Stream #364b7090-31b7-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
INFO [STREAM-OUT-/10.21.20.28:12062] 2018-03-28 05:20:33,946
StreamResultFuture.java:188 - [Stream #4e7caa20-31b3-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
INFO [Service Thread] 2018-03-28 05:23:43,480 StatusLogger.java:56 -
ViewMutationStage 0 0 0 0
0
ERROR [MessagingService-Incoming-/10.21.20.25] 2018-03-28 05:23:43,480
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
INFO [STREAM-OUT-/10.21.20.28:7878] 2018-03-28 05:24:10,052
StreamResultFuture.java:188 - [Stream #04f62810-31b6-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
ERROR [MessagingService-Incoming-/10.21.20.26] 2018-03-28 05:25:14,091
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
INFO [STREAM-OUT-/10.21.20.28:8733] 2018-03-28 05:27:21,114
StreamResultFuture.java:188 - [Stream #d32ebcd0-31b4-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
INFO [STREAM-OUT-/10.21.20.25:30439] 2018-03-28 05:40:32,712
StreamResultFuture.java:188 - [Stream #e3d69890-31ba-11e8-b768-f997b824a5a9]
Session with /10.21.20.25 is complete
INFO [STREAM-OUT-/10.21.20.28:40626] 2018-03-28 05:42:04,661
StreamResultFuture.java:188 - [Stream #e2a12a00-31b3-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
INFO [STREAM-OUT-/10.21.20.28:48176] 2018-03-28 05:41:35,488
StreamResultFuture.java:188 - [Stream #f1fe0c00-31b5-11e8-8fc5-39c988b353c2]
Session with /10.21.20.28 is complete
INFO [STREAM-OUT-/10.21.20.25:28491] 2018-03-28 05:44:47,779
StreamResultFuture.java:188 - [Stream #cbb931f0-31ba-11e8-b768-f997b824a5a9]
Session with /10.21.20.25 is complete
ERROR [InternalResponseStage:226] 2018-03-28 05:45:16,194
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
ERROR [GossipStage:1] 2018-03-28 06:34:18,404 CassandraDaemon.java:226 -
Exception in thread Thread[GossipStage:1,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [BatchlogTasks:1] 2018-03-28 06:35:31,441 CassandraDaemon.java:226 -
Exception in thread Thread[BatchlogTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-8] 2018-03-28 06:34:42,168 Slf4JLogger.java:151 -
An exceptionCaught() event was fired, and it reached at the tail of the
pipeline. It usually means the last handler in the pipeline did not handle the
exception.
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-5] 2018-03-28 06:35:05,695 Slf4JLogger.java:146 -
An exception 'java.lang.OutOfMemoryError: Java heap space' [enable DEBUG level
for full stacktrace] was thrown by a user handler's exceptionCaught() method
while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.27] 2018-03-28 06:39:05,020
JVMStabilityInspector.java:141 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
WARN [epollEventLoopGroup-2-7] 2018-03-28 06:37:24,403 Slf4JLogger.java:151 -
An exceptionCaught() event was fired, and it reached at the tail of the
pipeline. It usually means the last handler in the pipeline did not handle the
exception.
java.lang.OutOfMemoryError: Java heap space
ERROR [STREAM-OUT-/10.21.20.25:36579] 2018-03-28 06:36:42,564
StreamSession.java:533 - [Stream #d7455580-31bf-11e8-b768-f997b824a5a9]
Streaming error occurred on session with peer 10.21.20.25
java.lang.OutOfMemoryError: Java heap space
ERROR [MessagingService-Incoming-/10.21.20.27] 2018-03-28 06:36:42,204
CassandraDaemon.java:226 - Exception in thread
Thread[MessagingService-Incoming-/10.21.20.27,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [ACCEPT-/10.21.20.24] 2018-03-28 06:39:46,868 CassandraDaemon.java:226 -
Exception in thread Thread[ACCEPT-/10.21.20.24,5,main]
java.lang.OutOfMemoryError: Java heap space
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org
