[
https://issues.apache.org/jira/browse/CASSANDRA-12735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819544#comment-16819544
]
Shannon Carey commented on CASSANDRA-12735:
-------------------------------------------
Adjusting the cache validity period and refresh rate only helps so much. A
configuration should be exposed to allow control over the timeout duration for
this. If the timeout is too small, then it repeatedly tries reading the data,
making it even less likely to succeed.
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
> -------------------------------------------------------------------------
>
> Key: CASSANDRA-12735
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12735
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core, Local/Config
> Environment: Python 2.7.11, Datastax Cassandra 3.7.0
> Reporter: Rajesh Radhakrishnan
> Priority: Normal
> Fix For: 3.11.x
>
>
> We got a cluster of two nodes running Cassandra.3.7.0 and using client
> running Python 2.7.11 injecting lot of data from maybe 100 or so jobs.
> ------------------------------------------------------------------------------------------
> Cache setting can be seen from system.log:
> INFO [main] 2016-09-30 15:12:50,002 AuthCache.java:172 - (Re)initializing
> CredentialsCache (validity period/update interval/max entries)
> (2000/2000/1000)
> INFO [SharedPool-Worker-1] 2016-09-30 15:15:09,561 AuthCache.java:172 -
> (Re)initializing PermissionsCache (validity period/update interval/max
> entries) (10000/10000/1000)
> INFO [SharedPool-Worker-1] 2016-09-30 15:15:24,319 AuthCache.java:172 -
> (Re)initializing RolesCache (validity period/update interval/max entries)
> (5000/5000/1000)
> ===========================================================
> But I am getting the following exception :
> ERROR [SharedPool-Worker-90] 2016-09-30 15:17:20,883 ErrorMessage.java:338 -
> Unexpected exception during request
> com.google.common.util.concurrent.UncheckedExecutionException:
> com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
> received only 0 responses.
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.ClientState.authorize(ClientState.java:375)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:308)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:285)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:272)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:256)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.cql3.statements.BatchStatement.checkAccess(BatchStatement.java:137)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:502)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:495)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:217)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
> [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)
> [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:283)
> [netty-all-4.0.36.Final.jar:4.0.36.Final]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_73]
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-3.7.0.jar:3.7.0]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_73]
> Caused by: com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
> received only 0 responses.
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
> ~[guava-18.0.jar:na]
> at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:86)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.PermissionsCache.lambda$new$1(PermissionsCache.java:37)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[guava-18.0.jar:na]
> ... 26 common frames omitted
> Caused by: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out -
> received only 0 responses.
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:489)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:269)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.RolesCache.lambda$new$195(RolesCache.java:36)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> ~[guava-18.0.jar:na]
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> ~[guava-18.0.jar:na]
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ~[guava-18.0.jar:na]
> ... 40 common frames omitted
> Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation
> timed out - received only 0 responses.
> at
> org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1714)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1663)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1604)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1523)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:979)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:273)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:234)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:497)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> at
> org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:485)
> ~[apache-cassandra-3.7.0.jar:3.7.0]
> ... 47 common frames omitted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
