Jeffrey created USERGRID-507:
---------------------------------
Summary: Many Histrix timeouts though Cassndra is not under load
Key: USERGRID-507
URL: https://issues.apache.org/jira/browse/USERGRID-507
Project: Usergrid
Issue Type: Bug
Components: Stack
Reporter: Jeffrey
Priority: Blocker
Cassandra is not under heavy load but I am seeing a high number of Hystrix
timeouts. Should we increase connection pool size? Is there an operational
setting that can help this?
2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR
org.apache.usergrid.persistence.core.task.NamedTaskExecutorImpl- Unable to
execute task. Exception is
com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1
timed-out and no fallback available.
at
com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646)
at
com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103)
at
com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
at
com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047)
at
com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627)
at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522)
at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431)
at
org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69)
at
org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
at
org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy92.auditShard(Unknown Source)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.TimeoutException
... 25 more
2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl-
Unable to execute audit for shard of {}
com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1
timed-out and no fallback available.
at
com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646)
at
com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103)
at
com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
at
com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047)
at
com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627)
at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522)
at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431)
at
org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69)
at
org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
at
org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy92.auditShard(Unknown Source)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.TimeoutException
... 25 more
2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl-
Unable to perform audit. Exception is
com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1
timed-out and no fallback available.
at
com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646)
at
com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103)
at
com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37)
at
com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57)
at
com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047)
at
com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627)
at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522)
at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431)
at
org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69)
at
org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187)
at
org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208)
at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
at com.sun.proxy.$Proxy92.auditShard(Unknown Source)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368)
at
org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.TimeoutException
... 25 more
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
