Hi All :)
I have an application which use DataStax Java driver v2.0.2 to connect
to Cassandra Cluster (6 nodes, v 2.0.11).
Application is deployed in three copies on 3 different servers. From
time to time on random application server I get this exception:
2014-11-04 10:37:15,301 - ERROR: Servlet.service() for servlet [Unique
Identifier Service] in context with path [/uis] threw exception
[com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: [cassandra-11/10.0.0.11:9042,
/10.0.0.10:9042, /10.0.0.12:9042, /10.0.0.7:9042, /10.0.0.9:9042,
/10.0.0.8:9042] - use getErrors() for details)] with root cause
com.datastax.driver.core.exceptions.NoHostAvailableException: All
host(s) tried for query failed (tried: [cassandra-11/10.0.0.11:9042,
/10.0.0.10:9042, /10.0.0.12:9042, /10.0.0.7:9042, /10.0.0.9:9042,
/10.0.0.8:9042] - use getErrors() for details)
at
com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103)
at
com.datastax.driver.core.SessionManager.execute(SessionManager.java:418)
at
com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:454)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:104)
at
com.datastax.driver.core.SessionManager.execute(SessionManager.java:92)
at
eu.europeana.cloud.service.uis.database.dao.CassandraDataProviderDAO.getProviders(CassandraDataProviderDAO.java:92)
at
eu.europeana.cloud.service.uis.CassandraDataProviderService.getProviders(CassandraDataProviderService.java:34)
at
eu.europeana.cloud.service.uis.rest.DataProvidersResource.getProviders(DataProvidersResource.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:104)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:353)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:343)
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:318)
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:983)
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:359)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
2014-11-04 10:37:28,317 - ERROR: [Control connection] Cannot connect
to any host, scheduling retry
The error usually occurs on only one of the application nodes
(sometimes two) but application on other nodes is working fine.
I cannot see any errors in Cassandra logs, so it looks like if it was
working ok. Cassandra and application servers are in the same server
room with a high throughput network, so connection shouldn't be a
problem - and my admin claims that it is not. But even if there was an
error in connection the driver should retry to connect, right? (that's
what its documentation says and I'm using default connection
parameters)
I don't know where to look for the cause of the error. Is this a
matter of DataStax Driver or my code or rather Cassandra?
Any help appreciated.
Kind regards,
Ola
