[
https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989150#comment-12989150
]
Thibaut commented on CASSANDRA-2081:
------------------------------------
Brandon, I haven't yet run stress test. I can reproduce this error every single
time with a single thread accessing my idle cluster.
I also reverted to an older version of hector, but this won't help. As noted
before, this error doesn't occur running
apache-cassandra-2011-01-24_06-01-26.jar.
Here is the debug output of one of the nodes timing out in my application and
not returning an answer:
I only try to access (read/iterator) one single table "table_usersources".
The application runs on node intr1n5 (192.168.0.5) and I added the debug output
of intr1n19 (192.168.0.19). The node intr1n18(192.168.0.18) is down and not
responding.
Please let me know if you need more information in order to fix this bug.
Thanks!
Intr1n19:
INFO [HintedHandoff:1] 2011-02-01 11:47:39,772 HintedHandOffManager.java (line
249) Finished hinted handoff of 0 rows to endpoint /192.168.0.15
INFO [ScheduledTasks:1] 2011-02-01 11:47:40,773 Gossiper.java (line 205)
InetAddress /192.168.0.10 is now dead.
DEBUG [ScheduledTasks:1] 2011-02-01 11:47:40,773 MessagingService.java (line
176) Resetting pool for /192.168.0.10
INFO [HintedHandoff:1] 2011-02-01 11:47:40,775 HintedHandOffManager.java (line
192) Started hinted handoff for endpoint /192.168.0.10
INFO [GossipStage:1] 2011-02-01 11:47:40,775 Gossiper.java (line 579)
InetAddress /192.168.0.10 is now UP
INFO [HintedHandoff:1] 2011-02-01 11:47:40,775 HintedHandOffManager.java (line
249) Finished hinted handoff of 0 rows to endpoint /192.168.0.10
DEBUG [WRITE-intr1n4/192.168.0.4] 2011-02-01 11:47:41,479
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n4/192.168.0.4
DEBUG [WRITE-intr1n20/192.168.0.20] 2011-02-01 11:47:42,729
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n20/192.168.0.20
DEBUG [WRITE-intr1n11/192.168.0.11] 2011-02-01 11:47:42,776
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n11/192.168.0.11
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:47:46,781
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n18/192.168.0.18
DEBUG [WRITE-intr1n15/192.168.0.15] 2011-02-01 11:47:50,786
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n15/192.168.0.15
DEBUG [WRITE-intr1n10/192.168.0.10] 2011-02-01 11:47:53,698
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n10/192.168.0.10
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:15,413 GCInspector.java (line 135) GC
for ParNew: 32 ms, 124741344 reclaimed leaving 880060312 used; max is 3289776128
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,853 FileUtils.java (line 48)
Deleting LocationInfo-f-79-Index.db
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,853 FileUtils.java (line 48)
Deleting LocationInfo-f-79-Filter.db
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,854 FileUtils.java (line 48)
Deleting LocationInfo-f-79-Statistics.db
INFO [ScheduledTasks:1] 2011-02-01 11:48:24,854 SSTable.java (line 147)
Deleted /hd2/cassandra_md5/data/system/LocationInfo-f-79
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:28,820
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n18/192.168.0.18
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,936 CassandraServer.java (line 445)
range_slice
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,943 StorageProxy.java (line 514)
RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,], max_keys=250}
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,943 StorageProxy.java (line 705)
restricted ranges for query [,] are [[,0cc], (0cc,199], (199,266], (266,333],
(333,400], (400,4cc], (4cc,599], (599,666], (666,733], (733,7ff], (7ff,8cc],
(8cc,999], (999,a66], (a66,b33], (b33,c00], (c00,ccc], (ccc,d99], (d99,e66],
(e66,f33], (f33,ffffffffffffffff], (ffffffffffffffff,]]
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 ReadCallback.java (line 58)
ReadCallback blocking for 2 responses
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.1
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.2
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,950 StorageProxy.java (line 562)
reading RangeSliceCommand{keyspace='table_usersources',
column_family='table_usersources_meta', super_column=null,
predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88
cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.3
DEBUG [RequestResponseStage:1] 2011-02-01 11:48:28,954 ResponseVerbHandler.java
(line 48) Processing response on a callback from 269@/192.168.0.1
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:38,771 StorageLoadBalancer.java (line
349) Disseminating load info ...
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,950 CassandraServer.java (line 483)
... timed out
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,957 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:44,835
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n18/192.168.0.18
DEBUG [pool-1-thread-3] 2011-02-01 11:48:56,275 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-6] 2011-02-01 11:48:56,275 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-7] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-11] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-5] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-2] 2011-02-01 11:48:56,277 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-8] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-10] 2011-02-01 11:48:56,277 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-12] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-15] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-14] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-9] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-16] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-13] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-4] 2011-02-01 11:48:56,278 ClientState.java (line 91)
logged out: #<User allow_all groups=[]>
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:57,845
OutboundTcpConnection.java (line 159) attempting to connect to
intr1n18/192.168.0.18
Application:
11:48:25,616 INFO ~ Registering JMX
me.prettyprint.cassandra.service:ServiceType=hector,MonitorType=hector
11:48:25,652 INFO ~ get connection for table_lists: consistency: ONE
11:48:25,695 INFO ~ get connection for table_lists: consistency: ONE
11:48:25,825 INFO ~ Downed Host Retry service started with queue size -1 and
retry delay 10s
11:48:28,887 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException:
java.net.NoRouteToHostException: No route to host
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at
me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
at
me.prettyprint.cassandra.connection.ConcurrentHClientPool.<init>(ConcurrentHClientPool.java:44)
at
me.prettyprint.cassandra.connection.HConnectionManager.<init>(HConnectionManager.java:63)
at
me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:62)
at
me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:58)
at
me.prettyprint.cassandra.service.ThriftCluster.<init>(ThriftCluster.java:17)
at
me.prettyprint.hector.api.factory.HFactory.createCluster(HFactory.java:157)
at
me.prettyprint.hector.api.factory.HFactory.getOrCreateCluster(HFactory.java:136)
....
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 23 more
11:48:28,889 ERROR ~ Could not start connection pool for host
intr1n18(192.168.0.18):9160
11:48:28,889 INFO ~ Host detected as down was added to retry queue:
intr1n18(192.168.0.18):9160
11:48:28,897 INFO ~ get connection for table_usersources: consistency: QUORUM
11:48:38,014 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException:
java.net.NoRouteToHostException: No route to host
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at
me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
at
me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.verifyConnection(CassandraHostRetryService.java:116)
at
me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.run(CassandraHostRetryService.java:96)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 13 more
11:48:38,019 ERROR ~ Downed intr1n18(192.168.0.18):9160 host still appears to
be down: Unable to open transport to intr1n18(192.168.0.18):9160 ,
java.net.NoRouteToHostException: No route to host
11:48:38,020 INFO ~ Downed Host retry status false with host:
intr1n18(192.168.0.18):9160
11:48:38,956 ERROR ~ Could not fullfill request on this host
CassandraClient<intr1n19:9160-594>
11:48:38,956 ERROR ~ Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:159)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:165)
at
me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:63)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at
me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:62)
...
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:704)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:149)
... 23 more
11:48:48,961 ERROR ~ Could not fullfill request on this host
CassandraClient<intr1n17:9160-577>
11:48:48,961 ERROR ~ Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at
me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
at
me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at
me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:159)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:165)
at
me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:67)
at
me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:63)
at
me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at
me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at
me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:62)
...
Caused by: TimedOutException()
at
org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
at
org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:704)
at
me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:149)
... 23 more
11:48:51,025 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException:
java.net.NoRouteToHostException: No route to host
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at
org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at
me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
at
me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.verifyConnection(CassandraHostRetryService.java:116)
at
me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.run(CassandraHostRetryService.java:96)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 13 more
11:48:51,025 ERROR ~ Downed intr1n18(192.168.0.18):9160 host still appears to
be down: Unable to open transport to intr1n18(192.168.0.18):9160 ,
java.net.NoRouteToHostException: No route to host
11:48:51,025 INFO ~ Downed Host retry status false with host:
intr1n18(192.168.0.18):9160
Address Status State Load Owns Token
ffffffffffffffff
192.168.0.1 Up Normal 11.26 GB 5.00% 0cc
192.168.0.2 Up Normal 11.23 GB 5.00% 199
192.168.0.3 Up Normal 11.58 GB 5.00% 266
192.168.0.4 Up Normal 6.77 GB 5.00% 333
192.168.0.5 Up Normal 6.86 GB 5.00% 400
192.168.0.6 Up Normal 6.81 GB 5.00% 4cc
192.168.0.7 Up Normal 6.88 GB 5.00% 599
192.168.0.8 Up Normal 6.84 GB 5.00% 666
192.168.0.9 Up Normal 6.52 GB 5.00% 733
192.168.0.10 Up Normal 5.17 GB 5.00% 7ff
192.168.0.11 Up Normal 6.75 GB 5.00% 8cc
192.168.0.12 Up Normal 7.06 GB 5.00% 999
192.168.0.13 Up Normal 7.27 GB 5.00% a66
192.168.0.14 Up Normal 7.71 GB 5.00% b33
192.168.0.15 Up Normal 7.46 GB 5.00% c00
192.168.0.16 Up Normal 6.94 GB 5.00% ccc
192.168.0.17 Up Normal 6.45 GB 5.00% d99
192.168.0.18 Down Normal ? 5.00% e66
192.168.0.19 Up Normal 6.26 GB 5.00% f33
192.168.0.20 Up Normal 6.33 GB 5.00% ffffffffffffffff
> Consistency QUORUM does not work anymore (hector:Could not fullfill request
> on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read).
> Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision
> (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with
> consistency level QUORUM again, while still using
> apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed
> intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open
> transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException:
> No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host
> retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill
> request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the
> cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
