Hi
I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra
0.5 release.
All is ok util I begin to get error like on the client side :
UnavailableException()
at
org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892)
at
org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616)
at
org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591)
at
tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229)
at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59)
at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
On the server more interresting :
INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) InetAddress
/38.96.191.42 is now dead.
INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) InetAddress
/38.96.191.42 is now UP
INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line 209)
Completed flushing /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db
INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java
(line 875) Compacting
[org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')]
INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java
(line 943) Compacted to
/root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396
bytes for 110384 keys. Time: 94627ms.
INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) InetAddress
/38.96.191.40 is now dead.
WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java
(line 484) Problem reading from socket connected to :
java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000 remote=/
38.96.191.39:50133]
WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java
(line 485) Exception was generated at : 02/17/2010 11:21:50 on thread
MESSAGING-SERVICE-POOL:2
Reached an EOL or something bizzare occured. Reading from:
/38.96.191.39BufferSizeRemaining: 16
java.io.IOException: Reached an EOL or something bizzare occured. Reading
from: /38.96.191.39 BufferSizeRemaining: 16
at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44)
at org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39)
at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
at
org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
As you can see it's like after some time the communication between node just
gone the node is declared dead but after some millis the node is up again
between this short time all the insert just gone.
It occurs after some time meaning the data is loaded into the memtable.
If someone can help me to understand what going on. Is the machine itself
overloaded. It's a single machine with 4 virtual machine sharing the same
disk is it the cause ?
Thanks for any help
Richard
