Just guessing: are you sure that these virtual machines don't suffer from a broken time synchronization?
On Wed, Feb 17, 2010 at 12:26, Richard Grossman <richie...@gmail.com> wrote: > Hi > > I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra > 0.5 release. > All is ok util I begin to get error like on the client side : > UnavailableException() > at > org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892) > at > org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616) > at > org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591) > at > tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229) > at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59) > at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > On the server more interresting : > INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) InetAddress > /38.96.191.42 is now dead. > INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) InetAddress > /38.96.191.42 is now UP > INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line > 209) Completed flushing > /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db > INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java > (line 875) Compacting > [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')] > INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java > (line 943) Compacted to > /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396 > bytes for 110384 keys. Time: 94627ms. > INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) > InetAddress /38.96.191.40 is now dead. > WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java > (line 484) Problem reading from socket connected to : > java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/ > 38.96.191.39:50133] > WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java > (line 485) Exception was generated at : 02/17/2010 11:21:50 on thread > MESSAGING-SERVICE-POOL:2 > Reached an EOL or something bizzare occured. Reading from: > /38.96.191.39BufferSizeRemaining: 16 > java.io.IOException: Reached an EOL or something bizzare occured. Reading > from: /38.96.191.39 BufferSizeRemaining: 16 > at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44) > at > org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39) > at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95) > at > org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > As you can see it's like after some time the communication between node > just gone the node is declared dead but after some millis the node is up > again between this short time all the insert just gone. > It occurs after some time meaning the data is loaded into the memtable. > > If someone can help me to understand what going on. Is the machine itself > overloaded. It's a single machine with 4 virtual machine sharing the same > disk is it the cause ? > > Thanks for any help > Richard >
