Re: Get UnavailableException() when write to multiple Node
Hi What do you mean by broken time synchronization is it a know problem with virtualized os ? On Wed, Feb 17, 2010 at 7:34 PM, Wojciech Kaczmarek kaczmare...@gmail.comwrote: Just guessing: are you sure that these virtual machines don't suffer from a broken time synchronization? On Wed, Feb 17, 2010 at 12:26, Richard Grossman richie...@gmail.comwrote: Hi I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra 0.5 release. All is ok util I begin to get error like on the client side : UnavailableException() at org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892) at org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616) at org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591) at tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229) at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59) at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) On the server more interresting : INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) InetAddress /38.96.191.42 is now dead. INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) InetAddress /38.96.191.42 is now UP INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line 209) Completed flushing /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java (line 875) Compacting [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')] INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java (line 943) Compacted to /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396 bytes for 110384 keys. Time: 94627ms. INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) InetAddress /38.96.191.40 is now dead. WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java (line 484) Problem reading from socket connected to : java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/ 38.96.191.39:50133] WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java (line 485) Exception was generated at : 02/17/2010 11:21:50 on thread MESSAGING-SERVICE-POOL:2 Reached an EOL or something bizzare occured. Reading from: /38.96.191.39BufferSizeRemaining: 16 java.io.IOException: Reached an EOL or something bizzare occured. Reading from: /38.96.191.39 BufferSizeRemaining: 16 at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44) at org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39) at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95) at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) As you can see it's like after some time the communication between node just gone the node is declared dead but after some millis the node is up again between this short time all the insert just gone. It occurs after some time meaning the data is loaded into the memtable. If someone can help me to understand what going on. Is the machine itself overloaded. It's a single machine with 4 virtual machine sharing the same disk is it the cause ? Thanks for any help Richard
Re: Get UnavailableException() when write to multiple Node
Xen has had issues with it. If you're using it, google for xen independent_wallclock, xen ntp and see if the issues described fit your configuration. On Thu, Feb 18, 2010 at 09:03, Richard Grossman richie...@gmail.com wrote: Hi What do you mean by broken time synchronization is it a know problem with virtualized os ? On Wed, Feb 17, 2010 at 7:34 PM, Wojciech Kaczmarek kaczmare...@gmail.com wrote: Just guessing: are you sure that these virtual machines don't suffer from a broken time synchronization? On Wed, Feb 17, 2010 at 12:26, Richard Grossman richie...@gmail.comwrote: Hi I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra 0.5 release. All is ok util I begin to get error like on the client side : UnavailableException() at org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892) at org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616) at org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591) at tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229) at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59) at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) On the server more interresting : INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) InetAddress /38.96.191.42 is now dead. INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) InetAddress /38.96.191.42 is now UP INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line 209) Completed flushing /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java (line 875) Compacting [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')] INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java (line 943) Compacted to /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396 bytes for 110384 keys. Time: 94627ms. INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) InetAddress /38.96.191.40 is now dead. WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java (line 484) Problem reading from socket connected to : java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/ 38.96.191.39:50133] WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java (line 485) Exception was generated at : 02/17/2010 11:21:50 on thread MESSAGING-SERVICE-POOL:2 Reached an EOL or something bizzare occured. Reading from: /38.96.191.39BufferSizeRemaining: 16 java.io.IOException: Reached an EOL or something bizzare occured. Reading from: /38.96.191.39 BufferSizeRemaining: 16 at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44) at org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39) at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95) at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) As you can see it's like after some time the communication between node just gone the node is declared dead but after some millis the node is up again between this short time all the insert just gone. It occurs after some time meaning the data is loaded into the memtable. If someone can help me to understand what going on. Is the machine itself overloaded. It's a single machine with 4 virtual machine sharing the same disk is it the cause ? Thanks for any help Richard
Re: Get UnavailableException() when write to multiple Node
On Wed, Feb 17, 2010 at 5:26 AM, Richard Grossman richie...@gmail.com wrote: If someone can help me to understand what going on. Is the machine itself overloaded. It's a single machine with 4 virtual machine sharing the same disk is it the cause ? Probably.
Re: Get UnavailableException() when write to multiple Node
Just guessing: are you sure that these virtual machines don't suffer from a broken time synchronization? On Wed, Feb 17, 2010 at 12:26, Richard Grossman richie...@gmail.com wrote: Hi I've configured 4 Virtual Machines CentOS 4GB memory. Each run cassandra 0.5 release. All is ok util I begin to get error like on the client side : UnavailableException() at org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892) at org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616) at org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591) at tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229) at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59) at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) On the server more interresting : INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) InetAddress /38.96.191.42 is now dead. INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) InetAddress /38.96.191.42 is now UP INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line 209) Completed flushing /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java (line 875) Compacting [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')] INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java (line 943) Compacted to /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db. 80405396/80405396 bytes for 110384 keys. Time: 94627ms. INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194) InetAddress /38.96.191.40 is now dead. WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java (line 484) Problem reading from socket connected to : java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/ 38.96.191.39:50133] WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java (line 485) Exception was generated at : 02/17/2010 11:21:50 on thread MESSAGING-SERVICE-POOL:2 Reached an EOL or something bizzare occured. Reading from: /38.96.191.39BufferSizeRemaining: 16 java.io.IOException: Reached an EOL or something bizzare occured. Reading from: /38.96.191.39 BufferSizeRemaining: 16 at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44) at org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39) at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95) at org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) As you can see it's like after some time the communication between node just gone the node is declared dead but after some millis the node is up again between this short time all the insert just gone. It occurs after some time meaning the data is loaded into the memtable. If someone can help me to understand what going on. Is the machine itself overloaded. It's a single machine with 4 virtual machine sharing the same disk is it the cause ? Thanks for any help Richard
