Hi, There is an issue reported for this in different scenario [1]
[1]https://issues.apache.org/jira/browse/CASSANDRA-6536 Thanks Shammi On Fri, Feb 28, 2014 at 11:50 AM, Nuwan Silva <nuw...@wso2.com> wrote: > Hi All, > > When running a high load on a MB Cluster we have observed Cassandra > throwing the following error. > > Test Scenario: > 1 publisher publishing 1,000,000 messages > 3 consumers, consuming > > after some time 15-20 mins the consumers stop consuming and we have > observed the following error in Cassandra. We also observed > *me.prettyprint.hector.api.exceptions.HTimedOutException: > TimedOutException()* thrown in MB > > > Cassandra Version: 1.2.13 > > cassandra system.log: > ---------------------------------- > INFO [ScheduledTasks:1] 2014-02-27 21:56:59,928 GCInspector.java (line > 119) GC for ParNew: 241 ms for 1 collections, 1191010416 used; max is > 8375238656 > INFO [MemoryMeter:1] 2014-02-27 21:57:38,322 Memtable.java (line 516) > CFS(Keyspace='QpidKeySpace', ColumnFamily='QueueEntries') liveRatio is > 49.411764705882355 (just-counted was 49.411764705882355). calculation took > 0ms for 1 columns > ERROR [ReadStage:923] 2014-02-27 22:01:42,284 CassandraDaemon.java (line > 191) Exception in thread Thread[ReadStage:923,5,main] > java.lang.RuntimeException: > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: Corrupt (negative) value length encountered > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1614) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.IOException: Corrupt (negative) value length encountered > at > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:357) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:166) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:50) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:90) > at > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:171) > at > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:154) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:143) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:160) > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:136) > at > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:84) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:291) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1397) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1213) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1129) > at org.apache.cassandra.db.Table.getRow(Table.java:344) > at > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70) > at > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1058) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1610) > ... 3 more > Caused by: java.io.IOException: Corrupt (negative) value length encountered > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:352) > at > org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:102) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) > at > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:73) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:398) > at > org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:353) > ... 27 more > > > WSO2 MB: > ----------------- > [2014-02-27 22:18:12,335] WARN > {me.prettyprint.cassandra.connection.HConnectionManager} - Could not > fullfill request on this host CassandraClient<204.13.85.2:9160-49> > [2014-02-27 22:18:12,335] WARN > {me.prettyprint.cassandra.connection.HConnectionManager} - Exception: > me.prettyprint.hector.api.exceptions.HTimedOutException: > TimedOutException() > at > me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:285) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:268) > at > me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103) > at > me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl.getSlice(KeyspaceServiceImpl.java:289) > at > me.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:53) > at > me.prettyprint.cassandra.model.thrift.ThriftSliceQuery$1.doInKeyspace(ThriftSliceQuery.java:49) > at > me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20) > at > me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85) > at > me.prettyprint.cassandra.model.thrift.ThriftSliceQuery.execute(ThriftSliceQuery.java:48) > at > org.wso2.andes.server.store.util.CassandraDataAccessHelper.getMessagesFromQueue(CassandraDataAccessHelper.java:489) > at > org.wso2.andes.server.store.CassandraMessageStore.getMessagesFromGlobalQueue(CassandraMessageStore.java:1252) > at > org.wso2.andes.server.cluster.GlobalQueueWorker.run(GlobalQueueWorker.java:73) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: TimedOutException() > at > org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7296) > at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) > at > org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:552) > at > org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:536) > at > me.prettyprint.cassandra.service.KeyspaceServiceImpl$7.execute(KeyspaceServiceImpl.java:273) > ... 16 more > > > Thanks, > NuwanS. > > -- > > > *Nuwan Silva* > *Senior Software Engineer - QA* > Mobile: +94779804543 > > WSO2 Inc. > lean . enterprise . middlewear. > http://www.wso2.com > -- Best Regards, * Shammi Jayasinghe* Associate Tech Lead WSO2, Inc.; http://wso2.com, mobile: +94 71 4493085
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
