[
https://issues.apache.org/jira/browse/CASSANDRA-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965328#comment-13965328
]
Martin Bligh edited comment on CASSANDRA-6981 at 4/10/14 1:23 PM:
------------------------------------------------------------------
So this is a little tricky, because it's proprietary data and I've changed
things around a bit since then. Basically what I was doing was on a desktop
machine with 32GB of RAM and just one disk (regular HDD, not SSD), created
about 16 tables, all the same, each with about 5 text fields and 5 binary
fields. Most of those fields had a secondary index. Then insert into all the
tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall
over like this ....
PS. I've never seen this again since disabling mmap access, but am fairly sure
that's great for performance.
was (Author: mbligh):
So this is a little tricky, because it's proprietary data and I've changed
things around a bit since then. Basically what I was doing was on a desktop
machine with 32GB of RAM and just one disk (regular HDD, not SSD), created
about 16 tables, all the same, each with about 5 text fields and 5 binary
fields. Most of those fields had a secondary index. Then insert into all the
tables in parallel.
I'm aware this isn't a great design scheme, but it certainly shouldn't fall
over like this ....
> java.io.EOFException from Cassandra when doing select
> -----------------------------------------------------
>
> Key: CASSANDRA-6981
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6981
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra 2.06, Oracle Java version "1.7.0_51", Linux
> Mint 16
> Reporter: Martin Bligh
>
> Cassandra 2.06, Oracle Java version "1.7.0_51", Linux Mint 16
> I have a cassandra keyspace with about 12 tables that are all the same.
> If I load 100,000 rows or so into a couple of those tables in Cassandra, it
> works fine.
> If I load a larger dataset, after a while one of the tables won't do lookups
> any more (not always the same one).
> {noformat}
> SELECT recv_time,symbol from table6 where mid='S-AUR01-20140324A-1221';
> {noformat}
> results in "Request did not complete within rpc_timeout."
> where "mid" is the primary key (varchar). If I look at the logs, it has an
> EOFException ... presumably it's running out of some resource (it's
> definitely not out of disk space)
> Sometimes it does this on secondary indexes too: dropping and rebuilding the
> index will fix it for a while. When it's broken, it seems like only one
> particular lookup key causes timeouts (and the EOFException every time) -
> other lookups work fine. I presume the index is corrupt somehow.
> {noformat}
> ERROR [ReadStage:110] 2014-04-03 12:39:47,018 CassandraDaemon.java (line 196)
> Exception in thread Thread[ReadStage:110,5,main]
> java.io.IOError: java.io.EOFException
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:79)
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:64)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:88)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:157)
> at
> org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:140)
> at
> org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:200)
> at
> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
> at
> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:185)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1551)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1380)
> at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1341)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readFully(Unknown Source)
> at java.io.RandomAccessFile.readFully(Unknown Source)
> at
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:348)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355)
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:110)
> at
> org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:85)
> at org.apache.cassandra.db.Column$1.computeNext(Column.java:75)
> ... 28 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
