[
https://issues.apache.org/jira/browse/CASSANDRA-1423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12901631#action_12901631
]
Dominic Williams commented on CASSANDRA-1423:
---------------------------------------------
Looks like this bug might hopefully be a straightforward case of not cleaning
up file handles (or at least, always opening files afresh, and then not
handling the error when you can't open any more). I should have checked logs.
>From system.log:
ERROR [pool-1-thread-74] 2010-08-23 23:18:34,749 Cassandra.java (line 2651)
Internal error processing get_slice
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.io.IOError: java.io.FileNotFoundException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
at
org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:354)
at
org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:297)
at
org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:125)
at
org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:231)
at
org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:309)
at
org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:270)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:2643)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2499)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.ExecutionException: java.io.IOError:
java.io.FileNotFoundException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:350)
... 11 more
Caused by: java.io.IOError: java.io.FileNotFoundException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
at
org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:68)
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:545)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:71)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:48)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:64)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:76)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:956)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:851)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:826)
at org.apache.cassandra.db.Table.getRow(Table.java:330)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:71)
at
org.apache.cassandra.service.StorageProxy$weakReadLocalCallable.call(StorageProxy.java:816)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.io.FileNotFoundException:
/var/lib/cassandra/data/FightMyMonster/UK_Schools_FullTextIndex-e-2-Data.db
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
at
org.apache.cassandra.io.util.BufferedSegmentedFile.getSegment(BufferedSegmentedFile.java:62)
... 16 more
> Cassandra's internal state broke by getting column slices. Error
> org.apache.thrift.TApplicationException: Internal error processing get_slice
> afterwards.
> ---------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-1423
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1423
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Environment: Ubuntu 9.04
> Reporter: Dominic Williams
> Priority: Critical
> Fix For: 0.7 beta 2
>
> Attachments: fmm-add-schools.zip
>
>
> The attached program (with data that it can import) causes large column
> slices to be requested from Cassandra.
> The program itself uploads school address data to a Cassyndex full text
> index, and then allows you to search that. The program simulates someone
> typing a search into an active search box, which shows you the matches for
> the current term as you type. Thus when you enter a search term such as
> "cherwell school oxford" actually it performs the searches "c", "ch", "che",
> "cher", "cherw" etc
> You can configure the delay between the "keystrokes". If your delay allows
> the searches to complete sequentially, you are ok. But if you have a short
> delay, and searches are created in parallel, pretty quickly this error will
> arise - "org.apache.thrift.TApplicationException: Internal error processing
> get_slice".
> Once this has occurred all future attempts and getting slices of columns will
> return the same error, and your'e only option is to restart Cassandra.
> This looks like some kind of concurrency edge condition bug caused by
> requesting sufficiently large intersecting slices in parallel. It may be in
> other versions too.
> I've been testing on 0.7 B1 using an RP cluster.
> The attached maven project should pull down the scale7 libraries but if
> interested you can find the sources at http://github.com/s7
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
