Here you have the entire stack trace I just got (*now it's a different
error...*):
java.io.FileNotFoundException:
/mnt/bio4j_volume/bio4jdb/lucene/organism_scientific_name_index/_1a9e.fdx
(Too many open files)
org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.index(LuceneIndexBatchInserterImpl.java:186)
com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:467)
com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192)
com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16)
java.lang.RuntimeException: java.io.FileNotFoundException:
/mnt/bio4j_volume/bio4jdb/lucene/instit ute_name_index/_c7.frq
(Too many open files)
at
org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:273)
at
org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.shutdown(LuceneIndexBatchInserterImpl.java:214)
at
com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:552)
at
com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192)
at
com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
at
com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16)
Caused by: java.io.FileNotFoundException:
/mnt/bio4j_volume/bio4jdb/lucene/institute_name_index/_c7.frq (Too many open
files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180)
at
org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74)
at
org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47)
at
org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33)
at
org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:94)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:144)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:72)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64)
at
org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:583)
at
org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3598)
at
org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3509)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3500)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2271)
at
org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2249)
at
org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:267)
... 5 more
I posted another question to the mail list regarding optimize() indexservice
which I guess could have something
to do with this exception,
Thanks in advance
Pablo
On Tue, Feb 15, 2011 at 11:57 AM, Pablo Pareja <[email protected]> wrote:
> Ok, I remember there was a java.io.IOException afterwards, unfortunately I
> didn't copy
> that part thinking it was the consequence of the previous one.
> Anyways I think it said something about an index file that existed but
> couldn't be accessed for an unknown reason,
> is that possible?
> I'll try to reproduce it again,
>
> Pablo
>
>
> On Tue, Feb 15, 2011 at 10:45 AM, Tobias Ivarsson <
> [email protected]> wrote:
>
>> There should have been a cause at the end of that trace, looking something
>> like:
>>
>> caused by: java.io.IOException ...
>>
>> That is what I'm interested in.
>>
>> -tobias
>>
>> On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja <[email protected]> wrote:
>>
>> > I already put the stack trace in the first message:
>> >
>> > Unable to write record[244986398] @[32583190934]
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
>> >
>> org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115)
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
>> >
>> >
>> org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351)
>> >
>> > the missing lines are just from my own classes ending in a
>> > BatchInserter.shutdown() method.
>> >
>> > Cheers,
>> >
>> > Pablo
>> >
>> > On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson <
>> > [email protected]> wrote:
>> >
>> > > I had a second look at this, and realized that those memory mapping
>> > > settings
>> > > aren't used from the batch inserter. Everything is allocated in the
>> Java
>> > > Heap. 63G out of 64G RAM is still a bit much, but not as big of a
>> problem
>> > > as
>> > > I first thought.
>> > >
>> > > This also means that there could be another problem behind the
>> exception
>> > > you
>> > > receive. There should have been a causing exception for the one you
>> > listed.
>> > > Could you please provide a full stack trace?
>> > >
>> > > Cheers,
>> > > Tobias
>> > >
>> > > On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja <[email protected]>
>> wrote:
>> > >
>> > > > Hi all,
>> > > > I keep getting this exception and don't know what the reason could
>> be:
>> > > >
>> > > > Unable to write record[244986398] @[32583190934]
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611)
>> > > >
>> > > >
>> > >
>> >
>> org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351)
>> > > >
>> > > > I'm using a 65G RAM machine and the java process is launched with
>> > -Xmx63G
>> > > > the conf values I pass to the batch inserter are the following:
>> > > >
>> > > > neostore.nodestore.db.mapped_memory=10G
>> > > > neostore.relationshipstore.db.mapped_memory=10G
>> > > > neostore.propertystore.db.mapped_memory=10G
>> > > > neostore.propertystore.db.strings.mapped_memory=10G
>> > > > neostore.propertystore.db.arrays.mapped_memory=0M
>> > > >
>> > > > Besides, there were still 50G free space in the disk when this
>> > exception
>> > > > was
>> > > > thrown
>> > > >
>> > > > I'd really appreciate any help,
>> > > > Cheers
>> > > >
>> > > > --
>> > > > Pablo Pareja Tobes
>> > > > LinkedIn http://www.linkedin.com/in/pabloparejatobes
>> > > > Twitter http://www.twitter.com/pablopareja
>> > > >
>> > > > http://www.ohnosequences.com
>> > > > _______________________________________________
>> > > > Neo4j mailing list
>> > > > [email protected]
>> > > > https://lists.neo4j.org/mailman/listinfo/user
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Tobias Ivarsson <[email protected]>
>> > > Hacker, Neo Technology
>> > > www.neotechnology.com
>> > > Cellphone: +46 706 534857
>> > > _______________________________________________
>> > > Neo4j mailing list
>> > > [email protected]
>> > > https://lists.neo4j.org/mailman/listinfo/user
>> > >
>> >
>> >
>> >
>> > --
>> > Pablo Pareja Tobes
>> > LinkedIn http://www.linkedin.com/in/pabloparejatobes
>> > Twitter http://www.twitter.com/pablopareja
>> >
>> > http://www.ohnosequences.com
>> > _______________________________________________
>> > Neo4j mailing list
>> > [email protected]
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>>
>>
>>
>> --
>> Tobias Ivarsson <[email protected]>
>> Hacker, Neo Technology
>> www.neotechnology.com
>> Cellphone: +46 706 534857
>> _______________________________________________
>> Neo4j mailing list
>> [email protected]
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Pablo Pareja Tobes
> LinkedIn http://www.linkedin.com/in/pabloparejatobes
> Twitter http://www.twitter.com/pablopareja
>
> http://www.ohnosequences.com
>
>
--
Pablo Pareja Tobes
LinkedIn http://www.linkedin.com/in/pabloparejatobes
Twitter http://www.twitter.com/pablopareja
http://www.ohnosequences.com
_______________________________________________
Neo4j mailing list
[email protected]
https://lists.neo4j.org/mailman/listinfo/user
