Hi Joseph, The way the CSV importer works, for safety, is to process the whole CSV at once, and hold it all in memory (rather than reading and processing it line-by-line). Therefore is probably safer to just import one or two thousand records at a time. THe reason it is probably running out of memory half way through, is that as it goes through each line, it is building up its records of what has changed, so that it can display this information at the end. So it has enough memory to read your long CSV file, but not enough to hold that, and the details of all the changes it has made.
Unfortunately no extra information (e.g. about checking the write lock) can be given, as the error you are encountering is an 'out of memory error', and these are provided directly by Java, not DSpace, so we have no control over them. Thanks, Stuart Lewis Digital Development Manager Te Tumu Herenga The University of Auckland Library Auckland Mail Centre, Private Bag 92019, Auckland 1142, New Zealand Ph: +64 (0)9 373 7599 x81928 On 5/02/2011, at 6:47 AM, Joseph Rhoads wrote: > Dear Dspace-tech community, > > I'm curious if anyone else has had problems using the command line > metadata-import? > > I'm trying to import 12861 items and it dies about half-way through.(7582 > items). > Before I upped the JAVA_OPTS memory max it would die between 2k and 3k items. > > The error message is below, but an additional problem that the crash creates > is that it leaves a "write.lock" file in "[dspace home]/search/" > Which will create problems if you don't realize that it's there. > Perhaps something should be mentioned in the error message when that happens. > > Thank You, > Joseph > > Server Setup: > ------------------- > Using Dspace 1.7 > Ubuntu 10.04 Server > JAVA_OPTS=-Xmx512m > Postgresql-8.4 > > Error Message > -------------------------- > 12861 item(s) will be changed > > Do you want to make these changes? [y/n] y > Exception: Java heap space > java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.store.BufferedIndexOutput.<init>(BufferedIndexOutput.java:26) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:185) > at > org.apache.lucene.store.SimpleFSDirectory.createOutput(SimpleFSDirectory.java:59) > at > org.apache.lucene.index.FormatPostingsPositionsWriter.<init>(FormatPostingsPositionsWriter.java:42) > at > org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:55) > at > org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33) > at > org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59) > at > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:96) > at org.apache.lucene.index.TermsHash.flush(TermsHash.java:154) > at org.apache.lucene.index.DocInverter.flush(DocInverter.java:76) > at > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:63) > at > org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:634) > at > org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4374) > at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4264) > at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4255) > at > org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:2244) > at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2207) > at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:2171) > at org.dspace.search.DSIndexer.processIndexingTask(DSIndexer.java:714) > at org.dspace.search.DSIndexer.indexContent(DSIndexer.java:271) > at org.dspace.search.SearchConsumer.end(SearchConsumer.java:181) > at org.dspace.event.BasicDispatcher.dispatch(BasicDispatcher.java:149) > at org.dspace.core.Context.commit(Context.java:336) > at > org.dspace.app.bulkedit.MetadataImport.runImport(MetadataImport.java:254) > at > org.dspace.app.bulkedit.MetadataImport.main(MetadataImport.java:1060) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:183) > --------------------- > > > > <ATT00001..txt><ATT00002..txt> ------------------------------------------------------------------------------ The modern datacenter depends on network connectivity to access resources and provide services. The best practices for maximizing a physical server's connectivity to a physical network are well understood - see how these rules translate into the virtual world? http://p.sf.net/sfu/oracle-sfdevnlfb _______________________________________________ DSpace-tech mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/dspace-tech

