Make sure streaming is on. Try using autoCommit in solrconfig.xml. This will push documents out of memory onto disk at a regular interval.
On Thu, Mar 31, 2011 at 8:51 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: > Try splitting the files into smaller chunks. It'll help. > >> Hi, >> >> I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an >> OutOfMemoryError! >> >> I've got 2048mo of RAM on this server, obviously it's not enough... How >> much RAM is recomended for indexing big files? >> >> Thanks for your help >> >> >> Here is the error from DIH mode verbose: >> >> <str name="EXCEPTION">java.lang.ClassCastException: >> java.lang.OutOfMemoryError cannot be cast to java.lang.Exception >> at >> org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139) >> at >> org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java >> :422) at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java >> :383) at >> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:24 >> 2) at >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) >> at >> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.j >> ava:331) at >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:38 >> 9) at >> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(Data >> ImportHandler.java:203) at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase >> .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java: >> 338) at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java >> :241) at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio >> nFilterChain.java:235) at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC >> hain.java:206) at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j >> ava:233) at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j >> ava:191) at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12 >> 7) at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10 >> 2) at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav >> a:109) at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) >> at >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) >> at >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt >> p11Protocol.java:588) at >> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) >> at java.lang.Thread.run(Thread.java:636) >> </str> > -- Lance Norskog goks...@gmail.com