Re: DIH OutOfMemoryError?
Make sure streaming is on. --> how to check ? - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other Cores < 100.000 - Solr1 for Search-Requests - commit every Minute - 5GB Xmx - Solr2 for Update-Request - delta every Minute - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/DIH-OutOfMemoryError-tp2759013p2811270.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: DIH OutOfMemoryError?
Make sure streaming is on. Try using autoCommit in solrconfig.xml. This will push documents out of memory onto disk at a regular interval. On Thu, Mar 31, 2011 at 8:51 AM, Markus Jelsma wrote: > Try splitting the files into smaller chunks. It'll help. > >> Hi, >> >> I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an >> OutOfMemoryError! >> >> I've got 2048mo of RAM on this server, obviously it's not enough... How >> much RAM is recomended for indexing big files? >> >> Thanks for your help >> >> >> Here is the error from DIH mode verbose: >> >> java.lang.ClassCastException: >> java.lang.OutOfMemoryError cannot be cast to java.lang.Exception >> at >> org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139) >> at >> org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java >> :422) at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java >> :383) at >> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:24 >> 2) at >> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) >> at >> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.j >> ava:331) at >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:38 >> 9) at >> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(Data >> ImportHandler.java:203) at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase >> .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java: >> 338) at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java >> :241) at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio >> nFilterChain.java:235) at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC >> hain.java:206) at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j >> ava:233) at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j >> ava:191) at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12 >> 7) at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10 >> 2) at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav >> a:109) at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) >> at >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) >> at >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt >> p11Protocol.java:588) at >> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) >> at java.lang.Thread.run(Thread.java:636) >> > -- Lance Norskog goks...@gmail.com
Re: DIH OutOfMemoryError?
Try splitting the files into smaller chunks. It'll help. > Hi, > > I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an > OutOfMemoryError! > > I've got 2048mo of RAM on this server, obviously it's not enough... How > much RAM is recomended for indexing big files? > > Thanks for your help > > > Here is the error from DIH mode verbose: > > java.lang.ClassCastException: > java.lang.OutOfMemoryError cannot be cast to java.lang.Exception > at > org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139) > at > org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java > :422) at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java > :383) at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:24 > 2) at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.j > ava:331) at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:38 > 9) at > org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(Data > ImportHandler.java:203) at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase > .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java: > 338) at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java > :241) at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio > nFilterChain.java:235) at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC > hain.java:206) at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j > ava:233) at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j > ava:191) at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12 > 7) at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10 > 2) at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav > a:109) at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt > p11Protocol.java:588) at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:636) >
DIH OutOfMemoryError?
Hi, I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an OutOfMemoryError! I've got 2048mo of RAM on this server, obviously it's not enough... How much RAM is recomended for indexing big files? Thanks for your help Here is the error from DIH mode verbose: java.lang.ClassCastException: java.lang.OutOfMemoryError cannot be cast to java.lang.Exception at org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139) at org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:422) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:383) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:331) at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389) at org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:203) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:636)
