Re: DIH OutOfMemoryError?

[stockii]

Make sure streaming is on. 
-- how to check ? 

-
--- System 

One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 
1 Core with 31 Million Documents other Cores  100.000

- Solr1 for Search-Requests - commit every Minute  - 5GB Xmx
- Solr2 for Update-Request  - delta every Minute - 4GB Xmx
--
View this message in context: 
http://lucene.472066.n3.nabble.com/DIH-OutOfMemoryError-tp2759013p2811270.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH OutOfMemoryError?

[Lance Norskog]

Make sure streaming is on.
Try using autoCommit in solrconfig.xml. This will push documents out
of memory onto disk at a regular interval.

On Thu, Mar 31, 2011 at 8:51 AM, Markus Jelsma
markus.jel...@openindex.io wrote:
 Try splitting the files into smaller chunks. It'll help.

 Hi,

 I'm trying to index a big XML file (800Mo) using DIH, but i'm getting an
 OutOfMemoryError!

 I've got 2048mo of RAM on this server, obviously it's not enough... How
 much RAM is recomended for indexing big files?

 Thanks for your help


 Here is the error from DIH mode verbose:

 str name=EXCEPTIONjava.lang.ClassCastException:
 java.lang.OutOfMemoryError cannot be cast to java.lang.Exception
      at
 org.apache.solr.handler.dataimport.DebugLogger.log(DebugLogger.java:139)
      at
 org.apache.solr.handler.dataimport.SolrWriter.log(SolrWriter.java:237)
      at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java
 :422) at
 org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java
 :383) at
 org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:24
 2) at
 org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180)
      at
 org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.j
 ava:331) at
 org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:38
 9) at
 org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(Data
 ImportHandler.java:203) at
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase
 .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
      at
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:
 338) at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java
 :241) at
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
 nFilterChain.java:235) at
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
 hain.java:206) at
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j
 ava:233) at
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
 ava:191) at
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12
 7) at
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10
 2) at
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
 a:109) at
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
      at
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
      at
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt
 p11Protocol.java:588) at
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
      at java.lang.Thread.run(Thread.java:636)
 /str




-- 
Lance Norskog
goks...@gmail.com