Hi, 25*100MB=2.5GB will most likely fail with just 4GB of heap space. But consecutive single `pushes` as you call it, of 25MB documents should work fine. Heap memory will only drop after the garbage collector comes along.
Cheers, On Thursday 17 March 2011 17:12:46 Geeta Subramanian wrote: > Hi, > > I am very new to SOLR and facing a lot of issues when using SOLR to push > large documents. I have solr running in tomcat. I have allocated about 4gb > memory (-Xmx) but I am pushing about twenty five 100 mb documents and > gives heap space and fails. > > Also I tried pushing just 1 document. It went thru successfully, but the > tomcat memory does not come down. It consumes about a gig memory for just > one 100 mb document and does not release it. > > Please let me know if I am making any mistake in configuration/ or set up. > > Here is the stack trace: > SEVERE: java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2882) > at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java: > 100) at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515) at > java.lang.StringBuffer.append(StringBuffer.java:306) > at java.io.StringWriter.write(StringWriter.java:77) > at > com.sun.org.apache.xml.internal.serializer.ToStream.processDirty(ToStream. > java:1570) at > com.sun.org.apache.xml.internal.serializer.ToStream.characters(ToStream.ja > va:1488) at > com.sun.org.apache.xml.internal.serializer.ToHTMLStream.characters(ToHTMLS > tream.java:1529) at > com.sun.org.apache.xalan.internal.xsltc.trax.TransformerHandlerImpl.charac > ters(TransformerHandlerImpl.java:168) at > org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecor > ator.java:124) at > org.apache.tika.sax.SecureContentHandler.characters(SecureContentHandler.j > ava:153) at > org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecor > ator.java:124) at > org.apache.tika.sax.ContentHandlerDecorator.characters(ContentHandlerDecor > ator.java:124) at > org.apache.tika.sax.SafeContentHandler.access$001(SafeContentHandler.java: > 39) at > org.apache.tika.sax.SafeContentHandler$1.write(SafeContentHandler.java:61) > at > org.apache.tika.sax.SafeContentHandler.filter(SafeContentHandler.java:113) > at > org.apache.tika.sax.SafeContentHandler.characters(SafeContentHandler.java: > 151) at > org.apache.tika.sax.XHTMLContentHandler.characters(XHTMLContentHandler.jav > a:175) at org.apache.tika.parser.txt.TXTParser.parse(TXTParser.java:144) at > org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:142) at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:99) at > com.commvault.solr.handler.extraction.CVExtractingDocumentLoader.load(CVEx > tractingDocumentLoader.java:349) at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Content > StreamHandlerBase.java:54) at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBas > e.java:131) at > org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleReque > st(RequestHandlers.java:237) at > org.apache.solr.core.SolrCore.execute(SolrCore.java:1323) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java > :337) at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav > a:240) at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicati > onFilterChain.java:235) at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilter > Chain.java:206) at > filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.jav > a:122) > > > Thanks for help, > Geeta > > > > > > > > > > > > > > ******************Legal Disclaimer*************************** > "This communication may contain confidential and privileged material > for the sole use of the intended recipient. Any unauthorized review, > use or distribution by others is strictly prohibited. If you have > received the message in error, please advise the sender by reply > email and delete the message. Thank you." > **************************************************************** -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
