Hi, I am trying to index application log files and some database tables. Size of the log files range from 1 MB to 100 MB. Database tables also have few thousands of rows.
I have used termvector highlighter for the content of the log files as mentioned below: Heap size : 10 GB OS: Linux, 64 bit Solr version : 3.4.0 Thanks & Regards Anand Anand Nigam RBS Global Banking & Markets Office: +91 124 492 5506 -----Original Message----- From: Glen Newton [mailto:glen.new...@gmail.com] Sent: 19 September 2011 16:52 To: solr-user@lucene.apache.org Subject: Re: OutOfMemoryError coming from TermVectorsReader Please include information about your heap size, (and other Java command line arguments) as well a platform OS (version, swap size, etc), Java version, underlying hardware (RAM, etc) for us to better help you. >From the information you have given, increasing your heap size should help. Thanks, Glen http://zzzoot.blogspot.com/ On Mon, Sep 19, 2011 at 1:34 AM, <anand.ni...@rbs.com> wrote: > Hi, > > I am new to solr. I an trying to index text documents of large size. On > searching from indexed documents I am getting following OutOfMemoryError. > Please help me in resolving this issue. > > The field which stores file content is configured in schema.xml as below: > > > <field name="Content" type="text_token" indexed="true" stored="true" > omitNorms="true" termVectors="true" termPositions="true" > termOffsets="true" /> > > and Highlighting is configured as below: > > > <str name="hl">on</str> > > <str name="hl.fl">${all.fields.list}</str> > > <str name="f.Content.hl.fragsize">500</str> > > <str name="f.Content.hl.useFastVectorHighlighter">true</str> > > > > 2011-09-16 09:38:45.763 [http-thread-pool-9091(5)] ERROR - > java.lang.OutOfMemoryError: Java heap space > at > org.apache.lucene.index.TermVectorsReader.readTermVector(TermVectorsRe > ader.java:503) > at > org.apache.lucene.index.TermVectorsReader.get(TermVectorsReader.java:2 > 63) > at > org.apache.lucene.index.TermVectorsReader.get(TermVectorsReader.java:2 > 84) > at > org.apache.lucene.index.SegmentReader.getTermFreqVector(SegmentReader. > java:759) > at > org.apache.lucene.index.DirectoryReader.getTermFreqVector(DirectoryRea > der.java:510) > at > org.apache.solr.search.SolrIndexReader.getTermFreqVector(SolrIndexRead > er.java:234) > at > org.apache.lucene.search.vectorhighlight.FieldTermStack.<init>(FieldTe > rmStack.java:83) > at > org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFiel > dFragList(FastVectorHighlighter.java:175) > at > org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getBest > Fragments(FastVectorHighlighter.java:166) > at > org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByFastV > ectorHighlighter(DefaultSolrHighlighter.java:509) > at > org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(Defaul > tSolrHighlighter.java:376) > at > org.apache.solr.handler.component.HighlightComponent.process(Highlight > Component.java:116) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear > chHandler.java:194) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle > rBase.java:129) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter. > java:356) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter > .java:252) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilterChain.java:256) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.java:215) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > lve.java:279) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > lve.java:175) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.ja > va:655) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java > :595) > at > com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98) > at > com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessi > onLockingStandardPipeline.java:91) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > va:162) > at > org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.ja > va:326) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > :227) > at > com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerM > apper.java:170) > at > com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:82 > 2) > at > com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719) > at > com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013) > > Thanks & Regards > Anand Nigam > Developer > > > ********************************************************************** > ************* The Royal Bank of Scotland plc. Registered in Scotland > No 90312. > Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB. > Authorised and regulated by the Financial Services Authority. The > Royal Bank of Scotland N.V. is authorised and regulated by the De > Nederlandsche Bank and has its seat at Amsterdam, the Netherlands, and > is registered in the Commercial Register under number 33002587. > Registered Office: Gustav Mahlerlaan 350, Amsterdam, The Netherlands. > The Royal Bank of Scotland N.V. and The Royal Bank of Scotland plc are > authorised to act as agent for each other in certain jurisdictions. > > This e-mail message is confidential and for use by the addressee only. > If the message is received by anyone other than the addressee, please > return the message to the sender by replying to it and then delete the > message from your computer. Internet e-mails are not necessarily > secure. The Royal Bank of Scotland plc and The Royal Bank of Scotland > N.V. including its affiliates ("RBS group") does not accept > responsibility for changes made to this message after it was sent. For > the protection of RBS group and its clients and customers, and in > compliance with regulatory requirements, the contents of both incoming > and outgoing e-mail communications, which could include proprietary > information and Non-Public Personal Information, may be read by > authorised persons within RBS group other than the intended recipient(s). > > Whilst all reasonable care has been taken to avoid the transmission of > viruses, it is the responsibility of the recipient to ensure that the > onward transmission, opening or use of this message and any > attachments will not adversely affect its systems or data. No > responsibility is accepted by the RBS group in this regard and the > recipient should carry out such virus and other checks as it considers > appropriate. > > Visit our website at www.rbs.com > > ********************************************************************** > ************* > -- - -
