Anand, But do you really want the whole log file to be a single Solr document (from a cursory look at the thread it seems that is the case). Why not break up a log file into multiple documents? e.g. each log message could be one Solr document. Not only will that solve your memory issues, but I think it also makes more sense if the intention is for a person to do a search and then look at the matched log messages - much easier if you point a person to a short log doc than a giant ones through which the person then has to do a manual find.
Otis ---- Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ ----- Original Message ----- > From: "anand.ni...@rbs.com" <anand.ni...@rbs.com> > To: solr-user@lucene.apache.org > Cc: > Sent: Thursday, September 22, 2011 11:56 PM > Subject: RE: OutOfMemoryError coming from TermVectorsReader > > Hi, > > I am trying to index application log files and some database tables. Size of > the > log files range from 1 MB to 100 MB. Database tables also have few thousands > of > rows. > > I have used termvector highlighter for the content of the log files as > mentioned > below: > > Heap size : 10 GB > OS: Linux, 64 bit > Solr version : 3.4.0 > > Thanks & Regards > Anand > > > > Anand Nigam > RBS Global Banking & Markets > Office: +91 124 492 5506 > > -----Original Message----- > From: Glen Newton [mailto:glen.new...@gmail.com] > Sent: 19 September 2011 16:52 > To: solr-user@lucene.apache.org > Subject: Re: OutOfMemoryError coming from TermVectorsReader > > Please include information about your heap size, (and other Java command line > arguments) as well a platform OS (version, swap size, etc), Java version, > underlying hardware (RAM, etc) for us to better help you. > > From the information you have given, increasing your heap size should help. > > Thanks, > Glen > > http://zzzoot.blogspot.com/ > > > On Mon, Sep 19, 2011 at 1:34 AM, <anand.ni...@rbs.com> wrote: >> Hi, >> >> I am new to solr. I an trying to index text documents of large size. On > searching from indexed documents I am getting following OutOfMemoryError. > Please > help me in resolving this issue. >> >> The field which stores file content is configured in schema.xml as below: >> >> >> <field name="Content" type="text_token" > indexed="true" stored="true" >> omitNorms="true" termVectors="true" > termPositions="true" >> termOffsets="true" /> >> >> and Highlighting is configured as below: >> >> >> <str name="hl">on</str> >> >> <str name="hl.fl">${all.fields.list}</str> >> >> <str name="f.Content.hl.fragsize">500</str> >> >> <str > name="f.Content.hl.useFastVectorHighlighter">true</str> >> >> >> >> 2011-09-16 09:38:45.763 [http-thread-pool-9091(5)] ERROR - >> java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.lucene.index.TermVectorsReader.readTermVector(TermVectorsRe >> ader.java:503) >> at >> org.apache.lucene.index.TermVectorsReader.get(TermVectorsReader.java:2 >> 63) >> at >> org.apache.lucene.index.TermVectorsReader.get(TermVectorsReader.java:2 >> 84) >> at >> org.apache.lucene.index.SegmentReader.getTermFreqVector(SegmentReader. >> java:759) >> at >> org.apache.lucene.index.DirectoryReader.getTermFreqVector(DirectoryRea >> der.java:510) >> at >> org.apache.solr.search.SolrIndexReader.getTermFreqVector(SolrIndexRead >> er.java:234) >> at >> > org.apache.lucene.search.vectorhighlight.FieldTermStack.<init>(FieldTe >> rmStack.java:83) >> at >> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFiel >> dFragList(FastVectorHighlighter.java:175) >> at >> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getBest >> Fragments(FastVectorHighlighter.java:166) >> at >> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByFastV >> ectorHighlighter(DefaultSolrHighlighter.java:509) >> at >> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(Defaul >> tSolrHighlighter.java:376) >> at >> org.apache.solr.handler.component.HighlightComponent.process(Highlight >> Component.java:116) >> at >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear >> chHandler.java:194) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle >> rBase.java:129) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter. >> java:356) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter >> .java:252) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli >> cationFilterChain.java:256) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi >> lterChain.java:215) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa >> lve.java:279) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa >> lve.java:175) >> at >> org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.ja >> va:655) >> at >> org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java >> :595) >> at >> com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98) >> at >> com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessi >> onLockingStandardPipeline.java:91) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja >> va:162) >> at >> org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.ja >> va:326) >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java >> :227) >> at >> com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerM >> apper.java:170) >> at >> com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:82 >> 2) >> at >> com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719) >> at >> com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013) >> >> Thanks & Regards >> Anand Nigam >> Developer >> >> >> ********************************************************************** >> ************* The Royal Bank of Scotland plc. Registered in Scotland >> No 90312. >> Registered Office: 36 St Andrew Square, Edinburgh EH2 2YB. >> Authorised and regulated by the Financial Services Authority. The >> Royal Bank of Scotland N.V. is authorised and regulated by the De >> Nederlandsche Bank and has its seat at Amsterdam, the Netherlands, and >> is registered in the Commercial Register under number 33002587. >> Registered Office: Gustav Mahlerlaan 350, Amsterdam, The Netherlands. >> The Royal Bank of Scotland N.V. and The Royal Bank of Scotland plc are >> authorised to act as agent for each other in certain jurisdictions. >> >> This e-mail message is confidential and for use by the addressee only. >> If the message is received by anyone other than the addressee, please >> return the message to the sender by replying to it and then delete the >> message from your computer. Internet e-mails are not necessarily >> secure. The Royal Bank of Scotland plc and The Royal Bank of Scotland >> N.V. including its affiliates ("RBS group") does not accept >> responsibility for changes made to this message after it was sent. For >> the protection of RBS group and its clients and customers, and in >> compliance with regulatory requirements, the contents of both incoming >> and outgoing e-mail communications, which could include proprietary >> information and Non-Public Personal Information, may be read by >> authorised persons within RBS group other than the intended recipient(s). >> >> Whilst all reasonable care has been taken to avoid the transmission of >> viruses, it is the responsibility of the recipient to ensure that the >> onward transmission, opening or use of this message and any >> attachments will not adversely affect its systems or data. No >> responsibility is accepted by the RBS group in this regard and the >> recipient should carry out such virus and other checks as it considers > appropriate. >> >> Visit our website at www.rbs.com >> >> ********************************************************************** >> ************* >> > > > > -- > - > > - >
