Lucene won't load the whole index into memory. See http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
What version of lucene? How are you opening index readers? How are you searching? How much memory are you giving the jvm? What else in your app is using all the memory? What else is going on on your server? -- Ian. On Tue, Jan 29, 2013 at 5:54 AM, dizh <d...@neusoft.com> wrote: > Hi All: > > I hava a large Index repo, size is 47G, yes 47G , when I look up for > something in it , and then it is dead, I trace its execution as follows: > > at > org.apache.lucene.search.TopFieldCollector.add(TopFieldCollector.java:1178) > at > org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.collect(TopFieldCollector.java:87) > at org.apache.lucene.search.Scorer.score(Scorer.java:62) > at > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:238) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:588) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:555) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:507) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:484) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309) > > I think it is because my Index is too large , and all jvm heaps are used, and > I want to ask for some answers: > > How I can reduce the memory which lucene used, Linux Top command result is as > follow: > > top - 13:45:11 up 2:56, 3 users, load average: 0.59, 0.33, 0.51 > Tasks: 193 total, 1 running, 192 sleeping, 0 stopped, 0 zombie > Cpu(s): 13.2%us, 0.2%sy, 0.0%ni, 86.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st > Mem: 3933684k total, 3793900k used, 139784k free, 21704k buffers > Swap: 20482864k total, 490644k used, 19992220k free, 1408172k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 5318 root 21 0 47.7g 1.6g 267m S 99.9 41.4 0:50.84 java > 5536 root 15 0 596m 67m 5856 S 3.0 1.8 0:26.22 python > > it is clearly Lucene load all index into memory, and I also know after this > query the memory is returned to OS, but just in this query, it is hung. > > Could anyone give some suggestions about how to query large Index on a single > machine not distributed? > > > --------------------------------------------------------------------------------------------------- > Confidentiality Notice: The information contained in this e-mail and any > accompanying attachment(s) > is intended only for the use of the intended recipient and may be > confidential and/or privileged of > Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of > this communication is > not the intended recipient, unauthorized use, forwarding, printing, storing, > disclosure or copying > is strictly prohibited, and may be unlawful.If you have received this > communication in error,please > immediately notify the sender by return e-mail, and delete the original > message and all copies from > your system. Thank you. > --------------------------------------------------------------------------------------------------- --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
