There are 14,696,502 documents, we are doing a lot of funky stuff but I'm not sure which is most likely to cause an impact. We're sorting on a dynamic field there are about 1000 different variants of this field that look like "priority_sort_for_<client_id>", which is an integer field. I've heard that sorting can have a big impact on memory consumption, could that be it?
How do I find out about the number of unique words in a field? These aren't very large documents but there is a text field that contains user input, which may be html and javascript so lots of symbols in there. Thanks, -Jeff On Wed, Sep 29, 2010 at 9:07 PM, Lance Norskog <goks...@gmail.com> wrote: > How many documents are there? How many unique words are in a text > field? Both of these numbers can have a non-linear effect on the > amount of space used. > > But, usually a 22Gb index (on disk) might need 6-12G of ram total. > There is something odd going on here. > > Lance > > On Wed, Sep 29, 2010 at 4:34 PM, Jeff Moss <jm...@heavyobjects.com> wrote: > > My server has 128GB of ram, the index is 22GB large. It seems the memory > > consumption goes up on every query and the garbage collector will never > free > > up as much memory as I expect it to. The memory consumption looks like a > > curve, it eventually levels off but the old gen is always 60 or 70GB. I > have > > tried adjusting the cache settings but it doesn't seem to make any > > difference. > > > > Is there something I'm doing wrong or is this expected behavior? > > > > Here is a screenshot of what I see in jconsole after running for a few > > minutes: > > http://i51.tinypic.com/2qntca1.png > > > > Here is a 24 hour period of the same data taken from a custom jmx > monitor: > > http://i51.tinypic.com/2vcu9u8.png > > > > The server performs pretty much as good at the beginning of this cycle as > it > > does at the end so all of this memory accumulation seems to not be doing > > anything useful. > > > > I am running the 1.4 war but I was having this problem with 1.3 also. > Tomcat > > 6.0.18, Java 1.6.0. I haven't gone as far as doing any memory profiling > or > > java debugging because I'm inexperienced, but that will be the next thing > I > > try. Any help would be appreciated. > > > > Thanks, > > > > -Jeff > > > > > > -- > Lance Norskog > goks...@gmail.com >
