Another good link is http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html, which also includes details on iCMS, which is the Incremental Mode for CMS.
On 5/15/12 6:32 PM, "Lutz Fechner" <lfech...@hubwoo.com> wrote: >CMS is the concurrent mark sweep garbage collector. Instead of waiting >for the memory to fill up and being collected for memory to be freed up >again, it runs concurrently while the app threads are running. Usually >the JVM would call a full stop the world collection. All threads would >be on hold until GC is finished. CMS would prevent lengthy stops from >happening in trade for constant micro stops. > >This usually is a good option for apps that are sensitive (from a user >experience) from hanging during GC time. > >See http://docs.oracle.com/javase/6/docs/technotes/guides/vm/cms-6.html > >Best Regards > >Lutz > >-----Original Message----- >From: Chris Bamford [mailto:chris.bamf...@talktalk.net] >Sent: Dienstag, 15. Mai 2012 16:38 >To: java-user@lucene.apache.org >Subject: Re: Memory question > > > Hi John, > >Very interesting, thanks for the detailed explanation. It certainly >sounds like the same symptoms! >Can I please clarify a couple of things ? > >- I have googled CMS / iCMS as I wasn't familiar with those acronyms >(apart from 'code management system' and that didn't sound right!) > and am I right in thinking that it is some sort of monitoring code >pulled into your server via a jar? (I'm confused why it would have its' >own > GC cycle...) >- So are you suggesting I play with my own JVM's (Sun/Oracle) parameters >to achieve a similar effect ? > >Thanks again, > >- Chris > > > > > > >-----Original Message----- >From: Nader, John P <john.na...@cengage.com> >To: java-user@lucene.apache.org <java-user@lucene.apache.org> >Sent: Tue, 15 May 2012 21:12 >Subject: Re: Memory question > > >We've encountered this issue and came up with a fairly good approach to >address it. > >We are on Lucene 3.0.2 with Java 1.6.0_29. Our indices are about 35GB >in >size. Our JVM runs at 20GB of heap, with about 12GB of steady usage. >Our >server has 32GB total. > >What would happen in our case is that the linux would page in more and >more of the memory mapped index files into memory, forcing idle portions >of the JVM heap to be swapped out. This was not an issue until our CMS >GC >kicked in. This would force swapping in of all JVM memory to collect >unused references. I/O wait would shoot up and performance would >suffer. >Yes, even CMS can kill performance is you are swapping. The tell-tale >sign was a spike in inbound swap at the start of CMS. > >In our case, we addressed the situation using iCMS, which is Incremental >CMS. This takes the mark phase (and sweep too?) and does it >continuously >with a configurable duty cycle. The result was that swapping was >smoothed >out to be a small stead drag on the system instead of a hard spike. >There >was an small loss in performance, but a big gain in stability. > >This tuning may be an option for you. BTW, pmap will give you statistic >on total file size and how much is resident. The java heap shows up in >pmap as well on linux, so you can determine how much of that is in >memory >as well. > >John > > > >On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamf...@talktalk.net> wrote: > >>Thanks Uwe. >> >>What I'd like to understand is the implications of this on a server >which >>opens a large number of indexes over a long period. Will this non-heap >>memory continue to grow? Will gc be effective at spotting it and >>releasing it via references in the heap? >> >> I had an instance yesterday where a server swapped itself to a >>standstill and had to be restarted. The load average was through the >roof >>and I am trying to understand why. One of my recent changes is updating >>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap >>stuff which is now standard under the covers. >> >>My server caches indexsearchers and then closes them based on how full >>the heap is getting. My worry is that if the bulk of the memory is >being >>allocated outside the Jvm, how can I make sensible decisions? >> >>Thanks for any pointers / info. >> >>Chris >> >> >> >>-----Original Message----- >>From: u...@thetaphi.de >>To: java-user@lucene.apache.org >>Sent: Tue, 15 May 2012 18:10 >>Subject: RE: Memory question >> >> >> >>It mmaps the files into virtual memory if it runs on a 64 bit JVM. >Because >>of that you see the mmapped CFS files. This is outside Java Heap and is >>all >>*virtual* no RAM is explicitely occupied except the O/S cache. >> >>----- >>Uwe Schindler >>H.-H.-Meier-Allee 63, D-28213 Bremen >>http://www.thetaphi.de >>eMail: u...@thetaphi.de >> >>> -----Original Message----- >>> From: Chris Bamford [mailto:chris.bamf...@talktalk.net] >>> Sent: Tuesday, May 15, 2012 4:47 PM >>> To: java-user@lucene.apache.org >>> Subject: Memory question >>> >>> Hi >>> >>> Can anyone tell me what happens to the memory when Lucene opens an >>>index? >>> Is it loaded into the JVM's heap or is it mapped into virtual memory >>outside of >>> it? >>> I am running on Linux and if I use pmap on the PID of my JVM, I can >see >>lots of >>> entries for index cfs files. >>> >>> Does this mean that indexes are mapped into non-heap memory? If so, >how >>> can I monitor the space my process is using if I cache open >>IndexSearchers? >>> >>> The details are: >>> >>> Sun 64-bit JVM on Linux. >>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the >>process of >>> a migration to 3.6) >>> >>> Thanks, >>> >>> - Chris >> >>--------------------------------------------------------------------- >>To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >For additional commands, e-mail: java-user-h...@lucene.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
