Thanks for the tips Alexandru. I just switched to JDK1.7 and gave G1 a try. Anecdotally, it doesn't appear to help much, but I've got more testing to do. I've also tried both the parallel collector and concurrent mark and sweep.
Another thing that is puzzling to me is that when I run the application on my laptop (MacBook Pro Retina with a SSD, 16 GB RAM, JDK1.6) on a smaller amount of data it works very well. However, when I run it on the server (CentOS 5, also with a SSD, 320GB RAM, JDK1.6) on the same smaller data set, it pretty much grinds to a halt. I've been wondering if it's related to NUMA, or some system setting. When we installed CentOS, we used the performance settings (not factory settings). It just blows my mind that my laptop can be more performant than a beefy server. There are many differences between the environments (I know, this is a problem), but my understanding was that Mac OS X was generally behind on their JVM, so if it worked there, it would most likely work as well or better in another environment (all else being as close as possible). Given that I don't have the ability in the short term to change the differences in the environments, it seems like the first logical step is to understand the differences (architecture, chipset, OS settings, JVM version, etc...). There are so many degrees of freedom that it's hard to know where to start in unraveling this mystery. Cheers, Bruce On Thursday, 27 March 2014 10:36:15 UTC-4, Alexandru Nedelcu wrote: > > Also, I see that UseNUMA is available in combination with the parallel > collector, which is the default apparently - but that's awful. In case you > haven't, also try CMS (the concurrent one) and G1 (which is supposedly > built to be non-blocking and to work well on big heaps with many cores). It > can make for a big difference. > > For processing TB of data, ultimately you'll want to scale that > horizontally. > > > On Thu, Mar 27, 2014 at 4:30 PM, Alexandru Nedelcu > <[email protected]<javascript:> > > wrote: > >> What GC params are you using. Have you given G1 a spin? >> >> >> On Thu, Mar 27, 2014 at 4:06 PM, Bruce <[email protected] <javascript:>>wrote: >> >>> Hi, >>> >>> I've been struggling for some time with a large scale application, >>> trying to find optimal JVM settings to address massive GC pauses. The app >>> will process hundreds of GB of data (ultimately in the TB range), and >>> employ as many cores as possible on a single machine (currently 48, but >>> will be 160 soon). I have no problem using up all of the cores, all of the >>> time. This doesn't really help me since I have this GC anvil around my >>> neck. I've tried just about every combination of flags, but with not much >>> success. To try to alleviate the problem of large GC pauses, I've employed >>> an off-heap cache for the data (MapDB). This has helped a bit, but I'm >>> coming to the conclusion that the only way to crack this nut is to break >>> the problem into coarse granular units in separate JVMs, peg them to NUMA >>> nodes, and communicate between them. Is this what others would recommend, >>> or is there a way to do something like this within a single JVM? If I go >>> the multi-JVM route, what purpose would -XX:+UseNUMA serve? For that >>> matter, what purpose does -XX:+UseNUMA serve now?...it doesn't really seem >>> to do anything. Would love to hear your thoughts. >>> >>> Thanks, >>> Bruce >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To post to this group, send email to [email protected]<javascript:> >>> . >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Alexandru Nedelcu >> www.bionicspirit.com >> >> PGP Public Key: >> https://bionicspirit.com/key.aexpk >> > > > > -- > Alexandru Nedelcu > www.bionicspirit.com > > PGP Public Key: > https://bionicspirit.com/key.aexpk > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
