That's good advice, Mark. You're right on about the OOM killer bit. I actually had just realized that Tomcat will always be killed by OOM if OOM is invoked, because Tomcat of course is carving a substantial heap of memory for itself. :)
I just had a look at my DSpace Control panel and found that the memory usage is less than half of what I've allocated Tomcat, and because I've pinned -Xms and -Xmx, Java is using it all anyways. I know the DSpace guide recommends this, but I'm not sure how I feel about it. In any case, I've reduced my -Xmx for Tomcat now, and they are currently: JAVA_OPTS="-Djava.awt.headless=true -Xms1024m -Xms2048m -XX:MaxPermSize=320m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode" I didn't realize bin/dspace was setting a heap size, that's interesting. I wonder which memory settings my apps were using then that caused them to run away with memory and incur the wrath of the OOM killer (upon Tomcat's java!). I will now explicitly set them in /etc/environment I think, as well as start logging the output of each cron job so I can see what's going wrong. I'm not keen to start messing with memory options in individual scripts; it seems like a maintenance nightmare, haha (sure to forget what I set where, with different test servers laying around etc). Cheers, Alan On 05/20/2014 04:10 PM, Mark H. Wood wrote: > On Tue, May 20, 2014 at 11:33:48AM +0300, Alan Orth wrote: >> I'm curious if anyone sets memory limits for DSpace's various cron jobs? > > As delivered, DSpace does that itself. In bin/dspace, around line 69, > the script sets a maximum heap size of 256 MB, *unless* you already > have JAVA_OPTS set. One thing you might check is whether there is > already a JAVA_OPTS value which sets substantially higher limits for > any of Java's memory pools. > >> Lately we've been having Tomcat's java process get killed every morning >> around the same time, and all dmesg shows is that "java" was killed by >> the kernel's OOM killer. Catalina logs don't show any "SEVERE" errors, >> so I have to assume it's the cron jobs which are using up loads of >> memory and then confusing the kernel, which then identifies Tomcat's >> java as the memory hog and kills it. > > The kernel's notion of the best candidate for the OOM Killer probably > has less to do with fixing blame than with recovering a big hunk of > memory, making Tomcat a tempting target no matter which process you > think is at fault. > >> So I'm just curious if anyone has had these kinds of problems, and >> if/what they set their JAVA_OPTS to in crontab. > > If you set JAVA_OPTS globally in crontab, I would suggest moving that > to individual commands (or scripts that they run) as needed, and > tuning each one to its actual needs. I haven't tried to profile > memory usage on commandline Java app.s yet, so I have no specific > advice here. > > What limits have you set for Tomcat? It's easy to find poor advice on > this -- I've passed along some myself -- and give Tomcat far *more* > than it needs. I hope you're keeping an eye on Tomcat's actual memory > usage and tuning it appropriately. (I have a weekly reminder to snoop > on this with my favorite tool: Psi Probe.) > >> The long term plan of course is to move to a machine with more memory >> (currently 4GB). > > 4GB should be plenty, unless you have many instances running > concurrently or a truly enormous collection. Other applications, the > DBMS, etc. will have some impact on that, of course. > > > > ------------------------------------------------------------------------------ > "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE > Instantly run your Selenium tests across 300+ browser/OS combos. > Get unparalleled scalability from the best Selenium testing platform available > Simple to use. Nothing to install. Get started now for free." > http://p.sf.net/sfu/SauceLabs > > > > _______________________________________________ > DSpace-tech mailing list > DSpace-tech@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-tech > List Etiquette: > https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette > -- Alan Orth alan.o...@gmail.com http://alaninkenya.org http://mjanja.co.ke "I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone." -Bjarne Stroustrup, inventor of C++ GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0
signature.asc
Description: OpenPGP digital signature
------------------------------------------------------------------------------ "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs
_______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
