Do you have any of the cache logs when this is happening? I would turn the memory shrinker off (set the property to false), as a start. I generally don't run with the memory shrinker on. But I'm shooting in the dark.
Aaron --- Joshua Szmajda <[EMAIL PROTECTED]> wrote: > Ahh yes of course, it was the user requirement. Now > I have a nice bunch > of data. This is interesting, but I'm not sure what > the [B class is: > > num #instances #bytes class name > -------------------------------------- > 1: 31419 284852480 [B > 2: 2277 19760264 [I > 3: 57834 3865240 [C > 4: 29628 1896192 > org.apache.jcs.engine.ElementAttributes > 5: 57838 1388112 java.lang.String > ... > > Niall Gallagher wrote: > > Hmm :D > > > > I just did a bit of digging. I've used this script > on a few of our > > servers in the past (32 and 64bit server VMs), but > I just found a server > > which gave me the exact same error message you > got. That server it turns > > out runs Java under a different user account to > the one I was logged > > into however. > > > > Try running the script from the exact same user > account the JVM process > > is running from. Even running from root doesn't > work didn't work for me > > on that server, it had to be exact same user > account, which is > > surprising. > > > > By the way those tools are documented here: > > > http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jmap.html > > and > > > http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html > > > > -basically they're supposed to work on most > platforms except Windows and > > Linux Itanium so unless you've got Itanium cpus it > should work for you. > > > > > > On Wed, 2008-04-09 at 14:44 -0400, Joshua Szmajda > wrote: > > > > > >> Hey Niall, > >> > >> Thanks for your script, but I'm getting these > errors: > >> > >> ./capture-diagnostics.sh RemoteCacheServerFactory > >> Capturing diagnostics for Java process > "RemoteCacheServerFactory" (pid > >> 2007)... > >> 2007: Unable to open socket file: target process > not responding or > >> HotSpot VM not loaded > >> The -F option can be used when the target process > is not responding > >> 2007: Unable to open socket file: target process > not responding or > >> HotSpot VM not loaded > >> The -F option can be used when the target process > is not responding > >> Saved diagnostics for "RemoteCacheServerFactory" > to > >> "RemoteCacheServerFactory-diagnostics.txt" > >> > >> There must be something I'm missing when I'm > running the cache server. I > >> noticed it uses the 'server' VM by default, maybe > these debug commands > >> are only good for the client VM? > >> > >> Thanks! > >> -Josh > >> > >> Niall Gallagher wrote: > >> > >>> Hi Josh, > >>> > >>> Can you modify your cron job to capture > diagnostics before it restarts > >>> the cache server? > >>> > >>> Then you can post the diagnostics next time it > happens. The script below > >>> will capture diagnostics for you. We use > something like this in-house > >>> for troubleshooting (not specifically for JCS). > >>> > >>> You'll first have to run the JDK 'jps' command > from either root, or the > >>> user account which runs your cache server > instance. This gives you the > >>> "name" of your cache server JVM process, which > you need to supply to the > >>> diagnostics script as command-line parameter. > The script uses the name > >>> to attach to the relevant JVM process. > >>> > >>> I don't know what might be causing the problem > for you. It could be a > >>> bug in JCS, or it could be a memory issue. The > diagnostics will help > >>> identify the problem. > >>> > >>> Save this as "capture-diagnostics.sh"... > >>> ------- > >>> #!/bin/sh > >>> # Saves the stack traces and class memory usage > information for a > >>> # Java process running on the machine to a > diagnostics file. > >>> # > >>> # This script expects the name of the relevant > Java process to be > >>> # specified as a parameter. The name specified > should match a Java > >>> # process name as listed by running the JDK > 'jps' command. > >>> # > >>> # Usage: sh capture-diagnostics.sh <name of > process> > >>> > >>> APP_NAME="$1" > >>> JDK_LOCATION="/usr/java/default" > >>> DUMP_FILE="$APP_NAME-diagnostics.txt" > >>> > >>> APP_PID="`$JDK_LOCATION/bin/jps|grep $APP_NAME > 2> /dev/null|cut -d\ > >>> -f1`" > >>> if [ "$APP_PID" = "" ]; then > >>> echo "ERROR: Can't determine pid of Java process > name specified > >>> \"$APP_NAME\"" > >>> echo "Usage: sh capture-diagnostics.sh <name of > process as listed by jps > >>> command>" > >>> exit 20 > >>> fi > >>> echo "Capturing diagnostics for Java process > \"$APP_NAME\" (pid > >>> $APP_PID)..." > >>> echo -e "Diagnostics for Java process > \"$APP_NAME\" (pid $APP_PID) as at > >>> `date`:-" >> $DUMP_FILE > >>> echo -e "\nTop 30 memory-consuming classes:-" >> > $DUMP_FILE > >>> $JDK_LOCATION/bin/jmap -histo:live $APP_PID > |head -n33 >> $DUMP_FILE > >>> echo -e "\nThread stack traces:-" >> $DUMP_FILE > >>> $JDK_LOCATION/bin/jstack $APP_PID >> $DUMP_FILE > >>> echo -e "\n" >> $DUMP_FILE > >>> echo "Saved diagnostics for \"$APP_NAME\" to > \"$DUMP_FILE\"" > >>> ------- > >>> > >>> > >>> On Wed, 2008-04-09 at 10:11 -0400, Joshua > Szmajda wrote: > >>> > >>> > >>> > >>>> Hey all, > >>>> > >>>> I've got a JCS remote cache server running on a > machine and every now > >>>> and then it will spiral out of control and lock > the machine. I have no > >>>> idea yet what's causing this, I've just put > some extra measures in place > >>>> to capture the logs from when it happens. My > solution at this point is a > >>>> cron job that checks now and then for excessive > cpu usage and restarts > >>>> the cache server. I'd like to be able to not > worry about it, though :). > >>>> > >>>> Any suggestions? > >>>> > >>>> Thanks! > >>>> -Josh > >>>> > >>>> P.S. it's running on ubuntu-server (kernel > 2.6.22-14-server). > >>>> I have up to 16 remote listeners connecting to > any given region. > >>>> (probably 20 application instances in all). > >>>> Puts grow at a rate of about 400 per second. > >>>> I pass these options to java: "-Xms128m > -Xmx2000m" > >>>> And here's my simple remote.cache.ccf: > === message truncated === --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
