This sounds plausible. Running the profiler on our production machine for 30 minutes showed heavy memory spikes but no leakage when the memory shrinker was enabled (After a successfull GC it always seemed to go back to a normal level). I also checked the memory shrinker code manually, I didn't dig too deep into the spooling but suspected this might be causing problems simply because the rest of the memory shrinker code was so simple.
I would have liked to run the profiler for a longer period but Jprofiler I was using is not stable enough to leave unattended on a production system (it crashes the jvm sometimes). So I was unable to confirm if it was a slow leak or sudden spike. I have seen other libraries OutOfMemory due to heap usage spikes where we previously suspected a leak, so it's not uncommon (in an XML processing library). Good luck anyway, Matt. >-- Original Message -- >Reply-To: "Turbine JCS Users List" <[EMAIL PROTECTED]> >Date: Sun, 09 May 2004 15:15:05 -0500 >From: Aaron Smuts <[EMAIL PROTECTED]> >Subject: RE: Out of memory problem >To: 'Turbine JCS Users List' <[EMAIL PROTECTED]> > > >I've been profiling JCS for days. I still have more work to do, but I >have a few ideas. > >The shrinker does not maintain references to any objects. However, >there is a potential issue involved in using the shrinker, namely, it >causes sudden spikes in memory usage. > >The main purpose of the shrinker is to find elements that have exceeded >their maximum idle memory time and then to spool these to disk. It also >checks for expired elements in the memory cache and removes them. > >When the shrinker finds elements to spool it initiates the spooling >process. This invovles putting the items in purgatory and then they are >queued to be written for disk. The disk writing can take a while if you >have thousands of items. The shrinker will try to spool every item in >memory that has exceeded the maxmemory time. Basically, this can result >in thousands of items in the queue which results in a memory spike that >can exceed the maximum heap size. All my tests indicate that the memory >is reclaimed. > >I will add a configuration option to set the maximum number of items to >spool per run of the shrinker. This way the size of the spike can be >minimized. > >There is a potential memory issue invovled in using the indexed disk >cache. The disk cache stores the keys in memory. If an expired element >is pulled from the disk cache or an item is removed from the cache, the >key gets removed from the set held in memory by the disk cache. >However, if you create millions of entries in the cache that you never >use, many of these get pushed onto the disk, and they don't get removed, >the in memory key size of the disk cache can grow indefinitely. > >This would most likely be the result of improper cache usage, since you >shouldn't cache something that won't likely be used again. However, >there is a high usage scenario that could result in massive disk key >memory usage. Let's assume that you cache objects that might not get >used very frequently, and you set them to expire after a short duration, >let's say 30 seconds. If that item gets spooled to disk within 30 >seconds, because either you have a small max memory size or extremely >high usage, and there is never a get for that item, then the key will >sit in disk memory forever or until a get. If there is a get it will be >found on disk, checked for expiration, and then removed, but if there is >no get, it will just hang around. > >For now, the way to prevent this is to increase the memory size and >shrink the max life time. Alternatively, you can disable the disk cache >if most of your items are of this type. > >In the future some sort of disk cleanup should occur, or the key sotrage >for the disk cache could use a LRU algorithm that would limit it size. >I think this is actually the best option. I will do something like this >soon. > >I can't find any actual memory leaks, only the spike from the shrinker >and the potential key bloat in the disk cache. > >I'm still hunting for more. > >Aaron > > > > >> -----Original Message----- >> From: Estefano Eduardo [mailto:[EMAIL PROTECTED] >> Sent: Friday, May 07, 2004 2:43 AM >> To: Turbine JCS Users List >> Subject: RE: Out of memory problem >> >> We also started getting out of memory errors. Same situation, only >after >> we started using the jcs library (I have latest copy from main trunk). >> >> We did not pin point the error yet though. It did not happen straight >> away, only after the server was running for a few days. The solution >is >> to restart the server at the moment. We only use memory cache and we >do >> use schrinkers on some regions. >> >> -----Original Message----- >> From: Aaron Smuts [mailto:[EMAIL PROTECTED] >> Sent: Thursday, 06 May, 2004 19:52 >> To: 'Turbine JCS Users List' >> Subject: RE: Out of memory problem >> >> >> Matthew informs me that the problem seems to go away with the shrinker >> off. >> >> This suggests a problem with the memory shrinker. It could be that >the >> thread is maintaining a reference to the keys, the spooling from the >> shrinker is problematic, or there is a problem generating the list of >> keys. >> >> I don't think any of these things is happening, so I'm sort of at a >loss >> for an explanation. >> >> I think the shrinker just sleeps for an interval. Perhaps it would be >> better to use a timer and start the shrinker and then let it die. One >> concern with this is that we wouldn't want more than one shrinker >> running per region at any given time. This could happen say if the >> number of elements was huge and the interval between shrinks was very >> small. I can solve that by passing in a lock that will prevent a >second >> shrinker from running. >> >> Hmmn. I'll look into it again. >> >> Aaron >> >> > -----Original Message----- >> > From: Aaron Smuts [mailto:[EMAIL PROTECTED] >> > Sent: Thursday, May 06, 2004 9:43 AM >> > To: 'Turbine JCS Users List' >> > Subject: RE: Out of memory problem >> > >> > Send your cache.ccf and let me know how much memory you've allocated >> for >> > the jvm. >> > >> > Aaron >> > >> > > -----Original Message----- >> > > From: Parvathi Rajaraman [mailto:[EMAIL PROTECTED] >> > > Sent: Thursday, May 06, 2004 1:53 AM >> > > To: '[EMAIL PROTECTED]' >> > > Subject: Out of memory problem >> > > >> > > Hi, >> > > >> > > I have recently started using JCS implementation for Caching in my >> > > project. I have noticed that every few minutes, I keep getting a >> > > out-of >> memory >> > > error >> > > and this has happened only after I have started using JCS. So is >> this >> > > something to do with the way I have configured it. Can you help me >> > out. >> > > >> > > Regds >> > > Parvathi >> > > >> > > >> --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: >> > [EMAIL PROTECTED] >> > > For additional commands, e-mail: >> > [EMAIL PROTECTED] >> > >> > >> > >--------------------------------------------------------------------- >> > To unsubscribe, e-mail: >> [EMAIL PROTECTED] >> > For additional commands, e-mail: >> [EMAIL PROTECTED] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: >[EMAIL PROTECTED] >> For additional commands, e-mail: >> [EMAIL PROTECTED] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: >[EMAIL PROTECTED] >> For additional commands, e-mail: >[EMAIL PROTECTED] > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: [EMAIL PROTECTED] >For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
