That 6 byte estimate sounds far too low. It probably needs to be corrected, but 300 sounds a bit high.
The indexed disk cache stores all the keys in memory along with the disk offset. You are using ints as the key. The offset is a long. That's more than 6 there. Plus the hashtable overhead, it's definitely way over 6 bytes. If you have time, try running the test with the memory size set to 0. I want to know just the disk cache overhead without any other noise. You should call System.gc a couple times and wait a bit in between calls, just to be safe. Then run the same test using a hashtable instead of JCS. Use a 0 byte array or something small as the value. I'm curious to see how much a hashtable uses. I was trying to implement a more efficient hashtable a while back, but I don't remember the numbers. Thanks, Aaron --- On Wed, 9/3/08, Nick <[EMAIL PROTECTED]> wrote: > From: Nick <[EMAIL PROTECTED]> > Subject: JCS using lots of mem for objects written to disk? > To: jcs-users@jakarta.apache.org > Date: Wednesday, September 3, 2008, 1:35 PM > I want to use JCS as a cache for a large number of objects, > where I'll have all of the objects in the indexed disk > cache, and about 10% in memory. > > What I've found is that JCS is using about 300 bytes of > memory for each item in the disk cache. This is contrary to > my expectation based on > http://jakarta.apache.org/jcs/IndexedDiskAuxCache.html > "Depending on the key size, 500,000 disk entries will > probably only require about 3 MB of memory." which > implies about 6 bytes per item of overhead. > > Here is my test program. Note that it waits until after > the purgatory objects have been written to disk before doing > the GC and heap calc. > > /* $Header$ */ > > import org.apache.jcs.JCS; > > import java.io.File; > import java.io.Serializable; > import java.util.ArrayList; > import java.util.List; > import java.util.Random; > > public class JcsTest { > JcsTest() { > } > > public static class TestData implements Serializable { > int n; > String key; > String v1; > String v2; > String v3; > String v4; > String v5; > String v6; > public TestData(int n) { > this.n = n; > this.key = Integer.toString(n); > v1 = this.key; > v2 = this.key + this.key; > v3 = this.key + this.key + this.key; > v4 = this.key + this.key + this.key + this.key; > v5 = this.key + this.key + this.key + this.key > + this.key; > v6 = this.key + this.key + this.key + this.key > + this.key > + this.key; > } > public String getKey() { > return this.key; > } > } > > void lookupTest(JCS cache, int numObjs, > int nTrials, int nOps) throws Exception { > Random rand = new Random(); > int max = numObjs - 1; > for (int trial = 0; trial < nTrials; trial++) { > long startTime = System.currentTimeMillis(); > for (int i = 0; i < nOps; i++) { > int x = rand.nextInt(max); > Object item = > cache.get(Integer.toString(x)); > if (item == null) > throw new Exception("PANIC: > can't find item: " + x); > } > long elapsed = System.currentTimeMillis() - > startTime; > System.out.printf("TEST rand access trial > %d: %d in %d ms, %d / sec\n", > trial, nOps, elapsed, > (int)(((double)nOps / (double)elapsed) * > 1000.0)); > } > > for (int trial = 0; trial < nTrials; trial++) { > long startTime = System.currentTimeMillis(); > for (int i = 0; i < nOps; i++) { > Object item = cache.get(Integer.toString(i > % max)); > if (item == null) > throw new Exception("PANIC: > can't find item: " + i); > } > long elapsed = System.currentTimeMillis() - > startTime; > System.out.printf("TEST repeat access > trial %d: %d in %d ms, %d / sec\n", > trial, nOps, elapsed, > (int)(((double)nOps / (double)elapsed) * > 1000.0)); > } > } > > long memCheck(String msg, long baseline) { > System.gc(); > long currentMem = > (Runtime.getRuntime().totalMemory() > - Runtime.getRuntime().freeMemory()) / 1000; > if (baseline != -1) { > System.out.printf("TEST %s: %d KB, delta > %d KB\n", > msg, currentMem, currentMem - baseline); > } else { > System.out.printf("TEST %s: %d > KB\n", > msg, currentMem); > } > return currentMem; > } > > void doit(String configFilePath, int numObjs, int > numTrials, int numOps) > throws Exception { > long startMem = memCheck("STARTING heap > size", -1); > > long startTime = System.currentTimeMillis(); > JCS cache = > JCS.getInstance("testCache1"); > for (int i = 0; i < numObjs; i++) { > cache.put(Integer.toString(i), new > TestData(i)); > } > long elapsed = System.currentTimeMillis() - > startTime; > > System.out.println("TEST: CACHE STATS: " > + cache.getStats()); > System.out.println("TEST: Sleeping 15 sec to > let objs roll to disk"); > Thread.sleep(15000); > System.out.println("TEST: CACHE STATS: " > + cache.getStats()); > > long afterCacheLoadMem = memCheck("after cache > load", startMem); > > lookupTest(cache, numObjs, numTrials, numOps); > > cache.clear(); > cache.dispose(); > } > > public static void main(String[] args) { > try { > if (args.length != 4) { > System.err.println( > "usage: JcsTest configFile numObjs > numTrials numOps"); > System.exit(0); > } > JcsTest tester = new JcsTest(); > tester.doit(args[0], > Integer.valueOf(args[1]), > Integer.valueOf(args[2]), > Integer.valueOf(args[3])); > } catch (Exception e) { > System.out.println("JcsTest failed: " > + e); > e.printStackTrace(); > } > } > } > > Cache config file: > > ############################################################## > ##### Default Region Configuration > jcs.default=DC > jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes > jcs.default.cacheattributes.MaxObjects=10000 > jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache > # The JSC docs incorrectly name this property > DiskUsagePattern. > jcs.default.cacheattributes.DiskUsagePatternName=UPDATE > > ############################################################## > ##### AUXILIARY CACHES > # Indexed Disk Cache > jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory > jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes > jcs.auxiliary.DC.attributes.DiskPath=jcs_swap > jcs.auxiliary.DC.attributes.MaxPurgatorySize=1000000 > # MaxKeySize=0 means no limit on number of objects cached > on disk > jcs.auxiliary.DC.attributes.MaxKeySize=-1 > jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300000 > jcs.auxiliary.DC.attributes.OptimizeOnShutdown=true > jcs.auxiliary.DC.attributes.MaxRecycleBinSize=7500 > > heap size output for loading 10k objects into the cache: > > out-mem10k-objs10k:TEST STARTING heap size: 2787 KB > out-mem10k-objs10k:TEST after cache load: 11023 KB, delta > 8236 KB > > heap size output for loading 100k objects into the cache: > > out-mem10k-objs100k:TEST STARTING heap size: 2787 KB > out-mem10k-objs100k:TEST after cache load: 47087 KB, delta > 44300 KB > > heap size output for loading 200k objects into the cache: > > out-mem10k-objs200k:TEST STARTING heap size: 2787 KB > out-mem10k-objs200k:TEST after cache load: 66597 KB, delta > 63810 KB > > So using the 11023 KB as our baseline, adding 190K more > objects resulted in an increase in heap of 55574 KB, or > 292.5 bytes per object. > > This is using the latest JCS software, using Java 6 update > 3, on RedHat Linux. On and I rm the jcs-swap files before > each run. > > Am I missing something? Is there some magic knob I forgot > to twist that would get this overhead way down? > > Thanks, > > Nick > > PS: for absolute completeness, here is the full output from > the 200k object run: > > ========================================================================= > > > TEST STARTING heap size: 2787 KB > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheManager > configure > INFO: Creating cache manager from config file: /cache.ccf > Sep 3, 2008 3:45:51 PM > org.apache.jcs.utils.threadpool.ThreadPoolManager loadConfig > INFO: thread_pool.default PoolConfiguration = useBoundary = > [true] boundarySize = [2000] maximumPoolSize = [150] > minimumPoolSize = [4] keepAliveTime = [300000] > whenBlockedPolicy = [RUN] startUpSize = [4] > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > setDefaultAuxValues > INFO: Setting default auxiliaries to DC > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > setDefaultCompositeCacheAttributes > INFO: setting defaultCompositeCacheAttributes to [ > useLateral = true, useRemote = true, useDisk = true, maxObjs > = 10000, maxSpoolPerRun = -1, diskUsagePattern = 1 ] > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > parseElementAttributes > INFO: No special ElementAttribute class defined for key > [jcs.default.elementattributes], using default class. > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > setDefaultElementAttributes > INFO: setting defaultElementAttributes to [ IS_LATERAL = > true, IS_SPOOL = true, IS_REMOTE = true, IS_ETERNAL = true, > MaxLifeSeconds = -1, IdleTime = -1, CreateTime = > 1220471151384, LastAccessTime = 1220471151384, > getTimeToLiveSeconds() = -1, createTime = 1220471151384 ] > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > parseRegions > INFO: Parsed regions [] > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > doConfigure > INFO: Finished configuration in 17 ms. > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCacheConfigurator > parseElementAttributes > INFO: No special ElementAttribute class defined for key > [jcs.region.testCache1.elementattributes], using default > class. > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.memory.lru.LRUMemoryCache initialize > INFO: initialized LRUMemoryCache for testCache1 > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.control.CompositeCache <init> > INFO: Constructed cache with name [testCache1] and cache > attributes [ useLateral = true, useRemote = true, useDisk = > true, maxObjs = 10000, maxSpoolPerRun = -1, diskUsagePattern > = 1 ] > Sep 3, 2008 3:45:51 PM > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache > <init> > INFO: Region [testCache1] Cache file root directory: > jcs_swap > Sep 3, 2008 3:45:51 PM > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache > initKeyMap > INFO: Region [testCache1] Set maxKeySize to unlimited' > Sep 3, 2008 3:45:51 PM > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache > <init> > INFO: Region [testCache1] Indexed Disk Cache is alive. > Sep 3, 2008 3:45:51 PM > org.apache.jcs.engine.CacheEventQueue put > INFO: Cache event queue created: CacheEventQueue > [listenerId=700406631, cacheName=testCache1] > TEST: CACHE STATS: Region Name = testCache1 > HitCountRam = 0 > HitCountAux = 0 > ---------------------------LRU Memory Cache > List Size = 9998 > Map Size = 9998 > Put Count = 200000 > Hit Count = 0 > Miss Count = 0 > ---------------------------Indexed Disk Cache > Is Alive = true > Key Map Size = 33404 > Data File Length = 22937956 > Hit Count = 0 > Bytes Free = 0 > Optimize Operation Count = 0 > Times Optimized = 0 > Recycle Count = 0 > Recycle Bin Size = 0 > Startup Size = 0 > Purgatory Hits = 0 > Purgatory Size = 166594 > Working = true > Alive = true > Empty = false > Size = 166593 > TEST: Sleeping 15 sec to let objs roll to disk > TEST: CACHE STATS: Region Name = testCache1 > HitCountRam = 0 > HitCountAux = 0 > ---------------------------LRU Memory Cache > List Size = 9998 > Map Size = 9998 > Put Count = 200000 > Hit Count = 0 > Miss Count = 0 > ---------------------------Indexed Disk Cache > Is Alive = true > Key Map Size = 200000 > Data File Length = 140755580 > Hit Count = 0 > Bytes Free = 0 > Optimize Operation Count = 0 > Times Optimized = 0 > Recycle Count = 0 > Recycle Bin Size = 0 > Startup Size = 0 > Purgatory Hits = 0 > Purgatory Size = 0 > Working = true > Alive = true > Empty = true > Size = 0 > TEST after cache load: 66597 KB, delta 63810 KB > TEST rand access trial 0: 5000 in 1068 ms, 4681 / sec > TEST rand access trial 1: 5000 in 667 ms, 7496 / sec > TEST rand access trial 2: 5000 in 545 ms, 9174 / sec > TEST rand access trial 3: 5000 in 442 ms, 11312 / sec > TEST rand access trial 4: 5000 in 406 ms, 12315 / sec > TEST rand access trial 5: 5000 in 379 ms, 13192 / sec > TEST rand access trial 6: 5000 in 403 ms, 12406 / sec > TEST rand access trial 7: 5000 in 405 ms, 12345 / sec > TEST rand access trial 8: 5000 in 407 ms, 12285 / sec > TEST rand access trial 9: 5000 in 406 ms, 12315 / sec > TEST repeat access trial 0: 5000 in 367 ms, 13623 / sec > TEST repeat access trial 1: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 2: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 3: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 4: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 5: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 6: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 7: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 8: 5000 in 11 ms, 454545 / sec > TEST repeat access trial 9: 5000 in 11 ms, 454545 / sec > Sep 3, 2008 3:46:15 PM > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache reset > WARNING: Region [testCache1] Reseting cache > Sep 3, 2008 3:46:15 PM > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache > initKeyMap > INFO: Region [testCache1] Set maxKeySize to unlimited' > Sep 3, 2008 3:46:15 PM > org.apache.jcs.engine.control.CompositeCache dispose > INFO: In DISPOSE, [testCache1] fromRemote [false] > Region Name = testCache1 > HitCountRam = 47707 > HitCountAux = 52293 > ---------------------------LRU Memory Cache > List Size = 0 > Map Size = 0 > Put Count = 252293 > Hit Count = 47707 > Miss Count = 52293 > ---------------------------Indexed Disk Cache > Is Alive = true > Key Map Size = 0 > Data File Length = 0 > Hit Count = 52293 > Bytes Free = 0 > Optimize Operation Count = 0 > Times Optimized = 0 > Recycle Count = 0 > Recycle Bin Size = 0 > Startup Size = 0 > Purgatory Hits = 0 > Purgatory Size = 0 > Working = true > Alive = false > Empty = true > Size = 0 > Sep 3, 2008 3:46:15 PM > org.apache.jcs.engine.control.CompositeCache dispose > INFO: In DISPOSE, [testCache1] auxiliary > [EMAIL PROTECTED] > Sep 3, 2008 3:46:15 PM > org.apache.jcs.engine.control.CompositeCache dispose > INFO: In DISPOSE, [testCache1] put 0 into auxiliary > [EMAIL PROTECTED] > Sep 3, 2008 3:46:15 PM > org.apache.jcs.auxiliary.disk.AbstractDiskCache$1 run > INFO: No longer waiting for event queue to finish: Cache > Event Queue > Working = true > Alive = false > Empty = true > Size = 0 > Sep 3, 2008 3:46:15 PM > org.apache.jcs.auxiliary.disk.AbstractDiskCache dispose > INFO: In dispose, destroying event queue. > Sep 3, 2008 3:46:15 PM > org.apache.jcs.engine.CacheEventQueue destroy > INFO: Destroy was called after queue was destroyed. Doing > nothing. Stats = Cache Event Queue > Working = true > Alive = false > Empty = true > Size = 0 > Sep 3, 2008 3:46:15 PM > org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache > disposeInternal > INFO: Region [testCache1] Shutdown complete. > Sep 3, 2008 3:46:15 PM > org.apache.jcs.engine.control.CompositeCache dispose > INFO: In DISPOSE, [testCache1] disposing of memory cache. > Sep 3, 2008 3:46:15 PM > org.apache.jcs.engine.memory.AbstractMemoryCache dispose > INFO: Memory Cache dispose called. Shutting down shrinker > thread if it is running. > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > [EMAIL PROTECTED] > For additional commands, e-mail: > [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
