I want to use JCS as a cache for a large number of objects, where I'll have all of the objects in the indexed disk cache, and about 10% in memory.
What I've found is that JCS is using about 300 bytes of memory for each item in the disk cache. This is contrary to my expectation based on http://jakarta.apache.org/jcs/IndexedDiskAuxCache.html "Depending on the key size, 500,000 disk entries will probably only require about 3 MB of memory." which implies about 6 bytes per item of overhead. Here is my test program. Note that it waits until after the purgatory objects have been written to disk before doing the GC and heap calc. /* $Header$ */ import org.apache.jcs.JCS; import java.io.File; import java.io.Serializable; import java.util.ArrayList; import java.util.List; import java.util.Random; public class JcsTest { JcsTest() { } public static class TestData implements Serializable { int n; String key; String v1; String v2; String v3; String v4; String v5; String v6; public TestData(int n) { this.n = n; this.key = Integer.toString(n); v1 = this.key; v2 = this.key + this.key; v3 = this.key + this.key + this.key; v4 = this.key + this.key + this.key + this.key; v5 = this.key + this.key + this.key + this.key + this.key; v6 = this.key + this.key + this.key + this.key + this.key + this.key; } public String getKey() { return this.key; } } void lookupTest(JCS cache, int numObjs, int nTrials, int nOps) throws Exception { Random rand = new Random(); int max = numObjs - 1; for (int trial = 0; trial < nTrials; trial++) { long startTime = System.currentTimeMillis(); for (int i = 0; i < nOps; i++) { int x = rand.nextInt(max); Object item = cache.get(Integer.toString(x)); if (item == null) throw new Exception("PANIC: can't find item: " + x); } long elapsed = System.currentTimeMillis() - startTime; System.out.printf("TEST rand access trial %d: %d in %d ms, %d / sec\n", trial, nOps, elapsed, (int)(((double)nOps / (double)elapsed) * 1000.0)); } for (int trial = 0; trial < nTrials; trial++) { long startTime = System.currentTimeMillis(); for (int i = 0; i < nOps; i++) { Object item = cache.get(Integer.toString(i % max)); if (item == null) throw new Exception("PANIC: can't find item: " + i); } long elapsed = System.currentTimeMillis() - startTime; System.out.printf("TEST repeat access trial %d: %d in %d ms, %d / sec\n", trial, nOps, elapsed, (int)(((double)nOps / (double)elapsed) * 1000.0)); } } long memCheck(String msg, long baseline) { System.gc(); long currentMem = (Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory()) / 1000; if (baseline != -1) { System.out.printf("TEST %s: %d KB, delta %d KB\n", msg, currentMem, currentMem - baseline); } else { System.out.printf("TEST %s: %d KB\n", msg, currentMem); } return currentMem; } void doit(String configFilePath, int numObjs, int numTrials, int numOps) throws Exception { long startMem = memCheck("STARTING heap size", -1); long startTime = System.currentTimeMillis(); JCS cache = JCS.getInstance("testCache1"); for (int i = 0; i < numObjs; i++) { cache.put(Integer.toString(i), new TestData(i)); } long elapsed = System.currentTimeMillis() - startTime; System.out.println("TEST: CACHE STATS: " + cache.getStats()); System.out.println("TEST: Sleeping 15 sec to let objs roll to disk"); Thread.sleep(15000); System.out.println("TEST: CACHE STATS: " + cache.getStats()); long afterCacheLoadMem = memCheck("after cache load", startMem); lookupTest(cache, numObjs, numTrials, numOps); cache.clear(); cache.dispose(); } public static void main(String[] args) { try { if (args.length != 4) { System.err.println( "usage: JcsTest configFile numObjs numTrials numOps"); System.exit(0); } JcsTest tester = new JcsTest(); tester.doit(args[0], Integer.valueOf(args[1]), Integer.valueOf(args[2]), Integer.valueOf(args[3])); } catch (Exception e) { System.out.println("JcsTest failed: " + e); e.printStackTrace(); } } } Cache config file: ############################################################## ##### Default Region Configuration jcs.default=DC jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes jcs.default.cacheattributes.MaxObjects=10000 jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache # The JSC docs incorrectly name this property DiskUsagePattern. jcs.default.cacheattributes.DiskUsagePatternName=UPDATE ############################################################## ##### AUXILIARY CACHES # Indexed Disk Cache jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes jcs.auxiliary.DC.attributes.DiskPath=jcs_swap jcs.auxiliary.DC.attributes.MaxPurgatorySize=1000000 # MaxKeySize=0 means no limit on number of objects cached on disk jcs.auxiliary.DC.attributes.MaxKeySize=-1 jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=300000 jcs.auxiliary.DC.attributes.OptimizeOnShutdown=true jcs.auxiliary.DC.attributes.MaxRecycleBinSize=7500 heap size output for loading 10k objects into the cache: out-mem10k-objs10k:TEST STARTING heap size: 2787 KB out-mem10k-objs10k:TEST after cache load: 11023 KB, delta 8236 KB heap size output for loading 100k objects into the cache: out-mem10k-objs100k:TEST STARTING heap size: 2787 KB out-mem10k-objs100k:TEST after cache load: 47087 KB, delta 44300 KB heap size output for loading 200k objects into the cache: out-mem10k-objs200k:TEST STARTING heap size: 2787 KB out-mem10k-objs200k:TEST after cache load: 66597 KB, delta 63810 KB So using the 11023 KB as our baseline, adding 190K more objects resulted in an increase in heap of 55574 KB, or 292.5 bytes per object. This is using the latest JCS software, using Java 6 update 3, on RedHat Linux. On and I rm the jcs-swap files before each run. Am I missing something? Is there some magic knob I forgot to twist that would get this overhead way down? Thanks, Nick PS: for absolute completeness, here is the full output from the 200k object run: ========================================================================= TEST STARTING heap size: 2787 KB Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheManager configure INFO: Creating cache manager from config file: /cache.ccf Sep 3, 2008 3:45:51 PM org.apache.jcs.utils.threadpool.ThreadPoolManager loadConfig INFO: thread_pool.default PoolConfiguration = useBoundary = [true] boundarySize = [2000] maximumPoolSize = [150] minimumPoolSize = [4] keepAliveTime = [300000] whenBlockedPolicy = [RUN] startUpSize = [4] Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator setDefaultAuxValues INFO: Setting default auxiliaries to DC Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator setDefaultCompositeCacheAttributes INFO: setting defaultCompositeCacheAttributes to [ useLateral = true, useRemote = true, useDisk = true, maxObjs = 10000, maxSpoolPerRun = -1, diskUsagePattern = 1 ] Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator parseElementAttributes INFO: No special ElementAttribute class defined for key [jcs.default.elementattributes], using default class. Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator setDefaultElementAttributes INFO: setting defaultElementAttributes to [ IS_LATERAL = true, IS_SPOOL = true, IS_REMOTE = true, IS_ETERNAL = true, MaxLifeSeconds = -1, IdleTime = -1, CreateTime = 1220471151384, LastAccessTime = 1220471151384, getTimeToLiveSeconds() = -1, createTime = 1220471151384 ] Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator parseRegions INFO: Parsed regions [] Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator doConfigure INFO: Finished configuration in 17 ms. Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCacheConfigurator parseElementAttributes INFO: No special ElementAttribute class defined for key [jcs.region.testCache1.elementattributes], using default class. Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.memory.lru.LRUMemoryCache initialize INFO: initialized LRUMemoryCache for testCache1 Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.control.CompositeCache <init> INFO: Constructed cache with name [testCache1] and cache attributes [ useLateral = true, useRemote = true, useDisk = true, maxObjs = 10000, maxSpoolPerRun = -1, diskUsagePattern = 1 ] Sep 3, 2008 3:45:51 PM org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache <init> INFO: Region [testCache1] Cache file root directory: jcs_swap Sep 3, 2008 3:45:51 PM org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache initKeyMap INFO: Region [testCache1] Set maxKeySize to unlimited' Sep 3, 2008 3:45:51 PM org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache <init> INFO: Region [testCache1] Indexed Disk Cache is alive. Sep 3, 2008 3:45:51 PM org.apache.jcs.engine.CacheEventQueue put INFO: Cache event queue created: CacheEventQueue [listenerId=700406631, cacheName=testCache1] TEST: CACHE STATS: Region Name = testCache1 HitCountRam = 0 HitCountAux = 0 ---------------------------LRU Memory Cache List Size = 9998 Map Size = 9998 Put Count = 200000 Hit Count = 0 Miss Count = 0 ---------------------------Indexed Disk Cache Is Alive = true Key Map Size = 33404 Data File Length = 22937956 Hit Count = 0 Bytes Free = 0 Optimize Operation Count = 0 Times Optimized = 0 Recycle Count = 0 Recycle Bin Size = 0 Startup Size = 0 Purgatory Hits = 0 Purgatory Size = 166594 Working = true Alive = true Empty = false Size = 166593 TEST: Sleeping 15 sec to let objs roll to disk TEST: CACHE STATS: Region Name = testCache1 HitCountRam = 0 HitCountAux = 0 ---------------------------LRU Memory Cache List Size = 9998 Map Size = 9998 Put Count = 200000 Hit Count = 0 Miss Count = 0 ---------------------------Indexed Disk Cache Is Alive = true Key Map Size = 200000 Data File Length = 140755580 Hit Count = 0 Bytes Free = 0 Optimize Operation Count = 0 Times Optimized = 0 Recycle Count = 0 Recycle Bin Size = 0 Startup Size = 0 Purgatory Hits = 0 Purgatory Size = 0 Working = true Alive = true Empty = true Size = 0 TEST after cache load: 66597 KB, delta 63810 KB TEST rand access trial 0: 5000 in 1068 ms, 4681 / sec TEST rand access trial 1: 5000 in 667 ms, 7496 / sec TEST rand access trial 2: 5000 in 545 ms, 9174 / sec TEST rand access trial 3: 5000 in 442 ms, 11312 / sec TEST rand access trial 4: 5000 in 406 ms, 12315 / sec TEST rand access trial 5: 5000 in 379 ms, 13192 / sec TEST rand access trial 6: 5000 in 403 ms, 12406 / sec TEST rand access trial 7: 5000 in 405 ms, 12345 / sec TEST rand access trial 8: 5000 in 407 ms, 12285 / sec TEST rand access trial 9: 5000 in 406 ms, 12315 / sec TEST repeat access trial 0: 5000 in 367 ms, 13623 / sec TEST repeat access trial 1: 5000 in 11 ms, 454545 / sec TEST repeat access trial 2: 5000 in 11 ms, 454545 / sec TEST repeat access trial 3: 5000 in 11 ms, 454545 / sec TEST repeat access trial 4: 5000 in 11 ms, 454545 / sec TEST repeat access trial 5: 5000 in 11 ms, 454545 / sec TEST repeat access trial 6: 5000 in 11 ms, 454545 / sec TEST repeat access trial 7: 5000 in 11 ms, 454545 / sec TEST repeat access trial 8: 5000 in 11 ms, 454545 / sec TEST repeat access trial 9: 5000 in 11 ms, 454545 / sec Sep 3, 2008 3:46:15 PM org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache reset WARNING: Region [testCache1] Reseting cache Sep 3, 2008 3:46:15 PM org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache initKeyMap INFO: Region [testCache1] Set maxKeySize to unlimited' Sep 3, 2008 3:46:15 PM org.apache.jcs.engine.control.CompositeCache dispose INFO: In DISPOSE, [testCache1] fromRemote [false] Region Name = testCache1 HitCountRam = 47707 HitCountAux = 52293 ---------------------------LRU Memory Cache List Size = 0 Map Size = 0 Put Count = 252293 Hit Count = 47707 Miss Count = 52293 ---------------------------Indexed Disk Cache Is Alive = true Key Map Size = 0 Data File Length = 0 Hit Count = 52293 Bytes Free = 0 Optimize Operation Count = 0 Times Optimized = 0 Recycle Count = 0 Recycle Bin Size = 0 Startup Size = 0 Purgatory Hits = 0 Purgatory Size = 0 Working = true Alive = false Empty = true Size = 0 Sep 3, 2008 3:46:15 PM org.apache.jcs.engine.control.CompositeCache dispose INFO: In DISPOSE, [testCache1] auxiliary [EMAIL PROTECTED] Sep 3, 2008 3:46:15 PM org.apache.jcs.engine.control.CompositeCache dispose INFO: In DISPOSE, [testCache1] put 0 into auxiliary [EMAIL PROTECTED] Sep 3, 2008 3:46:15 PM org.apache.jcs.auxiliary.disk.AbstractDiskCache$1 run INFO: No longer waiting for event queue to finish: Cache Event Queue Working = true Alive = false Empty = true Size = 0 Sep 3, 2008 3:46:15 PM org.apache.jcs.auxiliary.disk.AbstractDiskCache dispose INFO: In dispose, destroying event queue. Sep 3, 2008 3:46:15 PM org.apache.jcs.engine.CacheEventQueue destroy INFO: Destroy was called after queue was destroyed. Doing nothing. Stats = Cache Event Queue Working = true Alive = false Empty = true Size = 0 Sep 3, 2008 3:46:15 PM org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCache disposeInternal INFO: Region [testCache1] Shutdown complete. Sep 3, 2008 3:46:15 PM org.apache.jcs.engine.control.CompositeCache dispose INFO: In DISPOSE, [testCache1] disposing of memory cache. Sep 3, 2008 3:46:15 PM org.apache.jcs.engine.memory.AbstractMemoryCache dispose INFO: Memory Cache dispose called. Shutting down shrinker thread if it is running. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
