Hi Gerhard, I've tested your latest Store Janitor improvements (using Optimizit and your debug messages) and although its a very great improvement, I've found some other problems :-(
As my Optimizit trial was running out I decided to do some urgent work on the Checker and now have a solution which appears to work well :-) Here is a summary of the problems with ideas for correcting them (some are nitty gritty, and some you already know): *) When demand for memory is high, it is possible to run out of java.memory before the store has been emptied. Might it be possible to make sure the stores are empty before the memory can run out? *) When memory is low, there is a risk of running out of it before the next check. Why not set the interval short enough to ensure there is no risk of running out of memory before the next check? *) Calling the gc once every check interval seriously undermines system performance because it can take several seconds to complete, which is comparable to the check interval. Why not call it only when some items are released from a store? Why not trust it and use its characteristics to advantage? *) When several instances of Cocoon are running (independent .WAR apps in the same servlet container), the calls to gc become very dominant. Would this affect the strategy for store-janitor settings? *) If the percent to reduce storage is set to 10%, it fails to remove any when the number of items are below 10. The number of items to be removed needs rounding upwards. Why not remove a fixed number of items instead of a percentage? (My idea and now I think it was wrong!!) *) It is difficult to get a good picture of what is happening by observing the debug output, so it is difficult to know if it is working and difficult use the debug output to optimise choice of janitor settings. Might it be possible to make this easy? The proposed, implemented and tested solution to the above works as follows: 1) Monitor the rate at which memory is consumed during each check interval. Remember the biggest rate that has been found so far, to use it later. My system logs this at 2.5 megabytes per second. 2) If heapIsBig() && freeIsLow(), then attempt to free storage. 2.1) 'heapIsBig' means that the total memory has exceeded the parameter 'maxheap'. 2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'. 2.3) Attempt to remove from the stores, the number of items specified by the parameter 'reduceby'. 2.4) To free storage, start at the next store and remove items, moving to the next again if necessary, until the specified number of items have been removed or all the stores are empty. 2.5) Then call the garbage collector, but only if some memory items were freed. 3) Sleep for an interval half that in which the memory could run out. 3.1) If the remaining heap could be more than half filled during the interval specified by the 'maxsleep' parameter (at the max rate of memory consumption) then sleep for the time it would take to only half fill (freememory/maxrate)/2. Otherwise sleep for the max sleep interval. The effect of the store-janitor parameters in cocoon.xconf are now slightly changed, so I've changed the names of the parameters to reflect this. For the moment the names are changed only inside StoreJanitorImpl.java. If the solution were to be adopted then the names in cocoon.xconf would also change. The changes, with notes, are as follows: 'maxheap' (was 'heapsize') The maximum total heap size below which stores are guaranteed to remain intact. 'minfree' (was 'freememory') The minimum free heap required for stores to remain intact. I have found this should be set to less than 10% of avaliable memory, so that the jvm will respond to low memory by allocating more heap (if its available), before stores are likely to be reduced. Default value might be 2000000. Setting it to zero or very low will effectively disable store reduction. 'maxsleep' (was 'cleanupthreadinterval') The maximum interval between checks. Should be short enough to ensure that 'rate of change' data is collected before free memory runs low. Suggest 5 secs as default. 'reduceby' (was 'percent_to_free') is the number of items to be removed from the stores, on each attempt to reduce the stores. Each successfull attempt to reduce stores results in a call to the garbage collector taking several seconds. This limits the rate a which the stores can be emptied. Removing several items at a time compensates for this. Using the debug output, choose a number of items which results in an increase of free memory between successive store reductions. Best tested with a sudden high demand for memory when the stores are full. In practice this setting is not critical. Suggested default is 10 items. Ideally 'reduceby' would be in bytes! Setting 'minfree' and 'reduceby' both too low can intensify gc activity if sudden high demand requires all stores to be completely freed and can result in memory running out before all the stores are free. Verify using debug output. 'priority' remains unchanged. The above solution is implemented and tested and it works very well indeed for me. The parameters do not seem to be at all critical - thank goodness! I see no reason why the default settings would need to be changed if more memory were available. I have attached the StoreJanitorImpl.java so others can test it. I find I can get a very good picture of what is happening by using my editor's repeat find command on the debug output in the core.log.000001 file. Optimizeit is no longer necessary to prove that it works effectively. Beware! - there might still be things I don't know about that I have not take into account. Ideally - low memory should be detected by some sort of interrupt or exception rather than by polling. If you are subsequently interested in committing these changes into Cocoon and you want me to do some more work to conform to design practice etc., then please let me know. Very humble appologies, Gerhard (and others), for intruding into your code. If you like my changes, I hope you feel that I have released your brilliance (not undermined it). I feel that by far the biggest part of any credit should very definately still be yours (I only fixed the checker) :-). Feedback very welcome please. Pete H. [EMAIL PROTECTED] wrote: > Hi Team, > I hope I fixed the problem now! > > I changed the StoreJanitorImpl, I try to explain in > prosa: > > <not to serious> > Cocoon reaches 1 million request per second and > memory is getting dangerously low. Time for the > StoreJanitor to kick in and do something. First > he forces the GC. Damn, memory is still low, > we have to remove object from the Store to get > some bytes free. Panic, which Store shall we take? > Ahh index is at -1 so we take the first at index 0 > in the StoreArray. Gosh this one is pretty full how > much objects shall we kick out? Stupid question, > take 10% from all as configured. Chacka, did ya see > them fly? > Next time we take the next store on index 1. > Yeah great work dudes, take a rest. > </not to serious> > > Test it! > > Gerhard > > ------------------------------------------------ > If patience is a virtue, and ignorance is bliss, > you can have a pretty good life if you're stupid > and willing to wait. > ------------------------------------------------ > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, email: [EMAIL PROTECTED] > >
/***************************************************************************** * Copyright (C) The Apache Software Foundation. All rights reserved. * * ------------------------------------------------------------------------- * * This software is published under the terms of the Apache Software License * * version 1.1, a copy of which has been included with this distribution in * * the LICENSE file. * *****************************************************************************/ package org.apache.cocoon.components.store; import org.apache.avalon.framework.activity.Startable; import org.apache.avalon.framework.configuration.Configurable; import org.apache.avalon.framework.configuration.Configuration; import org.apache.avalon.framework.configuration.ConfigurationException; import org.apache.avalon.framework.logger.AbstractLoggable; import org.apache.avalon.framework.parameters.Parameters; import org.apache.avalon.framework.thread.ThreadSafe; import java.util.ArrayList; import java.util.Enumeration; import java.util.Iterator; /** * This class is a implentation of a StoreJanitor. Store classes * can register to the StoreJanitor. When memory is too low, * the StoreJanitor frees the registered caches until memory is normal. * * @author <a href="mailto:[EMAIL PROTECTED]";>Christian Schmitt</a> * @author <a href="mailto:[EMAIL PROTECTED]";>Gerhard Froehlich</a> * @author <a href="mailto:[EMAIL PROTECTED]";>Peter Royal</a> * @author <a href="mailto:[EMAIL PROTECTED]";>Peter Hargreaves</a> */ public class StoreJanitorImpl extends AbstractLoggable implements StoreJanitor, Configurable, ThreadSafe, Runnable, Startable { private int freememory = -1; private int heapsize = -1; private int cleanupthreadinterval = -1; private int priority = -1; private Runtime jvm; private ArrayList storelist; private int index = -1; private static boolean doRun = false; private int m_percent; /** * Initialize the StoreJanitorImpl. * A few options can be used : * <UL> * <LI>freememory = how many bytes shall be always free in the jvm</LI> * <LI>heapsize = max. size of jvm memory consumption</LI> * <LI>cleanupthreadinterval = how often (sec) shall run the cleanup thread</LI> * <LI>threadpriority = priority of the thread (1-10). (Default: 10)</LI> * </UL> * * @param the Configuration of the application * @exception ConfigurationException */ public void configure(Configuration conf) throws ConfigurationException { if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Configure StoreJanitorImpl"); } this.setJVM(Runtime.getRuntime()); Parameters params = Parameters.fromConfiguration(conf); this.setFreememory(params.getParameterAsInteger("freememory",1000000)); this.setHeapsize(params.getParameterAsInteger("heapsize",60000000)); this.setCleanupthreadinterval(params.getParameterAsInteger("cleanupthreadinterval",10)); this.setPriority(params.getParameterAsInteger( "threadpriority", Thread.currentThread().getPriority())); this.m_percent = params.getParameterAsInteger( "percent_to_free",10); if ((this.getFreememory() < 1)) { throw new ConfigurationException("StoreJanitorImpl freememory parameter has to be greater then 1"); } if ((this.getHeapsize() < 1)) { throw new ConfigurationException("StoreJanitorImpl heapsize parameter has to be greater then 1"); } if ((this.getCleanupthreadinterval() < 1)) { throw new ConfigurationException("StoreJanitorImpl cleanupthreadinterval parameter has to be greater then 1"); } if ((this.getPriority() < 1)) { throw new ConfigurationException("StoreJanitorImpl threadpriority has to be greater then 1"); } if ((this.m_percent > 100 && this.m_percent < 1)) { throw new ConfigurationException("StoreJanitorImpl percent_to_free, has to be between 1 and 100"); } this.setStoreList(new ArrayList()); } public void start() { doRun = true; Thread checker = new Thread(this); if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Intializing checker thread"); } checker.setPriority(this.getPriority()); checker.setDaemon(true); checker.setName("checker"); checker.start(); } public void stop() { doRun = false; } private long inUseAfter; // Remember while sleeping. private long sleepPeriod; // Remember while sleeping. private long maxRateOfChange; // Remember while sleeping. /** * The "checker" thread checks if items should be removed from the stores. */ public void run() { // Initialise for a safe first calculation of rate of change. inUseAfter = inUseNow(); sleepPeriod = Long.MAX_VALUE; maxRateOfChange = 1; while (doRun) { // Monitor the rate of change of heap in use. long changeWhileSleeping = inUseNow() - inUseAfter; long rateOfChange = longDiv(changeWhileSleeping, sleepPeriod); // bpms (same as or kbps). if (maxRateOfChange < rateOfChange) maxRateOfChange = rateOfChange; // Output debug info. debug("Waking after " + sleepPeriod + "ms, in use change " + changeWhileSleeping + " to " + inUseNow() + ", rate " + rateOfChange + "kb/sec, max rate " + maxRateOfChange + "kb/sec"); debug("maxHeap=" + getMaxHeap() + ", totalHeap=" + jvm.totalMemory() + ", heapIsBig=" + heapIsBig()); debug("minFree=" + getMinFree() + ", freeHeap=" + jvm.freeMemory() + ", freeIsLow=" + freeIsLow()); // If the heap is big, and the free memory is low. if (heapIsBig() && freeIsLow()) { synchronized (this) { attemptToFreeStorage(); } } // Remember memory in use before sleeping in order to calc slope when waking. inUseAfter = inUseNow(); // If time to half fill could be less than max sleep, then sleep for half min time to fill (& remember it to calc slope next time). sleepPeriod = minTimeToFill()/2 < getMaxSleep() ? minTimeToFill()/2 : getMaxSleep(); debug("Store checker going to sleep for " + sleepPeriod + "ms, (max sleep=" + getMaxSleep() + "ms), with memory in use=" + inUseAfter); try { Thread.currentThread().sleep(sleepPeriod); } catch (InterruptedException ignore) {} } } /** * Starting at the next store, removes items, moving to the next again if necessary, until the specified number of items have been removed or all the stores are empty. */ private void attemptToFreeStorage() { int storeListSize = getStoreList().size(); int remove = getReduceBy(); debug("number of stores is " + storeListSize + ", number of items to be removed is " + remove + ", if possible!"); incIndex(); for (int cnt = 0; cnt < storeListSize; incIndex(), cnt++) { // Look in all stores if necessary. if ((remove = reduceStoreBy(remove)) == 0 ) break; // Keep looking till all items removed, } // or all stores are empty. if ( remove < getReduceBy() ) { // If items were removed call garbage collector. long gcTime = System.currentTimeMillis(); jvm.gc(); gcTime = System.currentTimeMillis() - gcTime; debug("items removed, so collecting garbage - took " + gcTime + "ms"); debug("minFree=" + getMinFree() + ", freeHeap=" + jvm.freeMemory() + ", freeIsLow=" + freeIsLow()); } } private void incIndex() { if (++index >= getStoreList().size()) index = 0; } // Increment the store index. /** * Reduce the current store by the number of items specified, if possible. * * @param remove The number of items to be removed. * @return the remaining count of items, that could not be removed. */ private int reduceStoreBy(int remove) { Store store = (Store)storelist.get(index); int sizeBefore = countSize(store); for (int i = 0; i < sizeBefore & remove > 0; i++, remove--) { store.free(); // Free the items falling within (or partly within) the reduce-by percentage. } int sizeAfter = countSize(store); debug("store index=" + index + ", size before=" + sizeBefore + ", size after=" + sizeAfter + ", removed=" + (sizeBefore - sizeAfter)); return remove; } /** * To check if total memory is big enough to allow stores to be reduced. * * @return true if big enough. */ private boolean heapIsBig() { return jvm.totalMemory() > getMaxHeap(); } /** * To check if free memory is small enough to start reducing stores. * * @return true if small enough. */ private boolean freeIsLow() { return jvm.freeMemory() < getMinFree(); } /** * To calculate the minimum time in which the memory could be filled at the maximum rate of use. * * @return the minimum time to fill. */ private long minTimeToFill() { return longDiv(jvm.freeMemory(), maxRateOfChange); } /** * Long division, guarding agains accidental divide by zero. * * @return the result of division. */ private long longDiv(long top, long bottom) { try { return top / bottom; } catch (Exception e) { return top > 0 ? Long.MAX_VALUE : Long.MIN_VALUE; } } /** * Calculate the jvm memory in use now. * * @return memory in use. */ private long inUseNow() { return jvm.totalMemory() - jvm.freeMemory(); } /** * Count the size of a store. * * @return the size of the store. */ private int countSize(Store store) { int size = 0; Enumeration enum = store.keys(); while(enum.hasMoreElements()) { size++; enum.nextElement(); } return size; } // Renaming of parameters due to new functionality. private int getMaxHeap() { return heapsize; } private int getMinFree() { return freememory; } private int getMaxSleep() { return cleanupthreadinterval*1000; } private int getReduceBy() { return m_percent; } /** * Shorten the call to print a debug message. */ private void debug(String message) { if (this.getLogger().isDebugEnabled()) { this.getLogger().debug(message); } } /** * This method register the stores * * @param the store to be registered */ public void register(Store store) { this.getStoreList().add(store); if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Registering store instance"); this.getLogger().debug("Size of StoreJanitor now:" + this.getStoreList().size()); } } /** * This method unregister the stores * * @param the store to be unregistered */ public void unregister(Store store) { this.getStoreList().remove(store); if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Unregister store instance"); this.getLogger().debug("Size of StoreJanitor now:" + this.getStoreList().size()); } } /** * This method return a java.util.Iterator of every registered stores * * <i>The iterators returned is fail-fast: if list is structurally * modified at any time after the iterator is created, in any way, the * iterator will throw a ConcurrentModificationException. Thus, in the * face of concurrent modification, the iterator fails quickly and * cleanly, rather than risking arbitrary, non-deterministic behavior at * an undetermined time in the future.</i> * * @return a java.util.Iterator */ public Iterator iterator() { return this.getStoreList().iterator(); } private int getFreememory() { return freememory; } private void setFreememory(int _freememory) { this.freememory = _freememory; } private int getHeapsize() { return this.heapsize; } private void setHeapsize(int _heapsize) { this.heapsize = _heapsize; } private int getCleanupthreadinterval() { return this.cleanupthreadinterval; } private void setCleanupthreadinterval(int _cleanupthreadinterval) { this.cleanupthreadinterval = _cleanupthreadinterval; } private int getPriority() { return this.priority; } private void setPriority(int _priority) { this.priority = _priority; } private Runtime getJVM() { return this.jvm; } private void setJVM(Runtime _runtime) { this.jvm = _runtime; } private ArrayList getStoreList() { return this.storelist; } private void setStoreList(ArrayList _storelist) { this.storelist = _storelist; } }
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
