Hello Maarten, There is quite something to know about how to optimize your stores, but I will try to help you out on every issue. If afterwards things are still not clear, let me know. The problem you are having is pretty straighforward, and not to hard to fix, though needs quite some understanding of cocoon's caching impl.
> > > Hi! > Please help me with the following problem, cause everything I > try doesn't work. > > We want to cache large xml-documents every night from an > exist-database with > cocoon 2.1.9. Those files are about 600K each and we want to > store them on > harddisk. The problem is that while genereating the cache, > java runs out of > memory, depending on how much memory java has. It gives an error: > java.lang.OutOfMemoryError: Java heap space. > I would think that if memory is full, things should > automatically be cleared > from cache in memory. But that doens't seem te happen. I've > tried several > settings for java-memory but that all doesn't make a difference. > > This is the code in the sitemap for creating the cache: > > <map:transform type="cinclude"> > <map:parameter name="source" value="d:/cache-dir"/> > <map:parameter name="support-caching" value="true"/> > <map:parameter name="expires" value="172800"/> > </map:transform> Never used the cinclude transformer like this (I always use the by default caching IncludeTransformer, but since you are connecting to db, it is not cacheable by default.) One thing I am not sure about, and I really need confirmation about it from you: Where do the cached items end up in? In the transient store, or in the default store? How do you know this? Just start your app, and only call the pipeline where this cinclude is in. Now, go to your status page (http://localhost:8888/status by default, so just the hostname:port/status), and look where cache keys are created. If they are added to transient store, we have to change that, if they are added to EHDefaultstore, it is ok > > > The cocoon.xconf has the following settings: > <transient-store logger="core.store.transient"> > <parameter name="maxobjects" value="1000"/> > <parameter name="use-persistent-cache" value="false"/> > </transient-store> > > > <store logger="core.store"> > <parameter name="maxobjects" value="1000"/> > <parameter name="eternal" value="false"/> > <parameter name="timeToLiveSeconds" value="0"/> > <!-- 1 day --> > <parameter name="timeToIdleSeconds" value="86400"/> > <parameter name="overflow-to-disk" value="true"/> > <parameter name="diskpersistent" value="true"/> > <parameter name="use-persistent-cache" value="true"/> > </store> As far as I know, use-persistent-cache is never used. diskpersistent means persistent between jvm restarts. > > And the following settings are in core.properties (the more > memory I give, the > longer it takes before the memory error, but memory is at > some point limited off > course): > store-janitor.freememory = 2000000 > store-janitor.heapsize = 200000000 > store-janitor.cleanup-thread-interval = 3 > store-janitor.percent-to-free = 10 So, what is happening: you are having a heapsize of for example 200000000 (around 200 Mb, little less). How much memory does your JVM has? The heapsize should *always* be smaller then the JVM maxmemory! Cocoon has a build in memory manager, the StoreJanitor, that tries to free memory when used memory by the JVM exceeds the heapsize (the it tries to free percent_to_free). In my opinion, when this StoreJanitor is needed to free memory, you will run into problems sooner or later anyway. So, the goal of tuning you app is, that the StoreJanitor never tries to free memory. > > I've also tried to change the jetty-settings, with no luck: > <Call name="addListener"> > <Arg> > <New class="org.mortbay.http.SocketListener"> > <Set name="Port"><SystemProperty name="jetty.port" > default="8888"/></Set> > <Set name="MinThreads">10</Set> > <Set name="MaxThreads">100</Set> > <Set name="MaxIdleTimeMs">30000</Set> > <Set name="LowResourcePersistTimeMs">5000</Set> > <Set name="bufferSize">8192</Set> > </New> > </Arg> > </Call> Do not look for a solution in the jetty-settings . > > EHcache is updated to ehcache-1.2.3.jar. > > I hope someone can give some thoughts about were to look. So, why do you get OOM? Suppose, you run with a JVM of 256 Mb max memory. Now, you are caching your pipelines (I suppose all of them). So, suppose, you have a page, like this: <map:pipeline type="caching"> <map:match pattern="mypage"> <map:aggregate> <map:part src="cocoon:/get-from-exist"/> <map:part src="cocoon:/someelse"/> </map:aggregate> <map:serialize type="xml"/> </map:match> Now, suppose, your cocoon:/get-from-exist is nicely cached, with the 600 kb file. Then, the block above here, is cached as well when cocoon:/someelse is cacheable, so, for example you already have ~1.2 Mb (2 times 600). But, it might be possible that cocoon caches it many more times. Anyway, even if cocoon caches it ones, you see that you have maxobjects in your store settings of 1000 items. So, suppose at some point, there are 500 cached items with the 600k files in it, that is 300 Mb! maxobjects simple means : "how many cached responses should be kpet in memory". If your cached items end up in the transient store, you cannot overflow them to disk (let me know, if in transientstore, then I can help you out with another way, without the cinclude). If in ehdefaultstore, you can overflow items to disk. This is what you want. Your maxobjects should be lowered in order to avoid your OOM. Bring them back to 150 or something, and try again (keep overflow-to-disk = true. when 150 is exceeded, cached responses are overflowed to disk) . You can optimize settings just by testing: run xenu (http://home.snafu.de/tilman/xenulink.html), a cralwer, and look at the status page what is happening with the memory. So, this is in short a little on caching. If you really want to have the best solution, you have to do something similar I lately did, but it is not really trivial. From our remote repository, I wanted binary data to be stored in a diskcache only, but normal files, like .xml documents, I want to be kept in memory, for performance. Therefor, I have multiple stores in my cocoon app. Since we were also using event cache, I had to build this, because it was not supported by default, but I think, when you are not using event cache, you can configure multiple EHDefaultStores without problems. But, you need to know how to configure all the roles, your map:pipes (multiple cache-roles) and your cocoon.xconf. Hopefully, you won't be needing all this, Regards Ard > > Regards, Maarten Couvreur > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
