Hi, This can be solved on Tomcat 4.x & Windows by tweaking the JAVA_OPTS in catalina.bat to include a -Xmx setting of -Xmx300000000 and a <store-janitor> heapsize parameter setting of 250000000 (in cocoon.xconf). There is no science to these figures, I just made them super large to avoid mucking around! I suspect there could be just a _little_ optimising done here ;-)
Please note that this applies to the use of the JispFilesystemStore as covered in the <cache-persistent> section of cocoon.xconf. IMHO this should be used in preference to the original FilesystemStore for lots of reasons, not least of which is that the original will invariably bomb on a Windows platform thanks to it generating hella long filenames. rgds Jeremy -----Original Message----- From: Peter Neumcke [mailto:pneumcke@;virbus.de] Sent: 04 November 2002 10:26 To: '[EMAIL PROTECTED]' Subject: AW: Lucene Problems Hi Jeremy, we had this problem too (on Linux)- solved it by setting the maximum number of open file descriptors to 2048: ulimit -n 2048 > -----Ursprungliche Nachricht----- > Von: Jeremy Quinn [mailto:jeremy@;media.demon.co.uk] > Gesendet: Montag, 4. November 2002 11:06 > An: [EMAIL PROTECTED] > Betreff: Re: Lucene Problems > > > > On Sunday, Nov 3, 2002, at 20:34 Europe/London, Bernhard Huber wrote: > > > hi, > > > > Many thanks for your reply, Bernhard. > > >> IOException in index() > > >> This causes the index to be incomplete, and causes further > problems > >> when you try to search using the index. > >> > >> If I use the Lucene Samples in Cocoon, and index my Cocoon > >> documentation, I always get this error. > > > > Hm, i once got this kind of error, too, > > but i can't remember exactly how i did solve it, > > perhaps you must set the mergefactor of lucene differently. > > As much as i remember lucene has some options about how > many files are > > kept opened in case of indexing, > > it later merges the indexes files, .... > > Ah Ha! I was beginning to think it was some weakness of > MacOSX JVM (or > maybe it still is ....). > > Have you any idea where this is configured? > > >> Before I started to have the indexing failures, I had another > >> problem, this time much more difficult to replicate. > >> > >> About every 10 searches, I would get a "Bad Resource" error from > >> LuceneGenerator. A simple reload of the page would give me > the search > >> hits I expected. > >> > >> Can anyone think of a Cocoon cause for these problems, or > should I > >> get onto the Lucene team about them? > > > > Do you have anymore detailed exception stack traces? > > I do for the indexing, but not for the intermittent fault, which has > not happened again since the indexing started failing. > > Here is the indexing exception report: > > Cocoon 2 - Internal server error > > type fatal > > message IOException in index() > > description org.apache.cocoon.ProcessingException: IOException in > index(): java.io.FileNotFoundException: > /Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon- > files/index/_1d.f30 (Too many open files) > > sender org.apache.cocoon.servlet.CocoonServlet > > source Cocoon servlet > > stack-trace > > org.apache.cocoon.ProcessingException: IOException in index(): > java.io.FileNotFoundException: > /Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon- > files/index/_1d.f30 (Too many open files) > at > org.apache.cocoon.components.search.SimpleLuceneCocoonIndexerI > mpl.index( > SimpleLuceneCocoonIndexerImpl.java:261) > at > org.apache.cocoon.www.samples.search.create_index_xsp.createIndex(/ > Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon- > files/org/apache/cocoon/www/samples/search/create_index_xsp.java:106) > at > org.apache.cocoon.www.samples.search.create_index_xsp.generate > (/Users/ > jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-f > iles/org/ > apache/cocoon/www/samples/search/create_index_xsp.java:190) > at > org.apache.cocoon.generation.ServerPagesGenerator.generate(Ser > verPagesGe > nerator.java:269) > at > org.apache.cocoon.components.pipeline.AbstractProcessingPipeli > ne.process > XMLPipeline(AbstractProcessingPipeline.java:512) > at > org.apache.cocoon.components.pipeline.impl.AbstractCachingProc > essingPipe > line.processXMLPipeline(AbstractCachingProcessingPipeline.java:204) > at > org.apache.cocoon.components.pipeline.AbstractProcessingPipeli > ne.process > (AbstractProcessingPipeline.java:483) > at > org.apache.cocoon.components.treeprocessor.sitemap.SerializeNo > de.invoke( > SerializeNode.java:149) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:85) > at > org.apache.cocoon.components.treeprocessor.sitemap.PreparableM > atchNode.i > nvoke(PreparableMatchNode.java:166) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod > e.invoke(P > ipelineNode.java:153) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo > de.invoke( > PipelinesNode.java:143) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:326) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:308) > at > org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i > nvoke(Moun > tNode.java:131) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:85) > at > org.apache.cocoon.components.treeprocessor.sitemap.PreparableM > atchNode.i > nvoke(PreparableMatchNode.java:166) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod > e.invoke(P > ipelineNode.java:153) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo > de.invoke( > PipelinesNode.java:143) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:326) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:308) > at > org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i > nvoke(Moun > tNode.java:131) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:85) > at > org.apache.cocoon.components.treeprocessor.sitemap.PreparableM > atchNode.i > nvoke(PreparableMatchNode.java:166) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod > e.invoke(P > ipelineNode.java:153) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo > de.invoke( > PipelinesNode.java:143) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:326) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:308) > at org.apache.cocoon.Cocoon.process(Cocoon.java:595) > at > org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet. > java:1069) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > er(Applica > tionFilterChain.java:247) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > cationFilt > erChain.java:193) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > rapperValv > e.java:260) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardC > ontextValv > e.java:191) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.catalina.core.StandardContext.invoke(StandardContex > t.java:239 > 6) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > Valve.java > :180) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDi > spatcherVa > lve.java:170) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:641) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > Valve.java > :172) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:641) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > gineValve. > java:174) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter. > java:223) > at > org.apache.coyote.http11.Http11Processor.process(Http11Process > or.java:40 > 5) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle > r.processC > onnection(Http11Protocol.java:380) > at > org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoi > nt.java:50 > 8) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( > ThreadPool > .java:533) > at java.lang.Thread.run(Thread.java:491) > java.io.FileNotFoundException: > /Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon- > files/index/_1d.f30 (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:93) > at java.io.RandomAccessFile.(RandomAccessFile.java:138) > at > org.apache.lucene.store.FSInputStream$Descriptor.(Unknown Source) > at org.apache.lucene.store.FSInputStream.(Unknown Source) > at org.apache.lucene.store.FSDirectory.openFile(Unknown Source) > at > org.apache.lucene.index.SegmentReader.openNorms(Unknown Source) > at org.apache.lucene.index.SegmentReader.(Unknown Source) > at > org.apache.lucene.index.IndexWriter.mergeSegments(Unknown Source) > at org.apache.lucene.index.IndexWriter.optimize(Unknown Source) > at > org.apache.cocoon.components.search.SimpleLuceneCocoonIndexerI > mpl.index( > SimpleLuceneCocoonIndexerImpl.java:259) > at > org.apache.cocoon.www.samples.search.create_index_xsp.createIndex(/ > Users/jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon- > files/org/apache/cocoon/www/samples/search/create_index_xsp.java:106) > at > org.apache.cocoon.www.samples.search.create_index_xsp.generate > (/Users/ > jermq/Library/TomCat/work/Standalone/localhost/cocoon/cocoon-f > iles/org/ > apache/cocoon/www/samples/search/create_index_xsp.java:190) > at > org.apache.cocoon.generation.ServerPagesGenerator.generate(Ser > verPagesGe > nerator.java:269) > at > org.apache.cocoon.components.pipeline.AbstractProcessingPipeli > ne.process > XMLPipeline(AbstractProcessingPipeline.java:512) > at > org.apache.cocoon.components.pipeline.impl.AbstractCachingProc > essingPipe > line.processXMLPipeline(AbstractCachingProcessingPipeline.java:204) > at > org.apache.cocoon.components.pipeline.AbstractProcessingPipeli > ne.process > (AbstractProcessingPipeline.java:483) > at > org.apache.cocoon.components.treeprocessor.sitemap.SerializeNo > de.invoke( > SerializeNode.java:149) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:85) > at > org.apache.cocoon.components.treeprocessor.sitemap.PreparableM > atchNode.i > nvoke(PreparableMatchNode.java:166) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod > e.invoke(P > ipelineNode.java:153) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo > de.invoke( > PipelinesNode.java:143) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:326) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:308) > at > org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i > nvoke(Moun > tNode.java:131) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:85) > at > org.apache.cocoon.components.treeprocessor.sitemap.PreparableM > atchNode.i > nvoke(PreparableMatchNode.java:166) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod > e.invoke(P > ipelineNode.java:153) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo > de.invoke( > PipelinesNode.java:143) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:326) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:308) > at > org.apache.cocoon.components.treeprocessor.sitemap.MountNode.i > nvoke(Moun > tNode.java:131) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:85) > at > org.apache.cocoon.components.treeprocessor.sitemap.PreparableM > atchNode.i > nvoke(PreparableMatchNode.java:166) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelineNod > e.invoke(P > ipelineNode.java:153) > at > org.apache.cocoon.components.treeprocessor.AbstractParentProce > ssingNode. > invokeNodes(AbstractParentProcessingNode.java:109) > at > org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNo > de.invoke( > PipelinesNode.java:143) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:326) > at > org.apache.cocoon.components.treeprocessor.TreeProcessor.proce > ss(TreePro > cessor.java:308) > at org.apache.cocoon.Cocoon.process(Cocoon.java:595) > at > org.apache.cocoon.servlet.CocoonServlet.service(CocoonServlet. > java:1069) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > er(Applica > tionFilterChain.java:247) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > cationFilt > erChain.java:193) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > rapperValv > e.java:260) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardC > ontextValv > e.java:191) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.catalina.core.StandardContext.invoke(StandardContex > t.java:239 > 6) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHost > Valve.java > :180) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDi > spatcherVa > lve.java:170) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:641) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReport > Valve.java > :172) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:641) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > gineValve. > java:174) > at > org.apache.catalina.core.StandardPipeline$StandardPipelineValv > eContext.i > nvokeNext(StandardPipeline.java:643) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipel > ine.java:4 > 80) > at > org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995) > at > org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter. > java:223) > at > org.apache.coyote.http11.Http11Processor.process(Http11Process > or.java:40 > 5) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandle > r.processC > onnection(Http11Protocol.java:380) > at > org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoi > nt.java:50 > 8) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( > ThreadPool > .java:533) > at java.lang.Thread.run(Thread.java:491) > > request-uri > > /cocoon/samples/search/create > > path-info > > samples/search/create > > > > Thanks again > > regards Jeremy > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, email: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED] __________________________________________________ Do You Yahoo!? Everything you'll ever need on one web page from News and Sport to Email and Music Charts http://uk.my.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, email: [EMAIL PROTECTED]
