Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when reading a huge resource - ASF JIRA
Robert To get an answer to such questions the cocoon developer list would be the right place (i'll do it for you this time as cc)! That's what the lists are for! See also [1] for cocoons mailing lists [1] http://cocoon.apache.org/1275_1_1.html We are encountering this same problem in Cocoon 2.1.11. What is the status of this issue? Is there a fix for it? It doesn't seem to exist in Cocoon 2.0.4. Obviously net yet, otherwise you could see it in the issue tracker. Maybe BufferedOutputStream was introduced after cocoon 2.0.4 so that version doesn't suffers this issue. Ciao Felix https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12568953#action_12568953 Sincerely, Robert La Ferla OMS
cocoon contractor £ 400 -500 a day
Hi I am a looking for another cocoon developer to work on contract for a 6 month rolling contract based in west London Kind regards Marcus Clemens Director Senior Consultant Mercator IT Ltd Tel: 01892 752730 Fax: 01892 750972 Email: [EMAIL PROTECTED] Address: The Studio, Eridge Park, Eridge Green, Tunbridge Wells, Kent TN3 9JS Registered in England no: 05755983 Registered office: The Studio, Eridge Park, Eridge Green, Tunbridge Wells, Kent TN3 9JS This email may contain privileged/confidential information and is for the intended addressee only. If you have received this message in error then you must not use, retain, disseminate or otherwise deal with it. Please notify the sender by return email and destroy. The views of the author may not necessarily constitute the views of Mercator IT Solutions Ltd. Nothing in this email shall bind Mercator IT Solutions Ltd in any contract or obligation. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 04 March 2008 12:12 To: dev@cocoon.apache.org Subject: ForrestBot build for cocoon-docs FAILED Automated build for cocoon-docs FAILED Log attached. -- Forrestbot run ended at 04 March 12:12 PM Using Forrest 0.9-dev Forrestbot administrator: ForrestBot -- [echo] ... Forrest render START 2008-03-04 12:12:10 ... Rendering docs in /export/home/config/forrestbot-trunk/conf/work/cocoon-docs check-java-version: [echo] This is apache-forrest-0.9-dev [echo] Using Java 1.4 from /usr/j2se/jre init-props: [mkdir] Created dir: /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp echo-settings: check-skin: init-proxy: fetch-skins-descriptors: fetch-skin: unpack-skins: init-skins: fetch-plugins-descriptors: [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [echo] Fetching plugins descriptor: http://forrest.apache.org/plugins/plugins.xml [get] Getting: http://forrest.apache.org/plugins/plugins.xml [get] To: /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp/plugins-1.xml [get] local file date : Tue Apr 10 05:50:24 GMT+00:00 2007 [get] . [get] last modified = Wed Apr 11 02:07:04 GMT+00:00 2007 [echo] Fetching plugins descriptor: http://forrest.apache.org/plugins/whiteboard-plugins.xml [get] Getting: http://forrest.apache.org/plugins/whiteboard-plugins.xml [get] To: /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp/plugins-2.xml [get] local file date : Fri Feb 15 13:06:17 GMT+00:00 2008 [get] Not modified - so not downloaded [echo] Plugin list loaded from http://forrest.apache.org/plugins/plugins.xml. [echo] Plugin list loaded from http://forrest.apache.org/plugins/whiteboard-plugins.xml. init-plugins: [mkdir] Created dir: /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/webapp/conf [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [copy] Copying 1 file to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp [echo] -- Installing plugin: org.apache.forrest.plugin.output.pdf -- check-plugin: [echo] org.apache.forrest.plugin.output.pdf is available in the build dir. Trying to update it... init-props: echo-settings: init-proxy: fetch-plugins-descriptors: fetch-plugin: [echo] Trying to find the description of org.apache.forrest.plugin.output.pdf in the different descriptor files [echo] Using the descriptor file /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp/plugins-1.xml... [xslt] Processing /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp/plugins-1.xml to /export/home/config/forrestbot-trunk/conf/work/cocoon-docs/tmp/pluginlist2fetchbuild.xml [xslt] Loading stylesheet /export/opt/forrest-trunk/main/var/pluginlist2fetch.xsl fetch-local-unversioned-plugin: get-local: [echo] Trying to locally get org.apache.forrest.plugin.output.pdf [echo] Looking in local /export/opt/forrest-trunk/plugins [echo] Found ! init-build-compiler: echo-init: init: compile: jar: local-deploy: [echo] Locally deploying org.apache.forrest.plugin.output.pdf build: [echo] Plugin org.apache.forrest.plugin.output.pdf deployed ! Ready to configure fetch-remote-unversioned-plugin-version-forrest:
[jira] Subscription: COCOON-open-with-patch
Issue Subscription Filter: COCOON-open-with-patch (106 issues) Subscriber: cocoon Key Summary COCOON-2168 ResourceReader produces Java Heap Overflow when reading a huge resource https://issues.apache.org/jira/browse/COCOON-2168 COCOON-2162 [PATCH] Fix for Paginator when accessing out of bounds Pagination page https://issues.apache.org/jira/browse/COCOON-2162 COCOON-2137 XSD Schemas for CForms Development https://issues.apache.org/jira/browse/COCOON-2137 COCOON-2114 fix sorting in TraversableGenerator https://issues.apache.org/jira/browse/COCOON-2114 COCOON-2109 Incorrent cleanup of expired continuations https://issues.apache.org/jira/browse/COCOON-2109 COCOON-2108 xmodule:flow-attr Does not accept document objects https://issues.apache.org/jira/browse/COCOON-2108 COCOON-2104 [PATCH] Add base URI fixup support to XIncludeTransformer https://issues.apache.org/jira/browse/COCOON-2104 COCOON-2100 Retrieving mimeType returned by pipeline executed from Flow https://issues.apache.org/jira/browse/COCOON-2100 COCOON-2071 Option to turn off pooling for components (probably faster on new JVMs and simpler debugging) https://issues.apache.org/jira/browse/COCOON-2071 COCOON-2063 NekoHTMLTransformer needs to set the default-encoding of the current system to work properly with UTF-8 https://issues.apache.org/jira/browse/COCOON-2063 COCOON-2041 WebDAV Returns improper status on PUT https://issues.apache.org/jira/browse/COCOON-2041 COCOON-2040 Union widget does not work with booleanfield set as case widget https://issues.apache.org/jira/browse/COCOON-2040 COCOON-2037 New DynamicGroup widget https://issues.apache.org/jira/browse/COCOON-2037 COCOON-2035 NPE in the sorter of the EnhancedRepeater https://issues.apache.org/jira/browse/COCOON-2035 COCOON-2032 [PATCH] Sort order in paginated repeater https://issues.apache.org/jira/browse/COCOON-2032 COCOON-2030 submit-on-change doesn't work for a multivaluefield with list-type=checkbox https://issues.apache.org/jira/browse/COCOON-2030 COCOON-2018 Use thread context class loader to load custom binding classes https://issues.apache.org/jira/browse/COCOON-2018 COCOON-2017 More output beautification options for serializers https://issues.apache.org/jira/browse/COCOON-2017 COCOON-2015 Doctype added twice because root element (html) is inlined https://issues.apache.org/jira/browse/COCOON-2015 COCOON-2002 HTML transformer only works with latin-1 characters https://issues.apache.org/jira/browse/COCOON-2002 COCOON-1985 AbstractCachingProcessingPipeline locking with IncludeTransformer may hang pipeline https://issues.apache.org/jira/browse/COCOON-1985 COCOON-1974 Donating ContextAttributeInputModule https://issues.apache.org/jira/browse/COCOON-1974 COCOON-1973 CaptchaValidator: allow case-insensitive matching https://issues.apache.org/jira/browse/COCOON-1973 COCOON-1964 Redirects inside a block called via the blocks protocol fail https://issues.apache.org/jira/browse/COCOON-1964 COCOON-1963 Add a redirect action to the browser update handler https://issues.apache.org/jira/browse/COCOON-1963 COCOON-1960 Pipeline errors for generator/reader already set should provide more information https://issues.apache.org/jira/browse/COCOON-1960 COCOON-1949 [PATCH] load flowscript from file into specified Rhino context object https://issues.apache.org/jira/browse/COCOON-1949 COCOON-1946 [PATCH] - Javaflow Sample errors trying to enhance Javaflow classes and showing cform templates https://issues.apache.org/jira/browse/COCOON-1946 COCOON-1943 [Patch] Parameters in blocks-protocol URIs get decoded too early https://issues.apache.org/jira/browse/COCOON-1943 COCOON-1932 [PATCH] correct styling of disabled suggestion lists https://issues.apache.org/jira/browse/COCOON-1932 COCOON-1929 [PATCH] Reloading classloader in Cocoon 2.2 https://issues.apache.org/jira/browse/COCOON-1929 COCOON-1917 Request Encoding problem: multipart/form vs. url encoded https://issues.apache.org/jira/browse/COCOON-1917 COCOON-1915 Nullable value with additional String or XMLizable in JavaSelectionList https://issues.apache.org/jira/browse/COCOON-1915 COCOON-1914 Text as XMLizable in EmptySelectionList https://issues.apache.org/jira/browse/COCOON-1914 COCOON-1899 [PATCH] Cocoon XML:DB Implementation should not depend on Xindice https://issues.apache.org/jira/browse/COCOON-1899 COCOON-1898 [PATCH] XPatch support for maven-cocoon-deployer-plugin https://issues.apache.org/jira/browse/COCOON-1898 COCOON-1893 XML-Binding: Problem creating a new element
[jira] Commented: (COCOON-2168) ResourceReader produces Java Heap Overflow when reading a huge resource
[
https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12575550#action_12575550
]
Jörg Heinicke commented on COCOON-2168:
---
I've looked into this issue and I'm against your fix. It stands completely
against the idea of buffering the whole pipeline content which is done for
error handlers. It should be possible for them to reset the OutputStream. Or
asked in other words? Why using the non-flushing BufferedOutputStream at all?
Now how to handle the issue with huge resources? Should be quite easy. From
what I see from the code (AbstractProcessingPipeline) it is possible to
configure and setup/parameterize a pipeline with outputBufferSize. This means
on both map:pipe and map:pipeline it should be possible to set an actual buffer
size. Only if none is set (and it defaults to -1) the non-flushing
BufferedOutputStream is used.
Root sitemap has this example (in map:components/map:pipes section):
map:pipe name=noncaching
src=org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline
pool-max=${noncaching-pipeline.pool-max}
!-- parameter name=outputBufferSize value=8192/ --
/map:pipe
The following should also work (in map:pipelines section):
map:pipeline
map:parameter name=outputBufferSize value=8192/
...
/map:pipeline
We could argue about another default value than -1 though. Something like
1024^2.
WDYT?
ResourceReader produces Java Heap Overflow when reading a huge resource
---
Key: COCOON-2168
URL: https://issues.apache.org/jira/browse/COCOON-2168
Project: Cocoon
Issue Type: Bug
Components: * Cocoon Core
Affects Versions: 2.2-dev (Current SVN)
Reporter: Felix Knecht
Attachments: ResourceReader.diff, test-case.tar.gz
When reading a huge resource (i.e. 700MB file) the ResourceReader produces an
overflow due to the BufferedOutputStream which is used (and forced to be used
via AbstractReader). The BufferedOutputStream flushes only at the end (or
when forced to), but overwrites the flush method to do nothing.
As I don't know exactly where the BufferedOutputStream is used and what kind
of impacts it will have to change it there I'm just going to fix the
ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when reading a huge resource - ASF JIRA
On 05.03.2008 11:19, Felix Knecht wrote:
We are encountering this same problem in Cocoon 2.1.11. What is the
status of this issue? Is there a fix for it? It doesn't seem to
exist in Cocoon 2.0.4.
Obviously net yet, otherwise you could see it in the issue tracker.
Maybe BufferedOutputStream was introduced after cocoon 2.0.4 so that
version doesn't suffers this issue.
I've looked into this issue and I'm against Felix' fix. It stands
completely against the idea of buffering the whole pipeline content
which is done for error handlers. It should be possible for them to
reset the OutputStream. Or asked in other words? Why using the
non-flushing BufferedOutputStream at all?
Now how to handle the issue with huge resources? Should be quite easy.
From what I see from the code (AbstractProcessingPipeline) it is
possible to configure and setup/parameterize a pipeline with
outputBufferSize. This means on both map:pipe and map:pipeline it
should be possible to set an actual buffer size. Only if none is set
(and it defaults to -1) the non-flushing BufferedOutputStream is used.
Root sitemap has this example (in map:components/map:pipes section):
map:pipe name=noncaching
src=org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline
pool-max=${noncaching-pipeline.pool-max}
!-- parameter name=outputBufferSize value=8192/ --
/map:pipe
The following should also work (in map:pipelines section):
map:pipeline
map:parameter name=outputBufferSize value=8192/
...
/map:pipeline
We could argue about another default value than -1 though. Something
like 1024^2.
WDYT?
Joerg
ResourceReader headers
When looking at the history of ResourceReader I also noticed something else: In rev 155087 [1] Vadim did the following change: Move response header initialization into the setup phase. Remove Last-Modified header initialization: this is done in the environment by request from AbstractProcessingPipeline. In the next revision 410112 [2] (but 15 months later) Antonio readded setting the Last-Modified header: Fix COCOON-1840 XMLFileModule: file-specific configuration ignored. The commit message seems to be wrong and the change was not included in the user-provided patch [3]. It rather seems to be COCOON-1266 [4] but the provided and applied patch is older than Vadim's change. I guess we should review this code. The revisions for trunk are 155099 [5] and 410113 [6]. Joerg [1] http://svn.apache.org/viewvc?view=revrevision=155087 [2] http://svn.apache.org/viewvc?view=revrevision=410112 [3] https://issues.apache.org/jira/browse/COCOON-1840 [4] https://issues.apache.org/jira/browse/COCOON-1266 [5] http://svn.apache.org/viewvc?view=revrevision=155099 [6] http://svn.apache.org/viewvc?view=revrevision=410113
Re: ResourceReader headers
Hi Joerg, Good catch! I confirm revision 410112 has a bad log, the correct log should be related to COCOON-1266. The diff [1] is just about 1 line: response.addHeader(Vary, Host); I dropped the Vary header because the vary header in IE is broken [2]. Hence using it to stop caching is just a hack. See also[3]. Hope this helps. Best Regards, Antonio Gallardo. [1] http://svn.apache.org/viewvc/cocoon/branches/BRANCH_2_1_X/src/java/org/apache/cocoon/reading/ResourceReader.java?r1=155087r2=410112diff_format=h [2] http://lists.over.net/pipermail/mod_gzip/2002-December/006826.html [3] https://issues.apache.org/jira/browse/COCOON-1424 Here is the diff between Vadim code and the Joerg Heinicke escribió: When looking at the history of ResourceReader I also noticed something else: In rev 155087 [1] Vadim did the following change: Move response header initialization into the setup phase. Remove Last-Modified header initialization: this is done in the environment by request from AbstractProcessingPipeline. In the next revision 410112 [2] (but 15 months later) Antonio readded setting the Last-Modified header: Fix COCOON-1840 XMLFileModule: file-specific configuration ignored. The commit message seems to be wrong and the change was not included in the user-provided patch [3]. It rather seems to be COCOON-1266 [4] but the provided and applied patch is older than Vadim's change. I guess we should review this code. The revisions for trunk are 155099 [5] and 410113 [6]. Joerg [1] http://svn.apache.org/viewvc?view=revrevision=155087 [2] http://svn.apache.org/viewvc?view=revrevision=410112 [3] https://issues.apache.org/jira/browse/COCOON-1840 [4] https://issues.apache.org/jira/browse/COCOON-1266 [5] http://svn.apache.org/viewvc?view=revrevision=155099 [6] http://svn.apache.org/viewvc?view=revrevision=410113
