[jira] Commented: (COCOON-1681) Generator directory: Caching too much
[ http://issues.apache.org/jira/browse/COCOON-1681?page=comments#action_12365255 ] Jean-Baptiste Quenot commented on COCOON-1681: -- Apparently, what is causing the two isValid() invocations is that you use map:aggregate along with a cocoon: URL. The latter is handled by SitemapSource which is known to call the pipeline twice, one time for getting the validity, and another time for retrieving pipeline result. So yes, we need to have a consistent way of handling isValid() in DirectoryGenerator. Checking lastModified() twice is not a big problem provided that this check is only performed when all of the following conditions are met: 1) the expiry time has been reached 2) a file was removed or modified Generator directory: Caching too much --- Key: COCOON-1681 URL: http://issues.apache.org/jira/browse/COCOON-1681 Project: Cocoon Type: Bug Components: * Cocoon Core Versions: 2.1.8, 2.1.7 Reporter: Antonio Fiol Assignee: Jean-Baptiste Quenot Attachments: DirectoryGenerator.java.patch, stack1, stack2, stack3 In some cases, an update to the directory is not detected by the DirectoryGenerator. Debugging the issue, I discovered that isValid() is called twice on the same DirValidity, but it returns different values (-1 the first time, 1 the second time). Apparently, the reason for the inconsistency would be solved by removing the first of the two lines that update the expiry time in the isValid() method in DirValidity, but I am not sure whether this could cause problems in other places. A possibility would be that a DirValidity stores the fact that it already detected it is invalid, and is changed so that it always return -1. But... Are DirValidity objects reused? Could this change cause problems? I have not tested, so I don't know. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COCOON-1681) Generator directory: Caching too much
[
http://issues.apache.org/jira/browse/COCOON-1681?page=comments#action_12364075
]
Antonio Fiol commented on COCOON-1681:
--
I just attached three stack traces. However, only two of them are relevant for
this bug, but I am not knowledgeable enough to know which one is not.
Why do I say that? We have a non-trivial sitemap, with 5 match sections implied
in this situation.
Called URL is: /corresponsales/propuestas/deportes/inicio.html
External sitemap part gets called:
!-- Step 1 --
map:match pattern=**.html
map:act type=session
map:parameter name=action value=create/
map:act type=session-propagator
map:parameter name=java:comp/env/skin value={skin;} /
map:generate src=cocoon:/interno/{../../1}/!-- This calls Step
2 --
map:call resource=xpage2html
map:parameter name=hoja value=pantalla/
map:parameter name=pagina value={../../1}/
/map:call
map:serialize status-code=200 /
/map:act
/map:act
/map:match
Here is the relevant part of the internal sitemap, which is mounted on
/interno and so its Step 2 at the bottom is called from Step 1:
!-- Step 5 --
map:match pattern=inclusion/**/lista-descendiente-*(*)
map:generate type=directory src={paginas;}/{1}
map:parameter name=include value=.*\.html|^[^.]*$
/!-- Directories that have no period and HTML files --
map:parameter name=exclude value=historico /
map:parameter name=reverse value=true /
map:parameter name=sort value={2} /
map:parameter name=depth value=3 /
/map:generate
map:transform type=paginate src=pagesheets/file6.xml!--
Gets the dir:file paged by groups of 6 --
map:parameter name=page value={3} /
/map:transform
map:transform src=xsl/xdoc/directory.xsl
map:parameter name=path value={1} /
map:parameter name=orden value={2} /
/map:transform
map:serialize/
/map:match
!-- Step 4 --
map:match pattern=inclusion/**/lista-descendiente-*
map:redirect-to
uri=cocoon:/inclusion/{1}/lista-descendiente-{2}(1) /!-- Calls Step 5 --
/map:match
!-- Step 3 --
map:match pattern=paginas/**
map:select type=resource-exists
map:when test={paginas;}/{1}.xdoc!-- This file
exists, so this is the one that gets called. See the file below --
map:generate src={paginas;}/{1}.xdoc/
/map:when
map:otherwise
map:generate type=html src={paginas;}/{1}.html/
map:transform src=xsl/xdoc/html.xsl/
/map:otherwise
/map:select
map:transform type='xinclude' /!-- This xincludes Step 4.
See the file below for URL --
map:serialize/
/map:match
!-- Step 2 --
map:match pattern=**
map:aggregate element=page
map:part label=contenido src=cocoon:/paginas/{1}/!--
This will call Step 3, which will ultimately call directory generator, but
calling isValid() twice --
map:part src=cocoon:/datos/version-pdf/{1} /!-- This
uses directory generator, and isValid() only gets called once, so sitemap not
posted. One of the stack traces comes from here. --
map:part src=cocoon:/navegacion/principal/
map:part src=cocoon:/navegacion/secundaria/{1}/
map:part src=cocoon:/datos/calendario/hoy/completo/
map:part src={dinamico;}/xml/buscadores.xml/
/map:aggregate
map:serialize/
/map:match
inicio.xdoc file contains (only relevant part shown):
xdoc xmlns:xi=http://www.w3.org/2001/XInclude;
head
title[...]/title
/head
body
[...]
xi:include
href=cocoon://interno/inclusion/corresponsales/propuestas/deportes/lista-descendiente-lastModified/!--
Calls Step 4 --
/body
/xdoc
I hope it is not too difficult to follow. :-(
Generator directory: Caching too much
---
Key: COCOON-1681
URL: http://issues.apache.org/jira/browse/COCOON-1681
Project: Cocoon
Type: Bug
Components: * Cocoon Core
Versions: 2.1.8, 2.1.7
Reporter: Antonio Fiol
Assignee: Jean-Baptiste Quenot
Attachments: DirectoryGenerator.java.patch, stack1, stack2, stack3
In some cases, an update to the directory is not detected by the
DirectoryGenerator.
Debugging the issue, I discovered that isValid() is called twice on the same
DirValidity, but it returns
[jira] Commented: (COCOON-1681) Generator directory: Caching too much
[ http://issues.apache.org/jira/browse/COCOON-1681?page=comments#action_12363933 ] Jean-Baptiste Quenot commented on COCOON-1681: -- Disabling the expiry check is not an option, because it would mean that we make a stat() system call for every generated file everytime the generator is invoked. It would be better to understand why isValid() is called twice by providing the two stack frames (copy/paste from the debugger). The first isValid() call returning INVALID should trigger the removal of the cached response. Generator directory: Caching too much --- Key: COCOON-1681 URL: http://issues.apache.org/jira/browse/COCOON-1681 Project: Cocoon Type: Bug Components: * Cocoon Core Versions: 2.1.8, 2.1.7 Reporter: Antonio Fiol Assignee: Cocoon Developers Team In some cases, an update to the directory is not detected by the DirectoryGenerator. Debugging the issue, I discovered that isValid() is called twice on the same DirValidity, but it returns different values (-1 the first time, 1 the second time). Apparently, the reason for the inconsistency would be solved by removing the first of the two lines that update the expiry time in the isValid() method in DirValidity, but I am not sure whether this could cause problems in other places. A possibility would be that a DirValidity stores the fact that it already detected it is invalid, and is changed so that it always return -1. But... Are DirValidity objects reused? Could this change cause problems? I have not tested, so I don't know. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COCOON-1681) Generator directory: Caching too much
[ http://issues.apache.org/jira/browse/COCOON-1681?page=comments#action_12364063 ] Antonio Fiol commented on COCOON-1681: -- No, I am not disabling the expiry check. This was a temporary workaround. The code in isValid says something like: if not yet expired (current time expiry time) return VALID expiry time = current time OFFENDING LINE check filesystem and return INVALID if it really is INVALID (as it is not invalid, and FS was already checked) expiry time = current time return VALID What happens is that if it is called twice on an expired directory, the first time will return INVALID, but will (incorrectly) update the expiry time. So the second call will incorrectly return VALID, because it is not expired. I agree that a single call to isValid() would be much better, but isValid should anyway be implemented in a secure (consistent) manner. With our patch, if the cached version is invalid, the FS is checked twice instead of the ideal once. But if it is not expired, no FS check is done, so it's quite different from disabling the expiry checks. In any case, I liked your suggestion, and will try to provide the two stack frames, and I will also post the patch today (we finished testing yesterday, and it works like a charm). Generator directory: Caching too much --- Key: COCOON-1681 URL: http://issues.apache.org/jira/browse/COCOON-1681 Project: Cocoon Type: Bug Components: * Cocoon Core Versions: 2.1.8, 2.1.7 Reporter: Antonio Fiol Assignee: Jean-Baptiste Quenot In some cases, an update to the directory is not detected by the DirectoryGenerator. Debugging the issue, I discovered that isValid() is called twice on the same DirValidity, but it returns different values (-1 the first time, 1 the second time). Apparently, the reason for the inconsistency would be solved by removing the first of the two lines that update the expiry time in the isValid() method in DirValidity, but I am not sure whether this could cause problems in other places. A possibility would be that a DirValidity stores the fact that it already detected it is invalid, and is changed so that it always return -1. But... Are DirValidity objects reused? Could this change cause problems? I have not tested, so I don't know. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COCOON-1681) Generator directory: Caching too much
[ http://issues.apache.org/jira/browse/COCOON-1681?page=comments#action_12363884 ] Antonio Fiol commented on COCOON-1681: -- Still an issue on 2.1.8, and still an issue in our project. We are testing --we started this week-- a patch doing exactly what is said in the description (one line removal). So far, it is working for us. Maybe it is not the most efficient solution, but has not shown performance problems either (in some parts of the project, we work with depth=6, with quite a lot of files and directories). Please bear with our long times. I hope I will confirm our results by the end of this week. Generator directory: Caching too much --- Key: COCOON-1681 URL: http://issues.apache.org/jira/browse/COCOON-1681 Project: Cocoon Type: Bug Components: * Cocoon Core Versions: 2.1.8, 2.1.7 Reporter: Antonio Fiol Assignee: Cocoon Developers Team In some cases, an update to the directory is not detected by the DirectoryGenerator. Debugging the issue, I discovered that isValid() is called twice on the same DirValidity, but it returns different values (-1 the first time, 1 the second time). Apparently, the reason for the inconsistency would be solved by removing the first of the two lines that update the expiry time in the isValid() method in DirValidity, but I am not sure whether this could cause problems in other places. A possibility would be that a DirValidity stores the fact that it already detected it is invalid, and is changed so that it always return -1. But... Are DirValidity objects reused? Could this change cause problems? I have not tested, so I don't know. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (COCOON-1681) Generator directory: Caching too much
[ http://issues.apache.org/jira/browse/COCOON-1681?page=comments#action_12356956 ] Antonio Fiol commented on COCOON-1681: -- Workaround: Set map:parameter name=refreshDelay value=-1 / so the expiry check is disabled. Generator directory: Caching too much --- Key: COCOON-1681 URL: http://issues.apache.org/jira/browse/COCOON-1681 Project: Cocoon Type: Bug Components: * Cocoon Core Versions: 2.1.8-dev (Current SVN), 2.1.7 Reporter: Antonio Fiol In some cases, an update to the directory is not detected by the DirectoryGenerator. Debugging the issue, I discovered that isValid() is called twice on the same DirValidity, but it returns different values (-1 the first time, 1 the second time). Apparently, the reason for the inconsistency would be solved by removing the first of the two lines that update the expiry time in the isValid() method in DirValidity, but I am not sure whether this could cause problems in other places. A possibility would be that a DirValidity stores the fact that it already detected it is invalid, and is changed so that it always return -1. But... Are DirValidity objects reused? Could this change cause problems? I have not tested, so I don't know. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
