[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194398#comment-14194398 ] Viktor Adam commented on SLING-3984: [~cziegeler] We've had a few deployment since my last update and we haven't had this problem so I think you can close this ticket. Thank you for your help! JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166846#comment-14166846 ] Carsten Ziegeler commented on SLING-3984: - [~vadam_ig] It seems this is all somehow related to taglibs. I have the feeling that while one jsp has successfully compiled a taglib, another jsp using the same taglib is invalidated (as it has been changed) and therefore the taglib class is removed. The first jsp can't load the taglib anymore, the second succeeds, and the first one succeeds on next access. Looking at the code, I could imagine that the problem is somewhere in this direction JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166851#comment-14166851 ] Viktor Adam commented on SLING-3984: [~cziegeler] Yes, I was also thinking something like this. My thought was that maybe one thread starts compiling a taglib then the second one tries to load it from the repository before it would be written there by the first thread. I don't know if something like this is possible? JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166922#comment-14166922 ] Carsten Ziegeler commented on SLING-3984: - [~vadam_ig] I'm not 100% sure, but I think that's not possible as the code goes through the TagProcessor and that one is serializing everything with the new lock in place. It could also be that processing of the modification events is still ongoing while the next requests asked already for the new jsp. in that case, we might deal with a partial broken setup which leads to these problems. The events are not handled atomically in JspScriptEngineFactory#handleEvent (or the JspRuntimeContext). Maybe we have to come up with a global lock for the runtime context JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161733#comment-14161733 ] Viktor Adam commented on SLING-3984: Thank you, [~cziegeler], I've patched the 2.0.28 version which we use and with those two commits it looks like it fixes our problems. JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157770#comment-14157770 ] Felix Meschberger commented on SLING-3984: -- While I appreciate this fix, it actually brings us further away from stock Jasper and thus makes it harder and harder to ever upgrade. Any thoughts on this ? JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157797#comment-14157797 ] Carsten Ziegeler commented on SLING-3984: - We're already miles away from the original jasper as we added a lot of stuff over time. So one more thing on top does not make a difference JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157806#comment-14157806 ] Carsten Ziegeler commented on SLING-3984: - [~vadam_ig] I've appled the part of your patch for the tag file compilation in rev. 1629144. I think, in combination with the first commit this should fix the problem JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156473#comment-14156473 ] Carsten Ziegeler commented on SLING-3984: - If I understand it correctly, the problem with the tag files was a result of the initial problem having two wrappers for the same jsp being processed/compiled in parallel. So I think we don't have to do anything there. JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154696#comment-14154696 ] Bertrand Delacretaz commented on SLING-3984: Can you share a scenario to reproduce the problem, a curl-based example or something similar? JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154708#comment-14154708 ] Carsten Ziegeler commented on SLING-3984: - First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. While the operation is not atomic, there should only be one wrapper returned from getJspWrapper for a JSP file. The context.addWrapper method should ensure this. So according to your comments this does not seem to be the case. JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154715#comment-14154715 ] Carsten Ziegeler commented on SLING-3984: - I think I start to see the problem, would be great to have some to reproduce the issue easily. JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154739#comment-14154739 ] Viktor Adam commented on SLING-3984: Yes, the wrapper only existed once in the ConcurrentHashMap at the end, but the same JSP file could have different wrappers at the same time and be compiled on multiple threads. To reproduce, I've used an access.log from our server and replayed the requests to Sling when the JSP cache was empty. To do this, I have deleted the compiled classes and gave the bundle a restart (just to drop any in-memory cached classes) and then started hitting the server (in our case, doing this on 15 different threads was enough). I think it should be relatively easy to reproduce the issue if you have several JSPs including other JSPs and tag files then hitting them simultaneously (forcing them to start compiling at the same time but on different threads). JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
