[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194398#comment-14194398 ] Viktor Adam commented on SLING-3984: [~cziegeler] We've had a few deployment since my last update and we haven't had this problem so I think you can close this ticket. Thank you for your help! JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166851#comment-14166851 ] Viktor Adam commented on SLING-3984: [~cziegeler] Yes, I was also thinking something like this. My thought was that maybe one thread starts compiling a taglib then the second one tries to load it from the repository before it would be written there by the first thread. I don't know if something like this is possible? JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Adam reopened SLING-3984: The bundle works much better now. However we've encountered some Exceptions again (see the attached zip file). So it looks like the compilation can still fail on some threads but eventually it can execute the code successfully (there were only a few exceptions in a short time then it seems to be fixed). (I have changed some class and file names in the stackstraces) JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14165357#comment-14165357 ] Viktor Adam edited comment on SLING-3984 at 10/9/14 4:51 PM: - The bundle works much better now. However we've encountered some Exceptions again (see the attached zip file). So it looks like the compilation can still fail on some threads but eventually it can execute the code successfully (there were only a few exceptions in a short time then it seems to be fixed). (I have changed some class and file names in the stackstraces) I'll try to find some time to experiment with this a bit more. was (Author: vadam_ig): The bundle works much better now. However we've encountered some Exceptions again (see the attached zip file). So it looks like the compilation can still fail on some threads but eventually it can execute the code successfully (there were only a few exceptions in a short time then it seems to be fixed). (I have changed some class and file names in the stackstraces) JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Adam updated SLING-3984: --- Attachment: jsp_exception.stacktraces.zip JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch, jsp_exception.stacktraces.zip We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161733#comment-14161733 ] Viktor Adam commented on SLING-3984: Thank you, [~cziegeler], I've patched the 2.0.28 version which we use and with those two commits it looks like it fixes our problems. JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Assignee: Carsten Ziegeler Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SLING-3984) JSP Compilation failure under heavy load
Viktor Adam created SLING-3984: -- Summary: JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Adam updated SLING-3984: --- Attachment: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch Patch to synchronize access for JSP wrappers JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SLING-3984) JSP Compilation failure under heavy load
[ https://issues.apache.org/jira/browse/SLING-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154739#comment-14154739 ] Viktor Adam commented on SLING-3984: Yes, the wrapper only existed once in the ConcurrentHashMap at the end, but the same JSP file could have different wrappers at the same time and be compiled on multiple threads. To reproduce, I've used an access.log from our server and replayed the requests to Sling when the JSP cache was empty. To do this, I have deleted the compiled classes and gave the bundle a restart (just to drop any in-memory cached classes) and then started hitting the server (in our case, doing this on 15 different threads was enough). I think it should be relatively easy to reproduce the issue if you have several JSPs including other JSPs and tag files then hitting them simultaneously (forcing them to start compiling at the same time but on different threads). JSP Compilation failure under heavy load Key: SLING-3984 URL: https://issues.apache.org/jira/browse/SLING-3984 Project: Sling Issue Type: Improvement Components: Scripting Affects Versions: Scripting JSP 2.0.28 Reporter: Viktor Adam Fix For: Scripting JSP 2.1.6 Attachments: Sling_scripting__workaround_for_concurrency_issues_when_compiling.patch We have seen problems with JSP compilation under heavy load. It looks like when multiple threads try to compile the same JSP or tag file the result can be NoClassDefFoundError, ClassNotFoundException or simply compilation failures (saying method return types not match the expected type, etc.) While I was investigating the issue I have found the following things: First: multiple wrapper instances can exist at the same time for a given JSP file because the caching of them (creating/storing) is not atomic. I have verified that multiple threads can execute the compilation inside the prepareServlet method at the same time. This was my initial prime suspect. Synchronizing access for this method didn't solve all the problems so I have looked into the taglib related parts as taglib related exceptions/errors showed up frequently in stack traces. Finally I've found out that the problem (for us) is around the loadTagFile method of the JspServletWrapper class. This method is invoked from the TagFileProcessor.loadTagFile method which - I believe - again, can have different JspServletWrapper instances for the same taglib at the same time and compiling them concurrently can lead to problems (so it seems). By adding a per-file locking around the body of this method I am no longer able to reproduce the compilation errors, JSP files are compiled correctly. I will attach a patch with my changes to the 2.0.28 version (I haven't found any noticeable changes around this area since) which solved the problems, for us at least. I think I may lock for a larger scope than it is absolutely necessary so maybe someone can advise me on this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
