Re: Too Many Files Wicket 1.4.1
I was able to distill the problem down to what I think is the core issue here. When an AJAX Wicket page is rendered it contains a reference to two .js files as Igor had mentioned, the Web URLs of these files look like this: App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-ajax.js These requests are of course fielded by the Wicket filter which ends up trying to load the resources using the UrlResourceStream. In order to actually load these files Wicket gets the JAR URL for each file, converts them to a URIs and then passes them to the File constructor. The problem is that the URI that is created is opaque - http://www.docjar.com/docs/api/java/net/URI.html see here for a good explanation in the case of wicket-event.js the URI string is like the following : jar:file:/absolute-path-tojar/wicket-1.4.1.jar!/org/apache/wicket/markup/html/wicket-event.js It is similar for the case of wicket-ajax.js A quick glance at the File constructor shows that an opaque URL cannot be used to create a File as it is considered non-hierarchical. This of course leaves you with lots of requests to load this file, none of which can be completed and results in a lot of open URL connections which can never be closed. I guess my question is, should this loading mechanism be working because it doesnt seem like it could the way it is currently written? Is it possible to configure how Wicket finds these files embedded in the jar? Have I missed something? Adam Bender-2 wrote: Thanks for the explanation I think that helps shed some light... The tests are actually JMeter tests driving load by emulating a web browser - the application the are testing is running in Deployment mode set up as though it were a production server. After a little more digging I have been able to determine that is only a problem on pages which use Ajax - and it looks like there is a related debug message coming from an exception handler regarding the wicket-event.js file not being accessible (URI is not hierarchical). This would actually explain why the problem only manifests when the JMeter tests are set to request embedded assets as wicket ajax pages embed requests for additional pieces of javascript - which in the case of wicket-event.js are causing exceptions to be thrown and the JVM bug you mentioned is probably preventing these resources from being cleaned up properly. Does that sound right? Adam On Oct 20, 2009, at 8:41 PM, Igor Vaynberg wrote: when you are requesting an embedded resource wicket needs to stream the file out of a jar, the way the jvm handles that is by creating a jar url connection object that streams the file. unfortunately, there is a bug in the jdk where the url connection does not have a close method and so wicket or any other java app cannot release the connection. this is addressed, afair, in jdk 7. i have many apps deployed in production and have not managed yet to run out of the handles with the limit set about 4K. not sure why this is different in your case. you mentioned tests, are those unit tests and is wicket there running in deployment mode? the resource watcher, which should be disabled, will cause the handles to run out because it continuously monitors markup files for changes (hot reloading in dev mode) and everytime it checks a markup file in a jar it creates the url connection and leaks it. by default it is disabled in deployment mode but if you manually set the poll frequency in your settings it will be reenabled. -igor On Tue, Oct 20, 2009 at 4:07 PM, Adam Bender a...@magpieti.com wrote: We have run with a limit as high as 10,000 files and our tests can bring it to the limit in 20 minutes, but that still doesn't explain why so many copies of the jar are needed - and only when we are also requesting embedded assets... On Oct 20, 2009, at 5:04 PM, Major Péter wrote: resolution - solution... Just set the ulimit in the initscript and run it with start-stop- daemon, it will make the problem disappear (but still there would be many open files...) Peter 2009-10-21 00:38 keltezéssel, Major Péter írta: Hi, we also had this issue, because (I think) we are running two different wicket application in only one glassfish domain. Our resolution was, that we are running now the glassfish domains with custom init scripts with ulimit settings. Maybe this will help for you. Best Regards, Peter 2009-10-21 00:14 keltezéssel, Martin Grigorov írta: Hi Adam, You may try to debug what is the problem with https://wiki.sdn.sap.com/wiki/display/Java/JPicus El mar, 20-10-2009 a las 15:39 -0600, Adam Bender escribió: Greetings all, Recently I have been performance testing a Wicket application (1.4.1) and I am running into the dreaded Too Many Files Open issue. I have
Re: Too Many Files Wicket 1.4.1
This issue looks like it is directly related to http://issues.apache.org/jira/browse/WICKET-438 and the access of the lastModified header. Every time a URLResourceStream is constructed the lastModified time is requested at line 85 and the number of file handles goes up by one. The solution to the jira issue indicated that upgrading the version of linux fixed the problem but it doesnt seem to be the case with OS X or Red Hat Enterprise Linux... adambender wrote: I was able to distill the problem down to what I think is the core issue here. When an AJAX Wicket page is rendered it contains a reference to two .js files as Igor had mentioned, the Web URLs of these files look like this: App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-ajax.js These requests are of course fielded by the Wicket filter which ends up trying to load the resources using the UrlResourceStream. In order to actually load these files Wicket gets the JAR URL for each file, converts them to a URIs and then passes them to the File constructor. The problem is that the URI that is created is opaque - http://www.docjar.com/docs/api/java/net/URI.html see here for a good explanation in the case of wicket-event.js the URI string is like the following : jar:file:/absolute-path-tojar/wicket-1.4.1.jar!/org/apache/wicket/markup/html/wicket-event.js It is similar for the case of wicket-ajax.js A quick glance at the File constructor shows that an opaque URL cannot be used to create a File as it is considered non-hierarchical. This of course leaves you with lots of requests to load this file, none of which can be completed and results in a lot of open URL connections which can never be closed. I guess my question is, should this loading mechanism be working because it doesnt seem like it could the way it is currently written? Is it possible to configure how Wicket finds these files embedded in the jar? Have I missed something? Adam Bender-2 wrote: Thanks for the explanation I think that helps shed some light... The tests are actually JMeter tests driving load by emulating a web browser - the application the are testing is running in Deployment mode set up as though it were a production server. After a little more digging I have been able to determine that is only a problem on pages which use Ajax - and it looks like there is a related debug message coming from an exception handler regarding the wicket-event.js file not being accessible (URI is not hierarchical). This would actually explain why the problem only manifests when the JMeter tests are set to request embedded assets as wicket ajax pages embed requests for additional pieces of javascript - which in the case of wicket-event.js are causing exceptions to be thrown and the JVM bug you mentioned is probably preventing these resources from being cleaned up properly. Does that sound right? Adam On Oct 20, 2009, at 8:41 PM, Igor Vaynberg wrote: when you are requesting an embedded resource wicket needs to stream the file out of a jar, the way the jvm handles that is by creating a jar url connection object that streams the file. unfortunately, there is a bug in the jdk where the url connection does not have a close method and so wicket or any other java app cannot release the connection. this is addressed, afair, in jdk 7. i have many apps deployed in production and have not managed yet to run out of the handles with the limit set about 4K. not sure why this is different in your case. you mentioned tests, are those unit tests and is wicket there running in deployment mode? the resource watcher, which should be disabled, will cause the handles to run out because it continuously monitors markup files for changes (hot reloading in dev mode) and everytime it checks a markup file in a jar it creates the url connection and leaks it. by default it is disabled in deployment mode but if you manually set the poll frequency in your settings it will be reenabled. -igor On Tue, Oct 20, 2009 at 4:07 PM, Adam Bender a...@magpieti.com wrote: We have run with a limit as high as 10,000 files and our tests can bring it to the limit in 20 minutes, but that still doesn't explain why so many copies of the jar are needed - and only when we are also requesting embedded assets... On Oct 20, 2009, at 5:04 PM, Major Péter wrote: resolution - solution... Just set the ulimit in the initscript and run it with start-stop- daemon, it will make the problem disappear (but still there would be many open files...) Peter 2009-10-21 00:38 keltezéssel, Major Péter írta: Hi, we also had this issue, because (I think) we are running two different wicket application in only one glassfish domain. Our resolution was, that we are running now
Re: Too Many Files Wicket 1.4.1
For what it is worth I saw a file being created in the URLResourceStream constructor at line 88 but it doesn't look like this is the source of any problems, its just a way to figure out how to check the lastModified time later on. While I can't claim any kind of expertise on the inner workings of Wicket it seems that this problem would be solved if it wasn't required to check the last modification time on those resources which are embedded in the Wicket jar. Of course I have no idea what else this would affect so it may not be so simple. It may also be possible to just cache the stream or the actual .js string for use by later requests so that it is not necessary to try and reload files like this from the jar every time they are requested. For our use what we are likely going to do is pull the .js files out and place them where httpd can serve them and then use some url rewriting magic to prevent Wicket from serving them. Adam igor.vaynberg wrote: i just dont see what we can do about this... we are calling connection.getLastModified(), we are not creating a file, etc... -igor On Wed, Oct 21, 2009 at 10:18 AM, adambender adamben...@gmail.com wrote: This issue looks like it is directly related to http://issues.apache.org/jira/browse/WICKET-438 and the access of the lastModified header. Every time a URLResourceStream is constructed the lastModified time is requested at line 85 and the number of file handles goes up by one. The solution to the jira issue indicated that upgrading the version of linux fixed the problem but it doesnt seem to be the case with OS X or Red Hat Enterprise Linux... adambender wrote: I was able to distill the problem down to what I think is the core issue here. When an AJAX Wicket page is rendered it contains a reference to two .js files as Igor had mentioned, the Web URLs of these files look like this: App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-ajax.js These requests are of course fielded by the Wicket filter which ends up trying to load the resources using the UrlResourceStream. In order to actually load these files Wicket gets the JAR URL for each file, converts them to a URIs and then passes them to the File constructor. The problem is that the URI that is created is opaque - http://www.docjar.com/docs/api/java/net/URI.html see here for a good explanation in the case of wicket-event.js the URI string is like the following : jar:file:/absolute-path-tojar/wicket-1.4.1.jar!/org/apache/wicket/markup/html/wicket-event.js It is similar for the case of wicket-ajax.js A quick glance at the File constructor shows that an opaque URL cannot be used to create a File as it is considered non-hierarchical. This of course leaves you with lots of requests to load this file, none of which can be completed and results in a lot of open URL connections which can never be closed. I guess my question is, should this loading mechanism be working because it doesnt seem like it could the way it is currently written? Is it possible to configure how Wicket finds these files embedded in the jar? Have I missed something? Adam Bender-2 wrote: Thanks for the explanation I think that helps shed some light... The tests are actually JMeter tests driving load by emulating a web browser - the application the are testing is running in Deployment mode set up as though it were a production server. After a little more digging I have been able to determine that is only a problem on pages which use Ajax - and it looks like there is a related debug message coming from an exception handler regarding the wicket-event.js file not being accessible (URI is not hierarchical). This would actually explain why the problem only manifests when the JMeter tests are set to request embedded assets as wicket ajax pages embed requests for additional pieces of javascript - which in the case of wicket-event.js are causing exceptions to be thrown and the JVM bug you mentioned is probably preventing these resources from being cleaned up properly. Does that sound right? Adam On Oct 20, 2009, at 8:41 PM, Igor Vaynberg wrote: when you are requesting an embedded resource wicket needs to stream the file out of a jar, the way the jvm handles that is by creating a jar url connection object that streams the file. unfortunately, there is a bug in the jdk where the url connection does not have a close method and so wicket or any other java app cannot release the connection. this is addressed, afair, in jdk 7. i have many apps deployed in production and have not managed yet to run out of the handles with the limit set about 4K. not sure why this is different in your case. you mentioned tests, are those unit tests and is wicket there running in deployment mode? the resource watcher, which should be disabled
Re: Too Many Files Wicket 1.4.1
Is it possible to use this http://www.mail-archive.com/wicket-u...@lists.sourceforge.net/msg20951.html workaround in the URLResourceStream constructor - similar to how it is done in URLResourceStream#lastModifiedTime()? adambender wrote: For what it is worth I saw a file being created in the URLResourceStream constructor at line 88 but it doesn't look like this is the source of any problems, its just a way to figure out how to check the lastModified time later on. While I can't claim any kind of expertise on the inner workings of Wicket it seems that this problem would be solved if it wasn't required to check the last modification time on those resources which are embedded in the Wicket jar. Of course I have no idea what else this would affect so it may not be so simple. It may also be possible to just cache the stream or the actual .js string for use by later requests so that it is not necessary to try and reload files like this from the jar every time they are requested. For our use what we are likely going to do is pull the .js files out and place them where httpd can serve them and then use some url rewriting magic to prevent Wicket from serving them. Adam igor.vaynberg wrote: i just dont see what we can do about this... we are calling connection.getLastModified(), we are not creating a file, etc... -igor On Wed, Oct 21, 2009 at 10:18 AM, adambender adamben...@gmail.com wrote: This issue looks like it is directly related to http://issues.apache.org/jira/browse/WICKET-438 and the access of the lastModified header. Every time a URLResourceStream is constructed the lastModified time is requested at line 85 and the number of file handles goes up by one. The solution to the jira issue indicated that upgrading the version of linux fixed the problem but it doesnt seem to be the case with OS X or Red Hat Enterprise Linux... adambender wrote: I was able to distill the problem down to what I think is the core issue here. When an AJAX Wicket page is rendered it contains a reference to two .js files as Igor had mentioned, the Web URLs of these files look like this: App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js App Context Root/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-ajax.js These requests are of course fielded by the Wicket filter which ends up trying to load the resources using the UrlResourceStream. In order to actually load these files Wicket gets the JAR URL for each file, converts them to a URIs and then passes them to the File constructor. The problem is that the URI that is created is opaque - http://www.docjar.com/docs/api/java/net/URI.html see here for a good explanation in the case of wicket-event.js the URI string is like the following : jar:file:/absolute-path-tojar/wicket-1.4.1.jar!/org/apache/wicket/markup/html/wicket-event.js It is similar for the case of wicket-ajax.js A quick glance at the File constructor shows that an opaque URL cannot be used to create a File as it is considered non-hierarchical. This of course leaves you with lots of requests to load this file, none of which can be completed and results in a lot of open URL connections which can never be closed. I guess my question is, should this loading mechanism be working because it doesnt seem like it could the way it is currently written? Is it possible to configure how Wicket finds these files embedded in the jar? Have I missed something? Adam Bender-2 wrote: Thanks for the explanation I think that helps shed some light... The tests are actually JMeter tests driving load by emulating a web browser - the application the are testing is running in Deployment mode set up as though it were a production server. After a little more digging I have been able to determine that is only a problem on pages which use Ajax - and it looks like there is a related debug message coming from an exception handler regarding the wicket-event.js file not being accessible (URI is not hierarchical). This would actually explain why the problem only manifests when the JMeter tests are set to request embedded assets as wicket ajax pages embed requests for additional pieces of javascript - which in the case of wicket-event.js are causing exceptions to be thrown and the JVM bug you mentioned is probably preventing these resources from being cleaned up properly. Does that sound right? Adam On Oct 20, 2009, at 8:41 PM, Igor Vaynberg wrote: when you are requesting an embedded resource wicket needs to stream the file out of a jar, the way the jvm handles that is by creating a jar url connection object that streams the file. unfortunately, there is a bug in the jdk where the url connection does not have a close method and so wicket or any other java app cannot release the connection. this is addressed, afair, in jdk 7. i have many apps deployed
Re: Too Many Files Wicket 1.4.1
Sorry to keep replying to myself but this fix does appear to work... It kept
my open file handles down to 5 or 6 for the duration of the load suite.
public UrlResourceStream(final URL url)
{
// Save URL
this.url = url;
URLConnection connection = null;
try
{
connection = url.openConnection();
contentLength = connection.getContentLength();
contentType = connection.getContentType();
if (connection instanceof JarURLConnection)
{
JarURLConnection jarUrlConnection =
(JarURLConnection)connection;
URL jarFileUrl =
jarUrlConnection.getJarFileURL();
URLConnection jarFileConnection =
jarFileUrl.openConnection();
try
{
lastModified =
jarFileConnection.getLastModified();
}
finally
{
jarFileConnection.getInputStream().close();
}
}
else
{
lastModified = connection.getLastModified();
}
try
{
file = new File(new URI(url.toExternalForm()));
}
catch (Exception ex)
{
log.debug(cannot convert url: + url + to
file ( + ex.getMessage()
+
), falling back to the inputstream for
polling);
}
if (file != null !file.exists())
{
file = null;
}
}
catch (IOException ex)
{
// It should be impossible to get here or the original
URL
// couldn't have been constructed. But we re-throw with
details
// anyway.
final IllegalArgumentException illegalArgumentException
= new
IllegalArgumentException(
Invalid URL parameter + url);
illegalArgumentException.initCause(ex);
throw illegalArgumentException;
}
finally
{
// if applicable, disconnect
if (connection != null)
{
if (connection instanceof HttpURLConnection)
{
((HttpURLConnection)connection).disconnect();
}
else
{
try
{
connection.getInputStream().close();
}
catch (Exception ex)
{
// ignore
}
}
}
}
}
adambender wrote:
Is it possible to use this
http://www.mail-archive.com/wicket-u...@lists.sourceforge.net/msg20951.html
workaround in the URLResourceStream constructor - similar to how it is
done in URLResourceStream#lastModifiedTime()?
adambender wrote:
For what it is worth I saw a file being created in the URLResourceStream
constructor at line 88 but it doesn't look like this is the source of any
problems, its just a way to figure out how to check the lastModified time
later on.
While I can't claim any kind of expertise on the inner workings of Wicket
it seems that this problem would be solved if it wasn't required to check
the last modification time on those resources which are embedded in the
Wicket jar. Of course I have no idea what else this would affect so it
may not be so simple. It may also be possible to just cache the stream or
the actual .js string for use by later requests so that it is not
necessary to try and reload files like this from the jar every time they
are requested.
For our use what we are likely going to do is pull the .js files out and
place them where httpd can serve them and then use some url rewriting
magic to prevent Wicket from serving them.
Adam
igor.vaynberg wrote:
i just dont see what we can do about this... we are calling
