please file a jira so we can track this.
-igor
On Wed, Oct 21, 2009 at 3:47 PM, adambender <adamben...@gmail.com> wrote:
>
> Sorry to keep replying to myself but this fix does appear to work... It kept
> my open file handles down to 5 or 6 for the duration of the load suite.
>
> public UrlResourceStream(final URL url)
> {
> // Save URL
> this.url = url;
> URLConnection connection = null;
> try
> {
> connection = url.openConnection();
> contentLength = connection.getContentLength();
> contentType = connection.getContentType();
> >>if (connection instanceof JarURLConnection)
> >>{
> >> JarURLConnection jarUrlConnection =
> (JarURLConnection)connection;
> >> URL jarFileUrl =
> jarUrlConnection.getJarFileURL();
> >> URLConnection jarFileConnection =
> jarFileUrl.openConnection();
> >> try
> >> {
> >> lastModified =
> jarFileConnection.getLastModified();
> >> }
> >> finally
> >> {
> >>
> jarFileConnection.getInputStream().close();
> >> }
> >>}
> >>else
> >>{
> >> lastModified = connection.getLastModified();
> >>}
> try
> {
> file = new File(new URI(url.toExternalForm()));
> }
> catch (Exception ex)
> {
> log.debug("cannot convert url: " + url + " to
> file (" + ex.getMessage()
> +
> "), falling back to the inputstream
> for polling");
> }
> if (file != null && !file.exists())
> {
> file = null;
> }
> }
> catch (IOException ex)
> {
> // It should be impossible to get here or the original
> URL
> // couldn't have been constructed. But we re-throw
> with details
> // anyway.
> final IllegalArgumentException
> illegalArgumentException = new
> IllegalArgumentException(
> "Invalid URL parameter " + url);
> illegalArgumentException.initCause(ex);
> throw illegalArgumentException;
> }
> finally
> {
> // if applicable, disconnect
> if (connection != null)
> {
> if (connection instanceof HttpURLConnection)
> {
>
> ((HttpURLConnection)connection).disconnect();
> }
> else
> {
> try
> {
>
> connection.getInputStream().close();
> }
> catch (Exception ex)
> {
> // ignore
> }
> }
> }
> }
> }
>
>
>
>
>
>
> adambender wrote:
>>
>> Is it possible to use this
>> http://www.mail-archive.com/wicket-u...@lists.sourceforge.net/msg20951.html
>> workaround in the URLResourceStream constructor - similar to how it is
>> done in URLResourceStream#lastModifiedTime()?
>>
>>
>>
>> adambender wrote:
>>>
>>> For what it is worth I saw a file being created in the URLResourceStream
>>> constructor at line 88 but it doesn't look like this is the source of any
>>> problems, its just a way to figure out how to check the lastModified time
>>> later on.
>>>
>>> While I can't claim any kind of expertise on the inner workings of Wicket
>>> it seems that this problem would be solved if it wasn't required to check
>>> the last modification time on those resources which are embedded in the
>>> Wicket jar. Of course I have no idea what else this would affect so it
>>> may not be so simple. It may also be possible to just cache the stream or
>>> the actual .js string for use by later requests so that it is not
>>> necessary to try and reload files like this from the jar every time they
>>> are requested.
>>>
>>> For our use what we are likely going to do is pull the .js files out and
>>> place them where httpd can serve them and then use some url rewriting
>>> magic to prevent Wicket from serving them.
>>>
>>>
>>> Adam
>>>
>>>
>>> igor.vaynberg wrote:
>>>>
>>>> i just dont see what we can do about this... we are calling
>>>> connection.getLastModified(), we are not creating a file, etc...
>>>>
>>>> -igor
>>>>
>>>> On Wed, Oct 21, 2009 at 10:18 AM, adambender <adamben...@gmail.com>
>>>> wrote:
>>>>>
>>>>> This issue looks like it is directly related to
>>>>> http://issues.apache.org/jira/browse/WICKET-438 and the access of the
>>>>> lastModified header. Every time a URLResourceStream is constructed the
>>>>> lastModified time is requested at line 85 and the number of file
>>>>> handles
>>>>> goes up by one. The solution to the jira issue indicated that upgrading
>>>>> the
>>>>> version of linux fixed the problem but it doesnt seem to be the case
>>>>> with OS
>>>>> X or Red Hat Enterprise Linux...
>>>>>
>>>>>
>>>>> adambender wrote:
>>>>>>
>>>>>> I was able to distill the problem down to what I think is the core
>>>>>> issue
>>>>>> here.
>>>>>>
>>>>>> When an AJAX Wicket page is rendered it contains a reference to two
>>>>>> .js
>>>>>> files as Igor had mentioned, the Web URLs of these files look like
>>>>>> this:
>>>>>>
>>>>>> <App Context
>>>>>> Root>/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-event.js
>>>>>> <App Context
>>>>>> Root>/resources/org.apache.wicket.markup.html.WicketEventReference/wicket-ajax.js
>>>>>>
>>>>>> These requests are of course fielded by the Wicket filter which ends
>>>>>> up
>>>>>> trying to load the resources using the UrlResourceStream. In order to
>>>>>> actually load these files Wicket gets the JAR URL for each file,
>>>>>> converts
>>>>>> them to a URIs and then passes them to the File constructor. The
>>>>>> problem
>>>>>> is that the URI that is created is opaque -
>>>>>> http://www.docjar.com/docs/api/java/net/URI.html see here for a good
>>>>>> explanation in the case of wicket-event.js the URI string is like the
>>>>>> following :
>>>>>>
>>>>>> jar:file:/<absolute-path-tojar>/wicket-1.4.1.jar!/org/apache/wicket/markup/html/wicket-event.js
>>>>>>
>>>>>> It is similar for the case of wicket-ajax.js
>>>>>>
>>>>>> A quick glance at the File constructor shows that an opaque URL cannot
>>>>>> be
>>>>>> used to create a File as it is considered non-hierarchical. This of
>>>>>> course
>>>>>> leaves you with lots of requests to load this file, none of which can
>>>>>> be
>>>>>> completed and results in a lot of open URL connections which can never
>>>>>> be
>>>>>> closed.
>>>>>>
>>>>>> I guess my question is, should this loading mechanism be working
>>>>>> because
>>>>>> it doesnt seem like it could the way it is currently written? Is it
>>>>>> possible to configure how Wicket finds these files embedded in the
>>>>>> jar?
>>>>>> Have I missed something?
>>>>>>
>>>>>>
>>>>>> Adam Bender-2 wrote:
>>>>>>>
>>>>>>> Thanks for the explanation I think that helps shed some light... The
>>>>>>> tests are actually JMeter tests driving load by emulating a web
>>>>>>> browser - the application the are testing is running in Deployment
>>>>>>> mode set up as though it were a production server. After a little
>>>>>>> more
>>>>>>> digging I have been able to determine that is only a problem on pages
>>>>>>> which use Ajax - and it looks like there is a related debug message
>>>>>>> coming from an exception handler regarding the wicket-event.js file
>>>>>>> not being accessible (URI is not hierarchical). This would actually
>>>>>>> explain why the problem only manifests when the JMeter tests are set
>>>>>>> to request embedded assets as wicket ajax pages embed requests for
>>>>>>> additional pieces of javascript - which in the case of
>>>>>>> wicket-event.js
>>>>>>> are causing exceptions to be thrown and the JVM bug you mentioned is
>>>>>>> probably preventing these resources from being cleaned up properly.
>>>>>>>
>>>>>>> Does that sound right?
>>>>>>>
>>>>>>> Adam
>>>>>>>
>>>>>>>
>>>>>>> On Oct 20, 2009, at 8:41 PM, Igor Vaynberg wrote:
>>>>>>>
>>>>>>>> when you are requesting an embedded resource wicket needs to stream
>>>>>>>> the file out of a jar, the way the jvm handles that is by creating a
>>>>>>>> jar url connection object that streams the file. unfortunately,
>>>>>>>> there
>>>>>>>> is a bug in the jdk where the url connection does not have a close
>>>>>>>> method and so wicket or any other java app cannot release the
>>>>>>>> connection. this is addressed, afair, in jdk 7.
>>>>>>>>
>>>>>>>> i have many apps deployed in production and have not managed yet to
>>>>>>>> run out of the handles with the limit set about 4K. not sure why
>>>>>>>> this
>>>>>>>> is different in your case. you mentioned "tests", are those unit
>>>>>>>> tests
>>>>>>>> and is wicket there running in deployment mode?
>>>>>>>>
>>>>>>>> the resource watcher, which should be disabled, will cause the
>>>>>>>> handles
>>>>>>>> to run out because it continuously monitors markup files for changes
>>>>>>>> (hot reloading in dev mode) and everytime it checks a markup file in
>>>>>>>> a
>>>>>>>> jar it creates the url connection and leaks it. by default it is
>>>>>>>> disabled in deployment mode but if you manually set the poll
>>>>>>>> frequency
>>>>>>>> in your settings it will be reenabled.
>>>>>>>>
>>>>>>>> -igor
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 20, 2009 at 4:07 PM, Adam Bender <a...@magpieti.com>
>>>>>>>> wrote:
>>>>>>>>> We have run with a limit as high as 10,000 files and our tests can
>>>>>>>>> bring it
>>>>>>>>> to the limit in 20 minutes, but that still doesn't explain why so
>>>>>>>>> many
>>>>>>>>> copies of the jar are needed - and only when we are also requesting
>>>>>>>>> embedded
>>>>>>>>> assets...
>>>>>>>>>
>>>>>>>>> On Oct 20, 2009, at 5:04 PM, Major Péter wrote:
>>>>>>>>>
>>>>>>>>>> resolution -> solution...
>>>>>>>>>> Just set the ulimit in the initscript and run it with start-stop-
>>>>>>>>>> daemon,
>>>>>>>>>> it will make the problem disappear (but still there would be many
>>>>>>>>>> open
>>>>>>>>>> files...)
>>>>>>>>>>
>>>>>>>>>> Peter
>>>>>>>>>>
>>>>>>>>>> 2009-10-21 00:38 keltezéssel, Major Péter írta:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> we also had this issue, because (I think) we are running two
>>>>>>>>>>> different
>>>>>>>>>>> wicket application in only one glassfish domain. Our resolution
>>>>>>>>>>> was,
>>>>>>>>>>> that we are running now the glassfish domains with custom init
>>>>>>>>>>> scripts
>>>>>>>>>>> with ulimit settings. Maybe this will help for you.
>>>>>>>>>>>
>>>>>>>>>>> Best Regards,
>>>>>>>>>>> Peter
>>>>>>>>>>>
>>>>>>>>>>> 2009-10-21 00:14 keltezéssel, Martin Grigorov írta:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Adam,
>>>>>>>>>>>>
>>>>>>>>>>>> You may try to debug what is the problem with
>>>>>>>>>>>> https://wiki.sdn.sap.com/wiki/display/Java/JPicus
>>>>>>>>>>>>
>>>>>>>>>>>> El mar, 20-10-2009 a las 15:39 -0600, Adam Bender escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Greetings all,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Recently I have been performance testing a Wicket application
>>>>>>>>>>>>> (1.4.1)
>>>>>>>>>>>>> and I
>>>>>>>>>>>>> am running into the dreaded Too Many Files Open issue. I have
>>>>>>>>>>>>> searched
>>>>>>>>>>>>> the
>>>>>>>>>>>>> mailing lists and most of the trouble around this issue seems
>>>>>>>>>>>>> to come
>>>>>>>>>>>>> from
>>>>>>>>>>>>> being in DEVELOPMENT mode so I made sure we were in DEPLOYMENT
>>>>>>>>>>>>> mode.
>>>>>>>>>>>>> When I
>>>>>>>>>>>>> run 'lsof -p xxxx | grep wicket-1.4.1.jar' I see over 1000
>>>>>>>>>>>>> entries and
>>>>>>>>>>>>> it's
>>>>>>>>>>>>> growing monotonically. This seems really unusual - why would
>>>>>>>>>>>>> wicket
>>>>>>>>>>>>> need
>>>>>>>>>>>>> 1000 copies of this jar open? An additional bit of weirdness
>>>>>>>>>>>>> came up
>>>>>>>>>>>>> when
>>>>>>>>>>>>> our load tests stopped loading the embedded assets (css, js and
>>>>>>>>>>>>> images)
>>>>>>>>>>>>> in
>>>>>>>>>>>>> each page - this seemed to cap the number of lsof entries at
>>>>>>>>>>>>> 5... This
>>>>>>>>>>>>> is
>>>>>>>>>>>>> even weirder because our app serves these static items from
>>>>>>>>>>>>> httpd
>>>>>>>>>>>>> without
>>>>>>>>>>>>> tomcat ever knowing about them. How could our loading of
>>>>>>>>>>>>> embedded items
>>>>>>>>>>>>> really affect the number of file handles wicket needs?
>>>>>>>>>>>>>
>>>>>>>>>>>>> For completeness we are running this app on Red Hat Enterprise
>>>>>>>>>>>>> Linx 5
>>>>>>>>>>>>> with
>>>>>>>>>>>>> Tomcat 6.0.20 and Java 1.6
>>>>>>>>>>>>>
>>>>>>>>>>>>> Adam
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://www.nabble.com/Too-Many-Files-Wicket-1.4.1-tp25983047p25996646.html
>>>>> Sent from the Wicket - User mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
>>>> For additional commands, e-mail: users-h...@wicket.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Too-Many-Files-Wicket-1.4.1-tp25983047p26001456.html
> Sent from the Wicket - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
> For additional commands, e-mail: users-h...@wicket.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
