It's hard for me to say, what the right way to fix this situation is. The number of open files (that look to the OS as wicket .jar files) will go up significantly and then go down (down - probably due to the garbage collection). It is during those spikes caused by many users hitting wicket pages around the same time that we got "too many open files" error. It almost looks like the only way to really eliminate those unnecessary open files to numerous Wicket resources (and to guarantee "too many open files" will never hit us) is to unpack those resources from the wicket .jar-s and put them into WEB-INF/classes directory instead. Once we did it, we didn't see all those dozens (and even hundreds) open files ever again...
Bob. Eelco Hillenius wrote: > > Too bad, > > Eelco > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: >> why would you cache? and which one? the url connection to an entry in a >> jar >> file (thats the JarUrlConnection) >> or (i guess) the FileUrlConnection (to the jar file itself) >> >> both don't make much sense to cache >> the first one we don't need to cache we only need to use it once by >> really >> loading the resource >> and i guess when it is finalized it is cleaned up. >> We already don't use it anymore for the last modified. Because there we >> use >> only the second one >> So the fileUrlConnection to the jarFile itself thats is inside the >> JarUrlConnection object. >> on that one we call last modified everytime, But that will not cause the >> file to open. (because it doesn't have to read the file itself) >> >> And we can't construct JarUrlConnections (for reading the jar entries) >> with >> the same file url connection because there is >> no way to initialize the jar url connection directly with the file url >> connection so they all would use the same. >> >> johan >> >> >> >> On 2/2/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > >> > Would it be possible and useful to cache the URL connection? Does it >> > have a time out and/ or does it use an exclusive lock? >> > >> > Eelco >> > >> > On 2/1/07, Johan Compagner <[EMAIL PROTECTED] > wrote: >> > > that is what you would think... But why generates a modification >> check >> one >> > > file handle for every check in the file? >> > > >> > > because UrlConnection.connect() has again a JarUrlConnection >> internally >> that >> > > makes a new connection to that jar file >> > > and UrlConnection does have a connect() but not a disconnect() so you >> can't >> > > clear it. >> > > >> > > johan >> > > >> > > >> > > >> > > On 2/2/07, Eelco Hillenius < [EMAIL PROTECTED]> wrote: >> > > > Yeah, but that would be always one fd for a jar, no matter how many >> > > > files in it that have to be read, right? >> > > > >> > > > Eelco >> > > > >> > > > >> > > > On 2/1/07, Johan Compagner <[EMAIL PROTECTED]> wrote: >> > > > > yes the modification checker. >> > > > > But we do need to really load the resource out of the jar file >> once. >> So >> > > that >> > > > > file handle will be used. >> > > > > >> > > > > johan >> > > > > >> > > > > >> > > > > >> > > > > On 2/1/07, Eelco Hillenius <[EMAIL PROTECTED]> wrote: >> > > > > > >> > > > > > Wasn't it optimized now so that it only hits jars once and only >> hits >> > > > > > per resource where normal files are involved? >> > > > > > >> > > > > > Eelco >> > > > > > >> > > > > > On 2/1/07, Johan Compagner < [EMAIL PROTECTED] > wrote: >> > > > > > > one per resource will i think not really change. >> > > > > > > >> > > > > > > >> > > > > > > johan >> > > > > > > >> > > > > > > >> > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED] > wrote: >> > > > > > > > >> > > > > > > > We will, when we are on production. Now that we are are >> still >> in >> > > alpha >> > > > > we >> > > > > > > > prefer "development" (hey, we wrote our first line of >> wicket >> code >> > > 5-6 >> > > > > > > weeks >> > > > > > > > ago!) >> > > > > > > > >> > > > > > > > Also, even in deployment mode 'lsof' still shows us a lot >> of >> open >> > > > > files >> > > > > > > (one >> > > > > > > > per resource) if we don't unpack resources from the .jar . >> It >> may >> > > be >> > > > > > > smaller >> > > > > > > > number than in development mode, but still... I imagine >> your >> > > > > "workaround" >> > > > > > > > would close those unnecessarily open files. >> > > > > > > > >> > > > > > > > If you can't backport it, please, tell me where it is in >> SVN. >> > > We'll >> > > > > "hack" >> > > > > > > > it in oursleves for now... >> > > > > > > > >> > > > > > > > Bob >> > > > > > > > >> > > > > > > > >> > > > > > > > Johan Compagner wrote: >> > > > > > > > > >> > > > > > > > > can't you run in deployment mode instead of development >> mode? >> > > > > > > > > then that shouldn't happen. >> > > > > > > > > >> > > > > > > > > I can see if we can backport it. >> > > > > > > > > >> > > > > > > > > johan >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > On 2/1/07, beboris < [EMAIL PROTECTED]> wrote: >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> We have found a pretty weird situation with "too many >> open >> > > files" >> > > > > error >> > > > > > > > >> on >> > > > > > > > >> our alpha-testing site. Further analysis showed that on >> each >> > > page >> > > > > > > refresh >> > > > > > > > >> the following resources get repeatedly obtained from the >> > > wicket's >> > > > > .jar >> > > > > > > > >> and >> > > > > > > > >> add to the number of open files: >> > > > > > > > >> 'wicket/ajax/wicket-ajax.js' >> > > > > > > > >> >> > > > > > > >> > > > > >> > > >> 'wicket/extensions/ajax/markup/html/modal/res/modal.css' >> > > > > > > > >> >> > > > > > > >> > > 'wicket/extensions/ajax/markup/html/modal/res/modal.js' >> > > > > > > > >> >> > > > > 'wicket/extensions/markup/html/tree/res/tree.css' >> > > > > > > > >> >> > > > > 'wicket/extensions/markup/html/tree/res/tree.js' >> > > > > > > > >> >> > > > > > > > >> Eventually (due to garbage collection?) the number of >> open >> > > files >> > > > > goes >> > > > > > > > >> down >> > > > > > > > >> again. But we wanted to know why those files stayed open >> in >> the >> > > > > first >> > > > > > > > >> place >> > > > > > > > >> and were not closed upon retrieving a resource. >> > > > > > > > >> >> > > > > > > > >> It looks like switching to the "deployment" mode from >> > > "development" >> > > > > one >> > > > > > > > >> would significantly reduce the peak numbers of the open >> > > > > files/streams >> > > > > > > to >> > > > > > > > >> wicket .jar-s, and extracting resources from the .jar >> would >> > > reduce >> > > > > it >> > > > > > > > >> even >> > > > > > > > >> better. We were told that the original reason for files >> staying >> > > > > open is >> > > > > > > a >> > > > > > > > >> Java bug (the fact that URLConnection doesnt have a >> .close), >> > > which >> > > > > > > causes >> > > > > > > > >> those nasty results when combined with development mode >> trying >> > > to >> > > > > > > monitor >> > > > > > > > >> [resource] files for changes and reloading them. >> > > > > > > > >> >> > > > > > > > >> We were also told that there is a workaround for that >> problem >> > > in >> > > > > SVN >> > > > > > > > >> somewhere, but it's probably not backported to 1.2.5 . I >> have 2 >> > > > > > > questions >> > > > > > > > >> in >> > > > > > > > >> that regard: >> > > > > > > > >> (1) Where can we find those workarounds in the code? >> > > > > > > > >> (2) Is it too much work to backport them to 1.2.5 so >> when >> it's >> > > > > released >> > > > > > > > >> it >> > > > > > > > >> doesn't contain the problem? >> > > > > > > > >> >> > > > > > > > >> Thanks, >> > > > > > > > >> Bob. >> > > > > > > > >> -- >> > > > > > > > >> View this message in context: >> > > > > > > > >> >> > > > > > > >> > > > > >> > > >> http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8743682 >> > > > > > > > >> Sent from the Wicket - User mailing list archive at >> Nabble.com >> > > . >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> > > > > >> > > >> ------------------------------------------------------------------------- >> > > > > > > > >> Using Tomcat but need to do more? Need to support web >> services, >> > > > > > > security? >> > > > > > > > >> Get stuff done quickly with pre-integrated technology to >> make >> > > your >> > > > > job >> > > > > > > > >> easier. >> > > > > > > > >> Download IBM WebSphere Application Server v.1.0.1 based >> on >> > > Apache >> > > > > > > > >> Geronimo >> > > > > > > > >> >> > > > > > > >> > > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > > > > > >> >> > > _______________________________________________ >> > > > > > > > >> Wicket-user mailing list >> > > > > > > > >> Wicket-user@lists.sourceforge.net >> > > > > > > > >> >> > > > > > > >> > > > > >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> ------------------------------------------------------------------------- >> > > > > > > > > Using Tomcat but need to do more? Need to support web >> services, >> > > > > > > security? >> > > > > > > > > Get stuff done quickly with pre-integrated technology to >> make >> > > your >> > > > > job >> > > > > > > > > easier. >> > > > > > > > > Download IBM WebSphere Application Server v.1.0.1 based >> on >> > > Apache >> > > > > > > Geronimo >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > > > > > > >> _______________________________________________ >> > > > > > > > > Wicket-user mailing list >> > > > > > > > > Wicket-user@lists.sourceforge.net >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > View this message in context: >> > > > > > > >> > > > > >> > > >> http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8751579 >> > > > > > > > Sent from the Wicket - User mailing list archive at >> Nabble.com >> . >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > >> ------------------------------------------------------------------------- >> > > > > > > > Using Tomcat but need to do more? Need to support web >> services, >> > > > > security? >> > > > > > > > Get stuff done quickly with pre-integrated technology to >> make >> your >> > > job >> > > > > > > easier. >> > > > > > > > Download IBM WebSphere Application Server v.1.0.1 based on >> Apache >> > > > > Geronimo >> > > > > > > > >> > > > > > > >> > > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > > > > > >> _______________________________________________ >> > > > > > > > Wicket-user mailing list >> > > > > > > > Wicket-user@lists.sourceforge.net >> > > > > > > > >> > > > > >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > >> > > >> ------------------------------------------------------------------------- >> > > > > > > Using Tomcat but need to do more? Need to support web >> services, >> > > > > security? >> > > > > > > Get stuff done quickly with pre-integrated technology to make >> your >> > > job >> > > > > > > easier. >> > > > > > > Download IBM WebSphere Application Server v.1.0.1 based on >> Apache >> > > > > Geronimo >> > > > > > > >> > > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > > > > >> > > > > > > _______________________________________________ >> > > > > > > Wicket-user mailing list >> > > > > > > Wicket-user@lists.sourceforge.net >> > > > > > > >> > > > > >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > >> > > >> ------------------------------------------------------------------------- >> > > > > > Using Tomcat but need to do more? Need to support web services, >> > > security? >> > > > > > Get stuff done quickly with pre-integrated technology to make >> your >> job >> > > > > easier. >> > > > > > Download IBM WebSphere Application Server v.1.0.1 based on >> Apache >> > > Geronimo >> > > > > > >> > > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > > > _______________________________________________ >> > > > > > Wicket-user mailing list >> > > > > > Wicket-user@lists.sourceforge.net >> > > > > > >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > > > >> > > > > >> > > > > >> > > > > >> > > >> ------------------------------------------------------------------------- >> > > > > Using Tomcat but need to do more? Need to support web services, >> > > security? >> > > > > Get stuff done quickly with pre-integrated technology to make >> your >> job >> > > > > easier. >> > > > > Download IBM WebSphere Application Server v.1.0.1 based on Apache >> > > Geronimo >> > > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > > _______________________________________________ >> > > > > Wicket-user mailing list >> > > > > Wicket-user@lists.sourceforge.net >> > > > > >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > > >> > > > > >> > > > >> > > > >> > > >> ------------------------------------------------------------------------- >> > > > Using Tomcat but need to do more? Need to support web services, >> security? >> > > > Get stuff done quickly with pre-integrated technology to make your >> job >> > > easier. >> > > > Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> > > > >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > > _______________________________________________ >> > > > Wicket-user mailing list >> > > > Wicket-user@lists.sourceforge.net >> > > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > > >> > > >> > > >> > > >> ------------------------------------------------------------------------- >> > > Using Tomcat but need to do more? Need to support web services, >> security? >> > > Get stuff done quickly with pre-integrated technology to make your >> job >> > > easier. >> > > Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> > > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > > _______________________________________________ >> > > Wicket-user mailing list >> > > Wicket-user@lists.sourceforge.net >> > > >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> > > >> > > >> > >> > >> ------------------------------------------------------------------------- >> > Using Tomcat but need to do more? Need to support web services, >> security? >> > Get stuff done quickly with pre-integrated technology to make your job >> easier. >> > Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> > >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> > _______________________________________________ >> > Wicket-user mailing list >> > Wicket-user@lists.sourceforge.net >> > https://lists.sourceforge.net/lists/listinfo/wicket-user >> > >> >> >> ------------------------------------------------------------------------- >> Using Tomcat but need to do more? Need to support web services, security? >> Get stuff done quickly with pre-integrated technology to make your job >> easier. >> Download IBM WebSphere Application Server v.1.0.1 based on Apache >> Geronimo >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 >> >> _______________________________________________ >> Wicket-user mailing list >> Wicket-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/wicket-user >> >> >> > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier. > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Wicket-user mailing list > Wicket-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/wicket-user > > -- View this message in context: http://www.nabble.com/Fighting-%22Too-many-open-files%22-problem-related-to-wicket-resource-files-tf3153256.html#a8760198 Sent from the Wicket - User mailing list archive at Nabble.com. ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Wicket-user mailing list Wicket-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/wicket-user
