Re: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space)
It's actually a vendor supplied application. I filed a bug/support request and they returned with a patch. I guess I was wondering if there were any jvm startup options I may have over looked that may help with this sort of issues. But it definitely seems to be a coding issue. Thanks, Chad On Thu, Oct 2, 2008 at 2:27 PM, Christopher Schultz [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chad, Chad Kellerman wrote: To get a better understanding... the application indexes uploaded files, the indexing process creates the temporary files then removes them. Can you share the code that actually does this? The javadoc for File.createTempFile states: To arrange for a file created by this method to be deleted automatically, use the deleteOnExit() method. If your engineers followed this advice, this may be your problem: the JVM is waiting until JVM shutdown to actually delete the files. Even if the code does an explicit File.delete(), the JVM is probably still hanging onto a reference to the file's directory entry in order to delete it on exit. That seems like a plausible explanation for why the files are marked as deleted in lsof, but are still being kept alive by the JVM. The solution here would be to simply not use File.deleteOnExit; instead, use File.delete explicitly. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjlEpIACgkQ9CaO5/Lv0PBo0wCfRhUitB/6/pxDQiGtbdo0Ctwk qj4AnjuQwMY3yduWoAN5/InlbZnX1CW6 =i6Uc -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chad, Chad Kellerman wrote: It's actually a vendor supplied application. I filed a bug/support request and they returned with a patch. Were you able to inspect the patch? I'm interested to see what they chose to do. I guess I was wondering if there were any jvm startup options I may have over looked that may help with this sort of issues. But it definitely seems to be a coding issue. No, I don't think that the JVM does anything like this unless you ask it to. I suspect that File.deleteOnExit is being used by programmers who didn't realize the implications on long-running processes. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjmMC0ACgkQ9CaO5/Lv0PBMHwCfZfzk7htsD9itW+sPLGwd4vgw ByUAoIEkRUOAkkdAHkD1WhXOf/Uo3zuI =OeI4 -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chad, Chad Kellerman wrote: To get a better understanding... the application indexes uploaded files, the indexing process creates the temporary files then removes them. Can you share the code that actually does this? The javadoc for File.createTempFile states: To arrange for a file created by this method to be deleted automatically, use the deleteOnExit() method. If your engineers followed this advice, this may be your problem: the JVM is waiting until JVM shutdown to actually delete the files. Even if the code does an explicit File.delete(), the JVM is probably still hanging onto a reference to the file's directory entry in order to delete it on exit. That seems like a plausible explanation for why the files are marked as deleted in lsof, but are still being kept alive by the JVM. The solution here would be to simply not use File.deleteOnExit; instead, use File.delete explicitly. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjlEpIACgkQ9CaO5/Lv0PBo0wCfRhUitB/6/pxDQiGtbdo0Ctwk qj4AnjuQwMY3yduWoAN5/InlbZnX1CW6 =i6Uc -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space)
Tomcat Users, Just wondering if anyone else has run into this problem. First let's get the specifics out of the way: Tomcat: 5.5.25 Java: Sun JDK 1.5.0_13 Linux RHEL 5 The Tomcat server is situated on it's own file system, with the applications on another. I got a page this weekend, saying that the disk space was at 95% on the application file system. Yet when I did a 'du' on the directories, it only showed about 4% used. But when I did a lsof I saw the rest of the used files in a (deleted) state. I tried a 'sync' to see if that would write the changes to disk, but the files were still listed as deleted and the disk space (df) was still at 95%. I had to resort to restarting the tomcat server. Everything freed up and the disk usage was back to normal. To get a better understanding... the application indexes uploaded files, the indexing process creates the temporary files then removes them. I am wondering if anyone has seen this before (on Tomcat, I've seen this with databases where the only this to do is restart the server, but never with Tomcat) and what they have done to help alleviate the issue. Thanks, -- sunckell
RE: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space)
From: Chad Kellerman [mailto:[EMAIL PROTECTED] Subject: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space) the application indexes uploaded files, the indexing process creates the temporary files then removes them. Sounds like your indexing process is hanging onto some form of reference to the temporary files, thereby keeping the busy counts for the inodes up. Space for a file won't be released until its busy count goes go zero, even if it is unlinked. Look for places in your code that hang on to references to objects of the type File, RandomAccessFile, some form of InputStream or OutputStream, etc. Not nulling out those references may keep the underlying inode busy. A heap profiler might help in tracking down those specific object types. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space)
- Original Message - From: Chad Kellerman [EMAIL PROTECTED] To: users@tomcat.apache.org Sent: Monday, September 29, 2008 4:39 PM Subject: Tomcat JVM not releasing deleted files (lsof show delete inodes taking up disk space) Tomcat Users, Just wondering if anyone else has run into this problem. First let's get the specifics out of the way: Tomcat: 5.5.25 Java: Sun JDK 1.5.0_13 Linux RHEL 5 The Tomcat server is situated on it's own file system, with the applications on another. I got a page this weekend, saying that the disk space was at 95% on the application file system. Yet when I did a 'du' on the directories, it only showed about 4% used. But when I did a lsof I saw the rest of the used files in a (deleted) state. I tried a 'sync' to see if that would write the changes to disk, but the files were still listed as deleted and the disk space (df) was still at 95%. I had to resort to restarting the tomcat server. Everything freed up and the disk usage was back to normal. To get a better understanding... the application indexes uploaded files, the indexing process creates the temporary files then removes them. I am wondering if anyone has seen this before (on Tomcat, I've seen this with databases where the only this to do is restart the server, but never with Tomcat) and what they have done to help alleviate the issue. Thanks, -- sunckell Chad, not sure my linux is really stale now, but I think you have answered it... the code is not releasing the file. Something like you up load the file, start the index in a thread, it deletes the file, but the thread is still holding a ref to the original file handle... the app is holding the file open somewhere. If you have a windows dev box around, get someone to run it from there... XP gets really upset if that happens and you'll probably find it. You disk usage is skew because linux is reporting that space available... but it cant let the file go, because something in the code is hanging on. ... I think... hope its your code ;) --- HARBOR : http://www.kewlstuff.co.za/index.htm The most powerful application server on earth. The only real POJO Application Server. See it in Action : http://www.kewlstuff.co.za/cd_tut_swf/whatisejb1.htm --- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
