Re: AW: AW: AW: AW: Filehandle left open when using sendfile
20 Jul 2022 12:09:46 Thomas Hoffmann (Speed4Trade GmbH) : Hello Mark, I briefly want to ask whether the internal discussion about the open JVM file handle when using sendfile/Memory-Mapped-Files resulted in any conclusions? We opted to document the risk of file locking and left the code unchanged. Mark Thanks in advance! Thomas -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 20. Juni 2022 22:13 An: users@tomcat.apache.org Betreff: Re: AW: AW: AW: Filehandle left open when using sendfile On 20/06/2022 11:39, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, thanks for your reply! -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 20. Juni 2022 12:06 An: users@tomcat.apache.org Betreff: Re: AW: AW: Filehandle left open when using sendfile On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: In the meantime I stumbled upon this bug-Report: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 So maybe the problem lies even deeper. Similar description here: https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java- caus es-problems.html Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory- Mapped files under Windows. I don’t know if there are other solutions. Your research looks to be exhaustive. I can't find any better ideas. Using the java.lang.ref.Cleaner looks to be a viable option. We know when the mapped file is no longer being used. However, that requires Java 12 onwards. This is only going to be required if the file locking is an issue. In read-only scenarios or when using an OS other than Windows it won't be an issue. So, what do we want to do? 1. Disable sendfile for HTTP/2 if running on Windows? 2. Document the potential issues with sendfile + HTTP/2 + Windows if resources are not read-only? 3. Use the JreCompat mechanism to clear the references if possible: - if running on Windows - on all OSes - if enabled via configuration Something else? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I did some further searching on this topic. Several posts disregard using java.lang.ref.Cleaner because if the buffer is used afterwards, it will crash the VM. But if used carefully it works. If we use this option, it should be possible to use it appropriately carefully. About your suggestions: 2) Documenting would be helpful, if lock can't be prevented. I also found documentation at e.g. https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.h tml#map-java.nio.channels.FileChannel.MapMode-long-long- " The buffer and the mapping that it represents will remain valid until the buffer itself is garbage-collected." Which is essentially the problem. Using the Cleaner would clean up the reference sooner. 3) As JreCompat is a bit risky, enabling via config sounds safe to me. JreCompat is perfectly safe. The jdk.internal.misc.Unsafe API is where the risk is and this is primarily the risk of the crash mentioned above that we should be able to avoid. Some other (theoretical?) options: 4) In an older version of Tomcat native lib there seemed to be a native Implementation of MMap: https://tomcat.apache.org/tomcat-10.0- doc/api/org/apache/tomcat/jni/Mmap.html I read that this was an alternative to the Java memory mapped file. But it was removed in newer versions. Maybe it can be resurrected for this case and used if native lib is available(?) Sorry, no. We are moving away from the native library. Eventually we will just use project Panama to wrap OpenSSL. Until then, we are removing everything that isn't required to support the use of OPenSSl with NIO and NIO2. The primary reason for this is stability. 5) Instead of FileChannel.map maybe a normal ByteBuffer with FileChannel.read(buffer) can be used (?) That is worth considering. The other sendfile implementations don't use a memory mapped file. I'll start a discussion on the dev list. One remaining question: I didn’t find FileChannel.map in the other connectors. Is Http2AsyncUpgradeHandler the only occurrence? In the main code base, yes. There is another usage in the test code but that is less of a concern. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AW: AW: Filehandle left open when using sendfile
Hello Mark, I briefly want to ask whether the internal discussion about the open JVM file handle when using sendfile/Memory-Mapped-Files resulted in any conclusions? Thanks in advance! Thomas > -Ursprüngliche Nachricht- > Von: Mark Thomas > Gesendet: Montag, 20. Juni 2022 22:13 > An: users@tomcat.apache.org > Betreff: Re: AW: AW: AW: Filehandle left open when using sendfile > > On 20/06/2022 11:39, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > Hello Mark, > > > > thanks for your reply! > > > >> -Ursprüngliche Nachricht- > >> Von: Mark Thomas > >> Gesendet: Montag, 20. Juni 2022 12:06 > >> An: users@tomcat.apache.org > >> Betreff: Re: AW: AW: Filehandle left open when using sendfile > >> > >> On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: > >> > >> > >> > >>> In the meantime I stumbled upon this bug-Report: > >>> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 > >>> So maybe the problem lies even deeper. > >>> Similar description here: > >>> https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java- > >> caus > >>> es-problems.html > >>> > >>> Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory- > >> Mapped files under Windows. > >>> I don’t know if there are other solutions. > >> > >> Your research looks to be exhaustive. I can't find any better ideas. > >> > >> Using the java.lang.ref.Cleaner looks to be a viable option. We know > >> when the mapped file is no longer being used. However, that requires > >> Java 12 onwards. > >> > >> This is only going to be required if the file locking is an issue. In > >> read-only scenarios or when using an OS other than Windows it won't be > an issue. > >> > >> So, what do we want to do? > >> > >> 1. Disable sendfile for HTTP/2 if running on Windows? > >> > >> 2. Document the potential issues with sendfile + HTTP/2 + Windows if > >> resources are not read-only? > >> > >> 3. Use the JreCompat mechanism to clear the references if possible: > >> - if running on Windows > >> - on all OSes > >> - if enabled via configuration > >> > >> Something else? > >> > >> Mark > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > I did some further searching on this topic. > > Several posts disregard using java.lang.ref.Cleaner because if the buffer is > used afterwards, it will crash the VM. But if used carefully it works. > > If we use this option, it should be possible to use it appropriately > carefully. > > > About your suggestions: > > 2) Documenting would be helpful, if lock can't be prevented. > > I also found documentation at e.g. > https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.h > tml#map-java.nio.channels.FileChannel.MapMode-long-long- > > " The buffer and the mapping that it represents will remain valid > > until > the buffer itself is garbage-collected." > > Which is essentially the problem. Using the Cleaner would clean up the > reference sooner. > > > 3) As JreCompat is a bit risky, enabling via config sounds safe to me. > > JreCompat is perfectly safe. The jdk.internal.misc.Unsafe API is where the > risk is and this is primarily the risk of the crash mentioned above that we > should be able to avoid. > > > Some other (theoretical?) options: > > 4) In an older version of Tomcat native lib there seemed to be a native > Implementation of MMap: https://tomcat.apache.org/tomcat-10.0- > doc/api/org/apache/tomcat/jni/Mmap.html > >I read that this was an alternative to the Java memory mapped > > file. But it was removed in newer versions. Maybe it can be > > resurrected for this case and used if native lib is available(?) > > Sorry, no. We are moving away from the native library. Eventually we will just > use project Panama to wrap OpenSSL. Until then, we are removing > everything that isn't required to support the use of OPenSSl with NIO and > NIO2. > > The primary reason for this is stability. > > > 5) Instead of FileChannel.map maybe a normal ByteBuffer with > > FileChannel.read(buffer) can be used (?) > > That is worth considering. The other sendfile implement
Re: AW: AW: AW: AW: Filehandle left open when using sendfile
On 6/20/22 23:59, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 20. Juni 2022 22:13 An: users@tomcat.apache.org Betreff: Re: AW: AW: AW: Filehandle left open when using sendfile On 20/06/2022 11:39, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, thanks for your reply! -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 20. Juni 2022 12:06 An: users@tomcat.apache.org Betreff: Re: AW: AW: Filehandle left open when using sendfile On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: In the meantime I stumbled upon this bug-Report: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 So maybe the problem lies even deeper. Similar description here: https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java- caus es-problems.html Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory- Mapped files under Windows. I don’t know if there are other solutions. Your research looks to be exhaustive. I can't find any better ideas. Using the java.lang.ref.Cleaner looks to be a viable option. We know when the mapped file is no longer being used. However, that requires Java 12 onwards. This is only going to be required if the file locking is an issue. In read-only scenarios or when using an OS other than Windows it won't be an issue. So, what do we want to do? 1. Disable sendfile for HTTP/2 if running on Windows? 2. Document the potential issues with sendfile + HTTP/2 + Windows if resources are not read-only? 3. Use the JreCompat mechanism to clear the references if possible: - if running on Windows - on all OSes - if enabled via configuration Something else? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I did some further searching on this topic. Several posts disregard using java.lang.ref.Cleaner because if the buffer is used afterwards, it will crash the VM. But if used carefully it works. If we use this option, it should be possible to use it appropriately carefully. About your suggestions: 2) Documenting would be helpful, if lock can't be prevented. I also found documentation at e.g. https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.h tml#map-java.nio.channels.FileChannel.MapMode-long-long- " The buffer and the mapping that it represents will remain valid until the buffer itself is garbage-collected." Which is essentially the problem. Using the Cleaner would clean up the reference sooner. 3) As JreCompat is a bit risky, enabling via config sounds safe to me. JreCompat is perfectly safe. The jdk.internal.misc.Unsafe API is where the risk is and this is primarily the risk of the crash mentioned above that we should be able to avoid. Some other (theoretical?) options: 4) In an older version of Tomcat native lib there seemed to be a native Implementation of MMap: https://tomcat.apache.org/tomcat-10.0- doc/api/org/apache/tomcat/jni/Mmap.html I read that this was an alternative to the Java memory mapped file. But it was removed in newer versions. Maybe it can be resurrected for this case and used if native lib is available(?) Sorry, no. We are moving away from the native library. Eventually we will just use project Panama to wrap OpenSSL. Until then, we are removing everything that isn't required to support the use of OPenSSl with NIO and NIO2. The primary reason for this is stability. 5) Instead of FileChannel.map maybe a normal ByteBuffer with FileChannel.read(buffer) can be used (?) That is worth considering. The other sendfile implementations don't use a memory mapped file. I'll start a discussion on the dev list. One remaining question: I didn’t find FileChannel.map in the other connectors. Is Http2AsyncUpgradeHandler the only occurrence? In the main code base, yes. There is another usage in the test code but that is less of a concern. Mark Just two thoughts / remarks: 3) New java versions provide java.lang.ref.Cleaner. In older java versions a similar class was sun.misc.Cleaner (though the usage looks a bit strange) 5) The memory mapped file approach is quite memory efficient as the file gets virtually mapped into the address space of the java process without loading all the data into (real) memory. As far as I understood, only the used parts of the file gets loaded by Windows OS. When reading the file into a Buffer, the file is read completely. This shouldn’t make a big difference with files around some KB or MB. However, for large files with several GB they might be handled well via memory mapped files but not with a ByteBuffer. I am not sure, if sendfile is popular and used for this use-cases. So maybe a configuration toggle between "memory-optimized" and &q
AW: AW: AW: AW: Filehandle left open when using sendfile
Hello Mark, > -Ursprüngliche Nachricht- > Von: Mark Thomas > Gesendet: Montag, 20. Juni 2022 22:13 > An: users@tomcat.apache.org > Betreff: Re: AW: AW: AW: Filehandle left open when using sendfile > > On 20/06/2022 11:39, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > Hello Mark, > > > > thanks for your reply! > > > >> -Ursprüngliche Nachricht- > >> Von: Mark Thomas > >> Gesendet: Montag, 20. Juni 2022 12:06 > >> An: users@tomcat.apache.org > >> Betreff: Re: AW: AW: Filehandle left open when using sendfile > >> > >> On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: > >> > >> > >> > >>> In the meantime I stumbled upon this bug-Report: > >>> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 > >>> So maybe the problem lies even deeper. > >>> Similar description here: > >>> https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java- > >> caus > >>> es-problems.html > >>> > >>> Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory- > >> Mapped files under Windows. > >>> I don’t know if there are other solutions. > >> > >> Your research looks to be exhaustive. I can't find any better ideas. > >> > >> Using the java.lang.ref.Cleaner looks to be a viable option. We know > >> when the mapped file is no longer being used. However, that requires > >> Java 12 onwards. > >> > >> This is only going to be required if the file locking is an issue. In > >> read-only scenarios or when using an OS other than Windows it won't be > an issue. > >> > >> So, what do we want to do? > >> > >> 1. Disable sendfile for HTTP/2 if running on Windows? > >> > >> 2. Document the potential issues with sendfile + HTTP/2 + Windows if > >> resources are not read-only? > >> > >> 3. Use the JreCompat mechanism to clear the references if possible: > >> - if running on Windows > >> - on all OSes > >> - if enabled via configuration > >> > >> Something else? > >> > >> Mark > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > I did some further searching on this topic. > > Several posts disregard using java.lang.ref.Cleaner because if the buffer is > used afterwards, it will crash the VM. But if used carefully it works. > > If we use this option, it should be possible to use it appropriately > carefully. > > > About your suggestions: > > 2) Documenting would be helpful, if lock can't be prevented. > > I also found documentation at e.g. > https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.h > tml#map-java.nio.channels.FileChannel.MapMode-long-long- > > " The buffer and the mapping that it represents will remain valid > > until > the buffer itself is garbage-collected." > > Which is essentially the problem. Using the Cleaner would clean up the > reference sooner. > > > 3) As JreCompat is a bit risky, enabling via config sounds safe to me. > > JreCompat is perfectly safe. The jdk.internal.misc.Unsafe API is where the > risk is and this is primarily the risk of the crash mentioned above that we > should be able to avoid. > > > Some other (theoretical?) options: > > 4) In an older version of Tomcat native lib there seemed to be a native > Implementation of MMap: https://tomcat.apache.org/tomcat-10.0- > doc/api/org/apache/tomcat/jni/Mmap.html > >I read that this was an alternative to the Java memory mapped > > file. But it was removed in newer versions. Maybe it can be > > resurrected for this case and used if native lib is available(?) > > Sorry, no. We are moving away from the native library. Eventually we will just > use project Panama to wrap OpenSSL. Until then, we are removing > everything that isn't required to support the use of OPenSSl with NIO and > NIO2. > > The primary reason for this is stability. > > > 5) Instead of FileChannel.map maybe a normal ByteBuffer with > > FileChannel.read(buffer) can be used (?) > > That is worth considering. The other sendfile implementations don't use a > memory mapped file. > > I'll start a discussion on the dev list. > > > One remaining question: > > I didn’t find FileChannel.map in the
Re: AW: AW: AW: Filehandle left open when using sendfile
On 20/06/2022 11:39, Thomas Hoffmann (Speed4Trade GmbH) wrote: Hello Mark, thanks for your reply! -Ursprüngliche Nachricht- Von: Mark Thomas Gesendet: Montag, 20. Juni 2022 12:06 An: users@tomcat.apache.org Betreff: Re: AW: AW: Filehandle left open when using sendfile On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: In the meantime I stumbled upon this bug-Report: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 So maybe the problem lies even deeper. Similar description here: https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java- caus es-problems.html Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory- Mapped files under Windows. I don’t know if there are other solutions. Your research looks to be exhaustive. I can't find any better ideas. Using the java.lang.ref.Cleaner looks to be a viable option. We know when the mapped file is no longer being used. However, that requires Java 12 onwards. This is only going to be required if the file locking is an issue. In read-only scenarios or when using an OS other than Windows it won't be an issue. So, what do we want to do? 1. Disable sendfile for HTTP/2 if running on Windows? 2. Document the potential issues with sendfile + HTTP/2 + Windows if resources are not read-only? 3. Use the JreCompat mechanism to clear the references if possible: - if running on Windows - on all OSes - if enabled via configuration Something else? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I did some further searching on this topic. Several posts disregard using java.lang.ref.Cleaner because if the buffer is used afterwards, it will crash the VM. But if used carefully it works. If we use this option, it should be possible to use it appropriately carefully. About your suggestions: 2) Documenting would be helpful, if lock can't be prevented. I also found documentation at e.g. https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.html#map-java.nio.channels.FileChannel.MapMode-long-long- " The buffer and the mapping that it represents will remain valid until the buffer itself is garbage-collected." Which is essentially the problem. Using the Cleaner would clean up the reference sooner. 3) As JreCompat is a bit risky, enabling via config sounds safe to me. JreCompat is perfectly safe. The jdk.internal.misc.Unsafe API is where the risk is and this is primarily the risk of the crash mentioned above that we should be able to avoid. Some other (theoretical?) options: 4) In an older version of Tomcat native lib there seemed to be a native Implementation of MMap: https://tomcat.apache.org/tomcat-10.0-doc/api/org/apache/tomcat/jni/Mmap.html I read that this was an alternative to the Java memory mapped file. But it was removed in newer versions. Maybe it can be resurrected for this case and used if native lib is available(?) Sorry, no. We are moving away from the native library. Eventually we will just use project Panama to wrap OpenSSL. Until then, we are removing everything that isn't required to support the use of OPenSSl with NIO and NIO2. The primary reason for this is stability. 5) Instead of FileChannel.map maybe a normal ByteBuffer with FileChannel.read(buffer) can be used (?) That is worth considering. The other sendfile implementations don't use a memory mapped file. I'll start a discussion on the dev list. One remaining question: I didn’t find FileChannel.map in the other connectors. Is Http2AsyncUpgradeHandler the only occurrence? In the main code base, yes. There is another usage in the test code but that is less of a concern. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AW: Filehandle left open when using sendfile
Hello Mark, thanks for your reply! > -Ursprüngliche Nachricht- > Von: Mark Thomas > Gesendet: Montag, 20. Juni 2022 12:06 > An: users@tomcat.apache.org > Betreff: Re: AW: AW: Filehandle left open when using sendfile > > On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > > > > In the meantime I stumbled upon this bug-Report: > > https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 > > So maybe the problem lies even deeper. > > Similar description here: > > https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java- > caus > > es-problems.html > > > > Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory- > Mapped files under Windows. > > I don’t know if there are other solutions. > > Your research looks to be exhaustive. I can't find any better ideas. > > Using the java.lang.ref.Cleaner looks to be a viable option. We know when > the mapped file is no longer being used. However, that requires Java 12 > onwards. > > This is only going to be required if the file locking is an issue. In > read-only > scenarios or when using an OS other than Windows it won't be an issue. > > So, what do we want to do? > > 1. Disable sendfile for HTTP/2 if running on Windows? > > 2. Document the potential issues with sendfile + HTTP/2 + Windows if > resources are not read-only? > > 3. Use the JreCompat mechanism to clear the references if possible: > - if running on Windows > - on all OSes > - if enabled via configuration > > Something else? > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org I did some further searching on this topic. Several posts disregard using java.lang.ref.Cleaner because if the buffer is used afterwards, it will crash the VM. But if used carefully it works. About your suggestions: 2) Documenting would be helpful, if lock can't be prevented. I also found documentation at e.g. https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.html#map-java.nio.channels.FileChannel.MapMode-long-long- " The buffer and the mapping that it represents will remain valid until the buffer itself is garbage-collected." 3) As JreCompat is a bit risky, enabling via config sounds safe to me. Some other (theoretical?) options: 4) In an older version of Tomcat native lib there seemed to be a native Implementation of MMap: https://tomcat.apache.org/tomcat-10.0-doc/api/org/apache/tomcat/jni/Mmap.html I read that this was an alternative to the Java memory mapped file. But it was removed in newer versions. Maybe it can be resurrected for this case and used if native lib is available(?) 5) Instead of FileChannel.map maybe a normal ByteBuffer with FileChannel.read(buffer) can be used (?) One remaining question: I didn’t find FileChannel.map in the other connectors. Is Http2AsyncUpgradeHandler the only occurrence? Thanks a lot! Thomas
Re: AW: AW: Filehandle left open when using sendfile
On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote: In the meantime I stumbled upon this bug-Report: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154 So maybe the problem lies even deeper. Similar description here: https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java-causes-problems.html Some ppl suggest to use java.lang.ref.Cleaner or don’t use Memory-Mapped files under Windows. I don’t know if there are other solutions. Your research looks to be exhaustive. I can't find any better ideas. Using the java.lang.ref.Cleaner looks to be a viable option. We know when the mapped file is no longer being used. However, that requires Java 12 onwards. This is only going to be required if the file locking is an issue. In read-only scenarios or when using an OS other than Windows it won't be an issue. So, what do we want to do? 1. Disable sendfile for HTTP/2 if running on Windows? 2. Document the potential issues with sendfile + HTTP/2 + Windows if resources are not read-only? 3. Use the JreCompat mechanism to clear the references if possible: - if running on Windows - on all OSes - if enabled via configuration Something else? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: Filehandle left open when using sendfile
Hello Chris,
> -Ursprüngliche Nachricht-
> Von: Christopher Schultz
> Gesendet: Donnerstag, 16. Juni 2022 05:38
> An: users@tomcat.apache.org
> Betreff: Re: AW: Filehandle left open when using sendfile
>
> Thomas,
>
> On 6/15/22 02:26, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > Hello Christopher,
> >
> >> -Ursprüngliche Nachricht-
> >> Von: Christopher Schultz
> >> Gesendet: Dienstag, 14. Juni 2022 20:26
> >> An: users@tomcat.apache.org
> >> Betreff: Re: Filehandle left open when using sendfile
> >>
> >> Thomas,
> >>
> >> On 6/14/22 13:52, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> >>> Hello,
> >>> we are using Tomcat 10.0.16 under windows.
> >>> For sending files to the browser, we are using sendfile by setting
> >>> the
> >> attribute "org.apache.tomcat.sendfile.filename".
> >>> Streaming an image to the browser works well in this way.
> >>> But we observed that if the user tries to delete the file
> >>> afterwards, there is
> >> still a file-handle on this file.
> >>
> >> Which user? Some admin on the server, or the user using the browser?
> >> (Dumb question, I know, but tb and ff had a bug for a while where
> >> downloads would leave dangling file handles, disallowing deleting
> >> those files after they were downloaded.)
> >>
> >>> I took a look at NioEndpoint.java -->
> >>>
> >>
> https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/uti
> >> l
> >>> /net/NioEndpoint.java
> >>>
> >>> At line 869 there is a stream opened:
> >>> FileInputStream fis = new FileInputStream(f);
> >>>
> >>> However, there is no close in this method. Maybe this might cause
> >>> the
> >> open file-handle?
> >>
> >> Did you read the line above that?
> >>
> >> @SuppressWarnings("resource") // Closed when channel is closed
> >> FileInputStream fis = new FileInputStream(f);
> >>
> >> The channel itself is cleaned-up properly and, with it, the file handle.
> >>
> >>
> https://github.com/apache/tomcat/blob/69b9a341062dc1930782941bb
> >> aa1880e9281/java/org/apache/tomcat/util/net/NioEndpoint.java#L1226
> >>
> >> (At least, I *think* this is where the cleanup happens. I'm not very
> >> well- versed in the connection-handling code in Tomcat.)
> >>
> >>> We are using protocol="org.apache.coyote.http11.Http11NioProtocol"..
> >>> I
> >> think there are different implementations for sendfile-feature.
> >>> The open file-handle can be observer via ProcessExplorer from
> Microsoft.
> >>
> >> How long did you wait for the file handle to close?
> >>
> >> Are you able to attach a debugger and see that the object hasn't been
> >> cleaned-up?
> >>
> >> -chris
> >>
> >
> > Thanks for taking a look at it.
> > Sorry, if the setup was not described in detail. Maybe you can imagine an
> image library, hosted server-side.
> > The user can upload pics and after upload the user can also decide to
> delete the image.
>
> Light bulb: are you using ImageIO?
ImageIO is not used. The pictures are saved as they are without modification.
>
> There have been some well-known leaks in there over the years. It's also
> *very* easy to leak file handles if you aren't managing your resources
> appropriately.
>
> Are you *sure* you have clean use of resources within your own code?
>
> If you are using *any* of Java's javax.imageio code, is it possible to skip
> that
> and re-try your test? Can you e.g. upload a plain text file (or even an image)
> and only stream the bytes from client -> disk without doing anything else?
>
> > Requests:
> > 1) POST-Request with image upload. Server saves the image in the file
> system and returns the URL.
> > 2) GET-Request, the browser requests the URL of the uploaded picture.
> > Server is sending the file via sendfile
> > 3) POST-Request with command to delete the picture (maybe user doesn’t
> > like the pic). Tomcat tries to delete the file on the server side but
> > fails because of the left handle in step 2
>
> Are you sure it's from step #2 and not step #1?
At first I was hunting the problem at step 1)
After a while I figured out, that it is indeed happening in 2)
The following sample reproduces the orphaned file handle under Windows:
<%@page import=&
Re: AW: Filehandle left open when using sendfile
Thomas,
On 6/15/22 02:26, Thomas Hoffmann (Speed4Trade GmbH) wrote:
Hello Christopher,
-Ursprüngliche Nachricht-
Von: Christopher Schultz
Gesendet: Dienstag, 14. Juni 2022 20:26
An: users@tomcat.apache.org
Betreff: Re: Filehandle left open when using sendfile
Thomas,
On 6/14/22 13:52, Thomas Hoffmann (Speed4Trade GmbH) wrote:
Hello,
we are using Tomcat 10.0.16 under windows.
For sending files to the browser, we are using sendfile by setting the
attribute "org.apache.tomcat.sendfile.filename".
Streaming an image to the browser works well in this way.
But we observed that if the user tries to delete the file afterwards, there is
still a file-handle on this file.
Which user? Some admin on the server, or the user using the browser?
(Dumb question, I know, but tb and ff had a bug for a while where
downloads would leave dangling file handles, disallowing deleting those files
after they were downloaded.)
I took a look at NioEndpoint.java -->
https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util
/net/NioEndpoint.java
At line 869 there is a stream opened:
FileInputStream fis = new FileInputStream(f);
However, there is no close in this method. Maybe this might cause the
open file-handle?
Did you read the line above that?
@SuppressWarnings("resource") // Closed when channel is closed
FileInputStream fis = new FileInputStream(f);
The channel itself is cleaned-up properly and, with it, the file handle.
https://github.com/apache/tomcat/blob/69b9a341062dc1930782941bb
aa1880e9281/java/org/apache/tomcat/util/net/NioEndpoint.java#L1226
(At least, I *think* this is where the cleanup happens. I'm not very well-
versed in the connection-handling code in Tomcat.)
We are using protocol="org.apache.coyote.http11.Http11NioProtocol".. I
think there are different implementations for sendfile-feature.
The open file-handle can be observer via ProcessExplorer from Microsoft.
How long did you wait for the file handle to close?
Are you able to attach a debugger and see that the object hasn't been
cleaned-up?
-chris
Thanks for taking a look at it.
Sorry, if the setup was not described in detail. Maybe you can imagine an image
library, hosted server-side.
The user can upload pics and after upload the user can also decide to delete
the image.
Light bulb: are you using ImageIO?
There have been some well-known leaks in there over the years. It's also
*very* easy to leak file handles if you aren't managing your resources
appropriately.
Are you *sure* you have clean use of resources within your own code?
If you are using *any* of Java's javax.imageio code, is it possible to
skip that and re-try your test? Can you e.g. upload a plain text file
(or even an image) and only stream the bytes from client -> disk without
doing anything else?
Requests:
1) POST-Request with image upload. Server saves the image in the file system
and returns the URL.
2) GET-Request, the browser requests the URL of the uploaded picture. Server is
sending the file via sendfile
3) POST-Request with command to delete the picture (maybe user doesn’t like the
pic). Tomcat tries to delete the file on the server side but fails because of
the left handle in step 2
Are you sure it's from step #2 and not step #1?
The handle stays for several minutes for sure.
It is hard to debug because when stepping through, the file-handle is gone
after processing the request.
That seems suspicious if you are saying that Tomcat is definitely
leaving the file handle open.
I think there are some if-statements whether the transmission is pending or
finished.
The mentioned line in the NIOEndpoint is not reached. So the file handle must
be left somewhere else. My first assumption was wrong.
Does the problem go away if you disable sendfile and stream the file
directly?
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AW: Filehandle left open when using sendfile
> -Ursprüngliche Nachricht-
> Von: Thomas Hoffmann (Speed4Trade GmbH)
>
> Gesendet: Mittwoch, 15. Juni 2022 08:26
> An: Tomcat Users List
> Betreff: AW: Filehandle left open when using sendfile
>
> Hello Christopher,
>
> > -Ursprüngliche Nachricht-
> > Von: Christopher Schultz
> > Gesendet: Dienstag, 14. Juni 2022 20:26
> > An: users@tomcat.apache.org
> > Betreff: Re: Filehandle left open when using sendfile
> >
> > Thomas,
> >
> > On 6/14/22 13:52, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > > Hello,
> > > we are using Tomcat 10.0.16 under windows.
> > > For sending files to the browser, we are using sendfile by setting
> > > the
> > attribute "org.apache.tomcat.sendfile.filename".
> > > Streaming an image to the browser works well in this way.
> > > But we observed that if the user tries to delete the file
> > > afterwards, there is
> > still a file-handle on this file.
> >
> > Which user? Some admin on the server, or the user using the browser?
> > (Dumb question, I know, but tb and ff had a bug for a while where
> > downloads would leave dangling file handles, disallowing deleting
> > those files after they were downloaded.)
> >
> > > I took a look at NioEndpoint.java -->
> > >
> >
> https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util
> > > /net/NioEndpoint.java
> > >
> > > At line 869 there is a stream opened:
> > > FileInputStream fis = new FileInputStream(f);
> > >
> > > However, there is no close in this method. Maybe this might cause
> > > the
> > open file-handle?
> >
> > Did you read the line above that?
> >
> > @SuppressWarnings("resource") // Closed when channel is closed
> > FileInputStream fis = new FileInputStream(f);
> >
> > The channel itself is cleaned-up properly and, with it, the file handle.
> >
> >
> https://github.com/apache/tomcat/blob/69b9a341062dc1930782941bb
> > aa1880e9281/java/org/apache/tomcat/util/net/NioEndpoint.java#L1226
> >
> > (At least, I *think* this is where the cleanup happens. I'm not very
> > well- versed in the connection-handling code in Tomcat.)
> >
> > > We are using protocol="org.apache.coyote.http11.Http11NioProtocol"..
> > > I
> > think there are different implementations for sendfile-feature.
> > > The open file-handle can be observer via ProcessExplorer from Microsoft.
> >
> > How long did you wait for the file handle to close?
> >
> > Are you able to attach a debugger and see that the object hasn't been
> > cleaned-up?
> >
> > -chris
> >
>
> Thanks for taking a look at it.
> Sorry, if the setup was not described in detail. Maybe you can imagine an
> image library, hosted server-side.
> The user can upload pics and after upload the user can also decide to delete
> the image.
>
> Requests:
> 1) POST-Request with image upload. Server saves the image in the file
> system and returns the URL.
> 2) GET-Request, the browser requests the URL of the uploaded picture.
> Server is sending the file via sendfile
> 3) POST-Request with command to delete the picture (maybe user doesn’t
> like the pic). Tomcat tries to delete the file on the server side but fails
> because of the left handle in step 2
>
> The handle stays for several minutes for sure.
>
> It is hard to debug because when stepping through, the file-handle is gone
> after processing the request.
> I think there are some if-statements whether the transmission is pending or
> finished.
> The mentioned line in the NIOEndpoint is not reached. So the file handle
> must be left somewhere else. My first assumption was wrong.
>
> Greetings, Thomas
Short update on this issue:
Today, I could further debug the issue.
Within Http2AsyncUpgradeHandler.java the following code creates the file handle:
protected SendfileState processSendfile(SendfileData sendfile) {
if (sendfile != null) {
try {
try (FileChannel channel = FileChannel.open(sendfile.path,
StandardOpenOption.READ)) {
sendfile.mappedBuffer = channel.map(MapMode.READ_ONLY,
sendfile.pos, sendfile.end - sendfile.pos);
}
..
Even with the try-with-resource block, the file handle is still there after
this code block.
Sometimes the handle vanishes quite quickly, sometimes the handle stays open
for many minutes.
It can be verified with a little jsp sample page:
<%@page import="java.io.*"%>
<%@page import="java.net.*&q
AW: Filehandle left open when using sendfile
Hello Christopher,
> -Ursprüngliche Nachricht-
> Von: Christopher Schultz
> Gesendet: Dienstag, 14. Juni 2022 20:26
> An: users@tomcat.apache.org
> Betreff: Re: Filehandle left open when using sendfile
>
> Thomas,
>
> On 6/14/22 13:52, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > Hello,
> > we are using Tomcat 10.0.16 under windows.
> > For sending files to the browser, we are using sendfile by setting the
> attribute "org.apache.tomcat.sendfile.filename".
> > Streaming an image to the browser works well in this way.
> > But we observed that if the user tries to delete the file afterwards, there
> > is
> still a file-handle on this file.
>
> Which user? Some admin on the server, or the user using the browser?
> (Dumb question, I know, but tb and ff had a bug for a while where
> downloads would leave dangling file handles, disallowing deleting those files
> after they were downloaded.)
>
> > I took a look at NioEndpoint.java -->
> >
> https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util
> > /net/NioEndpoint.java
> >
> > At line 869 there is a stream opened:
> > FileInputStream fis = new FileInputStream(f);
> >
> > However, there is no close in this method. Maybe this might cause the
> open file-handle?
>
> Did you read the line above that?
>
> @SuppressWarnings("resource") // Closed when channel is closed
> FileInputStream fis = new FileInputStream(f);
>
> The channel itself is cleaned-up properly and, with it, the file handle.
>
> https://github.com/apache/tomcat/blob/69b9a341062dc1930782941bb
> aa1880e9281/java/org/apache/tomcat/util/net/NioEndpoint.java#L1226
>
> (At least, I *think* this is where the cleanup happens. I'm not very well-
> versed in the connection-handling code in Tomcat.)
>
> > We are using protocol="org.apache.coyote.http11.Http11NioProtocol".. I
> think there are different implementations for sendfile-feature.
> > The open file-handle can be observer via ProcessExplorer from Microsoft.
>
> How long did you wait for the file handle to close?
>
> Are you able to attach a debugger and see that the object hasn't been
> cleaned-up?
>
> -chris
>
Thanks for taking a look at it.
Sorry, if the setup was not described in detail. Maybe you can imagine an image
library, hosted server-side.
The user can upload pics and after upload the user can also decide to delete
the image.
Requests:
1) POST-Request with image upload. Server saves the image in the file system
and returns the URL.
2) GET-Request, the browser requests the URL of the uploaded picture. Server is
sending the file via sendfile
3) POST-Request with command to delete the picture (maybe user doesn’t like the
pic). Tomcat tries to delete the file on the server side but fails because of
the left handle in step 2
The handle stays for several minutes for sure.
It is hard to debug because when stepping through, the file-handle is gone
after processing the request.
I think there are some if-statements whether the transmission is pending or
finished.
The mentioned line in the NIOEndpoint is not reached. So the file handle must
be left somewhere else. My first assumption was wrong.
Greetings, Thomas
Re: Filehandle left open when using sendfile
Thomas,
On 6/14/22 13:52, Thomas Hoffmann (Speed4Trade GmbH) wrote:
Hello,
we are using Tomcat 10.0.16 under windows.
For sending files to the browser, we are using sendfile by setting the attribute
"org.apache.tomcat.sendfile.filename".
Streaming an image to the browser works well in this way.
But we observed that if the user tries to delete the file afterwards, there is
still a file-handle on this file.
Which user? Some admin on the server, or the user using the browser?
(Dumb question, I know, but tb and ff had a bug for a while where
downloads would leave dangling file handles, disallowing deleting those
files after they were downloaded.)
I took a look at NioEndpoint.java -->
https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util/net/NioEndpoint.java
At line 869 there is a stream opened:
FileInputStream fis = new FileInputStream(f);
However, there is no close in this method. Maybe this might cause the open
file-handle?
Did you read the line above that?
@SuppressWarnings("resource") // Closed when channel is closed
FileInputStream fis = new FileInputStream(f);
The channel itself is cleaned-up properly and, with it, the file handle.
https://github.com/apache/tomcat/blob/69b9a341062dc1930782941bbaa1880e9281/java/org/apache/tomcat/util/net/NioEndpoint.java#L1226
(At least, I *think* this is where the cleanup happens. I'm not very
well-versed in the connection-handling code in Tomcat.)
We are using protocol="org.apache.coyote.http11.Http11NioProtocol".. I think
there are different implementations for sendfile-feature.
The open file-handle can be observer via ProcessExplorer from Microsoft.
How long did you wait for the file handle to close?
Are you able to attach a debugger and see that the object hasn't been
cleaned-up?
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Filehandle left open when using sendfile
Hello, we are using Tomcat 10.0.16 under windows. For sending files to the browser, we are using sendfile by setting the attribute "org.apache.tomcat.sendfile.filename". Streaming an image to the browser works well in this way. But we observed that if the user tries to delete the file afterwards, there is still a file-handle on this file. I took a look at NioEndpoint.java --> https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util/net/NioEndpoint.java At line 869 there is a stream opened: FileInputStream fis = new FileInputStream(f); However, there is no close in this method. Maybe this might cause the open file-handle? We are using protocol="org.apache.coyote.http11.Http11NioProtocol"... I think there are different implementations for sendfile-feature. The open file-handle can be observer via ProcessExplorer from Microsoft. Thanks in advance! Thomas - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Filehandle left open when using sendfile
Hello, we are using Tomcat 10.0.16 under windows. For sending files to the browser, we are using sendfile by setting the attribute "org.apache.tomcat.sendfile.filename". Streaming an image to the browser works well in this way. But we observed that if the user tries to delete the file afterwards, there is still a file-handle on this file. I took a look at NioEndpoint.java --> https://github.com/apache/tomcat/blob/main/java/org/apache/tomcat/util/net/NioEndpoint.java At line 869 there is a stream opened: FileInputStream fis = new FileInputStream(f); However, there is no close in this method. Maybe this might cause the open file-handle? We are using protocol="org.apache.coyote.http11.Http11NioProtocol".. I think there are different implementations for sendfile-feature. The open file-handle can be observer via ProcessExplorer from Microsoft. Thanks in advance! Thomas
