Re: OutOfMemory on large file download with AJP and cachingAllowed=false
On 26/04/2019 21:07, Olivier Jaquemet wrote: > PS : completely unrelated to this matter, I just find out that the page > https://tomcat.apache.org/svn.html contains outdated information and > should probably removed as it as been replaced with > https://tomcat.apache.org/source.html There is meant to be a rewrite rule in place for that. I'll see if I can figure out why it isn't working. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
On 26/04/2019 09:56, Mark Thomas wrote: There was an extra copy but Chris's suggestion got me thinking and I found a much better solution. The patch has been applied to 9.0.x and 8.5.x and will be in the next release of both. 7.0.x is not affected. The patch fixes the OutOfMemoryError and the ArrayIndexOutOfBoundsException. Mark Thank you Mark and Christopher for your work on this. As always you rocks. For the record, if anyone is looking for the corresponding issue/bug, none was entered (quite unfortunate from my point of view regarding defect tracking), but the corresponding commit is there : * master (9.x) https://github.com/apache/tomcat/commit/a8f1e96a456d8493a8e64dfe743a8ae663b28ce * 8.5 : https://github.com/apache/tomcat/commit/4ab58e9881ebdc039a657f5f77caf66b673f934b (and also two some commits added for improved getContent javadoc) Thanks again. Olivier PS : completely unrelated to this matter, I just find out that the page https://tomcat.apache.org/svn.html contains outdated information and should probably removed as it as been replaced with https://tomcat.apache.org/source.html
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
On 25/04/2019 21:37, Mark Thomas wrote: > On 25/04/2019 21:16, Christopher Schultz wrote: >> On 4/25/19 15:55, Mark Thomas wrote: >> If the resources are caching-aware, then I think the DefaultServlet >> can just always use Resource.getInputStream. >> >> Hmm. That might cause a lot of unnecessary IO if the bytes are >> actually available. > > That is a very tempting solution. The result is a LOT cleaner than the > patch I just wrote. CachingResource is smart enough to cache the bytes > and wrap them in a ByteArrayInputStream if Resource.getInputStream is > called. My only concern is I think this introduces and additional copy > of the data. I need to check that. There was an extra copy but Chris's suggestion got me thinking and I found a much better solution. The patch has been applied to 9.0.x and 8.5.x and will be in the next release of both. 7.0.x is not affected. The patch fixes the OutOfMemoryError and the ArrayIndexOutOfBoundsException. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
On 25/04/2019 21:16, Christopher Schultz wrote: > Mark, > > On 4/25/19 15:55, Mark Thomas wrote: >> On 23/04/2019 16:29, Olivier Jaquemet wrote: >>> On 23/04/2019 16:12, Christopher Schultz wrote: On 4/23/19 05:58, Olivier Jaquemet wrote: > >> > > * Add the following directive to context.xml : cachingAllowed="false" /> Okay. Why context.xml, by the way? >>> I don't even know (yet...) why this setting was added in the >>> first place in the environment where it was present... ! so why >>> this file... I don't know either :) > >> DefaultServlet is assuming caching is in place. If you disable it, >> you hit this issue. > > * Create a large file in the samples webapp, for example : > cd webapps/examples dd if=/dev/zero of=large.txt bs=1k > count=200 > >> > Reading the code for FileResource.getContent, it's clear that the entire file is being loaded into memory, which obviously isn't going to work, here. I'm wondering why that's happening since streaming is the correct behavior when caching=false. Also strange is that DefaultServlet will attempt to call FileResource.getContent() -- which returns a byte[] -- and, if that returns null, it will call FileResource.getInputStream which ... calls this.getContent. So this looks like a special-case for FileResource just trying to implement that interface in the simplest way possible. > >> It is assuming it is working with a CachedResource instance rather >> than directly with a FileResource instance. > FileResource seems to implement in-memory caching whether it's enabled or not. I can't understand why this doesn't fail for the other kind of connector. Everything else is the same? You have two separate connectors in one instance, or are you changing the connector between tests? >>> >>> Everything is exactly the same as I have only one instance with >>> two separate connectors (AJP+HTTP). > >> I suspect HTTP avoids it because sendfile is enabled. > >> The DefaultServlet logic needs a little refactoring. > > And maybe FileResource, too? > > I wasn't able to follow the logic of whether caching or not caching > was enabled. I only did cursory checking, but it seemed like none of > the resources implementations included any caching-aware code at all. > Was I looking in the wrong place? Don't think so. When caching is enabled everything gets wrapped in CachedResource. > If the resources are caching-aware, then I think the DefaultServlet > can just always use Resource.getInputStream. > > Hmm. That might cause a lot of unnecessary IO if the bytes are > actually available. That is a very tempting solution. The result is a LOT cleaner than the patch I just wrote. CachingResource is smart enough to cache the bytes and wrap them in a ByteArrayInputStream if Resource.getInputStream is called. My only concern is I think this introduces and additional copy of the data. I need to check that. > Maybe when caching is disabled, we need to wrap resources in an > UncachedResource object which always returns null from getContent() > and forces the use of an InputStream? My instinct is that would be too much but I'll keep it in mind in case I end up in a logic hole that that digs me out of. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 4/25/19 15:55, Mark Thomas wrote: > On 23/04/2019 16:29, Olivier Jaquemet wrote: >> On 23/04/2019 16:12, Christopher Schultz wrote: >>> On 4/23/19 05:58, Olivier Jaquemet wrote: > > > * Add the following directive to context.xml : >>> cachingAllowed="false" /> >>> Okay. Why context.xml, by the way? >> I don't even know (yet...) why this setting was added in the >> first place in the environment where it was present... ! so why >> this file... I don't know either :) > > DefaultServlet is assuming caching is in place. If you disable it, > you hit this issue. > * Create a large file in the samples webapp, for example : cd webapps/examples dd if=/dev/zero of=large.txt bs=1k count=200 > > > >>> Reading the code for FileResource.getContent, it's clear that >>> the entire file is being loaded into memory, which obviously >>> isn't going to work, here. I'm wondering why that's happening >>> since streaming is the correct behavior when caching=false. >>> Also strange is that DefaultServlet will attempt to call >>> FileResource.getContent() -- which returns a byte[] -- and, if >>> that returns null, it will call FileResource.getInputStream >>> which ... calls this.getContent. So this looks like a >>> special-case for FileResource just trying to implement that >>> interface in the simplest way possible. > > It is assuming it is working with a CachedResource instance rather > than directly with a FileResource instance. > >>> FileResource seems to implement in-memory caching whether it's >>> enabled or not. >>> >>> I can't understand why this doesn't fail for the other kind of >>> connector. Everything else is the same? You have two separate >>> connectors in one instance, or are you changing the connector >>> between tests? >> >> Everything is exactly the same as I have only one instance with >> two separate connectors (AJP+HTTP). > > I suspect HTTP avoids it because sendfile is enabled. > > The DefaultServlet logic needs a little refactoring. And maybe FileResource, too? I wasn't able to follow the logic of whether caching or not caching was enabled. I only did cursory checking, but it seemed like none of the resources implementations included any caching-aware code at all. Was I looking in the wrong place? If the resources are caching-aware, then I think the DefaultServlet can just always use Resource.getInputStream. Hmm. That might cause a lot of unnecessary IO if the bytes are actually available. Maybe when caching is disabled, we need to wrap resources in an UncachedResource object which always returns null from getContent() and forces the use of an InputStream? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlzCFbkACgkQHPApP6U8 pFjfng//YwlhPYoJLxXA1QOr9wQ3QvXusXqoqExQ7cQVlv9Z+in7Xl4/qHp65QBp wlAGqxGCvxbltQCiBuMmIoK3Lz8GEJcgAl+Rt7FCDPC22yfJ/dCQjFEgAQ8+BftD VohdH+7fJxP+KF2P29qh3Yb041VDC6L1NkkERUOf+4mDKA/B0RnmhuW/QVRCpB5a 6Ola0yR8VP6zfjubA+PkUjDtXD4nw0vIQ5MbvV3pqIIPE5bA0+GfgbslTbgNiPNZ u8tQlHJzUuvEUexzWCN9f6Ltu0pO8l1ovp4djP5CMLuY1PUm/ZUIkJw1wuB8Qft4 ByX/i5VzOPXxzbfw2cSboNe5PKhHk1LOokLNWB0UDfPsHWJI9Ef0G0mr5neljUGY uu9tAFG/G6G1LkglLRNVlBajmXi3wMy9I73l2Pkj7lE45kRwv+z185IobG/6034x ocX8u8UMNAmi4dWSc51x4PaXYTaI7lH9jmrPsyYDq0AHhrd5RxVT1Q9KT2sJexY2 6Qf/6mOXqzUf6H5aBrloOiVOcMOrtYRJG5mRK+UuoaaN+zngvubhlQIVHOa+8eP/ kDtKsQctLoLafvVWpTRsy7wMpOcfD2LkyDDhVI9QpwJtigQwvjuqGRUc+iITvejX +tIoaqqwoHb0MoHPHdQI+0XStCxEa6N5oK3EtbjhwKpAaZnsuiU= =drNE -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
On 23/04/2019 16:29, Olivier Jaquemet wrote: > On 23/04/2019 16:12, Christopher Schultz wrote: >> On 4/23/19 05:58, Olivier Jaquemet wrote: >>> * Add the following directive to context.xml : >> cachingAllowed="false" /> >> Okay. Why context.xml, by the way? > I don't even know (yet...) why this setting was added in the first place > in the environment where it was present... ! > so why this file... I don't know either :) DefaultServlet is assuming caching is in place. If you disable it, you hit this issue. >>> * Create a large file in the samples webapp, for example : cd >>> webapps/examples dd if=/dev/zero of=large.txt bs=1k count=200 >> Reading the code for FileResource.getContent, it's clear that the >> entire file is being loaded into memory, which obviously isn't going >> to work, here. I'm wondering why that's happening since streaming is >> the correct behavior when caching=false. Also strange is that >> DefaultServlet will attempt to call FileResource.getContent() -- which >> returns a byte[] -- and, if that returns null, it will call >> FileResource.getInputStream which ... calls this.getContent. So this >> looks like a special-case for FileResource just trying to implement >> that interface in the simplest way possible. It is assuming it is working with a CachedResource instance rather than directly with a FileResource instance. >> FileResource seems to implement in-memory caching whether it's enabled >> or not. >> >> I can't understand why this doesn't fail for the other kind of >> connector. Everything else is the same? You have two separate >> connectors in one instance, or are you changing the connector between >> tests? > > Everything is exactly the same as I have only one instance with two > separate connectors (AJP+HTTP). I suspect HTTP avoids it because sendfile is enabled. The DefaultServlet logic needs a little refactoring. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
On 23/04/2019 16:12, Christopher Schultz wrote: Olivier, Hi Christopher, Thanks for you answer. On 4/23/19 05:58, Olivier Jaquemet wrote: Hi all, We were able to reproduce a OutOfMemory error when using AJP and the Resources cachingAllowed=false directive. It looks like a bug of AJP connector(s), as it does not occurs with other HTTP connectors. Could you confirm the behavior described below is indeed bug ? (if you do, I'll create the issue on bugzilla) To reproduce : * Use latest tomcat 8.5 version (tested with Tomcat 8.5.40) * Add an AJP connector to server.xml nb: no compression nb: NIO connector is in use ; no APR (see stack trace for thread name) * Add the following directive to context.xml : Okay. Why context.xml, by the way? I don't even know (yet...) why this setting was added in the first place in the environment where it was present... ! so why this file... I don't know either :) * Create a large file in the samples webapp, for example : cd webapps/examples dd if=/dev/zero of=large.txt bs=1k count=200 ~2GiB static file * Start Tomcat with a 1024 mb heap size (JAVA_OPTS="-Xms1024m -Xmx1024m" * Configure Apache HTTPD to use mod_proxy_ajp, or mod_jk (both will have the same issue) [1] * Start Apache HTTPD * Download file through default HTTP connector http://localhost:8080/examples/large.txt --> OK * Download file through Apache/AJP http://localhost/examples/large.txt --> BUG : OutOfMemory error occurs Exception in thread "ajp-nio-8009-exec-10" java.lang.OutOfMemoryError: Java heap space at org.apache.catalina.webresources.FileResource.getContent(FileResource. java:207) at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServl et.java:992) at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java: 438) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ... That's ... interesting. What if you request the file more than once via the default connector? I would have expected these code paths to be relatively the same. There are no problems at all if I request the file many times in // using the default HTTP connector. Can you generate a heap dump when the OOME occurs? Use the following JVM startup options: - -XX:+HeapDumpOnOutOfMemoryError - -XX:HeapDumpPath=/path/to/heap-dump-file Sure. Here are heap dumps and log file from two different run : https://www.dropbox.com/sh/q6iqpe42fxvsvin/AAARR4uOnn-qJ4PCg8e_Ll8La?dl=0 If you generate a heap dump, are you able to navigate it? I've heard that Eclipse MAT is good. I've mostly used YourKit and it's quite good, able to pick-out the "big things" and help locate what kind of object(s) is/are taking up all the space. I've used MAT some times before, and indeed it's a really nice tool ! However, if you are familiar with YourKit, you might find the culprit faster than me... Can you have a look a the heap dump I shared above ? I did have a look at it with MAT... however I could not find any definitive conclusion ... There is a BufferedWriter of an AccessLogValve which reference a large char array with \0 (like the content of the large generated file...). There are many (leaking?) java.util.zip.ZipFile$ZipFileInflaterInputStream (however you said there were no compression, though here they are) ... Reading the code for FileResource.getContent, it's clear that the entire file is being loaded into memory, which obviously isn't going to work, here. I'm wondering why that's happening since streaming is the correct behavior when caching=false. Also strange is that DefaultServlet will attempt to call FileResource.getContent() -- which returns a byte[] -- and, if that returns null, it will call FileResource.getInputStream which ... calls this.getContent. So this looks like a special-case for FileResource just trying to implement that interface in the simplest way possible. FileResource seems to implement in-memory caching whether it's enabled or not. I can't understand why this doesn't fail for the other kind of connector. Everything else is the same? You have two separate connectors in one instance, or are you changing the connector between tests? Everything is exactly the same as I have only one instance with two separate connectors (AJP+HTTP). One last (confusing) information in form of an exception, that I could *not* reproduce our test environments, but that I saw on the server where the symptom first occurred. I don't think this is related to the OOM, but it might be another symptom of the same resource configuration. java.lang.ArrayIndexOutOfBoundsException: Unable to return [/path/to/file/being/downloaded.ext] as a byte array since the resource is [2,637,615,704] bytes in size which is larger than the maximum size of a byte array at org.apache.catalina.webresources.FileResource.getContent(FileResource.java:196) at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:1000) at
Re: OutOfMemory on large file download with AJP and cachingAllowed=false
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Olivier, On 4/23/19 05:58, Olivier Jaquemet wrote: > Hi all, > > We were able to reproduce a OutOfMemory error when using AJP and > the Resources cachingAllowed=false directive. It looks like a bug > of AJP connector(s), as it does not occurs with other HTTP > connectors. > > Could you confirm the behavior described below is indeed bug ? (if > you do, I'll create the issue on bugzilla) > > To reproduce : > > * Use latest tomcat 8.5 version (tested with Tomcat 8.5.40) * Add > an AJP connector to server.xml protocol="AJP/1.3" redirectPort="8443" /> nb: no compression nb: NIO connector is in use ; no APR (see stack trace for thread name) > * Add the following directive to context.xml : cachingAllowed="false" /> Okay. Why context.xml, by the way? > * Create a large file in the samples webapp, for example : cd > webapps/examples dd if=/dev/zero of=large.txt bs=1k count=200 ~2GiB static file > * Start Tomcat with a 1024 mb heap size (JAVA_OPTS="-Xms1024m > -Xmx1024m" * Configure Apache HTTPD to use mod_proxy_ajp, or mod_jk > (both will have the same issue) [1] * Start Apache HTTPD * Download > file through default HTTP connector > http://localhost:8080/examples/large.txt --> OK * Download file > through Apache/AJP http://localhost/examples/large.txt --> BUG : > OutOfMemory error occurs Exception in thread > "ajp-nio-8009-exec-10" java.lang.OutOfMemoryError: Java heap space > at > > org.apache.catalina.webresources.FileResource.getContent(FileResource. java:207) > > at > > org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServl et.java:992) > > at > > org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java: 438) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) > ... That's ... interesting. What if you request the file more than once via the default connector? I would have expected these code paths to be relatively the same. Can you generate a heap dump when the OOME occurs? Use the following JVM startup options: - -XX:+HeapDumpOnOutOfMemoryError - -XX:HeapDumpPath=/path/to/heap-dump-file If you generate a heap dump, are you able to navigate it? I've heard that Eclipse MAT is good. I've mostly used YourKit and it's quite good, able to pick-out the "big things" and help locate what kind of object(s) is/are taking up all the space. Reading the code for FileResource.getContent, it's clear that the entire file is being loaded into memory, which obviously isn't going to work, here. I'm wondering why that's happening since streaming is the correct behavior when caching=false. Also strange is that DefaultServlet will attempt to call FileResource.getContent() -- which returns a byte[] -- and, if that returns null, it will call FileResource.getInputStream which ... calls this.getContent. So this looks like a special-case for FileResource just trying to implement that interface in the simplest way possible. FileResource seems to implement in-memory caching whether it's enabled or not. I can't understand why this doesn't fail for the other kind of connector. Everything else is the same? You have two separate connectors in one instance, or are you changing the connector between tests? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAly/HVsACgkQHPApP6U8 pFgkLg//SGrPQ86VAyamI2nK0Ai67C1Y33+pBqEqlPP4UzPUu8cjnQd8cigvGqrv HvgJJJrsDEA5XRpS7I2Xw/rUMHHyXjalarCHr0dHQdYbL108HLDlXSzhOBD/viA+ PZIeMjWAlcpZfRk1Eg9pZtlAbzANi8X6EdbKutpkMo+6clUmAaAMLI4C3e/OInKA IqfUqqEseeS6Bdi51GZoNfm8yspWz5hEcBSoWI5H5kTQCbLZ4Wpf4D2RXFI5Cj1s rVaGG9ne47QQKa1dS5iQGBVFE8MwLWPIW7rkyIrRbVaNWd54mX18cL/hMmQuiOfd 9D3KrUm6j8TeP48OcD5LFW5uz8k92cOZ114cVlql5ndifBp1XdC2TXddGjiC4Uf8 8DhG94wGIiv6U51v1XEtF9u7/b4d8UMsEsqpzZ/EcN85Bs5ZHrzGzDrhgq/SrsC3 tutVODV78FWM5daxvilFhzXAtbaF+wqFzakGIeHQ8QefQFR9rEPvMKxrrMKJgvM2 2DLNNvKUmBZICYDEahzHTbcdaELVRwbQwO5cElk1rcBp66EhG4DY0wgvimeQbnjA iPjMG2VCSIXsVghV0qgLTPk8B8S00i/e18TYl46ciJpTvLE5tleDS/kRb75oj3Hm aWOZ+qAAYpAiP1vsUFyHQfWYBIqfK7h7fRjruWDfBAtabbPRRyg= =VhGf -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
OutOfMemory on large file download with AJP and cachingAllowed=false
Hi all, We were able to reproduce a OutOfMemory error when using AJP and the Resources cachingAllowed=false directive. It looks like a bug of AJP connector(s), as it does not occurs with other HTTP connectors. Could you confirm the behavior described below is indeed bug ? (if you do, I'll create the issue on bugzilla) To reproduce : * Use latest tomcat 8.5 version (tested with Tomcat 8.5.40) * Add an AJP connector to server.xml * Add the following directive to context.xml : * Create a large file in the samples webapp, for example : cd webapps/examples dd if=/dev/zero of=large.txt bs=1k count=200 * Start Tomcat with a 1024 mb heap size (JAVA_OPTS="-Xms1024m -Xmx1024m" * Configure Apache HTTPD to use mod_proxy_ajp, or mod_jk (both will have the same issue) [1] * Start Apache HTTPD * Download file through default HTTP connector http://localhost:8080/examples/large.txt --> OK * Download file through Apache/AJP http://localhost/examples/large.txt --> BUG : OutOfMemory error occurs Exception in thread "ajp-nio-8009-exec-10" java.lang.OutOfMemoryError: Java heap space at org.apache.catalina.webresources.FileResource.getContent(FileResource.java:207) at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:992) at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:438) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ... Additionnal informations : * The bug could be reproduced on both Linux/Windows * Tested with latest OpenJDK 8. Regards, Olivier [1] the HTTPD configuration : * if using mod_proxy_ajp (sudo a2enmod proxy proxy_ajp) ProxyPass / ajp://127.0.0.1:8009/ * if using mod_jk : (sudo a2enmod jk) JkWorkerProperty worker.list=tomcat JkWorkerProperty worker.tomcat.type=ajp13 JkWorkerProperty worker.tomcat.host=localhost JkWorkerProperty worker.tomcat.port=8009 JkMount / tomcat JkMount /* tomcat