RE: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
You are right. Actually, my concern is if useAsyncIO="true" gives better results in general then should it be good idea to wait for that as we are still assessing the HTTP/2 behavior in context of our application. Thanks, Kedar -Original Message- From: Mark Thomas Sent: Thursday, June 17, 2021 2:32 AM To: users@tomcat.apache.org Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client) On 16/06/2021 19:42, Deshmukh, Kedar wrote: > Thanks Mark for the quick update. > > Can you please provide how useAsyncIO="false" makes impact in terms of > performance, scalability (number of connections to the server) and > reliability ? Well, if you set useAsyncIO="false" it works. If you set useAsyncIO="true" it fails 90% of the time. I'd suggest that, with that failure rate, discussion of performance and scalability are somewhat irrelevant. With test cases that do not trigger this issue the difference between the two is marginal. Remy is more familiar with the details than me but from memory useAsyncIO="true" is a little better on older JREs. On modern JREs there isn't much in it. It is also likely to depend on application usage patterns, OS, etc. In short, you'd need to test the difference on your application with your hardware etc. I'd expect the difference to be hard to measure. Mark > > Regards, > Kedar > > > -Original Message- > From: Mark Thomas > Sent: Wednesday, June 16, 2021 11:41 PM > To: users@tomcat.apache.org > Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer > (server -> client) > > On 16/06/2021 18:47, Rémy Maucherat wrote: >> On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: >> >>> On 16/06/2021 18:01, Deshmukh, Kedar wrote: >>> I have one additional question at this point. How easy is this issue to >>> reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it >>> is reproducible when concurrency is 5 or more and file sizes are >>> between 1GB-5GB. >>> >>> Thanks for the confirmation. I have converted your test classes into >>> a Tomcat unit test (easy for me to work with) and the issue looks to >>> be repeatable on Linux with the latest 10.1.x code. >>> >>> I'm starting to look at this now. I'll post again when I have more >>> information. I'm not expecting this to be quick. > > Kedar, > > If you set useAsyncIO="false" on the Connector that should work around the > problem for now. The Servlet Async API will still be available. > Tomcat just uses a different code path to write to the network. > >> I did not expect it would be so easy to reproduce. Can you commit the test ? > > It is a bit of a hack at the moment. The code isn't particularly clean and I > have hard-coded some file paths for my system (I have a bunch of 5GB Windows > MSDN ISOs I am using for the large files. I also don't think we want test > cases that using multi-GB files running on every test run. > > If I clean things up a bit, parameterise the hard-coded paths bits and > disable the test by default it should be in a reasonable state to commit. > > It looks very much like the vectoredOperation and the associated semaphore is > where things are going wrong. > > I'm aiming to work on this some more tomorrow. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
Mark, On 6/16/21 17:02, Mark Thomas wrote: On 16/06/2021 19:42, Deshmukh, Kedar wrote: Thanks Mark for the quick update. Can you please provide how useAsyncIO="false" makes impact in terms of performance, scalability (number of connections to the server) and reliability ? Well, if you set useAsyncIO="false" it works. If you set useAsyncIO="true" it fails 90% of the time. I'd suggest that, with that failure rate, discussion of performance and scalability are somewhat irrelevant. Funny, but 100% true. -chris -Original Message- From: Mark Thomas Sent: Wednesday, June 16, 2021 11:41 PM To: users@tomcat.apache.org Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client) On 16/06/2021 18:47, Rémy Maucherat wrote: On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: On 16/06/2021 18:01, Deshmukh, Kedar wrote: I have one additional question at this point. How easy is this issue to reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it is reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB. Thanks for the confirmation. I have converted your test classes into a Tomcat unit test (easy for me to work with) and the issue looks to be repeatable on Linux with the latest 10.1.x code. I'm starting to look at this now. I'll post again when I have more information. I'm not expecting this to be quick. Kedar, If you set useAsyncIO="false" on the Connector that should work around the problem for now. The Servlet Async API will still be available. Tomcat just uses a different code path to write to the network. I did not expect it would be so easy to reproduce. Can you commit the test ? It is a bit of a hack at the moment. The code isn't particularly clean and I have hard-coded some file paths for my system (I have a bunch of 5GB Windows MSDN ISOs I am using for the large files. I also don't think we want test cases that using multi-GB files running on every test run. If I clean things up a bit, parameterise the hard-coded paths bits and disable the test by default it should be in a reasonable state to commit. It looks very much like the vectoredOperation and the associated semaphore is where things are going wrong. I'm aiming to work on this some more tomorrow. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: CATALINA_OPTS vs JAVA_OPTS
Thanks Chris! Dream * Excel * Explore * Inspire Jon McAlexander Infrastructure Engineer Asst Vice President Middleware Product Engineering Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@wellsfargo.com Upcoming PTO: 10/30/2020, 11/6/2020, 11/13/2020, 11/20/2020, 11/27/2020, 12/2/2020, 12/4/2020, 12/11/2020, 12/18/2020, 12/28/2020, 12/29/2020, 12/30/2020, 12/31/2020 This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. > -Original Message- > From: Christopher Schultz > Sent: Wednesday, June 16, 2021 3:48 PM > To: users@tomcat.apache.org > Subject: Re: CATALINA_OPTS vs JAVA_OPTS > Importance: Low > > Jon, > > On 6/16/21 14:31, jonmcalexan...@wellsfargo.com.INVALID wrote: > > Ok, so this is a really good explanation. However, when setting up in > > Windows as a Service, does the JAVA_OPTIONS in the Registry go in as > > JAVA_OPTS or CATALINA_OPTS? Is there a way to have separate > > CATALINA_OPTS for Tomcat Windows Services? > > It's more complicated than you are making it seem. > > If you use service.bat to create your Windows Service definition, then the > effective CATALINA_OPTS and JAVA_OPTS environment variable values > within the shell at the time you call bin\service.bat will have the intended > effect: they will combine and set the values which go into the registry for > that Service. When you launch the service, the (current) values of > CATALINA_OPTS and JAVA_OPTS, if they exist, are completely ignored > because the Service definition captured them at creation-time. > > If you want to alter them, you can either set their values again, and > delete/re-create the service using bin\service.bat or you can use > tomcatXw.exe to edit the service values directly. > > You have to remember that Windows, while supporting environment > variables nearly through-and-through, essentially encourages all self- > respecting Windows admins to ignore those environment variables. > Everything is done through the registry and so the Tomcat config scripts > attempt to bridge the gap between the gray beard UNIX admins who are > used to things like CATALINA_OPTS and JAVA_OPTS and whatnot and the > Windows admins who have an entirely different mental-model of system > services. > > It's a neat little hack IMHO: let the CLI warriors set environment variables > and > create a Service. The Service captures those values into the more > Windowsey paradigm, and off you go. And none of the Windows admins > have to bother themselves with environment variables. > > -chris > > >> -Original Message- From: Christopher Schultz > >> Sent: Wednesday, June 16, 2021 11:14 > >> AM To: users@tomcat.apache.org Subject: Re: CATALINA_OPTS vs > >> JAVA_OPTS > >> > >> Noelette, > >> > >> On 6/16/21 11:29, Noelette Stout wrote: > >>> Thanks! I was mostly trying to figure out if there was precedence or > >>> if it was additive (i.e. 2GB to tomcat itself and another 2GB to the > >>> apps). We're having some resource issues on one of our servers, so I > >>> wanted to make sure I understood how the resources were being > >> allocated. > >> > >> No additivity at all: the last one on the command-line wins. There is > >> no heap separation between Tomcat and the applications: it's > >> one(ish) big, happy heap. :) > >> > >> A note about CATALINA_OPTS versus JAVA_OPTS: when you use the > various > >> scripts provided by Tomcat, CATALINA_OPTS is only used when launching > >> a Tomcat instance. JAVA_OPTS is used when launching *any* Java > >> process. There are many Java processes those scripts will launch that > >> aren't actually launching Tomcat. Examples include: > >> > >> 1. catalina.sh configtest 2. catalina.sh stop (also shutdown.sh) 3. > >> catalina.sh version 4. tool-wrapper.sh [anything] > >> > >> In all of those cases, JAVA_OPTS will be passed to the JVM. > >> > >> Do you really need a 2 gig heap to send a "shutdown" command to a > >> running server? Probably not. > >> > >> -chris > >> > >>> On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent > >> wrote: > >>> > > > On 6/16/21 9:06 AM, Noelette Stout wrote: > > openjdk version "1.8.0_292" > > > > > > On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > > > wrote: > > > > > Both as for the same minimum so you should get 2G at start up. > I'm not sure which has precedency but I would be on java opt. > I don't have a catalina env, but you can see how CATALINA_OPTS is > used in relationship with JAVA_OPTS > > >
[ANN] Apache Tomcat 8.5.68 available
The Apache Tomcat team announces the immediate availability of Apache Tomcat 8.5.68. Apache Tomcat 8 is an open source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and Java Authentication Service Provider Interface for Containers technologies. Apache Tomcat 8.5.68 is a bugfix and feature release. (Please note that 8.5.67 was not released due to a problem with artifact-signing which was corrected in this 8.5.68 release). The notable changes compared to 8.5.66 include: - Improve robustness of HTTP/2 HPACK decoding - Improvements to the handling of the Transfer-Encoding header - Review code used to generate Java source from JSPs and tags and remove code found to be unnecessary. Please refer to the change log for the complete list of changes: http://tomcat.apache.org/tomcat-8.5-doc/changelog.html Downloads: http://tomcat.apache.org/download-80.cgi Migration guides from Apache Tomcat 7.x and 8.0.x: http://tomcat.apache.org/migration.html Enjoy! - The Apache Tomcat team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
On 16/06/2021 19:42, Deshmukh, Kedar wrote: Thanks Mark for the quick update. Can you please provide how useAsyncIO="false" makes impact in terms of performance, scalability (number of connections to the server) and reliability ? Well, if you set useAsyncIO="false" it works. If you set useAsyncIO="true" it fails 90% of the time. I'd suggest that, with that failure rate, discussion of performance and scalability are somewhat irrelevant. With test cases that do not trigger this issue the difference between the two is marginal. Remy is more familiar with the details than me but from memory useAsyncIO="true" is a little better on older JREs. On modern JREs there isn't much in it. It is also likely to depend on application usage patterns, OS, etc. In short, you'd need to test the difference on your application with your hardware etc. I'd expect the difference to be hard to measure. Mark Regards, Kedar -Original Message- From: Mark Thomas Sent: Wednesday, June 16, 2021 11:41 PM To: users@tomcat.apache.org Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client) On 16/06/2021 18:47, Rémy Maucherat wrote: On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: On 16/06/2021 18:01, Deshmukh, Kedar wrote: I have one additional question at this point. How easy is this issue to reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it is reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB. Thanks for the confirmation. I have converted your test classes into a Tomcat unit test (easy for me to work with) and the issue looks to be repeatable on Linux with the latest 10.1.x code. I'm starting to look at this now. I'll post again when I have more information. I'm not expecting this to be quick. Kedar, If you set useAsyncIO="false" on the Connector that should work around the problem for now. The Servlet Async API will still be available. Tomcat just uses a different code path to write to the network. I did not expect it would be so easy to reproduce. Can you commit the test ? It is a bit of a hack at the moment. The code isn't particularly clean and I have hard-coded some file paths for my system (I have a bunch of 5GB Windows MSDN ISOs I am using for the large files. I also don't think we want test cases that using multi-GB files running on every test run. If I clean things up a bit, parameterise the hard-coded paths bits and disable the test by default it should be in a reasonable state to commit. It looks very much like the vectoredOperation and the associated semaphore is where things are going wrong. I'm aiming to work on this some more tomorrow. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [ANN] Apache Tomcat 8.5.68 available
All, Apologies for the repeated messages. I'm getting bounces from the cross-posted announce@ mailing lists because I'm not emailing from @apache.org source. I think I just need one last try... -chris On 6/16/21 16:53, Christopher Schultz wrote: The Apache Tomcat team announces the immediate availability of Apache Tomcat 8.5.68. Apache Tomcat 8 is an open source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and Java Authentication Service Provider Interface for Containers technologies. Apache Tomcat 8.5.68 is a bugfix and feature release. (Please note that 8.5.67 was not released due to a problem with artifact-signing which was corrected in this 8.5.68 release). The notable changes compared to 8.5.66 include: - Improve robustness of HTTP/2 HPACK decoding - Improvements to the handling of the Transfer-Encoding header - Review code used to generate Java source from JSPs and tags and remove code found to be unnecessary. Please refer to the change log for the complete list of changes: http://tomcat.apache.org/tomcat-8.5-doc/changelog.html Downloads: http://tomcat.apache.org/download-80.cgi Migration guides from Apache Tomcat 7.x and 8.0.x: http://tomcat.apache.org/migration.html Enjoy! - The Apache Tomcat team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
On 16/06/2021 21:52, Christopher Schultz wrote: Mark, On 6/16/21 14:10, Mark Thomas wrote: On 16/06/2021 18:47, Rémy Maucherat wrote: On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: On 16/06/2021 18:01, Deshmukh, Kedar wrote: I have one additional question at this point. How easy is this issue to reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it is reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB. Thanks for the confirmation. I have converted your test classes into a Tomcat unit test (easy for me to work with) and the issue looks to be repeatable on Linux with the latest 10.1.x code. I'm starting to look at this now. I'll post again when I have more information. I'm not expecting this to be quick. Kedar, If you set useAsyncIO="false" on the Connector that should work around the problem for now. The Servlet Async API will still be available. Tomcat just uses a different code path to write to the network. I did not expect it would be so easy to reproduce. Can you commit the test ? It is a bit of a hack at the moment. The code isn't particularly clean and I have hard-coded some file paths for my system (I have a bunch of 5GB Windows MSDN ISOs I am using for the large files. I also don't think we want test cases that using multi-GB files running on every test run. If I clean things up a bit, parameterise the hard-coded paths bits and disable the test by default it should be in a reasonable state to commit. It looks very much like the vectoredOperation and the associated semaphore is where things are going wrong. I'm aiming to work on this some more tomorrow. Is it inadvisable to use a trivial JSP or Servlet that just generates X bytes? Or does this require the use of the DefaultServlet at the moment? Right now, I don't know. Is it still possible to reproduce using smaller window sizes and smaller total resource sizes? It would be nice if a unit-test didn't have to transfer 5GiB even through the loopback interface. I may have a better answer after I do some further investigation tomorrow. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
[ANN] Apache Tomcat 8.5.68 available
The Apache Tomcat team announces the immediate availability of Apache Tomcat 8.5.68. Apache Tomcat 8 is an open source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and Java Authentication Service Provider Interface for Containers technologies. Apache Tomcat 8.5.68 is a bugfix and feature release. (Please note that 8.5.67 was not released due to a problem with artifact-signing which was corrected in this 8.5.68 release). The notable changes compared to 8.5.66 include: - Improve robustness of HTTP/2 HPACK decoding - Improvements to the handling of the Transfer-Encoding header - Review code used to generate Java source from JSPs and tags and remove code found to be unnecessary. Please refer to the change log for the complete list of changes: http://tomcat.apache.org/tomcat-8.5-doc/changelog.html Downloads: http://tomcat.apache.org/download-80.cgi Migration guides from Apache Tomcat 7.x and 8.0.x: http://tomcat.apache.org/migration.html Enjoy! - The Apache Tomcat team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CATALINA_OPTS vs JAVA_OPTS
And to expand on one other point, the shutdown mechanism is different for the Windows Service (assuming you are using default jvm mode). You don't have to worry about creating a whole new Java process just to send the shutdown command to the shutdown port. Mark On 16/06/2021 21:47, Christopher Schultz wrote: Jon, On 6/16/21 14:31, jonmcalexan...@wellsfargo.com.INVALID wrote: Ok, so this is a really good explanation. However, when setting up in Windows as a Service, does the JAVA_OPTIONS in the Registry go in as JAVA_OPTS or CATALINA_OPTS? Is there a way to have separate CATALINA_OPTS for Tomcat Windows Services? It's more complicated than you are making it seem. If you use service.bat to create your Windows Service definition, then the effective CATALINA_OPTS and JAVA_OPTS environment variable values within the shell at the time you call bin\service.bat will have the intended effect: they will combine and set the values which go into the registry for that Service. When you launch the service, the (current) values of CATALINA_OPTS and JAVA_OPTS, if they exist, are completely ignored because the Service definition captured them at creation-time. If you want to alter them, you can either set their values again, and delete/re-create the service using bin\service.bat or you can use tomcatXw.exe to edit the service values directly. You have to remember that Windows, while supporting environment variables nearly through-and-through, essentially encourages all self-respecting Windows admins to ignore those environment variables. Everything is done through the registry and so the Tomcat config scripts attempt to bridge the gap between the gray beard UNIX admins who are used to things like CATALINA_OPTS and JAVA_OPTS and whatnot and the Windows admins who have an entirely different mental-model of system services. It's a neat little hack IMHO: let the CLI warriors set environment variables and create a Service. The Service captures those values into the more Windowsey paradigm, and off you go. And none of the Windows admins have to bother themselves with environment variables. -chris -Original Message- From: Christopher Schultz Sent: Wednesday, June 16, 2021 11:14 AM To: users@tomcat.apache.org Subject: Re: CATALINA_OPTS vs JAVA_OPTS Noelette, On 6/16/21 11:29, Noelette Stout wrote: Thanks! I was mostly trying to figure out if there was precedence or if it was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). We're having some resource issues on one of our servers, so I wanted to make sure I understood how the resources were being allocated. No additivity at all: the last one on the command-line wins. There is no heap separation between Tomcat and the applications: it's one(ish) big, happy heap. :) A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various scripts provided by Tomcat, CATALINA_OPTS is only used when launching a Tomcat instance. JAVA_OPTS is used when launching *any* Java process. There are many Java processes those scripts will launch that aren't actually launching Tomcat. Examples include: 1. catalina.sh configtest 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. tool-wrapper.sh [anything] In all of those cases, JAVA_OPTS will be passed to the JVM. Do you really need a 2 gig heap to send a "shutdown" command to a running server? Probably not. -chris On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent wrote: On 6/16/21 9:06 AM, Noelette Stout wrote: openjdk version "1.8.0_292" On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: Both as for the same minimum so you should get 2G at start up. I'm not sure which has precedency but I would be on java opt. I don't have a catalina env, but you can see how CATALINA_OPTS is used in relationship with JAVA_OPTS - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
Mark, On 6/16/21 14:10, Mark Thomas wrote: On 16/06/2021 18:47, Rémy Maucherat wrote: On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: On 16/06/2021 18:01, Deshmukh, Kedar wrote: I have one additional question at this point. How easy is this issue to reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it is reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB. Thanks for the confirmation. I have converted your test classes into a Tomcat unit test (easy for me to work with) and the issue looks to be repeatable on Linux with the latest 10.1.x code. I'm starting to look at this now. I'll post again when I have more information. I'm not expecting this to be quick. Kedar, If you set useAsyncIO="false" on the Connector that should work around the problem for now. The Servlet Async API will still be available. Tomcat just uses a different code path to write to the network. I did not expect it would be so easy to reproduce. Can you commit the test ? It is a bit of a hack at the moment. The code isn't particularly clean and I have hard-coded some file paths for my system (I have a bunch of 5GB Windows MSDN ISOs I am using for the large files. I also don't think we want test cases that using multi-GB files running on every test run. If I clean things up a bit, parameterise the hard-coded paths bits and disable the test by default it should be in a reasonable state to commit. It looks very much like the vectoredOperation and the associated semaphore is where things are going wrong. I'm aiming to work on this some more tomorrow. Is it inadvisable to use a trivial JSP or Servlet that just generates X bytes? Or does this require the use of the DefaultServlet at the moment? Is it still possible to reproduce using smaller window sizes and smaller total resource sizes? It would be nice if a unit-test didn't have to transfer 5GiB even through the loopback interface. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CATALINA_OPTS vs JAVA_OPTS
Jon, On 6/16/21 14:31, jonmcalexan...@wellsfargo.com.INVALID wrote: Ok, so this is a really good explanation. However, when setting up in Windows as a Service, does the JAVA_OPTIONS in the Registry go in as JAVA_OPTS or CATALINA_OPTS? Is there a way to have separate CATALINA_OPTS for Tomcat Windows Services? It's more complicated than you are making it seem. If you use service.bat to create your Windows Service definition, then the effective CATALINA_OPTS and JAVA_OPTS environment variable values within the shell at the time you call bin\service.bat will have the intended effect: they will combine and set the values which go into the registry for that Service. When you launch the service, the (current) values of CATALINA_OPTS and JAVA_OPTS, if they exist, are completely ignored because the Service definition captured them at creation-time. If you want to alter them, you can either set their values again, and delete/re-create the service using bin\service.bat or you can use tomcatXw.exe to edit the service values directly. You have to remember that Windows, while supporting environment variables nearly through-and-through, essentially encourages all self-respecting Windows admins to ignore those environment variables. Everything is done through the registry and so the Tomcat config scripts attempt to bridge the gap between the gray beard UNIX admins who are used to things like CATALINA_OPTS and JAVA_OPTS and whatnot and the Windows admins who have an entirely different mental-model of system services. It's a neat little hack IMHO: let the CLI warriors set environment variables and create a Service. The Service captures those values into the more Windowsey paradigm, and off you go. And none of the Windows admins have to bother themselves with environment variables. -chris -Original Message- From: Christopher Schultz Sent: Wednesday, June 16, 2021 11:14 AM To: users@tomcat.apache.org Subject: Re: CATALINA_OPTS vs JAVA_OPTS Noelette, On 6/16/21 11:29, Noelette Stout wrote: Thanks! I was mostly trying to figure out if there was precedence or if it was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). We're having some resource issues on one of our servers, so I wanted to make sure I understood how the resources were being allocated. No additivity at all: the last one on the command-line wins. There is no heap separation between Tomcat and the applications: it's one(ish) big, happy heap. :) A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various scripts provided by Tomcat, CATALINA_OPTS is only used when launching a Tomcat instance. JAVA_OPTS is used when launching *any* Java process. There are many Java processes those scripts will launch that aren't actually launching Tomcat. Examples include: 1. catalina.sh configtest 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. tool-wrapper.sh [anything] In all of those cases, JAVA_OPTS will be passed to the JVM. Do you really need a 2 gig heap to send a "shutdown" command to a running server? Probably not. -chris On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent wrote: On 6/16/21 9:06 AM, Noelette Stout wrote: openjdk version "1.8.0_292" On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: Both as for the same minimum so you should get 2G at start up. I'm not sure which has precedency but I would be on java opt. I don't have a catalina env, but you can see how CATALINA_OPTS is used in relationship with JAVA_OPTS - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CATALINA_OPTS vs JAVA_OPTS
Noelette, On 6/16/21 13:31, Noelette Stout wrote: ok, one more question for clarification :-) The vendor docs say this: "Define CATALINA_OPTS to configure the following JVM settings: CATALINA_OPTS=-server -Xms2048m -Xmx4g -XX:MaxMetaspaceSize=512m Note:If you are deploying multiple [vendor] applications to the same Tomcat server, -Xmx must be increased by 2g and -XX:MaxMetaspaceSize must be increased by 128m." Given that they are (ostensibly, based on the above statement) JVM settings, would these settings be better placed under JAVA_OPTS instead of CATALINA_OPTS? As others have said, no, you still want CATALINA_OPTS. I guess I wasn't absolutely clear before. Both CATALINA_OPTS *and* JAVA_OPTS will be used when launching Tomcat. Other things those scripts do will only use JAVA_OPTS. So you only want CATALINA_OPTS to include the things that you *really* want to affect Tomcat itself. If you set JAVA_OPTS to -Xms5g then you will get a 5 GiB heap allocated to do nothing more than this simple CLI command: $ echo SHUTDOWN | nc localhost:8005 CATALINA_OPTS is essentially JAVA_OPTS for when you are starting Tomcat, but not the other stuff (like shutting it down). -chris On Wed, Jun 16, 2021 at 10:29 AM Noelette Stout wrote: This confirms something that I was thinking originally, which is that we have no good reason to have both in setenv.sh. I'm pretty sure that these were both mentioned in various pieces of documentation from the application vendor. Their documentation tends to be less than stellar. Combine bad documentation with app admins that are completely unfamiliar with tomcat... and you get messy setups. Unfortunately, there are more of them than me, so I have to have ALL of my facts straight before I go into battle. Thanks for this additional information. It is much appreciated. On Wed, Jun 16, 2021 at 10:14 AM Christopher Schultz < ch...@christopherschultz.net> wrote: Noelette, On 6/16/21 11:29, Noelette Stout wrote: Thanks! I was mostly trying to figure out if there was precedence or if it was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). We're having some resource issues on one of our servers, so I wanted to make sure I understood how the resources were being allocated. No additivity at all: the last one on the command-line wins. There is no heap separation between Tomcat and the applications: it's one(ish) big, happy heap. :) A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various scripts provided by Tomcat, CATALINA_OPTS is only used when launching a Tomcat instance. JAVA_OPTS is used when launching *any* Java process. There are many Java processes those scripts will launch that aren't actually launching Tomcat. Examples include: 1. catalina.sh configtest 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. tool-wrapper.sh [anything] In all of those cases, JAVA_OPTS will be passed to the JVM. Do you really need a 2 gig heap to send a "shutdown" command to a running server? Probably not. -chris On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent wrote: On 6/16/21 9:06 AM, Noelette Stout wrote: openjdk version "1.8.0_292" On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: Both as for the same minimum so you should get 2G at start up. I'm not sure which has precedency but I would be on java opt. I don't have a catalina env, but you can see how CATALINA_OPTS is used in relationship with JAVA_OPTS - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: CATALINA_OPTS vs JAVA_OPTS
Yes, but ultimately it is running Java, so was curious. Is there even a need of both when running as a Windows service? These are probably "duh" questions, but just want to fully "get it". Thanks, Dream * Excel * Explore * Inspire Jon McAlexander Infrastructure Engineer Asst Vice President Middleware Product Engineering Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@wellsfargo.com Upcoming PTO: 10/30/2020, 11/6/2020, 11/13/2020, 11/20/2020, 11/27/2020, 12/2/2020, 12/4/2020, 12/11/2020, 12/18/2020, 12/28/2020, 12/29/2020, 12/30/2020, 12/31/2020 This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. > -Original Message- > From: Olaf Kock > Sent: Wednesday, June 16, 2021 1:45 PM > To: users@tomcat.apache.org > Subject: Re: CATALINA_OPTS vs JAVA_OPTS > > Service configurations are service configurations. You won't run the other > options as service, so those are for the JVM that is used for the service. And > I'm not aware that a service is stopped as the command line version. At least > I'd hope so - a standard JVM would be good enough, if the start/stop > mechanism not handled in the windows executable anyway. > > Note: This is dangerous half-knowledge. I haven't been on Windows for > ages, just read about it still being around from time to time. > > CATALINA_OPTS is utilized by the startup batch/script file, which is not used > for services /at all/. > > Olaf > > On 16.06.21 20:31, jonmcalexan...@wellsfargo.com.INVALID wrote: > > Ok, so this is a really good explanation. However, when setting up in > Windows as a Service, does the JAVA_OPTIONS in the Registry go in as > JAVA_OPTS or CATALINA_OPTS? Is there a way to have separate > CATALINA_OPTS for Tomcat Windows Services? > > > > Thanks, > > > > Dream * Excel * Explore * Inspire > > Jon McAlexander > > Infrastructure Engineer > > Asst Vice President > > > > Middleware Product Engineering > > Enterprise CIO | Platform Services | Middleware | Infrastructure > > Solutions > > > > 8080 Cobblestone Rd | Urbandale, IA 50322 > > MAC: F4469-010 > > Tel 515-988-2508 | Cell 515-988-2508 > > > > jonmcalexan...@wellsfargo.com > > > > Upcoming PTO: 10/30/2020, 11/6/2020, 11/13/2020, 11/20/2020, > > 11/27/2020, 12/2/2020, 12/4/2020, 12/11/2020, 12/18/2020, 12/28/2020, > 12/29/2020, 12/30/2020, 12/31/2020 This message may contain confidential > and/or privileged information. If you are not the addressee or authorized to > receive this for the addressee, you must not use, copy, disclose, or take any > action based on this message or any information herein. If you have received > this message in error, please advise the sender immediately by reply e-mail > and delete this message. Thank you for your cooperation. > > > >> -Original Message- > >> From: Christopher Schultz > >> Sent: Wednesday, June 16, 2021 11:14 AM > >> To: users@tomcat.apache.org > >> Subject: Re: CATALINA_OPTS vs JAVA_OPTS > >> > >> Noelette, > >> > >> On 6/16/21 11:29, Noelette Stout wrote: > >>> Thanks! I was mostly trying to figure out if there was precedence or > >>> if it was additive (i.e. 2GB to tomcat itself and another 2GB to the > >>> apps). We're having some resource issues on one of our servers, so I > >>> wanted to make sure I understood how the resources were being > >> allocated. > >> > >> No additivity at all: the last one on the command-line wins. There is > >> no heap separation between Tomcat and the applications: it's one(ish) > >> big, happy heap. :) > >> > >> A note about CATALINA_OPTS versus JAVA_OPTS: when you use the > various > >> scripts provided by Tomcat, CATALINA_OPTS is only used when launching > >> a Tomcat instance. JAVA_OPTS is used when launching *any* Java > process. > >> There are many Java processes those scripts will launch that aren't > >> actually launching Tomcat. Examples include: > >> > >> 1. catalina.sh configtest > >> 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. > >> tool-wrapper.sh [anything] > >> > >> In all of those cases, JAVA_OPTS will be passed to the JVM. > >> > >> Do you really need a 2 gig heap to send a "shutdown" command to a > >> running server? Probably not. > >> > >> -chris > >> > >>> On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent > >> wrote: > > On 6/16/21 9:06 AM, Noelette Stout wrote: > > openjdk version "1.8.0_292" > > > > > > On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > > > wrote: > > > Both as for the same minimum so you should get 2G at start up. I'm >
Re: CATALINA_OPTS vs JAVA_OPTS
Service configurations are service configurations. You won't run the other options as service, so those are for the JVM that is used for the service. And I'm not aware that a service is stopped as the command line version. At least I'd hope so - a standard JVM would be good enough, if the start/stop mechanism not handled in the windows executable anyway. Note: This is dangerous half-knowledge. I haven't been on Windows for ages, just read about it still being around from time to time. CATALINA_OPTS is utilized by the startup batch/script file, which is not used for services /at all/. Olaf On 16.06.21 20:31, jonmcalexan...@wellsfargo.com.INVALID wrote: > Ok, so this is a really good explanation. However, when setting up in Windows > as a Service, does the JAVA_OPTIONS in the Registry go in as JAVA_OPTS or > CATALINA_OPTS? Is there a way to have separate CATALINA_OPTS for Tomcat > Windows Services? > > Thanks, > > Dream * Excel * Explore * Inspire > Jon McAlexander > Infrastructure Engineer > Asst Vice President > > Middleware Product Engineering > Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions > > 8080 Cobblestone Rd | Urbandale, IA 50322 > MAC: F4469-010 > Tel 515-988-2508 | Cell 515-988-2508 > > jonmcalexan...@wellsfargo.com > > Upcoming PTO: 10/30/2020, 11/6/2020, 11/13/2020, 11/20/2020, 11/27/2020, > 12/2/2020, 12/4/2020, 12/11/2020, 12/18/2020, 12/28/2020, 12/29/2020, > 12/30/2020, 12/31/2020 > This message may contain confidential and/or privileged information. If you > are not the addressee or authorized to receive this for the addressee, you > must not use, copy, disclose, or take any action based on this message or any > information herein. If you have received this message in error, please advise > the sender immediately by reply e-mail and delete this message. Thank you for > your cooperation. > >> -Original Message- >> From: Christopher Schultz >> Sent: Wednesday, June 16, 2021 11:14 AM >> To: users@tomcat.apache.org >> Subject: Re: CATALINA_OPTS vs JAVA_OPTS >> >> Noelette, >> >> On 6/16/21 11:29, Noelette Stout wrote: >>> Thanks! I was mostly trying to figure out if there was precedence or >>> if it was additive (i.e. 2GB to tomcat itself and another 2GB to the >>> apps). We're having some resource issues on one of our servers, so I >>> wanted to make sure I understood how the resources were being >> allocated. >> >> No additivity at all: the last one on the command-line wins. There is no heap >> separation between Tomcat and the applications: it's one(ish) big, happy >> heap. :) >> >> A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various >> scripts provided by Tomcat, CATALINA_OPTS is only used when launching a >> Tomcat instance. JAVA_OPTS is used when launching *any* Java process. >> There are many Java processes those scripts will launch that aren't actually >> launching Tomcat. Examples include: >> >> 1. catalina.sh configtest >> 2. catalina.sh stop (also shutdown.sh) >> 3. catalina.sh version >> 4. tool-wrapper.sh [anything] >> >> In all of those cases, JAVA_OPTS will be passed to the JVM. >> >> Do you really need a 2 gig heap to send a "shutdown" command to a running >> server? Probably not. >> >> -chris >> >>> On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent >> wrote: On 6/16/21 9:06 AM, Noelette Stout wrote: > openjdk version "1.8.0_292" > > > On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: > Both as for the same minimum so you should get 2G at start up. I'm not sure which has precedency but I would be on java opt. I don't have a catalina env, but you can see how CATALINA_OPTS is used in relationship with JAVA_OPTS - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
Thanks Mark for the quick update. Can you please provide how useAsyncIO="false" makes impact in terms of performance, scalability (number of connections to the server) and reliability ? Regards, Kedar -Original Message- From: Mark Thomas Sent: Wednesday, June 16, 2021 11:41 PM To: users@tomcat.apache.org Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client) On 16/06/2021 18:47, Rémy Maucherat wrote: > On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: > >> On 16/06/2021 18:01, Deshmukh, Kedar wrote: >> >>> I have one additional question at this point. How easy is this issue >>> to >> reproduce? Does it happen every time? In 10% of requests? 1% ? >>> >>> [Kedar] It is reproducible 9/10 times in my environment. So 90% time >>> it >> is reproducible when concurrency is 5 or more and file sizes are >> between 1GB-5GB. >> >> Thanks for the confirmation. I have converted your test classes into >> a Tomcat unit test (easy for me to work with) and the issue looks to >> be repeatable on Linux with the latest 10.1.x code. >> >> I'm starting to look at this now. I'll post again when I have more >> information. I'm not expecting this to be quick. Kedar, If you set useAsyncIO="false" on the Connector that should work around the problem for now. The Servlet Async API will still be available. Tomcat just uses a different code path to write to the network. > I did not expect it would be so easy to reproduce. Can you commit the test ? It is a bit of a hack at the moment. The code isn't particularly clean and I have hard-coded some file paths for my system (I have a bunch of 5GB Windows MSDN ISOs I am using for the large files. I also don't think we want test cases that using multi-GB files running on every test run. If I clean things up a bit, parameterise the hard-coded paths bits and disable the test by default it should be in a reasonable state to commit. It looks very much like the vectoredOperation and the associated semaphore is where things are going wrong. I'm aiming to work on this some more tomorrow. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [Possible Spam] Re: CATALINA_OPTS vs JAVA_OPTS
While on the subject of CATALINA_OPTS. In 9.0.46, the catalina.sh script displays these when it runs. I have several options in the CATALINA_OPTS, which makes this pretty long. I don't remember it being this way in the past. I would imagine that if this was changed to be like this, there was a good reason, but it makes my startup process look broken. Would having this go to the catalina.out file be a good solution? Or it is critical for others that it is output to stdout? If this is critical, should JAVA_OPTS also be displayed to stdout? - ( In the setenv.sh file, we use enough options that we use line continuation to make the file more readable which makes the CATALINA_OPTS line less readable because of the whitespace. -Xms128M \ -Xmx512M \ ... ) Mark Claassen Senior Software Engineer Donnell Systems, Inc. 130 South Main Street Leighton Plaza Suite 375 South Bend, IN 46601 E-mail: mailto:mclaas...@ocie.net Voice: (574)232-3784 Fax: (574)232-4014 Disclaimer: The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting. -Original Message- From: Noelette Stout Sent: Wednesday, June 16, 2021 1:47 PM To: Tomcat Users List Subject: [Possible Spam] Re: CATALINA_OPTS vs JAVA_OPTS Importance: Low Thanks! That clears things up a lot. I think my admins got confused and did both because there are some instructions that say to add these in JAVA_OPTS, but those instructions are specifically for Windows servers (we're on RHEL). Thanks everyone for all the info and help. On Wed, Jun 16, 2021 at 11:42 AM Mark Thomas wrote: > On 16/06/2021 18:31, Noelette Stout wrote: > > ok, one more question for clarification :-) The vendor docs say > > this: > > "Define CATALINA_OPTS to configure the following JVM settings: > > CATALINA_OPTS=-server -Xms2048m -Xmx4g -XX:MaxMetaspaceSize=512m > > Note:If you are deploying multiple [vendor] applications to the same > Tomcat > > server, -Xmx must be increased by 2g and -XX:MaxMetaspaceSize must > > be increased by 128m." > > > > Given that they are (ostensibly, based on the above statement) JVM > > settings, would these settings be better placed under JAVA_OPTS > > instead > of > > CATALINA_OPTS? > > No. CATALINA_OPTS. > > 99 times out of a 100 you want to put any JVM settings in CATALINA_OPTS. > > Mark > > > > > > > On Wed, Jun 16, 2021 at 10:29 AM Noelette Stout > > > > wrote: > > > >> This confirms something that I was thinking originally, which is > >> that we have no good reason to have both in setenv.sh. I'm pretty > >> sure that > these > >> were both mentioned in various pieces of documentation from the > application > >> vendor. Their documentation tends to be less than stellar. Combine > >> bad documentation with app admins that are completely unfamiliar > >> with > tomcat... > >> and you get messy setups. Unfortunately, there are more of them > >> than > me, so > >> I have to have ALL of my facts straight before I go into battle. > >> > >> Thanks for this additional information. It is much appreciated. > >> > >> On Wed, Jun 16, 2021 at 10:14 AM Christopher Schultz < > >> ch...@christopherschultz.net> wrote: > >> > >>> Noelette, > >>> > >>> On 6/16/21 11:29, Noelette Stout wrote: > Thanks! I was mostly trying to figure out if there was precedence > or > if > >>> it > was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). > >>> We're > having some resource issues on one of our servers, so I wanted to > make > >>> sure > I understood how the resources were being allocated. > >>> > >>> No additivity at all: the last one on the command-line wins. There > >>> is > no > >>> heap separation between Tomcat and the applications: it's one(ish) > >>> big, happy heap. :) > >>> > >>> A note about CATALINA_OPTS versus JAVA_OPTS: when you use the > >>> various scripts provided by Tomcat, CATALINA_OPTS is only used > >>> when launching a Tomcat instance. JAVA_OPTS is used when launching *any* > >>> Java process. > >>> There are many Java processes those scripts will launch that > >>> aren't actually launching Tomcat. Examples include: > >>> > >>> 1. catalina.sh configtest > >>> 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. > >>> tool-wrapper.sh [anything] > >>> > >>> In all of those cases, JAVA_OPTS will be passed to the JVM. > >>> > >>> Do you really need a 2 gig heap to send a "shutdown" command to a > >>> running server? Probably not. > >>> > >>> -chris > >>> > On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent > > >>> wrote: > > > > > > > On 6/16/21 9:06 AM, Noelette Stout wrote: > >> openjdk version "1.8.0_292" > >> > >> > >> On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > >> > > wrote: > >> > >> > > Both as for the same minimum so you should get 2G at
RE: CATALINA_OPTS vs JAVA_OPTS
Ok, so this is a really good explanation. However, when setting up in Windows as a Service, does the JAVA_OPTIONS in the Registry go in as JAVA_OPTS or CATALINA_OPTS? Is there a way to have separate CATALINA_OPTS for Tomcat Windows Services? Thanks, Dream * Excel * Explore * Inspire Jon McAlexander Infrastructure Engineer Asst Vice President Middleware Product Engineering Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@wellsfargo.com Upcoming PTO: 10/30/2020, 11/6/2020, 11/13/2020, 11/20/2020, 11/27/2020, 12/2/2020, 12/4/2020, 12/11/2020, 12/18/2020, 12/28/2020, 12/29/2020, 12/30/2020, 12/31/2020 This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. > -Original Message- > From: Christopher Schultz > Sent: Wednesday, June 16, 2021 11:14 AM > To: users@tomcat.apache.org > Subject: Re: CATALINA_OPTS vs JAVA_OPTS > > Noelette, > > On 6/16/21 11:29, Noelette Stout wrote: > > Thanks! I was mostly trying to figure out if there was precedence or > > if it was additive (i.e. 2GB to tomcat itself and another 2GB to the > > apps). We're having some resource issues on one of our servers, so I > > wanted to make sure I understood how the resources were being > allocated. > > No additivity at all: the last one on the command-line wins. There is no heap > separation between Tomcat and the applications: it's one(ish) big, happy > heap. :) > > A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various > scripts provided by Tomcat, CATALINA_OPTS is only used when launching a > Tomcat instance. JAVA_OPTS is used when launching *any* Java process. > There are many Java processes those scripts will launch that aren't actually > launching Tomcat. Examples include: > > 1. catalina.sh configtest > 2. catalina.sh stop (also shutdown.sh) > 3. catalina.sh version > 4. tool-wrapper.sh [anything] > > In all of those cases, JAVA_OPTS will be passed to the JVM. > > Do you really need a 2 gig heap to send a "shutdown" command to a running > server? Probably not. > > -chris > > > On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent > wrote: > > > >> > >> > >> On 6/16/21 9:06 AM, Noelette Stout wrote: > >>> openjdk version "1.8.0_292" > >>> > >>> > >>> On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > >> wrote: > >>> > >>> > >> Both as for the same minimum so you should get 2G at start up. I'm > >> not sure which has precedency but I would be on java opt. I don't > >> have a catalina env, but you can see how CATALINA_OPTS is used in > >> relationship with JAVA_OPTS > >> > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
On 16/06/2021 18:47, Rémy Maucherat wrote: On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: On 16/06/2021 18:01, Deshmukh, Kedar wrote: I have one additional question at this point. How easy is this issue to reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it is reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB. Thanks for the confirmation. I have converted your test classes into a Tomcat unit test (easy for me to work with) and the issue looks to be repeatable on Linux with the latest 10.1.x code. I'm starting to look at this now. I'll post again when I have more information. I'm not expecting this to be quick. Kedar, If you set useAsyncIO="false" on the Connector that should work around the problem for now. The Servlet Async API will still be available. Tomcat just uses a different code path to write to the network. I did not expect it would be so easy to reproduce. Can you commit the test ? It is a bit of a hack at the moment. The code isn't particularly clean and I have hard-coded some file paths for my system (I have a bunch of 5GB Windows MSDN ISOs I am using for the large files. I also don't think we want test cases that using multi-GB files running on every test run. If I clean things up a bit, parameterise the hard-coded paths bits and disable the test by default it should be in a reasonable state to commit. It looks very much like the vectoredOperation and the associated semaphore is where things are going wrong. I'm aiming to work on this some more tomorrow. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CATALINA_OPTS vs JAVA_OPTS
Thanks! That clears things up a lot. I think my admins got confused and did both because there are some instructions that say to add these in JAVA_OPTS, but those instructions are specifically for Windows servers (we're on RHEL). Thanks everyone for all the info and help. On Wed, Jun 16, 2021 at 11:42 AM Mark Thomas wrote: > On 16/06/2021 18:31, Noelette Stout wrote: > > ok, one more question for clarification :-) > > The vendor docs say this: > > "Define CATALINA_OPTS to configure the following JVM settings: > > CATALINA_OPTS=-server -Xms2048m -Xmx4g -XX:MaxMetaspaceSize=512m > > Note:If you are deploying multiple [vendor] applications to the same > Tomcat > > server, -Xmx must be increased by 2g and -XX:MaxMetaspaceSize must be > > increased by 128m." > > > > Given that they are (ostensibly, based on the above statement) JVM > > settings, would these settings be better placed under JAVA_OPTS instead > of > > CATALINA_OPTS? > > No. CATALINA_OPTS. > > 99 times out of a 100 you want to put any JVM settings in CATALINA_OPTS. > > Mark > > > > > > > On Wed, Jun 16, 2021 at 10:29 AM Noelette Stout > > wrote: > > > >> This confirms something that I was thinking originally, which is that we > >> have no good reason to have both in setenv.sh. I'm pretty sure that > these > >> were both mentioned in various pieces of documentation from the > application > >> vendor. Their documentation tends to be less than stellar. Combine bad > >> documentation with app admins that are completely unfamiliar with > tomcat... > >> and you get messy setups. Unfortunately, there are more of them than > me, so > >> I have to have ALL of my facts straight before I go into battle. > >> > >> Thanks for this additional information. It is much appreciated. > >> > >> On Wed, Jun 16, 2021 at 10:14 AM Christopher Schultz < > >> ch...@christopherschultz.net> wrote: > >> > >>> Noelette, > >>> > >>> On 6/16/21 11:29, Noelette Stout wrote: > Thanks! I was mostly trying to figure out if there was precedence or > if > >>> it > was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). > >>> We're > having some resource issues on one of our servers, so I wanted to make > >>> sure > I understood how the resources were being allocated. > >>> > >>> No additivity at all: the last one on the command-line wins. There is > no > >>> heap separation between Tomcat and the applications: it's one(ish) big, > >>> happy heap. :) > >>> > >>> A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various > >>> scripts provided by Tomcat, CATALINA_OPTS is only used when launching a > >>> Tomcat instance. JAVA_OPTS is used when launching *any* Java process. > >>> There are many Java processes those scripts will launch that aren't > >>> actually launching Tomcat. Examples include: > >>> > >>> 1. catalina.sh configtest > >>> 2. catalina.sh stop (also shutdown.sh) > >>> 3. catalina.sh version > >>> 4. tool-wrapper.sh [anything] > >>> > >>> In all of those cases, JAVA_OPTS will be passed to the JVM. > >>> > >>> Do you really need a 2 gig heap to send a "shutdown" command to a > >>> running server? Probably not. > >>> > >>> -chris > >>> > On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent > >>> wrote: > > > > > > > On 6/16/21 9:06 AM, Noelette Stout wrote: > >> openjdk version "1.8.0_292" > >> > >> > >> On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > > wrote: > >> > >> > > Both as for the same minimum so you should get 2G at start up. I'm > not > > sure which has precedency but I would be on java opt. I don't have a > > catalina env, but you can see how CATALINA_OPTS is used in > relationship > > with JAVA_OPTS > > > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > > >>> > >>> - > >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>> > >>> > >> > >> -- > >> Noelette Stout > >> > >> ITS Enterprise Applications - Application Administrator - Senior > >> > >> Business Administration Building, Rm 109L > >> 921 South 8th Ave 8037 > >> Idaho State University > >> Pocatello ID 83209 > >> E-mail: stounoel "at" isu "dot" edu > >> Desk: 208-282-2554 > >> > > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
On Wed, Jun 16, 2021 at 7:36 PM Mark Thomas wrote: > On 16/06/2021 18:01, Deshmukh, Kedar wrote: > > > I have one additional question at this point. How easy is this issue to > reproduce? Does it happen every time? In 10% of requests? 1% ? > > > > [Kedar] It is reproducible 9/10 times in my environment. So 90% time it > is reproducible when concurrency is 5 or more and file sizes are between > 1GB-5GB. > > Thanks for the confirmation. I have converted your test classes into a > Tomcat unit test (easy for me to work with) and the issue looks to be > repeatable on Linux with the latest 10.1.x code. > > I'm starting to look at this now. I'll post again when I have more > information. I'm not expecting this to be quick. > I did not expect it would be so easy to reproduce. Can you commit the test ? Rémy
Re: CATALINA_OPTS vs JAVA_OPTS
On 16/06/2021 18:31, Noelette Stout wrote: ok, one more question for clarification :-) The vendor docs say this: "Define CATALINA_OPTS to configure the following JVM settings: CATALINA_OPTS=-server -Xms2048m -Xmx4g -XX:MaxMetaspaceSize=512m Note:If you are deploying multiple [vendor] applications to the same Tomcat server, -Xmx must be increased by 2g and -XX:MaxMetaspaceSize must be increased by 128m." Given that they are (ostensibly, based on the above statement) JVM settings, would these settings be better placed under JAVA_OPTS instead of CATALINA_OPTS? No. CATALINA_OPTS. 99 times out of a 100 you want to put any JVM settings in CATALINA_OPTS. Mark On Wed, Jun 16, 2021 at 10:29 AM Noelette Stout wrote: This confirms something that I was thinking originally, which is that we have no good reason to have both in setenv.sh. I'm pretty sure that these were both mentioned in various pieces of documentation from the application vendor. Their documentation tends to be less than stellar. Combine bad documentation with app admins that are completely unfamiliar with tomcat... and you get messy setups. Unfortunately, there are more of them than me, so I have to have ALL of my facts straight before I go into battle. Thanks for this additional information. It is much appreciated. On Wed, Jun 16, 2021 at 10:14 AM Christopher Schultz < ch...@christopherschultz.net> wrote: Noelette, On 6/16/21 11:29, Noelette Stout wrote: Thanks! I was mostly trying to figure out if there was precedence or if it was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). We're having some resource issues on one of our servers, so I wanted to make sure I understood how the resources were being allocated. No additivity at all: the last one on the command-line wins. There is no heap separation between Tomcat and the applications: it's one(ish) big, happy heap. :) A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various scripts provided by Tomcat, CATALINA_OPTS is only used when launching a Tomcat instance. JAVA_OPTS is used when launching *any* Java process. There are many Java processes those scripts will launch that aren't actually launching Tomcat. Examples include: 1. catalina.sh configtest 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. tool-wrapper.sh [anything] In all of those cases, JAVA_OPTS will be passed to the JVM. Do you really need a 2 gig heap to send a "shutdown" command to a running server? Probably not. -chris On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent wrote: On 6/16/21 9:06 AM, Noelette Stout wrote: openjdk version "1.8.0_292" On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: Both as for the same minimum so you should get 2G at start up. I'm not sure which has precedency but I would be on java opt. I don't have a catalina env, but you can see how CATALINA_OPTS is used in relationship with JAVA_OPTS - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
On 16/06/2021 18:01, Deshmukh, Kedar wrote: I have one additional question at this point. How easy is this issue to reproduce? Does it happen every time? In 10% of requests? 1% ? [Kedar] It is reproducible 9/10 times in my environment. So 90% time it is reproducible when concurrency is 5 or more and file sizes are between 1GB-5GB. Thanks for the confirmation. I have converted your test classes into a Tomcat unit test (easy for me to work with) and the issue looks to be repeatable on Linux with the latest 10.1.x code. I'm starting to look at this now. I'll post again when I have more information. I'm not expecting this to be quick. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CATALINA_OPTS vs JAVA_OPTS
ok, one more question for clarification :-) The vendor docs say this: "Define CATALINA_OPTS to configure the following JVM settings: CATALINA_OPTS=-server -Xms2048m -Xmx4g -XX:MaxMetaspaceSize=512m Note:If you are deploying multiple [vendor] applications to the same Tomcat server, -Xmx must be increased by 2g and -XX:MaxMetaspaceSize must be increased by 128m." Given that they are (ostensibly, based on the above statement) JVM settings, would these settings be better placed under JAVA_OPTS instead of CATALINA_OPTS? On Wed, Jun 16, 2021 at 10:29 AM Noelette Stout wrote: > This confirms something that I was thinking originally, which is that we > have no good reason to have both in setenv.sh. I'm pretty sure that these > were both mentioned in various pieces of documentation from the application > vendor. Their documentation tends to be less than stellar. Combine bad > documentation with app admins that are completely unfamiliar with tomcat... > and you get messy setups. Unfortunately, there are more of them than me, so > I have to have ALL of my facts straight before I go into battle. > > Thanks for this additional information. It is much appreciated. > > On Wed, Jun 16, 2021 at 10:14 AM Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> Noelette, >> >> On 6/16/21 11:29, Noelette Stout wrote: >> > Thanks! I was mostly trying to figure out if there was precedence or if >> it >> > was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). >> We're >> > having some resource issues on one of our servers, so I wanted to make >> sure >> > I understood how the resources were being allocated. >> >> No additivity at all: the last one on the command-line wins. There is no >> heap separation between Tomcat and the applications: it's one(ish) big, >> happy heap. :) >> >> A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various >> scripts provided by Tomcat, CATALINA_OPTS is only used when launching a >> Tomcat instance. JAVA_OPTS is used when launching *any* Java process. >> There are many Java processes those scripts will launch that aren't >> actually launching Tomcat. Examples include: >> >> 1. catalina.sh configtest >> 2. catalina.sh stop (also shutdown.sh) >> 3. catalina.sh version >> 4. tool-wrapper.sh [anything] >> >> In all of those cases, JAVA_OPTS will be passed to the JVM. >> >> Do you really need a 2 gig heap to send a "shutdown" command to a >> running server? Probably not. >> >> -chris >> >> > On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent >> wrote: >> > >> >> >> >> >> >> On 6/16/21 9:06 AM, Noelette Stout wrote: >> >>> openjdk version "1.8.0_292" >> >>> >> >>> >> >>> On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent >> >> wrote: >> >>> >> >>> >> >> Both as for the same minimum so you should get 2G at start up. I'm not >> >> sure which has precedency but I would be on java opt. I don't have a >> >> catalina env, but you can see how CATALINA_OPTS is used in relationship >> >> with JAVA_OPTS >> >> >> >> >> >> - >> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >> >> >> > >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > -- > Noelette Stout > > ITS Enterprise Applications - Application Administrator - Senior > > Business Administration Building, Rm 109L > 921 South 8th Ave 8037 > Idaho State University > Pocatello ID 83209 > E-mail: stounoel "at" isu "dot" edu > Desk: 208-282-2554 > -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
RE: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
-Original Message- From: Mark Thomas Sent: Wednesday, June 16, 2021 9:29 PM To: users@tomcat.apache.org Subject: Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client) On 16/06/2021 15:05, Deshmukh, Kedar wrote: > Dear Tomcat users/dev team, > > We are understanding the impact of HTTP/2 in our application as HTTP/2 > provides better throughput and performance. I'd be wary of making such sweeping statements. HTTP/2 has some advantages and some disadvantages. Generally, the advantages will outweigh the disadvantages but that will not always be the case. [Kedar] - Okay. > Before directly tuning > HTTP/2 in application, we thought of analyzing certain use cases which > our application demands in standalone environment. > > Our use case is very simple. Java based standalone client is making > simple POST request to the server and server read the file name from > the request and push the requested file to client. Here, client can > request multiple files same time by sending multiple requests > concurrently, so server should be able to send multiple files > concurrently depending on configured concurrency level. > > Currently, in this test only single client is making requests to the > server with concurrency 5. Server is not overloaded and not performing > any other tasks. Machine has more than 500GB empty space and not > running any heavy applications. > > Test: > > We used different set of files for this test. Files with sizes between > 1GB - 5GB and concurrency > 5. We are using traditional connector > protocol HTTP11NIOProtocol with HTTP/2 is turned on. > > Observations: > > HTTP/1.1 - With HTTP/1.1 given sample code works fine. Only drawback > here is it opens multiple TCP connections to satisfy HTTP/1.1 > > HTTP/2 - With HTTP/2, it is expected to be only one TCP connection and > multiple streams to handle the traffic. Tomcat HTTP/2 debug logs > suggest that only one connection being used and multiple streams are > spawned as expected. So far everything is fine. But sample code does > not work consistently with higher concurrency (> 3). We captured the > stack trace of tomcat process which is attached here. Couple of tomcat > threads are waiting to acquire semaphore for socket write operation. > When write operation is stuck servlet is not able to push any data to > client and client is also stuck waiting for more data. I don't see any > error/exception at the client/server. That looks / sounds like there is a code path - probably an error condition ? - where the semaphore isn't released. [Kedar] Semaphore never released. As mentioned in sample code concurrency level is 5 so If you observe attached tomcat stack trace, you would find following very similar 4 threads which are waiting on semaphore which was acquired by other thread "Thread-9". "Thread-7" #58 daemon prio=5 os_prio=0 cpu=218.75ms elapsed=105.35s tid=0x01844997a800 nid=0x4bc0 waiting on condition [0x0027cc7fe000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method) - parking to wait for <0x0006158518b0> (a java.util.concurrent.Semaphore$NonfairSync) at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.10/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(java.base@11.0.10/AbstractQueuedSynchronizer.java:1079) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(java.base@11.0.10/AbstractQueuedSynchronizer.java:1369) at java.util.concurrent.Semaphore.tryAcquire(java.base@11.0.10/Semaphore.java:415) at org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1426) "Thread-9" #59 daemon prio=5 os_prio=0 cpu=187.50ms elapsed=105.35s tid=0x01844997b800 nid=0x4b48 in Object.wait() [0x0027cc9fe000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting on <0x00061ed36de0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState) at org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1457) The other possibility is related to the HTTP/2 flow control windows. If something goes wrong with the management of the flow control window for the connection it would block everything. [Kedar] - Okay. But I did not see anything in the logs or WINDOWS_UPDATE frame related errors. > streamReadTimeout and streamWriteTimeout are configured as -1 so they > are infinitely waiting for the write semaphore. That is generally a bad idea. By all means set it high but an infinite timeout is going to cause problems - particularly if clients just drop off the network. [Kedar] Okay. I will look into this, however I don't think this change would make impact on the
Re: CATALINA_OPTS vs JAVA_OPTS
This confirms something that I was thinking originally, which is that we have no good reason to have both in setenv.sh. I'm pretty sure that these were both mentioned in various pieces of documentation from the application vendor. Their documentation tends to be less than stellar. Combine bad documentation with app admins that are completely unfamiliar with tomcat... and you get messy setups. Unfortunately, there are more of them than me, so I have to have ALL of my facts straight before I go into battle. Thanks for this additional information. It is much appreciated. On Wed, Jun 16, 2021 at 10:14 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > Noelette, > > On 6/16/21 11:29, Noelette Stout wrote: > > Thanks! I was mostly trying to figure out if there was precedence or if > it > > was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). > We're > > having some resource issues on one of our servers, so I wanted to make > sure > > I understood how the resources were being allocated. > > No additivity at all: the last one on the command-line wins. There is no > heap separation between Tomcat and the applications: it's one(ish) big, > happy heap. :) > > A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various > scripts provided by Tomcat, CATALINA_OPTS is only used when launching a > Tomcat instance. JAVA_OPTS is used when launching *any* Java process. > There are many Java processes those scripts will launch that aren't > actually launching Tomcat. Examples include: > > 1. catalina.sh configtest > 2. catalina.sh stop (also shutdown.sh) > 3. catalina.sh version > 4. tool-wrapper.sh [anything] > > In all of those cases, JAVA_OPTS will be passed to the JVM. > > Do you really need a 2 gig heap to send a "shutdown" command to a > running server? Probably not. > > -chris > > > On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent > wrote: > > > >> > >> > >> On 6/16/21 9:06 AM, Noelette Stout wrote: > >>> openjdk version "1.8.0_292" > >>> > >>> > >>> On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > >> wrote: > >>> > >>> > >> Both as for the same minimum so you should get 2G at start up. I'm not > >> sure which has precedency but I would be on java opt. I don't have a > >> catalina env, but you can see how CATALINA_OPTS is used in relationship > >> with JAVA_OPTS > >> > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
Re: CATALINA_OPTS vs JAVA_OPTS
Noelette, On 6/16/21 11:29, Noelette Stout wrote: Thanks! I was mostly trying to figure out if there was precedence or if it was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). We're having some resource issues on one of our servers, so I wanted to make sure I understood how the resources were being allocated. No additivity at all: the last one on the command-line wins. There is no heap separation between Tomcat and the applications: it's one(ish) big, happy heap. :) A note about CATALINA_OPTS versus JAVA_OPTS: when you use the various scripts provided by Tomcat, CATALINA_OPTS is only used when launching a Tomcat instance. JAVA_OPTS is used when launching *any* Java process. There are many Java processes those scripts will launch that aren't actually launching Tomcat. Examples include: 1. catalina.sh configtest 2. catalina.sh stop (also shutdown.sh) 3. catalina.sh version 4. tool-wrapper.sh [anything] In all of those cases, JAVA_OPTS will be passed to the JVM. Do you really need a 2 gig heap to send a "shutdown" command to a running server? Probably not. -chris On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent wrote: On 6/16/21 9:06 AM, Noelette Stout wrote: openjdk version "1.8.0_292" On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: Both as for the same minimum so you should get 2G at start up. I'm not sure which has precedency but I would be on java opt. I don't have a catalina env, but you can see how CATALINA_OPTS is used in relationship with JAVA_OPTS - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: client write waits on postgresql RDS
Ayub,
On 6/15/21 18:43, Ayub Khan wrote:
Chris,
below is the method responsible for returning the data to the service api,
and its returned from service to the Rest controller
public MenuList getMenuMobileListNormalizedCombo(long a, long b, boolean c,
long d) throws Exception {
Connection con = null;
CallableStatement callableStatement = null;
ResultSet rs = null;
ResultSet rs1 = null;
MenuList menuList = new MenuList();
try {
con = connectionManager.getConnetion();
con.setAutoCommit(false);
callableStatement = con.prepareCall("call
appschema.menu_pkg$get_menu_items(?,?,?,?,?,?)");
callableStatement.setLong(1, a);
if (c == 0)
callableStatement.setNull(2, Types.BIGINT);
else
callableStatement.setLong(2, b);
callableStatement.setString(3,"Y");
if(d == 0)
callableStatement.setString(4, null);
else
callableStatement.setLong(4, d);
callableStatement.setNull(5, Types.OTHER);
callableStatement.setNull(6, Types.OTHER);
callableStatement.registerOutParameter(5, Types.OTHER);
callableStatement.registerOutParameter(6, Types.OTHER);
callableStatement.setFetchSize(5);
callableStatement.execute();
rs = (ResultSet) callableStatement.getObject(5);
rs1 = (ResultSet) callableStatement.getObject(6);
MenuMobile menuMobile;
try {
while (rs.next()) {
menuMobile = new MenuMobile();
//setters removed
menuMobileList.add(menuMobile);
}
MenuCombo menuCombo;
while (rs1.next()) {
//setters removed
menuComboList.add(menuCombo);
}
menuList.setMenuComboList(menuComboList);
menuList.setMenuMobileList(menuMobileList);
} catch (SQLException e) {
LOG.error(e.getLocalizedMessage(), e);
}
con.commit();
con.setAutoCommit(true);
} catch (SQLException e) {
LOG.error(e.getLocalizedMessage(), e);
throw e;
} finally {
if (rs != null)
rs.close();
You need try/catch blocks around your close() calls, too. It's unlikely
they will fail, but they can fail, and if they do, your other
resource-management tasks will be skipped.
if (rs1 != null)
rs1.close();
if(callableStatement!=null)callableStatement.close();
if(con!=null)con.close();
}
return menuList;
}
Below is the signature of the rest controller api method
@RequestMapping(value = "/m_listAllItems", method = RequestMethod.GET,
produces = {"application/json;charset=UTF-8"})
@ResponseBody
public Object m_listAllMenus(@RequestParam("a") long a,
@RequestParam("b") String b,
@RequestParam("c") String c,
@RequestParam(value
= "d",defaultValue ="0") String d) {
Object menuList = new Object();
try {
menuList = menuManagmentService.listAll(a, b, c, d); //this
service calls the dao in the service class
} catch (Exception e) {
LOG.error(e.getLocalizedMessage(), e);
}
return menuList;
}
How long does your CALL statement take to execute in the database? How
many records is it returning? How many bytes of JSON are you generating?
Calling JSON.toString(Object) isn't free. With any luck, your framework
is streaming the JSON out to the HttpServletResponse instead of doing this:
String json = JSON.toString(menuList);
response.write(json);
But if a StringBuilder/StringBuffer/String is being guilt on the server,
it can take some time. And a bunch of memory.
-chris
On Thu, Jun 10, 2021 at 8:10 AM Ayub Khan wrote:
Seeing client write waits on postgresql as attached in the image. Is there
any bottle neck which is causing the client write waits on postgresql ?
Below is the test setup
Jmeter-->(load balanced tomcat on ec2 instances)>rds read replicas
All these are running on different ec2 instances in AWS cloud in the same
region
below is the config of the http connector on tomcat:
Below are the specs of the server:
Ec2 instance which is running tomcat 8.5
c5.9x large
36 vpcu
72GB memory
10GBPS network
EBS band width 9500
postgresql RDS db.r6g.16xlarge
512 GB memory
64 VCPU
25 Gibs network
AWS Gravitron cpu
--Ayub
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
On 16/06/2021 15:05, Deshmukh, Kedar wrote: Dear Tomcat users/dev team, We are understanding the impact of HTTP/2 in our application as HTTP/2 provides better throughput and performance. I'd be wary of making such sweeping statements. HTTP/2 has some advantages and some disadvantages. Generally, the advantages will outweigh the disadvantages but that will not always be the case. Before directly tuning HTTP/2 in application, we thought of analyzing certain use cases which our application demands in standalone environment. Our use case is very simple. Java based standalone client is making simple POST request to the server and server read the file name from the request and push the requested file to client. Here, client can request multiple files same time by sending multiple requests concurrently, so server should be able to send multiple files concurrently depending on configured concurrency level. Currently, in this test only single client is making requests to the server with concurrency 5. Server is not overloaded and not performing any other tasks. Machine has more than 500GB empty space and not running any heavy applications. Test: We used different set of files for this test. Files with sizes between 1GB – 5GB and concurrency > 5. We are using traditional connector protocol HTTP11NIOProtocol with HTTP/2 is turned on. Observations: HTTP/1.1 - With HTTP/1.1 given sample code works fine. Only drawback here is it opens multiple TCP connections to satisfy HTTP/1.1 HTTP/2 - With HTTP/2, it is expected to be only one TCP connection and multiple streams to handle the traffic. Tomcat HTTP/2 debug logs suggest that only one connection being used and multiple streams are spawned as expected. So far everything is fine. But sample code does not work consistently with higher concurrency (> 3). We captured the stack trace of tomcat process which is attached here. Couple of tomcat threads are waiting to acquire semaphore for socket write operation. When write operation is stuck servlet is not able to push any data to client and client is also stuck waiting for more data. I don’t see any error/exception at the client/server. That looks / sounds like there is a code path - probably an error condition ? - where the semaphore isn't released. The other possibility is related to the HTTP/2 flow control windows. If something goes wrong with the management of the flow control window for the connection it would block everything. streamReadTimeout and streamWriteTimeout are configured as -1 so they are infinitely waiting for the write semaphore. That is generally a bad idea. By all means set it high but an infinite timeout is going to cause problems - particularly if clients just drop off the network. Outcome of this is client is able to receive only partial data from server and at some point server stuck to send any more data. We also tried IOUtils file transfer related APIs still it didn't help. I have also tried with Async non-blocking IO but the observations are same. Generally, the simpler you keep the test case, the easier it is for us to work with. Non-async and no external IO libraries is better. Our actual requirement is very similar where java based http client would request bulk data concurrently from server and server should push that without any trouble. But, it is not limited to files only. Server can push serialized java bulk objects over the stream concurrently. The content type should not make any difference to Tomcat. Static files vs dynamic content would make a difference. Note that sample code works fine most of the time if I enable HTTP/2 logs either in client or tomcat. So I would suggest not to turn on HTTP/2 debug logs to conclude anything. That suggests a timing issue of some sort. HTTP/2 is significantly more complex than HTTP/1.1 because you have multiple independent application threads all trying to write to the same socket and Tomcat has to track and allocate flow control window allocations both for individual streams and the overall connection. Following components are used in sample code for the test 1. Client - Java 11.0.10 httpclient - (client\Client.java) 2. Server - Tomcat 9.0.46 3. Servlet - AsyncServlet - (server\Server.java) 4. Operating system - Windows 10 5. Machine specifications – 32GB RAM and 500GB open space. 5. Latency - None, client and server are running on same machine 6. Set of files - You can use any random files whose sizes are between 1GB-5GB to reproduce the issue. Refer attachment for 1. Client side code 2. Server side servlet 3. server.xml 4. Tomcat Stacktrace 5. Tomcat server logs Thanks. That is all useful information. Could you please go through sample code along with server.xml. Here are my few questions 1. Why HTTP/2 is failing for such use case where large files are concurrently pushing to the client. I believe this is a very common use case must have be
Re: CATALINA_OPTS vs JAVA_OPTS
Thanks! I was mostly trying to figure out if there was precedence or if it was additive (i.e. 2GB to tomcat itself and another 2GB to the apps). We're having some resource issues on one of our servers, so I wanted to make sure I understood how the resources were being allocated. On Wed, Jun 16, 2021 at 9:17 AM Rob Sargent wrote: > > > On 6/16/21 9:06 AM, Noelette Stout wrote: > > openjdk version "1.8.0_292" > > > > > > On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent > wrote: > > > > > Both as for the same minimum so you should get 2G at start up. I'm not > sure which has precedency but I would be on java opt. I don't have a > catalina env, but you can see how CATALINA_OPTS is used in relationship > with JAVA_OPTS > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
Re: CATALINA_OPTS vs JAVA_OPTS
openjdk version "1.8.0_292" On Wed, Jun 16, 2021 at 9:04 AM Rob Sargent wrote: > > > > On Jun 16, 2021, at 8:36 AM, Noelette Stout > wrote: > > > > If I have a setenv.sh file that contains: > > CATALINA_OPTS="-server -Xms2048m -Xmx5g" > > JAVA_OPTS="... -Xms2048m -Xmx4g .." > > > > How much memory is actually being allocated to tomcat and the > applications > > it is serving up? > > > > -- > > Noelette Stout > > Which version of Java? > > > > ITS Enterprise Applications - Application Administrator - Senior > > > > Business Administration Building, Rm 109L > > 921 South 8th Ave 8037 > > Idaho State University > > Pocatello ID 83209 > > E-mail: stounoel "at" isu "dot" edu > > Desk: 208-282-2554 > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
Re: CATALINA_OPTS vs JAVA_OPTS
> On Jun 16, 2021, at 8:36 AM, Noelette Stout wrote: > > If I have a setenv.sh file that contains: > CATALINA_OPTS="-server -Xms2048m -Xmx5g" > JAVA_OPTS="... -Xms2048m -Xmx4g .." > > How much memory is actually being allocated to tomcat and the applications > it is serving up? > > -- > Noelette Stout Which version of Java? > > ITS Enterprise Applications - Application Administrator - Senior > > Business Administration Building, Rm 109L > 921 South 8th Ave 8037 > Idaho State University > Pocatello ID 83209 > E-mail: stounoel "at" isu "dot" edu > Desk: 208-282-2554 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
CATALINA_OPTS vs JAVA_OPTS
If I have a setenv.sh file that contains: CATALINA_OPTS="-server -Xms2048m -Xmx5g" JAVA_OPTS="... -Xms2048m -Xmx4g .." How much memory is actually being allocated to tomcat and the applications it is serving up? -- Noelette Stout ITS Enterprise Applications - Application Administrator - Senior Business Administration Building, Rm 109L 921 South 8th Ave 8037 Idaho State University Pocatello ID 83209 E-mail: stounoel "at" isu "dot" edu Desk: 208-282-2554
[ANN] Apache Tomcat 8.5.68 available
The Apache Tomcat team announces the immediate availability of Apache Tomcat 8.5.68. Apache Tomcat 8 is an open source software implementation of the Java Servlet, JavaServer Pages, Java Unified Expression Language, Java WebSocket and Java Authentication Service Provider Interface for Containers technologies. Apache Tomcat 8.5.68 is a bugfix and feature release. (Please note that 8.5.67 was not released due to a problem with artifact-signing which was corrected in this 8.5.68 release). The notable changes compared to 8.5.66 include: - Improve robustness of HTTP/2 HPACK decoding - Improvements to the handling of the Transfer-Encoding header - Review code used to generate Java source from JSPs and tags and remove code found to be unnecessary. Please refer to the change log for the complete list of changes: http://tomcat.apache.org/tomcat-8.5-doc/changelog.html Downloads: http://tomcat.apache.org/download-80.cgi Migration guides from Apache Tomcat 7.x and 8.0.x: http://tomcat.apache.org/migration.html Enjoy! - The Apache Tomcat team - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Trouble with HTTP/2 during concurrent bulk data transfer (server -> client)
Dear Tomcat users/dev team, We are understanding the impact of HTTP/2 in our application as HTTP/2 provides better throughput and performance. Before directly tuning HTTP/2 in application, we thought of analyzing certain use cases which our application demands in standalone environment. Our use case is very simple. Java based standalone client is making simple POST request to the server and server read the file name from the request and push the requested file to client. Here, client can request multiple files same time by sending multiple requests concurrently, so server should be able to send multiple files concurrently depending on configured concurrency level. Currently, in this test only single client is making requests to the server with concurrency 5. Server is not overloaded and not performing any other tasks. Machine has more than 500GB empty space and not running any heavy applications. Test: We used different set of files for this test. Files with sizes between 1GB - 5GB and concurrency > 5. We are using traditional connector protocol HTTP11NIOProtocol with HTTP/2 is turned on. Observations: HTTP/1.1 - With HTTP/1.1 given sample code works fine. Only drawback here is it opens multiple TCP connections to satisfy HTTP/1.1 HTTP/2 - With HTTP/2, it is expected to be only one TCP connection and multiple streams to handle the traffic. Tomcat HTTP/2 debug logs suggest that only one connection being used and multiple streams are spawned as expected. So far everything is fine. But sample code does not work consistently with higher concurrency (> 3). We captured the stack trace of tomcat process which is attached here. Couple of tomcat threads are waiting to acquire semaphore for socket write operation. When write operation is stuck servlet is not able to push any data to client and client is also stuck waiting for more data. I don't see any error/exception at the client/server. streamReadTimeout and streamWriteTimeout are configured as -1 so they are infinitely waiting for the write semaphore. Outcome of this is client is able to receive only partial data from server and at some point server stuck to send any more data. We also tried IOUtils file transfer related APIs still it didn't help. I have also tried with Async non-blocking IO but the observations are same. Our actual requirement is very similar where java based http client would request bulk data concurrently from server and server should push that without any trouble. But, it is not limited to files only. Server can push serialized java bulk objects over the stream concurrently. Note that sample code works fine most of the time if I enable HTTP/2 logs either in client or tomcat. So I would suggest not to turn on HTTP/2 debug logs to conclude anything. Following components are used in sample code for the test 1. Client - Java 11.0.10 httpclient - (client\Client.java) 2. Server - Tomcat 9.0.46 3. Servlet - AsyncServlet - (server\Server.java) 4. Operating system - Windows 10 5. Machine specifications - 32GB RAM and 500GB open space. 5. Latency - None, client and server are running on same machine 6. Set of files - You can use any random files whose sizes are between 1GB-5GB to reproduce the issue. Refer attachment for 1. Client side code 2. Server side servlet 3. server.xml 4. Tomcat Stacktrace 5. Tomcat server logs Could you please go through sample code along with server.xml. Here are my few questions 1. Why HTTP/2 is failing for such use case where large files are concurrently pushing to the client. I believe this is a very common use case must have be assessed earlier. 2. Connector configuration in server.xml is correct for HTTP/2 ? 3. Does AsyncServlet is written properly for such type of use case ? If not, then please suggest correct way to manage it. 4. If you are already aware of similar problem, can you point out any alternatives or recommendations. Thanks in advance for your support. Regards, Kedar Deshmukh <> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
