Re: Tomcat testing

[Martin Grigorov]

HI,

On Wed, Dec 8, 2021 at 11:20 AM Mark Thomas  wrote:

> On 08/12/2021 08:44, Hiran CHAUDHURI wrote:
> > Hello there.
> >
> > My organization requires that Tomcat releases - especially patches - get
> tested before we propagate them into production environments.
> > For sure similar tests are run by the ASF before releasing the software
> at all.
> >
> > Is there a way to run the very same tests within our organization? Where
> can I find out what automatic testing is already performed? Or would I have
> to develop a test suite from scratch?
>
> https://github.com/apache/tomcat/blob/main/BUILDING.txt


Here is what we do at Travis CI:
-
https://github.com/apache/tomcat/blob/2c5cea02157c3730ca498e0e26422dcaea5b8d4b/.travis.yml
-
https://github.com/apache/tomcat/blob/2c5cea02157c3730ca498e0e26422dcaea5b8d4b/.travis/antTest.sh


>
>
> Section 7.
>
> There is also the various CI systems:
> https://tomcat.apache.org/ci.html
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Enhancement: Additional user attributes queried by (some) realms

[Martin Grigorov]

On Tue, Jun 1, 2021 at 12:46 PM Carsten Klein  wrote:

>
>
> On 01/06/2021 10:18, Mark Thomas wrote:
>
> > I don't know if you can. I suspect not. By all means see if you can. I'm
> > mildly curious to find out the answer. Whether you can or not, you don't
> > need to.
>
> I found nothing to re-trigger the Travis CI build so far. However, now
> the CI test is 'successful'. Did you do anything about it?
>

I've restarted it few hours ago.
Only members of github.com/apache/tomcat team can restart builds via the
TravisCI UI. And most probably Apache Infra team members.


>
> Carsten
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Emdbedded Tomcat Issues

[Martin Grigorov]

On Mon, Apr 26, 2021 at 4:18 AM dana whitelow 
wrote:

>
> Embedded Tomcat 9.0.0.M6
>

Please use newer version. 9.0.0.M6 is released May 2016.
Latest 9.x release is 9.0.45.


>
>
>   Issue 1
>   //ctx.setAltDDName(
> "/home/user/microservices/AccountingApp/src/main/webapp/WEB-INF/web.xml");
> ctx.getServletContext().setAttribute(Globals.ALT_DD_ATTR,
> "/home/user/microservices/AccountingApp/src/main/webapp/WEB-INF/web.xml");
>
> Neither of the above  programmatic settings seems to have any
> effect on the message INFO: No global web.xml found
>
>
> INFO: Starting Servlet Engine: Apache Tomcat/9.0.0.M6
> Apr 26, 2021 1:58:59 AM org.apache.catalina.startup.ContextConfig
> getDefaultWebXmlFragment
> INFO: No global web.xml found
> Apr 26, 2021 1:58:59 AM org.omnifaces.ApplicationInitializer
> logOmniFacesVersion
> INFO: Using OmniFaces version 3.6.1
> Apr 26, 2021 1:58:59 AM org.apache.catalina.core.ApplicationContext log
> INFO: Marking servlet Faces Servlet as unavailable
> Apr 26, 2021 1:58:59 AM org.apache.catalina.core.StandardContext
> loadOnStartup
> SEVERE: Servlet [Faces Servlet] in web application [] threw load()
> exception
> java.util.MissingResourceException: Can't find javax.faces.LogStrings
> bundle from
>
> Issue 2
>
> from error message I think tomcat is looking for the file
> WebMessages.properties and WebMessages_es.properties from the
> faces-config.xml
> which I have in the src/main/resources/hacc/web/ and  in  directory
> src/main/resources/META-INF/resources/hacc/web ,
> but still it is giving me the error
> SEVERE: Servlet [Faces Servlet] in web application [] threw load()
> exception
> java.util.MissingResourceException: Can't find javax.faces.LogStrings
> bundle from
>
> at
> java.logging/java.util.logging.Logger.setupResourceInfo(Logger.java:2289)
> at java.logging/java.util.logging.Logger.(Logger.java:565)
> at
> java.logging/java.util.logging.LogManager.demandLogger(LogManager.java:522)
>
> 

Re: Tomcat 9.0

[Martin Grigorov]

Hi,

On Thu, Apr 8, 2021 at 3:25 AM Mohamed Eliyas Abdul Kadar <
mohamed.ka...@neogenomics.com> wrote:

> Hi All
> I am planning to use Tomcat for my development server. Initially we
> planned to go with version Tomcat 9.0.41. Now I see newer versions are
> release on top of that and see the latest version is Tomcat 9.0.45. Please
> let me know if there is any major fix of Tomcat 9.0.41 made on higher
> versions or we are good with Tomcat 9.0.44 as Tomcat 9.0.45 is not having
> any release date.
>

Please consult with https://tomcat.apache.org/tomcat-9.0-doc/changelog.html
for the changes in each version.

Martin


>
> Regards
> Eliyas
> This communication and its attachments contain confidential information
> and is intended only for the named addressee. If you are not the named
> addressee you should not disseminate, distribute or copy this
> communication. Please notify the sender immediately if you have received
> this communication by mistake and delete or destroy this communication.
> Communications cannot be guaranteed to be secured or error-free as
> information could be intercepted, corrupted, lost, destroyed, arrive late
> or incomplete, or contain viruses. The sender therefore does not accept
> liability for any errors or omissions in the contents of this communication
> which arise as a result of transmission. If verification is required please
> request a hard-copy version. NeoGenomics Laboratories, 12701 Commonwealth
> Dr, Fort Myers, FL 33913, http://www.neogenomics.com (2021)
>

Re: Tomcat Error: VersionLoggerListener

[Martin Grigorov]

Hi,

On Thu, Apr 1, 2021 at 1:42 PM KISHOR S.JADHAV 
wrote:

> Hi Team,
>  I am trying to run my war file in tomcat version 8.5 is not working so I
> have decrease the tomcat version and using the Tomcat8  but in this, I am
> getting the below error,
>
> [image: image.png]
>
>
> [image: image.png]
>

Please copy/paste the text instead of creating screenshots.
Attachments are not allowed in the mailing lists.


>
>  Getting this error *org.apache.catalina.startup.VersionLoggerListener
> log*
>  I am not able to find the correct solution for this issue, Request you
> to can you please help me to solve this issue
>
> Thanks and Regards,
> Kishor Jadhav
>
>

Re: Wait... NULL address in java.net.BindException: Address already in use (Bind failed) ???

[Martin Grigorov]

On Wed, Mar 17, 2021, 20:27 Eric Robinson  wrote:

> > From: Martin Grigorov 
> > Sent: Wednesday, March 17, 2021 12:45 PM
> > To: Tomcat Users List 
> > Subject: Re: Wait... NULL address in java.net.BindException: Address
> already
> > in use (Bind failed)  ???
> >
> > Hi,
> >
> > On Wed, Mar 17, 2021, 19:34 Eric Robinson 
> > wrote:
> >
> > > Getting error:
> > >
> > > java.net.BindException: Address already in use (Bind failed)
> > > :3787
> > >
> >
> > Please paste more lines of the exception.
> > Also please tell us which version of JDK/JRE you use.
> > This exception is very cryptic and does not usually tell which address
> is in use.
> > I.e. 3787 is not the port, as you might think. Most probably it is a
> line in some
> > class.
> >
>
> Tomcat: Apache Tomcat/8.5.51
> JVM: 1.8.0_241-b08
>
> The following error appears in catalina.out under tomcat 8. It does not
> mention the null. We tried it under tomcat 7 as well, and that is where it
> mentions the null.
>
> 17-Mar-2021 11:12:54.039 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio-3787"]
> 17-Mar-2021 11:12:54.048 SEVERE [main]
> org.apache.catalina.core.StandardService.initInternal Failed to initialize
> connector [Connector[HTTP/1.1-3787]]
>

This line says that 3787 is the port indeed.
Are you sure it is not bound?


org.apache.catalina.LifecycleException: Protocol handler
> initialization failed
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1032)
> at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
> at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:848)
> at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.startup.Catalina.load(Catalina.java:639)
> at
> org.apache.catalina.startup.Catalina.load(Catalina.java:662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:303)
> at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch
> .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch
> .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.tomcat.util.net
> .NioEndpoint.bind(NioEndpoint.java:221)
> at org.apache.tomcat.util.net
> .AbstractEndpoint.init(AbstractEndpoint.java:1118)
> at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:223)
> at
> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:587)
> at
> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1030)
> ... 13 more
>
>
> >
> > > I know how to fix the infamous "Address already in use (Bind failed)"
> > > problem when there is another process already listening on a port.
> > > However, I have confirmed with netstat and fuser that there is no
> > > other process listening on that port. Could the problem be that the
> > > host address is null for some reason? I don't recall seeing that
> > > before, and Google diving came up dry.
> > >
> > > -Eric
> > >
> > >
> > >
> > > Disclaimer : This email and any files transmitted with it are
> > > confidential and intended solely for intended recipients. If you are
> > > not the named addressee you should not disseminate, distribute, copy or
> > alter this email.
> > > A

Re: Wait... NULL address in java.net.BindException: Address already in use (Bind failed) ???

[Martin Grigorov]

Hi,

On Wed, Mar 17, 2021, 19:34 Eric Robinson  wrote:

> Getting error:
>
> java.net.BindException: Address already in use (Bind failed) :3787
>

Please paste more lines of the exception.
Also please tell us which version of JDK/JRE you use.
This exception is very cryptic and does not usually tell which address is
in use. I.e. 3787 is not the port, as you might think. Most probably it is
a line in some class.


> I know how to fix the infamous "Address already in use (Bind failed)"
> problem when there is another process already listening on a port. However,
> I have confirmed with netstat and fuser that there is no other process
> listening on that port. Could the problem be that the host address is null
> for some reason? I don't recall seeing that before, and Google diving came
> up dry.
>
> -Eric
>
>
>
> Disclaimer : This email and any files transmitted with it are confidential
> and intended solely for intended recipients. If you are not the named
> addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the company
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments.
>

Re: Keeping a session active with ajax

[Martin Grigorov]

Chris,

On Mon, Mar 15, 2021, 18:28 Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Martin,
>
> On 3/15/21 04:08, Martin Grigorov wrote:
> > Hi Maxim,
> >
> > On Mon, Mar 15, 2021 at 9:46 AM Maxim Solodovnik 
> > wrote:
> >
> >> Maybe offtopic
> >> but websocket requests seems to not warming up the HttpSession
> >>
> >
> > The websocket messages do not bring the JSESSIONID cookie/request
> parameter
> > and there is no way to lookup the http session from the web container
> > (Tomcat).
> > Wicket creates a copy of the HttpSession at WebSocket connection creation
> > time and uses it until the closing of the connection.
> > So, yes, the real http session may expire if you use only websocket based
> > communication.
>
> Yes, but that's not AJAX, as the OP mentioned. Perhaps "AJAX" just means
> "communication using javascript without 'traditional' navigation".
>

Did you read my first answer and Maxim's email?


> Thanks,
> -chris
> >> Tomcat 9.0.44
> >> Wicket 9.2.0 :)
> >>
> >> On Mon, 15 Mar 2021 at 14:43, Martin Grigorov 
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Which version of Tomcat ?
> >>>
> >>> On Mon, Mar 15, 2021 at 8:25 AM Jerry Malcolm 
> >>> wrote:
> >>>
> >>>> In my app a logged in user will access a page.  But then all of the
> >>>> interaction on the page is done back and forth with ajax calls to the
> >>>> server.  The ajax activities within the page could last well over the
> >> 30
> >>>> minute session timeout. Even though the user has been actively working
> >>>> for 30-45 minutes with lots of server calls, all of a sudden they
> >> aren't
> >>>> logged in any more.  The ajax calls are definitely successfully
> >>>> accessing the session since the ajax APIs utilize the logged on user,
> >>>> roles, etc.  They just aren't pinging the keep-alive bit in the
> session
> >>>> for some reason.  I'm not really sure what would be different in the
> >>>> calls from a regular page call from the browser vs an ajax call.  They
> >>>> both make a request that goes to a servlet or jsp and they both get
> >>>> response data back from tomcat.  But the session times out if I don't
> >> do
> >>>> a real page call.  What am I missing?
> >>>>
> >>>
> >>> Nothing!
> >>> Any request that uses the HttpSession
> >>> (HttpServletRequest.getHttpSession(true)) will "touch" the session and
> >>> update its "lastUsed" time, so it should not time out during active
> usage
> >>> of the application.
> >>> There must be something else that invalidates the http session.
> >>>
> >>>
> >>>>
> >>>> I really don't want to change the 'inactivity' timeout.  I just need
> to
> >>>> figure out how to tell tomcat with all of the ajax calls that the
> >>>> session is still very active.  Is there an easy solution?
> >>>>
> >>>> Thx
> >>>>
> >>>> Jerry
> >>>>
> >>>> BTW... TC 8.5
> >>>>
> >>>>
> >>>> -
> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>>
> >>>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Maxim
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Keeping a session active with ajax

[Martin Grigorov]

Hi Maxim,

On Mon, Mar 15, 2021 at 9:46 AM Maxim Solodovnik 
wrote:

> Maybe offtopic
> but websocket requests seems to not warming up the HttpSession
>

The websocket messages do not bring the JSESSIONID cookie/request parameter
and there is no way to lookup the http session from the web container
(Tomcat).
Wicket creates a copy of the HttpSession at WebSocket connection creation
time and uses it until the closing of the connection.
So, yes, the real http session may expire if you use only websocket based
communication.

Martin

P.S. Congrats for Apache OpenMeetings 6.0!


>
> Tomcat 9.0.44
> Wicket 9.2.0 :)
>
> On Mon, 15 Mar 2021 at 14:43, Martin Grigorov 
> wrote:
> >
> > Hi,
> >
> > Which version of Tomcat ?
> >
> > On Mon, Mar 15, 2021 at 8:25 AM Jerry Malcolm 
> > wrote:
> >
> > > In my app a logged in user will access a page.  But then all of the
> > > interaction on the page is done back and forth with ajax calls to the
> > > server.  The ajax activities within the page could last well over the
> 30
> > > minute session timeout. Even though the user has been actively working
> > > for 30-45 minutes with lots of server calls, all of a sudden they
> aren't
> > > logged in any more.  The ajax calls are definitely successfully
> > > accessing the session since the ajax APIs utilize the logged on user,
> > > roles, etc.  They just aren't pinging the keep-alive bit in the session
> > > for some reason.  I'm not really sure what would be different in the
> > > calls from a regular page call from the browser vs an ajax call.  They
> > > both make a request that goes to a servlet or jsp and they both get
> > > response data back from tomcat.  But the session times out if I don't
> do
> > > a real page call.  What am I missing?
> > >
> >
> > Nothing!
> > Any request that uses the HttpSession
> > (HttpServletRequest.getHttpSession(true)) will "touch" the session and
> > update its "lastUsed" time, so it should not time out during active usage
> > of the application.
> > There must be something else that invalidates the http session.
> >
> >
> > >
> > > I really don't want to change the 'inactivity' timeout.  I just need to
> > > figure out how to tell tomcat with all of the ajax calls that the
> > > session is still very active.  Is there an easy solution?
> > >
> > > Thx
> > >
> > > Jerry
> > >
> > > BTW... TC 8.5
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
> > >
>
>
>
> --
> Best regards,
> Maxim
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Keeping a session active with ajax

[Martin Grigorov]

Hi,

Which version of Tomcat ?

On Mon, Mar 15, 2021 at 8:25 AM Jerry Malcolm 
wrote:

> In my app a logged in user will access a page.  But then all of the
> interaction on the page is done back and forth with ajax calls to the
> server.  The ajax activities within the page could last well over the 30
> minute session timeout. Even though the user has been actively working
> for 30-45 minutes with lots of server calls, all of a sudden they aren't
> logged in any more.  The ajax calls are definitely successfully
> accessing the session since the ajax APIs utilize the logged on user,
> roles, etc.  They just aren't pinging the keep-alive bit in the session
> for some reason.  I'm not really sure what would be different in the
> calls from a regular page call from the browser vs an ajax call.  They
> both make a request that goes to a servlet or jsp and they both get
> response data back from tomcat.  But the session times out if I don't do
> a real page call.  What am I missing?
>

Nothing!
Any request that uses the HttpSession
(HttpServletRequest.getHttpSession(true)) will "touch" the session and
update its "lastUsed" time, so it should not time out during active usage
of the application.
There must be something else that invalidates the http session.


>
> I really don't want to change the 'inactivity' timeout.  I just need to
> figure out how to tell tomcat with all of the ajax calls that the
> session is still very active.  Is there an easy solution?
>
> Thx
>
> Jerry
>
> BTW... TC 8.5
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Issue in CSS render with tomcat version 8.5.58 onwards

[Martin Grigorov]

Hi,

On Sat, Mar 13, 2021, 08:16 Shiv Garg  wrote:

> Hi Team,
> we have clients reporting issues with tomcat 8.5.58 version onwards,
> stating that CSS is not loading properly in chrome/FF browser.
>

Please explain more what exactly does not work.
Even better create a simple application that demonstrates the issue.


> Can you please help me check if there is any change in this release
> onwards which could impact it.
>
> in change logs i can see below change , kindly verify if this can make any
> impact.
>
>- [image: Fix:] Extracted CSS styles from the Manager we application
>for better code maintenance and replaced the GIF logo with SVG. (isapir)
>
>
No, this is not the problem.
This changes modified the Manager application that you may use to
deploy/start/stop your applications.

Regards,
Martin


> Regards
> Shiv Garg
>

Re: Is ARM64 architecture officially supported ?

[Martin Grigorov]

Hi Chris,

On Thu, Feb 11, 2021 at 12:13 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Martin,
>
> On 4/20/20 03:28, Martin Grigorov wrote:
> > Hi Michael,
> >
> > On Sun, Apr 19, 2020 at 9:08 PM Michael Osipov 
> wrote:
> >
> >> Am 2020-04-19 um 19:51 schrieb Martin Grigorov:
> >>> Hi Emilio,
> >>>
> >>> On Fri, Apr 17, 2020 at 2:14 PM Emilio Fernandes <
> >>> emilio.fernande...@gmail.com> wrote:
> >>>
> >>>> Hola Tomcat community!
> >>>>
> >>>> We consider using AWS Graviton [1] based instances which use ARM64
> >>>> processors for our backend services.
> >>>> I've googled around and found [2] saying that Tomcat is being tested
> on
> >>>> ARM64 architecture at TravisCI! This is great!
> >>>> Does this mean that Tomcat is officially supported on ARM64 ? I was
> not
> >>>> able to find any specific documentation listing which platforms are
> >>>> officially supported.
> >>>>
> >>>> Does anyone from the community have any experience with Tomcat/HTTPD
> on
> >>>> ARM64 in production ?
> >>>>
> >>>
> >>> I work a lot with ARM64 servers lately!
> >>> So far we didn't face any issue related to ARM64 & Tomcat in our
> >>> application!
> >>> I've introduced the testing on TravisCI that you have found. In
> addition
> >>> for my daily job we have created a nightly test setup:
> >>> - build and test Tomcat 9.x (as at Travis)
> >>> - run some smoke tests with our application
> >>> - run some performance tests on x86_64 and ARM64 VMs with similar
> >> hardware
> >>> specs. The results for both architectures are very similar. One thing
> >> that
> >>> still bothers me is that the throughput on HTTPS is 3 times less than
> >> HTTP.
> >>
> >> OpenSSL or SunJSSE?
> >>
> >
> > Both.
> > With OpenSSL it is slightly faster than with JSSE, but again around 3
> times
> > slower than HTTP.
> >
> > Here are some details:
> > Tomcat: 9.x nightly build
> > tcnative: nightly build from master branch
> > JDK: AdoptJDK 14
> > OpenSSL 1.1.1  11 Sep 2018 (this comes from Ubuntu 18.04 repos. I will
> try
> > with a newer build)
> > JMeter: 5.2.1
> > The application is Spring Boot 2.2.6 with REST GET/PUT/POST/DELETE
> > endpoints that simply sets/gets data to/from Memcached. No usage of
> Spring
> > Security! No redirects from HTTP to HTTPS!
> >
> > Let me know if you need more information!
>
> Coming back to this almost a year later. :)
>

Thanks for doing it! I've forgot to update this thread with the problem and
the solution!


>
> My guess about the TLS performance is that your x86 chips have built-in
> hardware support for crypto primitives used by the cipher suite(s) you
> are testing, and your ARM64 chips do not.
>
> Either that, or neither OpenSSL nor Java are able to use those ARM64
> hardware features because they are not aware of them (yet, I would guess).
>
> You would have to get the specs of the chips to find out whether
> hardware acceleration is even a possibility, and then approach the
> crypto providers (e.g. Oracle, OpenSSL) with questions about how to
> enable those hardware-supported routines.
>

At the end the problem was in Apache JMeter 5.2.1 - the tool I used for the
load testing back then.
As I expained here
<https://martin-grigorov.medium.com/compare-apache-tomcat-performance-on-x86-64-and-arm64-cpu-architectures-aacfbb0b5bb6>
one needs to add  --jmeterproperty
httpclient4.validate_after_inactivity=4900 and --jmeterproperty
httpclient4.time_to_live=12 to JMeter properties, otherwise it was
closing the connection and due to the TLS handshakes it was giving terrible
results. Those settings are default in JMeter 5.3.0+.

Apart from that there is also https://github.com/kunpengcompute/KAE for
HiSilicon/Kunpeng chips, to get even more from OpenSSL on this hardware.

Martin


>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat on Amazon's Graviton2

[Martin Grigorov]

Hi Chen,

On Wed, Feb 10, 2021 at 6:00 PM Chen Levy  wrote:

> Hello Everyone
>
> We've been running our Staging environment on ARM64 based servers (Amazon
> EC2 T4g) for a while now, and plan to use similar machines to replace our
> Production environment's x86-64 based servers in the near future.
> All of our servers are based on Amazon Linux 2 with AdoptOpenJDK 15.0.2
> and Tomcat 9.
> Measured with our application, performance between x86-64 and ARM64 with
> two similarly sized machines (e.g. T3.large vs T4g.large) is up by an
> average of 22% on the T4g's.
> These numbers are largely academic, and will stay that way until we
> actually deploy a few machines in Production.
> The ARM64 machines are also cheaper by about 20%, so if all goes well,
> there aren't real downsides to the move.
>
> I read this correspondence about a year ago:
> https://marc.info/?l=tomcat-user=2=1=arm64=b
>
> I'd like to ask the community if anyone has production experience with
> Tomcat and ARM64/Graviton2, and whether there are specific issues I should
> beware of.
>

I have experience on ARM64 architecture but on Huawei HiSilicon chips
(Kunpeng), not Graviton2.
TLDR - everything runs just fine! :-)

In the beginning we had some bad performance results but they were caused
by the load testing tools -
https://martin-grigorov.medium.com/compare-apache-tomcat-performance-on-x86-64-and-arm64-cpu-architectures-aacfbb0b5bb6
Initially I used Apache JMeter  5.2.1 and it
had the issue of closing the connection after each request. This caused
very bad results for HTTPS. JMeter team helped me by suggesting two
settings which fixed the problem. Those settings are now enabled by default
in 5.3.0+
Then I replaced JMeter with wrk  - it gives much
better throughput results than JMeter! But at some point I realized (with
the help of Poul Henning Kamp from Varnish Cache team) that it has a nasty
bug in calculating the statistics (
https://martin-grigorov.medium.com/compare-varnish-cache-performance-on-x86-64-and-aarch64-cpu-architectures-cef5ad5fee5f
)
So I moved to Vegeta  ! It is almost
perfect for my needs. It has just one issue
 - Golang http library sends
RST_STREAM frame after the END_STREAM for HTTP/2. But Mark Thomas improved
Tomcat to keep lightweight versions of the HTTP2 streams for some more time
and this worked around this issue.

For my tests I used two *similar* VMs - one x86_64 and one aarch64. The
have almost identical hardware but due to the different architectures not
exactly the same.
https://martin-grigorov.medium.com/comparing-the-performance-of-several-http2-web-servers-fb5f3787532a
is
my latest blog article related to Tomcat and there x86_64 still won. In the
same tests Netty performed better on aarch64 (same hardware, JDK and
OpenSSL versions!). I guess the reason is in the custom EPoll impl that
Netty uses, but I am not sure.

Apart from Tomcat most of the problems related to moving to ARM64 we found
were related to file format (Protobuf, Parquet) and file compression
(Brotli) libraries because they use native libraries.
The only big problem with pure Java (well, Scala in this case) I am aware
of was in Scala Cats library:
- https://github.com/typelevel/cats-effect/releases/tag/v3.0.0-M4
- code with explanation from OpenJDK expert:
https://gist.github.com/RaasAhsan/8e3554a41e07068536425ca0de46c9e8

Good luck!

Martin


>
> Thanks
> Chen
>
>
>

Re: How to completely deactivate JSESSIONID cookie entry (in Chrome's Cookie list)

[Martin Grigorov]

On Fri, Jan 29, 2021 at 12:03 PM Tillmann Schulz
 wrote:

> Hi there,
>
> I am using Tomcat 8.5.58 and have a problem with JSESIONID cookie.
> It should be possible to completly deactivate the jsessionid cookie with
> the following code:
>
> <%@ page session="false" %>
>

This says that this particular JSP does not create an HTTP session but if
any other JSP of your app creates one then it will be transfered for all
requests until it is either deleted or it expires.


>
> If you do that and call the JSP, there is in no entry under cookies in
> google chrome's cookie list (if you press F12 in chrome) .
> But in chrome's cookie list there is a host entry for the hostname of the
> JSP.
>

What is the name of this cookie ?


> This is confusing, because there are no cookies set.
>
> I am not sure if this is a tomcat issue, but under firefox it also looks
> simular.
>
> Is it possible to completly deactivate cookies? There shouldn't be any
> hostname visible in the cookie view.
>
> Thank you for your help in advance
>
> Tillmann
>
>
>
>

Re: Jars in Tomcat lib directory

[Martin Grigorov]

Hi,

On Mon, Jan 18, 2021 at 10:38 PM Jonnalagadda, Swathi (External) <
swathi.jonnalaga...@xerox.com> wrote:

> The application was an age old application and was built using ant which
> means all required jars will be packaged.
>
> As part of testing the application after upgrading to Java 1.8 and
> deployed in tomcat 9, I realized that couple of additional jars are
> required during runtime.
>
> I then added these jars to Tomcat lib directory and observed that it is
> not picking up these unless I add it to classpath
>

Classes in jar files in $CATALINA_BASE/lib are available for all web
applications!
How do you "add to the classpath" ? I have the feeling you do something in
your bin/setenv.sh that breaks the default configuration.


>
> Regards
> Swathi
>
> -Original Message-
> From: Mounika Reddy [mailto:spidermai...@gmail.com]
> Sent: Monday, January 18, 2021 11:49 PM
> To: Tomcat Users List
> Subject: Re: Jars in Tomcat lib directory
>
> CAUTION:   This email originated from outside the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
> It means your tomcat setup is screwed up. You can actually package in your
> application as a dependency and see if it picks up. If not, then post the
> logs here to understand what's going on
>
> On Mon, Jan 18, 2021, 1:09 PM Jonnalagadda, Swathi (External) <
> swathi.jonnalaga...@xerox.com> wrote:
>
> > I am not sure if I am missing some configuration setting but what I
> > see is unless I add externally added jars under tomcat/lib to
> > classpath I don’t see them considerd by Tomcat irrespective of wether
> they are needed or not.
> >
> > For example I added ojdbc14.jar to Tomcat lib directory assuming that
> > it will consider it in classpath but until I specified the path of the
> > jar in classpath, the server wasn’t considering the jar.
> >
> > Please advise.
> >
> > -Original Message-
> > From: Mounika Reddy [mailto:spidermai...@gmail.com]
> > Sent: Monday, January 18, 2021 6:32 PM
> > To: Tomcat Users List
> > Subject: Re: Jars in Tomcat lib directory
> >
> > CAUTION:   This email originated from outside the organization. Do not
> > click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
> > It will provided if your application requires.
> >
> > On Mon, Jan 18, 2021, 7:54 AM Jonnalagadda, Swathi (External) <
> > swathi.jonnalaga...@xerox.com> wrote:
> >
> > > Hi
> > >
> > > Doesn’t tomcat consider jar files under lib directory in classpath.
> > > Do we have to do any additional configuration for that?
> > >
> > >
> > > Thanks
> > > Swathi
> > >
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Not able to make JSESSIONID cookie secure

[Martin Grigorov]

Hi Amit,

On Wed, Jan 6, 2021 at 11:15 AM Amit Khosla 
wrote:

> Hi,
>
> Thanks for the reply.
>
> We tried the settings on multiple machines. And found that the same
> configuration machines gave different results.
> We are getting multiple jsessionid cookies being created. In our
> application, we have a multi tenant application.
> For each tenant we have an nginx running calling the application url.
> URL being hit on browser is like
> *.myapp.com *
> We are able to see 2 JSESSIONID cookies being generated for this call on
> some machines.
> 1. domain: /myapp  which is having secure flag.
> 2. domain: /myapp/ which is NOT having secure flag.
>
> Strangely, in most machines, we are finding the second cookie being
> generated but the first cookie not generated.
> Only when we saw on one of the machines, the first cookie, we found that
> the secure jsessionid cookie being generated on that particular
> environment.
>
> The environment having 2 cookies and the one with one cookie are identical.
> Can you please help me fix the creation of the second cookie and also how
> to ensure that the first cookie is generated in all environments?
>

I think you issue is in Nginx config.
Please test first only with Tomcat and see whether it behaves propertly.
If it does then ask for more help at Nginx forums.
If it doesn't then please explain in more details how exactly you test it,
with configs, urls, etc.


>
>
> On Tue, Jan 5, 2021 at 1:24 AM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Mark,
> >
> > On 1/4/21 03:17, Mark Thomas wrote:
> > > On 04/01/2021 06:02, Amit Khosla wrote:
> > >> Hi,
> > >>
> > >> We are still facing this issue. Can someone please help us?
> > >
> > > In a clean 8.5.x install, session cookies are only marked as secure if
> > > the request that triggered the session creation is made over a secure
> > > channel (typically HTTPS).
> > >
> > > If you amend the session configuration in $CATALINA_BASE/conf/web.xml
> > from:
> > >
> > > 
> > >  30
> > > 
> > >
> > > to
> > >
> > > 
> > >  30
> > >  
> > >  true
> > >  
> > > 
> > >
> > > then session cookies will be generated with the secure flag whether or
> > > not the request that triggered the the session creation was made over a
> > > secure channel.
> > >
> > > Reviewing the thread:
> > >
> > > Are you sure you are amending the correct web.xml file? One way to
> check
> > > this is to make a deliberate error in the file and confirm that this
> > > error is reported when Tomcat starts.
> > >
> > > Note that you can only use  once in a web.xml file. If
> > > the web.xml file already contains a  element you must
> > > add to that existing element.
> > >
> > > Configuration in the application's web.xml file will override the
> global
> > > web.xml file. Make sure that the application's web.xml either does not
> > > specify a value for secure or specifies true.
> > >
> > > If you still have issues:
> > > - start with a clean Tomcat 8.5.x install
> > > - confirm that
> > >http://localhost:8080/examples/servlets/servlet/SessionExample
> > >generates a set-cookie header without the secure attribute
> > > - stop Tomcat
> > > - close the browser
> > > - amend conf/web.xml as above
> > > - start Tomcat
> > > - confirm that
> > >http://localhost:8080/examples/servlets/servlet/SessionExample
> > >generates a set-cookie header with the secure attribute
> > > - retest with your application
> > >
> > > You must close the browser between each request you expect to generate
> a
> > > session cookie to prevent any existing session from being used.
> >
> > You may be able to avoid this with either of:
> >
> > 1. "Private" browsing mode: use a new "private" tab/window each time
> >
> > 2. Under developer tools / Storage (you may have to poke around for
> > this), you should be able to see the cookies for the host and, if
> > appropriate, delete them.
> >
> > > If this test fails then you'll need to check the application source
> > > code. It is possible that the application is overriding your attempts
> to
> > > make the session cookie secure.
> >
> > +1
> >
> > Other possibilities include a reverse proxy where the client is using
> > HTTPS to communicate with the proxy, but HTTPS is not being used between
> > the proxy and the Tomcat server.
> >
> > -chris
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>
> --
> Thanks & Regards
> Amit Khosla
>

Re: Resource interpreted as Stylesheet but transferred with MIME type text/html: "".

[Martin Grigorov]

On Mon, Nov 23, 2020 at 7:26 AM Huynh Nghia  wrote:

> - Request Headers : Accep:text/css
>

"Accep" header name misses "t" at the end


> - But Reponse : Content-type:text/html
> [image: Screenshot_8.png]
>
> Vào Th 2, 23 thg 11, 2020 vào lúc 11:47 Huynh Nghia <
> hhnghiai...@gmail.com> đã viết:
>
>>
>>
>> Vào Th 2, 23 thg 11, 2020 vào lúc 11:35 Huynh Nghia <
>> hhnghiai...@gmail.com> đã viết:
>>
>>>
>>>
>>> -- Forwarded message -
>>> Từ: Huynh Nghia 
>>> Date: Th 2, 23 thg 11, 2020 vào lúc 11:21
>>> Subject: Resource interpreted as Stylesheet but transferred with MIME
>>> type text/html: "".
>>> To: 
>>>
>>>
>>> Hi Tomcat dev,
>>>
>>> - I got an error with my app running on tomcat v9 64bit last modified
>>> (Win 10):
>>>   Example:
>>>
>>>  v9.0.27/ 
>>> 2019-10-11 08:46-   [image: [DIR]] v9.0.29/ 
>>> 
>>> 2019-11-21 09:11-   [image: [DIR]] v9.0.30/ 
>>> 
>>> 2019-12-12 08:22-   [image: [DIR]] v9.0.31/ 
>>> 
>>> 2020-03-10 23:16-   [image: [DIR]] v9.0.33/ 
>>> 
>>> 2020-03-16 09:37-   [image: [DIR]] v9.0.34/ 
>>> 
>>> 2020-04-08 15:43-   [image: [DIR]] v9.0.35/ 
>>> 
>>> 2020-05-11 11:19-   [image: [DIR]] v9.0.36/ 
>>> 
>>> 2020-06-07 19:01-   [image: [DIR]] v9.0.37/ 
>>> 
>>> 2020-07-05 18:31-   [image: [DIR]] v9.0.38/ 
>>> 
>>> 2020-11-05 18:47-
>>>
>>> [image: [DIR]] v9.0.39/
>>> 
>>>   2020-11-05 18:47 -
>>>
>>> v9.0.40/  
>>>2020-11-17 14:23
>>>
>>>
>>> [image: Screenshot_4.png]
>>> [image: Screenshot_5.png]
>>> [image: Screenshot_6.png]
>>>
>>>
>>> - But my app work with tomcat v9 last modified :
>>>   Example:
>>>
>>> v9.0.4/    
>>>   2018-01-22 14:51
>>>
>>> v9.0.5/    
>>>   2018-02-11 20:11
>>>
>>>
>>> - Technology : Angularjs 1.8, Jquery 3.5.1,Java Sevlet version 1.7 : 51.0
>>>
>>> - Please help me workaround this error or fix it on version tomcat
>>> v9.0.40/ 
>>>
>>> Thanks you so much.
>>>
>>> Best Regards,
>>>
>>> Nghia Huynh
>>>
>>>
>>>
>>>
>>>
>>>
>>>

Re: NGINX + tomcat 8.0.35 (110: Connection timed out)

[Martin Grigorov]

On Thu, Nov 12, 2020 at 2:40 PM Ayub Khan  wrote:

> Martin,
>
> Could you provide me a command which you want me to run and provide you the
> results which might help you to debug this issue ?
>

1) start your app and click around to load the usual FDs
2) lsof -p `cat /var/run/tomcat8.pid` > after_start.txt
3) load your app
4) lsof -p `cat /var/run/tomcat8.pid` > after_load.txt

you can analyze the differences in the files yourself before sending them
to us :-)


>
>
> On Thu, Nov 12, 2020 at 1:36 PM Martin Grigorov 
> wrote:
>
> > On Thu, Nov 12, 2020 at 10:37 AM Ayub Khan  wrote:
> >
> > > Martin,
> > >
> > > These are file descriptors, some are related to the jar files which are
> > > included in the web application and some are related to the sockets
> from
> > > nginx to tomcat and some are related to database connections. I use the
> > > below command to count the open file descriptors
> > >
> >
> > which type of connections increase ?
> > the sockets ? the DB ones ?
> >
> >
> > >
> > > watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
> > >
> >
> > you can also use lsof command
> >
> >
> > >
> > >
> > >
> > > On Thu, Nov 12, 2020 at 10:56 AM Martin Grigorov  >
> > > wrote:
> > >
> > > > On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan 
> wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > I was load testing using the ec2 load balancer dns. I have
> increased
> > > the
> > > > > connector timeout to 6000 and also gave 32gig to the JVM of
> tomcat. I
> > > am
> > > > > not seeing connection timeout in nginx logs now. No errors in
> > > kernel.log
> > > > I
> > > > > am not seeing any errors in tomcat catalina.out.
> > > > > During regular operations when the request count is between 4 to 6k
> > > > > requests per minute the open files count for the tomcat process is
> > > > between
> > > > > 200 to 350. Responses from tomcat are within 5 seconds.
> > > > > If the requests count goes beyond 6.5 k open files slowly move up
> to
> > > > 2300
> > > > > to 3000 and the request responses from tomcat become slow.
> > > > >
> > > > > I am not concerned about high open files as I do not see any errors
> > > > related
> > > > > to open files. Only side effect of  open files going above 700 is
> the
> > > > > response from tomcat is slow. I checked if this is caused from
> > elastic
> > > > > search, aws cloud watch shows elastic search response is within 5
> > > > > milliseconds.
> > > > >
> > > > > what might be the reason that when the open files goes beyond 600,
> it
> > > > slows
> > > > > down the response time for tomcat. I tried with tomcat 9 and it's
> the
> > > > same
> > > > > behavior
> > > > >
> > > >
> > > > Do you know what kind of files are being opened ?
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> > > > > ch...@christopherschultz.net> wrote:
> > > > >
> > > > > > Ayub,
> > > > > >
> > > > > > On 11/3/20 10:56, Ayub Khan wrote:
> > > > > > > *I'm curious about why you are using all of cloudflare and ALB
> > and
> > > > > > > nginx.Seems like any one of those could provide what you are
> > > getting
> > > > > from
> > > > > > > all3 of them. *
> > > > > > >
> > > > > > > Cloudflare is doing just the DNS and nginx is doing ssl
> > termination
> > > > > >
> > > > > > What do you mean "Cloudflare is doing just the DNS?"
> > > > > >
> > > > > > So what is ALB doing, then?
> > > > > >
> > > > > > > *What is the maximum number of simultaneous requests that one
> > > > > > nginxinstance
> > > > > > > will accept? What is the maximum number of simultaneous
> > > > proxiedrequests
> > > > > > one
> > >

Re: NGINX + tomcat 8.0.35 (110: Connection timed out)

[Martin Grigorov]

On Thu, Nov 12, 2020 at 10:37 AM Ayub Khan  wrote:

> Martin,
>
> These are file descriptors, some are related to the jar files which are
> included in the web application and some are related to the sockets from
> nginx to tomcat and some are related to database connections. I use the
> below command to count the open file descriptors
>

which type of connections increase ?
the sockets ? the DB ones ?


>
> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>

you can also use lsof command


>
>
>
> On Thu, Nov 12, 2020 at 10:56 AM Martin Grigorov 
> wrote:
>
> > On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan  wrote:
> >
> > > Chris,
> > >
> > > I was load testing using the ec2 load balancer dns. I have increased
> the
> > > connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I
> am
> > > not seeing connection timeout in nginx logs now. No errors in
> kernel.log
> > I
> > > am not seeing any errors in tomcat catalina.out.
> > > During regular operations when the request count is between 4 to 6k
> > > requests per minute the open files count for the tomcat process is
> > between
> > > 200 to 350. Responses from tomcat are within 5 seconds.
> > > If the requests count goes beyond 6.5 k open files slowly move up  to
> > 2300
> > > to 3000 and the request responses from tomcat become slow.
> > >
> > > I am not concerned about high open files as I do not see any errors
> > related
> > > to open files. Only side effect of  open files going above 700 is the
> > > response from tomcat is slow. I checked if this is caused from elastic
> > > search, aws cloud watch shows elastic search response is within 5
> > > milliseconds.
> > >
> > > what might be the reason that when the open files goes beyond 600, it
> > slows
> > > down the response time for tomcat. I tried with tomcat 9 and it's the
> > same
> > > behavior
> > >
> >
> > Do you know what kind of files are being opened ?
> >
> >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> > > ch...@christopherschultz.net> wrote:
> > >
> > > > Ayub,
> > > >
> > > > On 11/3/20 10:56, Ayub Khan wrote:
> > > > > *I'm curious about why you are using all of cloudflare and ALB and
> > > > > nginx.Seems like any one of those could provide what you are
> getting
> > > from
> > > > > all3 of them. *
> > > > >
> > > > > Cloudflare is doing just the DNS and nginx is doing ssl termination
> > > >
> > > > What do you mean "Cloudflare is doing just the DNS?"
> > > >
> > > > So what is ALB doing, then?
> > > >
> > > > > *What is the maximum number of simultaneous requests that one
> > > > nginxinstance
> > > > > will accept? What is the maximum number of simultaneous
> > proxiedrequests
> > > > one
> > > > > nginx instance will make to a back-end Tomcat node? Howmany nginx
> > nodes
> > > > do
> > > > > you have? How many Tomcat nodes?  *
> > > > >
> > > > > We have 4 vms each having nginx and tomcat running on them and each
> > > > tomcat
> > > > > has nginx in front of them to proxy the requests. So it's one Nginx
> > > > > proxying to a dedicated tomcat on the same VM.
> > > >
> > > > Okay.
> > > >
> > > > > below is the tomcat connector configuration
> > > > >
> > > > >  > > > > connectionTimeout="6" maxThreads="2000"
> > > > >
>  protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > URIEncoding="UTF-8"
> > > > > redirectPort="8443" />
> > > >
> > > > 60 seconds is a *long* time for a connection timeout.
> > > >
> > > > Do you actually need 2000 threads? That's a lot, though not insane.
> > 2000
> > > > threads means you expect to handle 2000 concurrent (non-async,
> > > > non-Wewbsocket) requests. Do you need that (per node)? Are you
> > expecting
> > > > 8000 concurrent requests? Does your load-balancer understand the
> > > > topography and current-load on any given

Re: NGINX + tomcat 8.0.35 (110: Connection timed out)

[Martin Grigorov]

On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan  wrote:

> Chris,
>
> I was load testing using the ec2 load balancer dns. I have increased the
> connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I am
> not seeing connection timeout in nginx logs now. No errors in kernel.log I
> am not seeing any errors in tomcat catalina.out.
> During regular operations when the request count is between 4 to 6k
> requests per minute the open files count for the tomcat process is between
> 200 to 350. Responses from tomcat are within 5 seconds.
> If the requests count goes beyond 6.5 k open files slowly move up  to 2300
> to 3000 and the request responses from tomcat become slow.
>
> I am not concerned about high open files as I do not see any errors related
> to open files. Only side effect of  open files going above 700 is the
> response from tomcat is slow. I checked if this is caused from elastic
> search, aws cloud watch shows elastic search response is within 5
> milliseconds.
>
> what might be the reason that when the open files goes beyond 600, it slows
> down the response time for tomcat. I tried with tomcat 9 and it's the same
> behavior
>

Do you know what kind of files are being opened ?


>
>
>
>
>
>
> On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Ayub,
> >
> > On 11/3/20 10:56, Ayub Khan wrote:
> > > *I'm curious about why you are using all of cloudflare and ALB and
> > > nginx.Seems like any one of those could provide what you are getting
> from
> > > all3 of them. *
> > >
> > > Cloudflare is doing just the DNS and nginx is doing ssl termination
> >
> > What do you mean "Cloudflare is doing just the DNS?"
> >
> > So what is ALB doing, then?
> >
> > > *What is the maximum number of simultaneous requests that one
> > nginxinstance
> > > will accept? What is the maximum number of simultaneous proxiedrequests
> > one
> > > nginx instance will make to a back-end Tomcat node? Howmany nginx nodes
> > do
> > > you have? How many Tomcat nodes?  *
> > >
> > > We have 4 vms each having nginx and tomcat running on them and each
> > tomcat
> > > has nginx in front of them to proxy the requests. So it's one Nginx
> > > proxying to a dedicated tomcat on the same VM.
> >
> > Okay.
> >
> > > below is the tomcat connector configuration
> > >
> > >  > > connectionTimeout="6" maxThreads="2000"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > URIEncoding="UTF-8"
> > > redirectPort="8443" />
> >
> > 60 seconds is a *long* time for a connection timeout.
> >
> > Do you actually need 2000 threads? That's a lot, though not insane. 2000
> > threads means you expect to handle 2000 concurrent (non-async,
> > non-Wewbsocket) requests. Do you need that (per node)? Are you expecting
> > 8000 concurrent requests? Does your load-balancer understand the
> > topography and current-load on any given node?
> >
> > > When I am doing a load test of 2000 concurrent users I see the open
> files
> > > increase to 10,320 and when I take thread dump I see the threads are
> in a
> > > waiting state.Slowly as the requests are completed I see the open files
> > > come down to normal levels.
> >
> > Are you performing your load-test against the CF/ALB/nginx/Tomcat stack,
> > or just hitting Tomcat (or nginx) directly?
> >
> > Are you using HTTP keepalive in your load-test (from the client to
> > whichever server is being contacted)?
> >
> > > The output of the below command is
> > > sudo cat /proc/sys/kernel/pid_max
> > > 131072
> > >
> > > I am testing this on a c4.8xlarge VM in AWS.
> > >
> > > below is the config I changed in nginx.conf file
> > >
> > > events {
> > >  worker_connections 5;
> > >  # multi_accept on;
> > > }
> >
> > This will allow 50k incoming connections, and Tomcat will accept an
> > unbounded number of connections (for NIO connector). So limiting your
> > threads to 2000 only means that the work of each request will be done in
> > groups of 2000.
> >
> > > worker_rlimit_nofile 3;
> >
> > I'm not sure how many connections are handled by a single nginx worker.
> > If you accept 50k connections and only allow 30k file handles, you may
> > have a problem if that's all being done by a single worker.
> >
> > > What would be the ideal config for tomcat and Nginx so this setup on
> > > c4.8xlarge vm could serve at least 5k or 10k requests simultaneously
> > > without causing the open files to spike to 10K.
> >
> > You will never be able to serve 10k simultaneous requests without having
> > 10k open files on the server. If you mean 10k requests across the whole
> > 4-node environment, then I'd expect 10k requests to open (roughly) 2500
> > open files on each server. And of course, you need all kinds of other
> > files open as well, from JAR files to DB connections or other network
> > connections.
> >
> > But each connection needs a file descriptor, full stop. If you need 

Re: Timeout waiting to read data from client

[Martin Grigorov]

On Thu, Nov 12, 2020 at 12:34 AM Jerry Malcolm 
wrote:

> On 11/9/2020 11:05 AM, Jerry Malcolm wrote:
> >
> > On 11/9/2020 3:10 AM, Mark Thomas wrote:
> >> On 08/11/2020 01:33, Jerry Malcolm wrote:
> >>> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
>  Jerry,
> 
>  On 11/6/20 19:49, Jerry Malcolm wrote:
> > I have a relatively new environment with a standalone tomcat (8.5)
> > running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP. Its a direct
> > connection to port 443.  (Well technically, I have firewallD in the
> > flow in order to route the protected port 80 to port 8080 and 443 to
> > 8443 for TC).
> >
> > I am doing some stress testing on the server and failing miserably.
> > I am sending around 130 ajax calls in rapid succession using HTTP/2.
> > These are all very simple small page (JSP) requests.  Not a lot of
> > processing required. The first ~75 requests process normally.  Then
> > everything hangs up.  In the tomcat logs I'm getting a bunch of
> > "Timeout waiting to read data from client" exceptions. And in the
> > stacktrace for these exceptions, they are all occurring when I'm
> > trying to access a parameter from the request.  Looking at the
> > request network timing in the browser console, I see a bunch of the
> > requests returning in typical time of a few milliseconds. Then
> > another large block of requests that all start returning around 4
> > seconds, then another block that wait until 8 seconds to return.
> > I've tried firefox and chrome with the same results.
> >
> > I've been using httpd in front of TC for years.  So this is the first
> > time I'm running TC standalone.  It is very likely I've got some
> > parameters set horribly wrong.  But I have no clue where to start.
> > This is not a tiny EC2, and my internet connection is not showing any
> > signs of problems.  So I really don't think this is a
> > performance-related problem.  The problem is very consistent and
> > reproducible with the same counts of success/failure calls. What
> > could be causing the "Timeout waiting to read data from client" after
> > 75 calls, and then cause blocks of calls to wait 4 seconds, 8
> > seconds, etc before responding?  I really need to handle more
> > simultaneous load that this is currently allowing.
> >
> > Thanks in advance for the education.
>  Are you using HTTP Keepalives on your connections? Are you actually
>  re-using those connections in your test? What is your keepalive
>  timeout on your . Actually, what is your whole 
>  configuration?
> 
>  -chris
> 
> >>> Hi Chris, here are my two connector definitions from server.xml:
> >>>
> >>>   >>>port="8080"
> >>>protocol="HTTP/1.1"
> >>>connectionTimeout="2"
> >>>redirectPort="443" />
> >>>
> >>>   >>>port="8443"
> >>>maxThreads="150"
> >>>connectionTimeout="2"
> >>>SSLEnabled="true"
> >>>scheme="https"
> >>>secure="true"
> >>>clientAuth="false"
> >>>SSLCertificateFile="ssl/a.com/cert.pem"
> >>>SSLCertificateChainFile="ssl/a.com/chain.pem"
> >>>SSLCertificateKeyFile="ssl/a.com/privkey.pem">
> >>>   >>> className="org.apache.coyote.http2.Http2Protocol" />
> >>>  
> >> How are you stress testing this? All on a single HTTP/2 connection or
> >> multiple connections? With which tool?
> >>
> >> You might want to test HTTP/1.1 requests (with and without TLS) to see
> >> if the problem is specific to HTTP/2 or TLS as that should help narrow
> >> down the root cause.
> >>
> >> Mark
> >
> > Hi Mark, technically it's not a 'designed' stress test.  It's real
> > production code that just happens to stress the server more than
> > usual.  It's just a page that makes a bunch of ajax calls, and the
> > responses to each of those issue a second ajax call.
> >
> > If you don't see anything obvious in my configuration, we will
> > definitely pursue the http/1.1 options, etc.  I just wanted to
> > eliminate the chance of obvious 'pilot error' before digging deeper.
> >
> > Specifically, where is that error detected in the TC flow? In my logs
> > it fails on getting request parameters.  It sounds like the input
> > reader for the request is getting blocked.But the first part of
> > the request is getting in since it does route to the appropriate JSP.
> > Just seems strange that the http/2 or ssl layers would let half of the
> > request in and then block the rest of the request.  The browser
> > appears to be sending everything.  And it fails the same using firefox
> > or chrome.  Any ideas?
> >
> > Thx
> >
> >
> Update on this.  One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM
> after things locked up.  I removed the http2 'upgrade protocol' line
> from my connector, and everything works.  So it's apparently something

Re: Tomcat - Sentry configuration problem

[Martin Grigorov]

On Wed, Nov 11, 2020 at 3:16 PM Tamás Barta  wrote:

> Hi,
>
> > Why do you use 1.7.29 ?
>
> Because our company has a Sentry server supports this version of clients.
>
> > Do you specify this class name in logging.properties ? Is
> this class in 1.7.29 ?
>
> Yes and yes. If I add sentry jars to classpath, sentry works well.
> Exceptions are sent to the server and visible in web interface.
>
> > You just need to add it as a normal Maven dependency.
>
> Maven dependency is needed only If I would like to use Sentry in code. But
> I don't.
>
> > If you need to really put it on the bootstrap.
>
> When I say bootstrap I mean Tomcat bootstrap:
> https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html
> Which is standard java classpath, I see.
>
> BTW if I add jars to java bootclasspath, I got the same error.
>
> Just a very interesting test: if I add an empty JAR (no class files in it)
> file to setenv.sh (no Sentry jars at all) that I got the same error.
> It seems that adding an empty jar to classpath case this classloading
> problem :(
>

I've just tested this on 10.0.0-M9

Using CATALINA_BASE:   /home/martin/devel/apache-tomcat-10.0.0-M9
Using CATALINA_HOME:   /home/martin/devel/apache-tomcat-10.0.0-M9
Using CATALINA_TMPDIR: /home/martin/devel/apache-tomcat-10.0.0-M9/temp
Using JRE_HOME:/home/martin/devel/jdk-15
Using CLASSPATH:
:/home/martin/devel/apache-tomcat-10.0.0-M9/empty.jar:/home/martin/devel/apache-tomcat-10.0.0-M9/bin/bootstrap.jar:/home/martin/devel/apache-tomcat-10.0.0-M9/bin/tomcat-juli.jar
Using CATALINA_OPTS:
NOTE: Picked up JDK_JAVA_OPTIONS:
 --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

$ cat bin/setenv.sh
export CLASSPATH=$CLASSPATH:$CATALINA_BASE/empty.jar

$ ll empty.jar
-rw-r- 1 martin martin 22 ное 11 15:48 empty.jar

$ jar --list -f empty.jar


There is no problem with loading this empty jar and there are no class
loading issues either.



>
> Regards, Tamas
>
>
>
>
>
>
> On Wed, Nov 11, 2020 at 12:33 PM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta 
> wrote:
> >
> > > Hi,
> > >
> > > The ways you wrote is not usable, because they cause the following
> > problem:
> > >
> > > Handler error
> > > java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > at
> > >
> > >
> >
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582)
> > > at
> > >
> > >
> >
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524)
> > > at
> > >
> > >
> >
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322)
> > > at java.util.logging.LogManager$3.run(LogManager.java:399)
> > > at java.util.logging.LogManager$3.run(LogManager.java:396)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at
> > >
> > >
> >
> java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
> > > at java.util.logging.LogManager.access$800(LogManager.java:145)
> > > at java.util.logging.LogManager$2.run(LogManager.java:345)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at
> > >
> > >
> >
> java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
> > > at
> > java.util.logging.LogManager.getLogManager(LogManager.java:378)
> > > at java.util.logging.Logger.demandLogger(Logger.java:448)
> > > at java.util.logging.Logger.getLogger(Logger.java:502)
> > > at
> > > org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67)
> > > at
> > > org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187)
> > > at
> > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117)
> > > at
> > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141)
> > > at
> org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196)

Re: Tomcat - Sentry configuration problem

[Martin Grigorov]

Hi,

On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta  wrote:

> Hi,
>
> The ways you wrote is not usable, because they cause the following problem:
>
> Handler error
> java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
>
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582)
> at
>
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524)
> at
>
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322)
> at java.util.logging.LogManager$3.run(LogManager.java:399)
> at java.util.logging.LogManager$3.run(LogManager.java:396)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
> java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
> at java.util.logging.LogManager.access$800(LogManager.java:145)
> at java.util.logging.LogManager$2.run(LogManager.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
> java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
> at java.util.logging.LogManager.getLogManager(LogManager.java:378)
> at java.util.logging.Logger.demandLogger(Logger.java:448)
> at java.util.logging.Logger.getLogger(Logger.java:502)
> at
> org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67)
> at
> org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187)
> at
> org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117)
> at
> org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141)
> at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196)
> at
> org.apache.catalina.startup.Bootstrap.(Bootstrap.java:52)
> 10-Nov-2020 17:44:11.137 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server version:
>  Apache Tomcat/8.5.23
> 10-Nov-2020 17:44:11.139 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server built:
>  Sep 28 2017 10:30:11 UTC
> ...
>
> The Sentry log handler must be in Boostrap jars, it must be loaded before
> web application and before jars in $CATALINA_BASE/lib. If I am right, the
> best way to do that is by setting it in setenv.sh.
>

I haven't used Sentry before so I just looked it up.
https://docs.sentry.io/platforms/java/usage/ says that the current version
is 3.1.1. Why do you use 1.7.29 ? There is no io.sentry.jul.SentryHandler
class in 3.1.1. Do you specify this class name in logging.properties ? Is
this class in 1.7.29 ?
There is nothing about "bootstrap" on
https://docs.sentry.io/platforms/java/usage/ or at
https://docs.sentry.io/platforms/java/guides/spring-boot/. You just need to
add it as a normal Maven dependency.
If you need to really put it on the bootstrap path then you need to use
-Xbootclasspath:...
By using CLASSPATH=$CLASSPATH:... you actually *append* jars to the
classpath, while you sound like you need to *prepend* to it.


>
> I can't imagine how could it be possible without setting sentry jar the web
> application works well, there is no problem, but if I add sentry-1.7.29.jar
> to classpath then ServletContainerInitializer won't be found by Catalina.
> This class is only in $CATALINA_BASE/lib/servlet-api.jar.
>
> Thanks, Tamás
>
> On Wed, Nov 11, 2020 at 9:46 AM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta 
> wrote:
> >
> > > Hi,
> > >
> > > Now it is a Spring Boot web application project without embedded
> tomcat.
> > > There is no library in war with javax.servlet api classes. There error
> is
> > > the same:
> > >
> > > 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1]
> > > org.apache.catalina.core.ContainerBase.addChildInternal
> > > ContainerBase.addChild: start:
> > >  org.apache.catalina.LifecycleException: Failed to start component
> > >
> > >
> >
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]]
> > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
> > > at
> > >
> > >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
> > > at

Re: Tomcat - Sentry configuration problem

[Martin Grigorov]

Hi,

On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta  wrote:

> Hi,
>
> Now it is a Spring Boot web application project without embedded tomcat.
> There is no library in war with javax.servlet api classes. There error is
> the same:
>
> 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1]
> org.apache.catalina.core.ContainerBase.addChildInternal
> ContainerBase.addChild: start:
>  org.apache.catalina.LifecycleException: Failed to start component
>
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]]
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> javax/servlet/ServletContainerInitializer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1277)
> at
>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
>
> org.apache.catalina.startup.WebappServiceLoader.loadServices(WebappServiceLoader.java:188)
> at
>
> org.apache.catalina.startup.WebappServiceLoader.load(WebappServiceLoader.java:159)
> at
>
> org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1622)
> at
>
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1135)
> at
>
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775)
> at
>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
>
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5105)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> ... 10 more
> Caused by: java.lang.ClassNotFoundException:
> javax.servlet.ServletContainerInitializer
>

Definitely you mess up the classpath and there are none or more than one
servlet-api.jar in the classpath.

What happens if you put your sentry related jars in the .war file
(/WEB-INF/lib/) ? I.e. add them as proper Maven dependencies
Or if you put them in $CATALINA_BASE/lib ?
Why did you go with editing setenv.sh in first place ?


> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 38 more
>
> On Tue, Nov 10, 2020 at 11:28 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Tamás,
> >
> > On 11/10/20 15:42, Tamás Barta wrote:
> > > This is the content of setenv.sh:
> > >
> > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/sentry-1.7.29.jar
> > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/slf4j-api-1.7.24.jar
> > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/jackson-core-2.8.10.jar
> > >
> > > The web application without this sentry configuration works well for
> > ages.
> > > But if I add the above configuration, it won't.
> > >
> > > Yes, it is a maven project, now I checked and I see that there are
> Tomcat
> > > jar-s in it:
> > >
> > > tomcat-annotations-api-8.5.23.jar  tomcat-embed-core-8.5.23.jar
> > >   tomcat-embed-el-8.5.23.jar  

Re: Can Tomcat 9 be FIPS compliant without OpenSSL?

[Martin Grigorov]

Hi,

On Fri, Nov 6, 2020 at 8:57 AM Avik Ray  wrote:

> Dear team,
> Sending this query again after subscribing to the mailing list. Sent
> it originally 3 days back, but just saw an error response in the spam
> folder asking to subscribe first.
>
> We are using Tomcat 9.0.37 x64 on Windows Server 2016 OS and the NIO
> connector with JSSE, without an underlying OpenSSL.
>
> As per Tomcat 9 docs, the only mention of FIPS compliant operation I
> see is in the config of APR lifecycle listener, with the expectation
> of an underlying OpenSSL implementation that can be set to FIPS
> enabled mode. Ref:
> https://tomcat.apache.org/tomcat-9.0-doc/config/listeners.html
>
> Is it possible to be FIPS compliant with the usage of Tomcat, without
> the above setting? We were thinking of using BouncyCastle FIPS as the
> underlying Java crypto provider instead of OpenSSL for multiple
> reasons.
>
> Are there any other dependencies Tomcat has on the underlying stack,
> besides that provided by a Java crypto provider like BC-FIPS, having a
> bearing on FIPS compliance?
>
> Please advise, as this is urgent for a FIPS compliance decision.
>

Please check the README of this project -
https://github.com/amitlpande/tomcat-9-fips
Amit Pande recently shared it here at users@.

Regards,
Martin


>
> Thanks,
> Avik Ray
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: GraalVM native image for embedded Tomcat

[Martin Grigorov]

On Fri, Oct 16, 2020 at 3:43 PM Martin Grigorov 
wrote:

> Hi,
>
> I am trying to create a GraalVM native image for an application that uses
> tomcat-embed-core but it fails with the following error:
>
> [INFO] --- native-image-maven-plugin:20.2.0:native-image (default) @
> tomcat-embedded ---
> [INFO] ImageClasspath Entry:
> org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.39:compile
> (file:///home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar)
> [INFO] ImageClasspath Entry:
> org.apache.tomcat:tomcat-annotations-api:jar:9.0.39:compile
> (file:///home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar)
> [INFO] ImageClasspath Entry:
> info.mgsolutions:tomcat-embedded:jar:1.0-SNAPSHOT
> (file:///home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar)
> [INFO] Executing:
> /home/martin/devel/graalvm-ce-java11-20.2.0/bin/native-image -cp
> /home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar:/home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar:/home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar
> -H:Class=info.mgsolutions.tomcat.TomcatEmbedded
> [info.mgsolutions.tomcat.tomcatembedded:568255]classlist:   2,047.64
> ms,  0.96 GB
> [info.mgsolutions.tomcat.tomcatembedded:568255](cap): 852.60
> ms,  0.96 GB
> [info.mgsolutions.tomcat.tomcatembedded:568255]setup:   2,323.40
> ms,  0.96 GB
> Fatal error:java.util.MissingResourceException: Can't find bundle for base
> name org.apache.tomcat.util.descriptor.tld.LocalStrings, locale en_US
> at
> java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
> at
> java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
> at
> java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
> at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284)
> at
> com.oracle.svm.util.ModuleSupport.getResourceBundleFallback(ModuleSupport.java:81)
> at
> com.oracle.svm.util.ModuleSupport.getResourceBundle(ModuleSupport.java:62)
> at
> com.oracle.svm.core.jdk.LocalizationFeature.addBundleToCache(LocalizationFeature.java:266)
> at
> com.oracle.svm.hosted.ResourcesFeature$ResourcesRegistryImpl.addResourceBundles(ResourcesFeature.java:92)
> at
> com.oracle.svm.core.configure.ResourceConfigurationParser.parseEntry(ResourceConfigurationParser.java:90)
> at
> com.oracle.svm.core.configure.ResourceConfigurationParser.parseTopLevelObject(ResourceConfigurationParser.java:71)
> at
> com.oracle.svm.core.configure.ResourceConfigurationParser.parseAndRegister(ResourceConfigurationParser.java:47)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:132)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:119)
> at
> java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:108)
> at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:326)
> at
> java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
> at
> java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
> at
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
> at
> java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491)
> at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:125)
> at
> com.oracle.svm

GraalVM native image for embedded Tomcat

[Martin Grigorov]

Hi,

I am trying to create a GraalVM native image for an application that uses
tomcat-embed-core but it fails with the following error:

[INFO] --- native-image-maven-plugin:20.2.0:native-image (default) @
tomcat-embedded ---
[INFO] ImageClasspath Entry:
org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.39:compile
(file:///home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar)
[INFO] ImageClasspath Entry:
org.apache.tomcat:tomcat-annotations-api:jar:9.0.39:compile
(file:///home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar)
[INFO] ImageClasspath Entry:
info.mgsolutions:tomcat-embedded:jar:1.0-SNAPSHOT
(file:///home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar)
[INFO] Executing:
/home/martin/devel/graalvm-ce-java11-20.2.0/bin/native-image -cp
/home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar:/home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar:/home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar
-H:Class=info.mgsolutions.tomcat.TomcatEmbedded
[info.mgsolutions.tomcat.tomcatembedded:568255]classlist:   2,047.64
ms,  0.96 GB
[info.mgsolutions.tomcat.tomcatembedded:568255](cap): 852.60
ms,  0.96 GB
[info.mgsolutions.tomcat.tomcatembedded:568255]setup:   2,323.40
ms,  0.96 GB
Fatal error:java.util.MissingResourceException: Can't find bundle for base
name org.apache.tomcat.util.descriptor.tld.LocalStrings, locale en_US
at
java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
at
java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
at
java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284)
at
com.oracle.svm.util.ModuleSupport.getResourceBundleFallback(ModuleSupport.java:81)
at
com.oracle.svm.util.ModuleSupport.getResourceBundle(ModuleSupport.java:62)
at
com.oracle.svm.core.jdk.LocalizationFeature.addBundleToCache(LocalizationFeature.java:266)
at
com.oracle.svm.hosted.ResourcesFeature$ResourcesRegistryImpl.addResourceBundles(ResourcesFeature.java:92)
at
com.oracle.svm.core.configure.ResourceConfigurationParser.parseEntry(ResourceConfigurationParser.java:90)
at
com.oracle.svm.core.configure.ResourceConfigurationParser.parseTopLevelObject(ResourceConfigurationParser.java:71)
at
com.oracle.svm.core.configure.ResourceConfigurationParser.parseAndRegister(ResourceConfigurationParser.java:47)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:132)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:119)
at
java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:108)
at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:326)
at
java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at
java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491)
at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:125)
at
com.oracle.svm.hosted.ResourcesFeature.beforeAnalysis(ResourcesFeature.java:105)
at
com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$7(NativeImageGenerator.java:693)
at
com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70)
at
com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:693)
at
com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:555)
at
com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:468)

Re: Deploying war, Negative Date exception

[Martin Grigorov]

Hi Peter,

On Mon, Oct 12, 2020 at 2:20 PM Peter Henderson 
wrote:

> Hello fellow tomcat users.
>
> My environment.
> Tomcat: 9.0.39
> Java: openjdk 11.0.8 2020-07-14
> OS: Ubuntu 18.04.5 LTS
>
> Source code [0]
>
> When deploying this war [1], by copying it into the webapps directory,
> I get this exception. [2]
> java.lang.IllegalArgumentException: Negative time
>
>
> I only started seeing this exception when I upgraded my projects build tool
> version
> from
> sbt.version=1.3.10
> to
> sbt.version=1.4.0
>
>
> Is this a tomcat bug, a build tool bug or most likely something I'm doing
> wrong?
>
> Thanks
> Peter.
>
>
> [0]
> https://github.com/bollinger/NegativeDate
>
> [1]
> https://github.com/bollinger/NegativeDate/blob/master/Negative.war
>
> [2]
> 2020-10-12 11:41:35.932 SEVERE oacs.HostConfig Error deploying web
> application archive [/home/peter/apache-tomcat-9.0.39/webapps/Negative.war]
> java.lang.IllegalStateException: Error starting child
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1848)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at
>
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
>
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.catalina.LifecycleException: Failed to start
> component
>
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Negative]]
> at
>
> org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
> ... 24 more
> Caused by: java.lang.IllegalArgumentException: Negative time
> at java.base/java.io.File.setLastModified(File.java:1441)
>

The negative time comes from jarEntry.getTime() -
https://github.com/apache/tomcat/blob/d6e5d838a9565c0c4bfeca4396e6266d21a80288/java/org/apache/catalina/startup/ExpandWar.java#L167
I guess SBT creates the JAR/WAR so most probably SBT 1.4.0 is to blame here.

Martin


> at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:169)
> at
>
> org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:820)
> at
>
> org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:958)
> at
>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
>
> org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
> ... 25 more
>
> --
> Peter Henderson
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

Hi Arshiya,


On Fri, Oct 9, 2020 at 2:33 PM Arshiya Shariff
 wrote:

> Hi,
>
> Mark , with the test runs that I performed over clean 9.0.x branch I was
> not able to reproduce this. But with 9.0.38 and the jars built from 9.0.x
> with hash: c8ec2d4cde3a31b0e9df9a30e7915d77ba725545  , with 700 or 1000
> users (connections) and on sending 1000 Requests per second (or even
> lesser) , payload of 16K  from JMeter I can see that this Exception occurs
> within few minutes of starting the test . The maxThreads configured in
> tomcat is 200 .
>
> How often do you see these errors in your test run?
> Randomly, at times 2 or 3 such traces.
>
> Do you have the other end of that stack trace?
> It is only the two lines that is recursively printed till the end about
> ~500 times in one trace  :
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>
> I see the trace starting with :
> Exception in thread "http-nio-x.y.z-1090-exec-107"
> java.lang.StackOverflowError
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>
> (OR)
>
> Exception in thread "http-nio-x.y.z-1090-exec-87"
> java.lang.StackOverflowError
> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> .
> .
> .
> .
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>
> Is there anything that was fixed around this in latest 9.0.x branch ?
>

9.0.39 is being voted now at dev@ mailing list:

It can be obtained from:
https://dist.apache.org/repos/dist/dev/tomcat/tomcat-9/v9.0.39/
The Maven staging repo is:
https://repository.apache.org/content/repositories/orgapachetomcat-1281/

Give it a try and vote!


> Thanks and Regards
> Arshiya Shariff
>
> -Original Message-
> From: Mark Thomas 
> Sent: Monday, October 5, 2020 9:52 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to
> 1000/2000 (Embedded tomcat 9.0.38)
>
> On 05/10/2020 10:56, Arshiya Shariff wrote:
> > Hi All,
> >
> > Thank you so much Mark .
> > We tested the jars built from latest 9.0.x  with 2000 / 5000 users
> > (connections) from JMeter , We see a very good improvement with the
> > heap usage
>
> Good news. As is the fact that the other errors have been cleared up.
>
> > But I see this exception printed multiple times , I am not sure why this
> occurs :
> > Exception in thread "http-nio-x.y.z-1234-exec-213"
> java.lang.StackOverflowError
> > at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> > at org.apache.tomcat.util.net
> .NioChannel.read(NioChannel.java:174)
> > at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> > at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> > at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> > at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> > at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> > at org.apache.tomcat.util.net
> 

Re: Fwd: Re: At least one JAR was scanned for TLDs yet contained no TLDs.

[Martin Grigorov]

On Thu, Oct 8, 2020 at 3:31 PM Raivo Rebane  wrote:

> Hello
>
> if I start standalone tomcat program looks like:
>
> 17868 ?Sl 0:02 /usr/lib/jvm/default-java/bin/java
> -Djava.util.logging.config.file=/opt/tomcat/latest/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M
> -Xmx1024M -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath
> /opt/tomcat/latest/bin/bootstrap.jar:/opt/tomcat/latest/bin/tomcat-juli.jar
>
> -Dcatalina.base=/opt/tomcat/latest -Dcatalina.home=/opt/tomcat/latest
> -Djava.io.tmpdir=/opt/tomcat/latest/temp
> org.apache.catalina.startup.Bootstrap start
>
> No jars is added to classpath
>

What are you trying to do?
Why do you start it this way and what issue do you think you have with the
TLDs ?


>
> Raivo
>
> On 08.10.20 13:59, Raivo Rebane wrote:
>
> >
> >
> >
> >  Forwarded Message 
> > Subject: Re: At least one JAR was scanned for TLDs yet contained
> > no TLDs.
> > Date: Thu, 8 Oct 2020 13:37:49 +0300
> > From: Raivo Rebane 
> > To: Martin Grigorov ,
> > users-get.123_...@tomcat.apache.org
> >
> >
> >
> > Hello
> >
> > I debaged the jars list and got following:
> >
> > 08-Oct-2020 13:27:07.240 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-parser.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-transcoder.jar].
> > Consider adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-gui-util.jar].
> > Consider adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-svggen.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-xml.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-awt-util.jar].
> > Consider adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.242 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-util.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.242 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-script.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.242 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-brid

Re: Re: At least one JAR was scanned for TLDs yet contained no TLDs.

[Martin Grigorov]

On Thu, Oct 8, 2020, 14:00 Raivo Rebane  wrote:

>
>
>
>  Forwarded Message 
> Subject:Re: At least one JAR was scanned for TLDs yet contained no
> TLDs.
> Date:   Thu, 8 Oct 2020 13:37:49 +0300
> From:   Raivo Rebane 
> To: Martin Grigorov ,
> users-get.123_...@tomcat.apache.org
>
>
>
> Hello
>
> I debaged the jars list and got following:
>
> 08-Oct-2020 13:27:07.240 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-parser.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-transcoder.jar].
> Consider adding the JAR to the
> tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-gui-util.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-svggen.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-xml.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-awt-util.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-util.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-script.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-bridge.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-dom.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-gvt.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-swing.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-constants.jar].
> Consider adding the JAR to the
> tomcat.util.scan.StandardJarScanFilter.jarsT

Re: At least one JAR was scanned for TLDs yet contained no TLDs.

[Martin Grigorov]

Hi,

On Wed, Oct 7, 2020 at 9:46 PM Raivo Rebane  wrote:

> Hello
>
> If I start Tomcat9 I got following:
>
> [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR
> was scanned for TLDs yet contained no TLDs. Enable debug logging for
> this logger for a complete list of JARs that were scanned but no TLDs
> were found in them.
>
>   I would like to know which JARS is mentioned
>
> I add following lines to catalina.properies:
>
> org.apache.jasper.compiler.TldLocationsCache.level = ALL
> org.apache.jasper.servlet.TldScanner.level = ALL
>
> But I dont have list of these JARS
>
> And how to add these JARS to classpath anyway ?
>

Enable debug for org.apache.jasper.servlet.TldScanner


>
> Regards
>
> Raivo
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Is Tomcat7 supports HTTP2

[Martin Grigorov]

Hi,

On Thu, Oct 8, 2020 at 9:32 AM Tosh, Bibhuti Bhusan (Bibhuti) <
bt...@avaya.com> wrote:

> HI All,
> I am an user of tomcat7 version. I wanted to know this version tomcat
> 7.0.105 supports HTTP2 and CVE-2020-11996 is still applicable to tis
> version. I did not any reference of tomcat7 supporting HTTP2 and so asking
> this questions to community. Thanks in advance.
>

No, HTTP/2 is available in 8.5.+


>
> Regards,
> --Bibhuti
>
>

Re: VersionLoggerListener

[Martin Grigorov]

On Wed, Oct 7, 2020 at 2:19 PM Raivo Rebane  wrote:

> Hello
>
> I removed  org.apache.catalina.startup.VersionLoggerListener
>
> but I got following error:
>
> java.lang.NoSuchMethodError: 'org.apache.catalina.Engine
> org.apache.catalina.Service.getContainer()'
>

I'd recommend you to download a new .zip/.tgz, or re-install Tomcat.

Martin


> Regards
>
> Raivo
>
> On 07.10.20 14:01, Martin Grigorov wrote:
> > Hi,
> >
> > On Wed, Oct 7, 2020 at 1:51 PM Raivo Rebane  wrote:
> >
> >> Hello
> >>
> >> My tomcat9 don't starts and gives error:
> >>
> >>java.lang.ClassNotFoundException:
> >> org.apache.catalina.startup.VersionLoggerListener
> >>
> >> How to resolve this problem ?
> >>
> > Download Tomcat again. Your installation must be corrupted somehow.
> > Or remove org.apache.catalina.startup.VersionLoggerListener from
> > conf/server.xml
> >
> >
> >> Regards
> >>
> >> Raivo
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
>

Re: VersionLoggerListener

[Martin Grigorov]

Hi,

On Wed, Oct 7, 2020 at 1:51 PM Raivo Rebane  wrote:

> Hello
>
> My tomcat9 don't starts and gives error:
>
>   java.lang.ClassNotFoundException:
> org.apache.catalina.startup.VersionLoggerListener
>
> How to resolve this problem ?
>

Download Tomcat again. Your installation must be corrupted somehow.
Or remove org.apache.catalina.startup.VersionLoggerListener from
conf/server.xml


>
> Regards
>
> Raivo
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:47 PM Mark Thomas  wrote:

> On 06/10/2020 10:35, Martin Grigorov wrote:
> > On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov 
> > wrote:
> >
> >>
> >>
> >> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:
> >>
> >>> On 06/10/2020 07:30, Martin Grigorov wrote:
> >>>> Hi,
> >>>>
> >>>> I face an issue with the NIO2 protocol.
> >>>> When I increase the request rate to more than 500 requests per second
> it
> >>>> starts failing with:
> >>>
> >>> I'm unable to reproduce this with a 10.0.x build requesting a simple
> >>> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
> >>>
> >>> I see around 15k req/s and no failures.
> >>>
> >>
> >> I use latest 9.0.x.
> >> Let me try with non-embedded Tomcat!
> >>
> >
> > Same build of Tomcat started with ./bin/startup.sh has no such problems!
> > It seems
> >
> https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java
> > either misses something or does something more than what conf/server.xml
> > does.
> > I've tried to minify server.xml by removing all listeners, valves and
> JNDI
> > related stuff but it still works fine.
>
> maxThreads="8" setting? That looks very low to me.
>

Yep! This was the reason! Thanks!
The EOFs happen with standalone Tomcat too when maxThreads is 8.

I've reduced it because otherwise the throughput of NIO is lower. Async
Profiler (perf) says that there are many context switches.


>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov 
wrote:

>
>
> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:
>
>> On 06/10/2020 07:30, Martin Grigorov wrote:
>> > Hi,
>> >
>> > I face an issue with the NIO2 protocol.
>> > When I increase the request rate to more than 500 requests per second it
>> > starts failing with:
>>
>> I'm unable to reproduce this with a 10.0.x build requesting a simple
>> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
>>
>> I see around 15k req/s and no failures.
>>
>
> I use latest 9.0.x.
> Let me try with non-embedded Tomcat!
>

Same build of Tomcat started with ./bin/startup.sh has no such problems!
It seems
https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java
either misses something or does something more than what conf/server.xml
does.
I've tried to minify server.xml by removing all listeners, valves and JNDI
related stuff but it still works fine.


>
>
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:13 PM Rémy Maucherat  wrote:

> On Tue, Oct 6, 2020 at 8:31 AM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > I face an issue with the NIO2 protocol.
> > When I increase the request rate to more than 500 requests per second it
> > starts failing with:
> >
> > 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
> > org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State
> > [NEW]
> > 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
> > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
> > Connection [1], Stream [0], Frame type [null], Error
> > java.io.EOFException
> > at
> > org.apache.tomcat.util.net
> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
> >
>
> Also, it is a real EOF as the error says, caused by the client
> disconnecting for whatever reason (good or bad).
>

The client is exactly the same for Tomcat NIO/NIO2/APR, Golang, Node.js,
Rust and Netty.
It works for all but NIO2.


>
> Rémy
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:

> On 06/10/2020 07:30, Martin Grigorov wrote:
> > Hi,
> >
> > I face an issue with the NIO2 protocol.
> > When I increase the request rate to more than 500 requests per second it
> > starts failing with:
>
> I'm unable to reproduce this with a 10.0.x build requesting a simple
> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
>
> I see around 15k req/s and no failures.
>

I use latest 9.0.x.
Let me try with non-embedded Tomcat!


>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Unclear sentence in FailedRequestFilter's javadoc

[Martin Grigorov]

Hi Konstantin,

On Mon, Oct 5, 2020 at 2:53 PM Konstantin Kolinko 
wrote:

> пн, 5 окт. 2020 г. в 14:16, Martin Grigorov :
> >
> > Hi,
> >
> > What is the meaning of "addresses" at
> >
> https://github.com/apache/tomcat/blob/a0fb5f4b42c593fa566878dddfa213e5f3c9c499/java/org/apache/catalina/filters/FailedRequestFilter.java#L40
> > ?
> >
> > "Parameter parsing does check content type of the request, so there
> should
> > not be problems with *addresses* that use
> > request.getInputStream() and
> request.getReader()"
> >
> > I am not English native speaker, so it might be correct, but it looks
> like
> > the wrong word is used here.
>
> It looks odd for me as well, although from the commit history it looks
> that it was my commit that added that sentence [1], 9 years ago.
>
> Essentially, the text should be the same as in description of the
> filter in Configuration Reference [2][3]. It talks about "the servlet
> protected by this filter".
>
> [1] https://svn.apache.org/viewvc?view=revision=1198707
> [2] https://svn.apache.org/viewvc?view=revision=1200107
> [3]
> http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#Failed_Request_Filter


Thanks for the answer!
I've updated it to use the same text as at the website docs.


>
>
> It looks as if you are the first person who has read that Javadoc in 9
> years.
>

Yeah, usually people read the code, not the docs :-)

Cheers!
Martin


> Best regards,
> Konstantin Kolinko
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 9:30 AM Martin Grigorov  wrote:

> Hi,
>
> I face an issue with the NIO2 protocol.
> When I increase the request rate to more than 500 requests per second it
> starts failing with:
>
> 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
> org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State [NEW]
> 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
> Connection [1], Stream [0], Frame type [null], Error
> java.io.EOFException
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> at
> java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1126)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1054)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1112)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> at
> java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1038)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
> at
> org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1124)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.start(Nio2Endpoint.java:1004)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1450)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1293)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1265)
> at
> org.apache.coyote.http2.Http2AsyncParser.readConnectionPreface(Http2AsyncParser.java:55)
> at
> org.apache.coyote.http2.Http2UpgradeHandler.init(Http2UpgradeHandler.java:241)
> at
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.init(Http2AsyncUpgradeHandler.java:40)
> at
> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:316)
> at
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:40)
> at
> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1679)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at
> org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
> at
> java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadP

A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

Hi,

I face an issue with the NIO2 protocol.
When I increase the request rate to more than 500 requests per second it
starts failing with:

06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State [NEW]
06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
Connection [1], Stream [0], Frame type [null], Error
java.io.EOFException
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
at
java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
at
org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1126)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1054)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1112)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
at
java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1038)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
at
org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1124)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.start(Nio2Endpoint.java:1004)
at
org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1450)
at
org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1293)
at
org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1265)
at
org.apache.coyote.http2.Http2AsyncParser.readConnectionPreface(Http2AsyncParser.java:55)
at
org.apache.coyote.http2.Http2UpgradeHandler.init(Http2UpgradeHandler.java:241)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.init(Http2AsyncUpgradeHandler.java:40)
at
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:316)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:40)
at
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1679)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104)
at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99)
at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
at
java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:832)
06-Oct-2020 09:18:19.980 FINE [https-jsse-nio2-8080-exec-6]
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Entry,
Connection [1], SocketStatus [ERROR]

If I just change the protocol to org.apache.coyote.http11.Http11NioProtocol
then everything is OK.

To reproduce:
1) git clone https://github.com/martin-g/http2-server-perf-tests.git
2) 

Unclear sentence in FailedRequestFilter's javadoc

[Martin Grigorov]

Hi,

What is the meaning of "addresses" at
https://github.com/apache/tomcat/blob/a0fb5f4b42c593fa566878dddfa213e5f3c9c499/java/org/apache/catalina/filters/FailedRequestFilter.java#L40
?

"Parameter parsing does check content type of the request, so there should
not be problems with *addresses* that use
request.getInputStream() and request.getReader()"

I am not English native speaker, so it might be correct, but it looks like
the wrong word is used here.

Martin

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

On Thu, Oct 1, 2020 at 12:29 PM Mark Thomas  wrote:

> On 30/09/2020 18:47, Martin Grigorov wrote:
> > On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas  wrote:
> >> On 30/09/2020 16:17, Mark Thomas wrote:
>
> 
>
> >>> That is helpful. Looks like you have found a way to reproduce the
> buffer
> >>> issues reported in
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
> >>
> >> Can you share the command you used to trigger those errors please.
> >>
> >
> > The Vegeta command I used is:
> >
> > jq -ncM '{"method": "POST", "url": "
> https://localhost:8080/testbed/plaintext;,
> > "body":"payload=Some
> >
> sdgggwwsdgssfshffheeepayload"
> > | @base64, header: {"Content-Type":
> > ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> > -http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta encode >
> > /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
> >
> > The app is at
> >
> https://github.com/martin-g/http2-server-perf-tests/tree/master/java/tomcat
> .
> > Just start EmbeddedTomcat#main() with -Dtomcat.http2=true
>
> Definitely timing related as I am unable to reproduce the problem with
> that command or some variations.
>
> However, I think I have managed to track down the root cause. The good
> news is that the BufferOverflowException is largely harmless. It is a
> side-effect of the connection being closed due to an error. My guess is
> that the error was a combination of vegeta sending an unexpected reset
> frame and Tomcat maintaining state for a very small number of streams in
> some circumstances.
>
> If you could retest with the latest 9.0.x that would be very helpful.
> The memory usage, stream state maintenance and this
> BufferOverflowException should all be fixed.
>

Yesterday it was very easy to reproduce it here.
It looks good now - both exception types didn't happen in several runs!

But something new broke:


SEVERE: Servlet.service() for servlet [plaintext] in context with path []
threw exception
java.lang.NullPointerException: Cannot throw exception because "ioe" is null
at
org.apache.coyote.http2.Http2UpgradeHandler.handleAppInitiatedIOException(Http2UpgradeHandler.java:797)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.handleAsyncException(Http2AsyncUpgradeHandler.java:276)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.writeWindowUpdate(Http2AsyncUpgradeHandler.java:252)
at org.apache.coyote.http2.Stream$StreamInputBuffer.doRead(Stream.java:1088)
at org.apache.coyote.Request.doRead(Request.java:555)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:336)
at
org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:632)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:362)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132)
at org.apache.catalina.connector.Request.readPostBody(Request.java:3308)
at org.apache.catalina.connector.Request.parseParameters(Request.java:3241)
at org.apache.catalina.connector.Request.getParameter(Request.java:1124)
at
org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381)
at info.mgsolutions.tomcat.PlainTextServlet.doPost(PlainTextServlet.java:41)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

I will improve it!


> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas  wrote:

> On 30/09/2020 16:17, Mark Thomas wrote:
> > On 30/09/2020 13:53, Martin Grigorov wrote:
> >> On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov 
> >
> >
> > 
> >
> >> When I load test HTTP2 with POST (with big bodies) there are many errors
> >> like:
> >>
> >> 1)
> >> Exception in thread "https-jsse-nio-8080-exec-5"
> >> java.nio.BufferOverflowException
> >> at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
> >> at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
> >> at
> >> org.apache.tomcat.util.net
> .SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
> >> at
> >> org.apache.tomcat.util.net
> .SocketWrapperBase.unRead(SocketWrapperBase.java:401)
> >> at
> >>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
> >> at
> >>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
> >> at
> >> org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
> >> at
> >> org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> >> at
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> >> at
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> >> at
> >>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >> at java.base/java.lang.Thread.run(Thread.java:832)
> >>
> >> 2)
> >> Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller
> events
> >> SEVERE: Failed to register socket with selector from poller
> >> java.nio.channels.ClosedChannelException
> >> at
> >>
> java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
> >> at
> >> org.apache.tomcat.util.net
> .NioEndpoint$Poller.events(NioEndpoint.java:609)
> >> at org.apache.tomcat.util.net
> .NioEndpoint$Poller.run(NioEndpoint.java:703)
> >> at java.base/java.lang.Thread.run(Thread.java:832)
> >
> > That is helpful. Looks like you have found a way to reproduce the buffer
> > issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>
> Can you share the command you used to trigger those errors please.
>

The Vegeta command I used is:

jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext;,
"body":"payload=Some
sdgggwwsdgssfshffheeepayload"
| @base64, header: {"Content-Type":
["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
-http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta encode >
/tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .

The app is at
https://github.com/martin-g/http2-server-perf-tests/tree/master/java/tomcat.
Just start EmbeddedTomcat#main() with -Dtomcat.http2=true

Martin


>
> Thanks,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov 
wrote:

> Hi,
>
> On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas  wrote:
>
>> On 30/09/2020 06:42, Arshiya Shariff wrote:
>> > Hi Martin ,
>> >
>> > Thank you for the response.
>> >
>> > With a payload of 200 bytes we were able to send 20K requests/sec with
>> 200 users from Jmeter without any memory issue . On increasing the payload
>> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
>> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
>> memory is cleared in the G1 mixed GC itself , but with 1000 users the
>> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
>> seconds to clear the memory. These cases were executed with maxThreads 200
>> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
>> still GC was struggling .
>> >
>> > When we tried with 10 instances of JMeter , each with 100 users , where
>> each instance was started with a delay of 1 minute we were able to see 1000
>> connections created in tomcat without any memory issues. But when 1000
>> users are created using single instance of JMeter in 20 seconds , tomcat's
>> memory is filling fast- 20GB in 2 minutes.
>> > We suspect that the burst of connections being opened has a problem .
>> Please help us with the same .
>> >
>> > On analyzing the heap dump we see
>> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93%
>> of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.
>>
>> You can't have high throughput, low GC pauses and small heap sizes.
>> Broadly you can have any two of those three at the expense of the third.
>>
>> The way Tomcat currently retains information about completed h2 streams
>> means you are likely to need a large heap under heavy load. There are
>> some changes already in 10.0.x that I plan to back-port to 9.0.x and
>> 8.5.x later today that should significantly reduce the heap requirements.
>>
>
> Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from
> 10.0.x with Vegeta for 3 mins:
> https://pasteboard.co/JtshrAs.png
> As you can see the GC is properly cleaning the heap. At the end the memory
> is not released until the GC kicks.
>
> Note: this is with a GET request without a body! I'm going to start a new
> email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for
> very low load.
>

When I load test HTTP2 with POST (with big bodies) there are many errors
like:

1)
Exception in thread "https-jsse-nio-8080-exec-5"
java.nio.BufferOverflowException
at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
at
org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
at
org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:832)

2)
Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events
SEVERE: Failed to register socket with selector from poller
java.nio.channels.ClosedChannelException
at
java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703)
at java.base/java.lang.Thread.run(Thread.java:832)

And the client gets these kind of error responses:

"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=7, ErrCode=PROTOCOL_ERROR,
debug=\"Connection [9354], Stream [7], The content length header value
[255] does not agree with the size of the data received [0]\"",
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=31,
ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [9355], Too much 

Re: HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM

[Martin Grigorov]

On Wed, Sep 30, 2020 at 1:15 PM Martin Grigorov 
wrote:

>
>
> On Wed, Sep 30, 2020 at 1:01 PM Martin Grigorov 
> wrote:
>
>> Hi,
>>
>> I've tried to test the scenario of Arshiya Shariff:
>> "With a payload of 200 bytes we were able to send 20K requests/sec with
>> 200 users from Jmeter without any memory issue . On increasing the payload
>> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
>> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
>> memory is cleared in the G1 mixed GC itself , but with 1000 users the
>> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
>> seconds to clear the memory. These cases were executed with maxThreads 200
>> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
>> still GC was struggling ."
>>
>> My Servlet looks like this:
>> https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
>>
>> And the Vegeta command I issue is:
>>
>> jq -ncM '{"method": "POST", "url": "
>> https://localhost:8080/testbed/plaintext;, "body":"payload=Some payload"
>> | @base64, header: {"Content-Type":
>> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
>> -http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
>> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>>
>> The outcome is:
>>
>> {
>>   "latencies": {
>> "total": 114369584,
>> "mean": 38123194,
>> "50th": 45737841,
>> "90th": 62258803,
>> "95th": 62258803,
>> "99th": 62258803,
>> "max": 62258803,
>> "min": 6372940
>>   },
>>   "bytes_in": {
>> "total": 24,
>> "mean": 8
>>   },
>>   "bytes_out": {
>> "total": 40,
>> "mean": 13.334
>>   },
>>   "earliest": "2020-09-30T12:40:15.208111926+03:00",
>>   "latest": "2020-09-30T12:40:17.208170426+03:00",
>>   "end": "2020-09-30T12:40:17.253908267+03:00",
>>   "duration": 258500,
>>   "wait": 45737841,
>>   "requests": 3,
>>   "rate": 1.499956126283306,
>>   "throughput": 0.9776144183650195,
>>   "success": 0.,
>>   "status_codes": {
>> "0": 1,
>> "200": 2
>>   },
>>   "errors": [
>> "Post \"https://localhost:8080/testbed/plaintext\": http2: server
>> sent GOAWAY and closed the connection; LastStreamID=3,
>> ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
>> connection will be closed\""
>>   ]
>> }
>>
>> I.e. it sends 1 POST request every second for 3 seconds and this leads to
>> GOAWAY+ENHANCE_YOUR_CALM !
>>
>> From my understanding on "overhead" this should happen if there are more
>> SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
>>
>> I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
>> so I will try to debug it or to reproduce it with Firefox (to be able to
>> use Wireframe).
>>
>> Any ideas why this happens ?
>>
>
> The reason is here:
> https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L1448
> overheadThreshold is 1024 and average is 20
> It goes twice in this method and then fails at
> https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L352
>

https://bz.apache.org/bugzilla/show_bug.cgi?id=63690#c7 explains the issue.
No idea why but Vegeta/Golang sends two DATA frames and the second one
(with END_STREAM) is empty:

30-Sep-2020 14:45:35.646 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream.emitHeader Connection [1], Stream [1], HTTP
header [accept-encoding], Value [gzip]
30-Sep-2020 14:45:35.646 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream.emitHeader Connection [1], Stream [1], HTTP
header [user-agent], Value [Go-http-client/2.0]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.swallow Connection [1], Stream [1],
Swallowed [0] bytes
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.validateFrame Connection [1], Stream
[1], Frame type [DATA], Flags [0], Payload size [20]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.readDataFrame Connection [1], Stream
[1], Data length, [20], Padding length [none]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream$StreamInputBuffer.onDataAvailable Data added
to inBuffer when read thread is waiting. Signalling that thread to continue
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.validateFrame Connection [1], Stream
[1], Frame type [DATA], Flags [1], Payload size [0]



>
>
>>
>> Martin
>>
>

Re: HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM

[Martin Grigorov]

On Wed, Sep 30, 2020 at 1:01 PM Martin Grigorov 
wrote:

> Hi,
>
> I've tried to test the scenario of Arshiya Shariff:
> "With a payload of 200 bytes we were able to send 20K requests/sec with
> 200 users from Jmeter without any memory issue . On increasing the payload
> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
> memory is cleared in the G1 mixed GC itself , but with 1000 users the
> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
> seconds to clear the memory. These cases were executed with maxThreads 200
> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
> still GC was struggling ."
>
> My Servlet looks like this:
> https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
>
> And the Vegeta command I issue is:
>
> jq -ncM '{"method": "POST", "url": "
> https://localhost:8080/testbed/plaintext;, "body":"payload=Some payload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>
> The outcome is:
>
> {
>   "latencies": {
> "total": 114369584,
> "mean": 38123194,
> "50th": 45737841,
> "90th": 62258803,
> "95th": 62258803,
> "99th": 62258803,
> "max": 62258803,
> "min": 6372940
>   },
>   "bytes_in": {
> "total": 24,
> "mean": 8
>   },
>   "bytes_out": {
> "total": 40,
> "mean": 13.334
>   },
>   "earliest": "2020-09-30T12:40:15.208111926+03:00",
>   "latest": "2020-09-30T12:40:17.208170426+03:00",
>   "end": "2020-09-30T12:40:17.253908267+03:00",
>   "duration": 258500,
>   "wait": 45737841,
>   "requests": 3,
>   "rate": 1.499956126283306,
>   "throughput": 0.9776144183650195,
>   "success": 0.,
>   "status_codes": {
> "0": 1,
> "200": 2
>   },
>   "errors": [
> "Post \"https://localhost:8080/testbed/plaintext\": http2: server
> sent GOAWAY and closed the connection; LastStreamID=3,
> ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
> connection will be closed\""
>   ]
> }
>
> I.e. it sends 1 POST request every second for 3 seconds and this leads to
> GOAWAY+ENHANCE_YOUR_CALM !
>
> From my understanding on "overhead" this should happen if there are more
> SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
>
> I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
> so I will try to debug it or to reproduce it with Firefox (to be able to
> use Wireframe).
>
> Any ideas why this happens ?
>

The reason is here:
https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L1448
overheadThreshold is 1024 and average is 20
It goes twice in this method and then fails at
https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L352


>
> Martin
>

HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM

[Martin Grigorov]

Hi,

I've tried to test the scenario of Arshiya Shariff:
"With a payload of 200 bytes we were able to send 20K requests/sec with 200
users from Jmeter without any memory issue . On increasing the payload to
5Kb and the number of users to 1000 in Jmeter and sending 1000 requests per
second , the heap of 20GB got filled in 2 minutes . With 200 users the
memory is cleared in the G1 mixed GC itself , but with 1000 users the
memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
seconds to clear the memory. These cases were executed with maxThreads 200
in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
still GC was struggling ."

My Servlet looks like this:
https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54

And the Vegeta command I issue is:

jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext;,
"body":"payload=Some payload" | @base64, header: {"Content-Type":
["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
-http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
/tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .

The outcome is:

{
  "latencies": {
"total": 114369584,
"mean": 38123194,
"50th": 45737841,
"90th": 62258803,
"95th": 62258803,
"99th": 62258803,
"max": 62258803,
"min": 6372940
  },
  "bytes_in": {
"total": 24,
"mean": 8
  },
  "bytes_out": {
"total": 40,
"mean": 13.334
  },
  "earliest": "2020-09-30T12:40:15.208111926+03:00",
  "latest": "2020-09-30T12:40:17.208170426+03:00",
  "end": "2020-09-30T12:40:17.253908267+03:00",
  "duration": 258500,
  "wait": 45737841,
  "requests": 3,
  "rate": 1.499956126283306,
  "throughput": 0.9776144183650195,
  "success": 0.,
  "status_codes": {
"0": 1,
"200": 2
  },
  "errors": [
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=3,
ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
connection will be closed\""
  ]
}

I.e. it sends 1 POST request every second for 3 seconds and this leads to
GOAWAY+ENHANCE_YOUR_CALM !

>From my understanding on "overhead" this should happen if there are more
SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.

I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
so I will try to debug it or to reproduce it with Firefox (to be able to
use Wireframe).

Any ideas why this happens ?

Martin

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

Hi,

On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas  wrote:

> On 30/09/2020 06:42, Arshiya Shariff wrote:
> > Hi Martin ,
> >
> > Thank you for the response.
> >
> > With a payload of 200 bytes we were able to send 20K requests/sec with
> 200 users from Jmeter without any memory issue . On increasing the payload
> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
> memory is cleared in the G1 mixed GC itself , but with 1000 users the
> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
> seconds to clear the memory. These cases were executed with maxThreads 200
> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
> still GC was struggling .
> >
> > When we tried with 10 instances of JMeter , each with 100 users , where
> each instance was started with a delay of 1 minute we were able to see 1000
> connections created in tomcat without any memory issues. But when 1000
> users are created using single instance of JMeter in 20 seconds , tomcat's
> memory is filling fast- 20GB in 2 minutes.
> > We suspect that the burst of connections being opened has a problem .
> Please help us with the same .
> >
> > On analyzing the heap dump we see
> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93%
> of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.
>
> You can't have high throughput, low GC pauses and small heap sizes.
> Broadly you can have any two of those three at the expense of the third.
>
> The way Tomcat currently retains information about completed h2 streams
> means you are likely to need a large heap under heavy load. There are
> some changes already in 10.0.x that I plan to back-port to 9.0.x and
> 8.5.x later today that should significantly reduce the heap requirements.
>

Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from
10.0.x with Vegeta for 3 mins:
https://pasteboard.co/JtshrAs.png
As you can see the GC is properly cleaning the heap. At the end the memory
is not released until the GC kicks.

Note: this is with a GET request without a body! I'm going to start a new
email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for
very low load.

Martin


>
> Mark
>
>
> >
> > Thanks and Regards
> > Arshiya Shariff
> >
> > -Original Message-
> > From: Martin Grigorov 
> > Sent: Monday, September 28, 2020 11:44 PM
> > To: Tomcat Users List 
> > Subject: Re: HTTP2: memory filled up fast on increasing the connections
> to 1000/2000 (Embedded tomcat 9.0.38)
> >
> > Hi Arshiya,
> >
> >
> > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff <
> arshiya.shar...@ericsson.com.invalid> wrote:
> >
> >> Hi All,
> >> With 200 threads(users) , ramp up duration of 2 seconds , loop count
> >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an
> >> embedded tomcat application we did not observe any memory issue , but
> >> on sending
> >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the
> >> application's heap space of 20 GB is occupied in 2 minutes and after 2
> >> full GCs the memory clears and comes down to 4GB (expected) .
> >>
> >
> > I am not sure whether you follow the other discussions at users@.
> > In another email thread we discuss load testing Tomcat HTTP2 and we are
> able to make around 12K reqs/s with another load testing tool -
> https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f=1=6a9c569d-7da1-4394-a9ac-bf72724992fa=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta
> > For me JMeter itself failed with OOM when increasing the number of the
> virtual users above 2K.
> > There are several improvements in Tomcat master and 9.0.x in the HTTP2
> area. Some of the changes are not yet downported to 9.0.x. We still test
> them, trying to avoid introducing regressions in 9.0.x.
> >
> >
> >>
> >> Embedded tomcat Version:9.0.38
> >> Max Threads : 200
> >>
> >
> > The number of threads should be less if you do only CPU calculations
> without IO/network. If your app blocks on IO/network calls then you need
> more spare threads.
> > With more threads there will be more context switches and less
> throughput.
> > That's why there is no one golden rule that applies to all applications.
> > 200 is a good default that works for most of the applications. But you
> need to test with different values to see which one gives the best
> performance for your scena

Re: ApacheCon @ Home Tomcat Track Schedule

[Martin Grigorov]

Hi,

A friendly reminder: the conference already started and the Tomcat sessions
will start in less than 6 hours from now!

On Fri, Aug 14, 2020 at 6:49 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> All,
>
> I'm happy to announce that the Apache Tomcat track schedule has been
> posted for ApacheCon @ Home, our virtual conference to replace
> "ApacheCon North America 2020". If you use social media to discuss
> this event, please use #ACAH2020 and tag @apachecon.
>
> You can find the schedule here, as well as links to the other 27 (!)
> tracks that will be held at the conference:
> https://www.apachecon.com/acah2020/tracks/tomcat.html
>
> If you'd like to register (zero cost!), you can do that here:
> https://hopin.to/events/apachecon-home
>
> Note that some tracks are happening at different times; please check
> the times for each presentation in each track for your timezone.
>
> I hope to "see" many of you at the conference this year.
>
> Thanks,
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl82sk4ACgkQHPApP6U8
> pFiypw/+M6Sz1wUtXk6A1CO5Lbz0jMU6/Ky53+zCb3skp4wNSVXaUuJ4xCb2hWg/
> cl8KkgP6N0e4stwB9M/XbL403Ic6NGYBvS2ZcMmm2rJqO99t5UvTpUY/DzzNwaXf
> d5z2h8p+aSU4ph6UZavi/Tydxa83WxQSRDuRww61xs6yHv7myELicv+fATJGY4fb
> slMLJt2Hzmh9C7t7+xkQDTpEUNz+oIV2yyQwuvkAS6WMHJbrqAkuqgQq6eAYm5Qc
> 79pb4e/G+E71Sr2AzeuFSpLQCUzYIyOkXFoLfr9L6Wb7iwwiu5bFP7+WAbJvI200
> tYjokn9NdZOwf4P3uY0xfCk3aq2jI9DZAcYjh7n0nhHCN+8vrqodQ/CZtoBYNQR5
> /HwMgszCyimfkegwb3CcuayFUko/Oa+klSOuh9XqFM8jb9XobyXx7dprZk3Ly0Kq
> oL/7h3+uBvz4rzR6FVdcuJ7stY4JpBhG+ZsjkiMktYcZovD9IP20RIE+omvqc0/S
> L1d/3XyRk3fM6JuQq2Y9yqTcgsbBrzEtcwIAOv2r9TQaRrwSbO2nTUiPVx8RTwQ4
> dtE60qVUvZ933I8aV4akqGkeUSBbxiv0WSfpV3T2F1latoqfEmlzueJNWrdgwXFC
> 8hSVyx29AsTiDGAXAswzxQDyNFlaY9zZZKi467doMvhJBGzTYfk=
> =pleB
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

Hi Arshiya,


On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff
 wrote:

> Hi All,
> With 200 threads(users) , ramp up duration of 2 seconds , loop count 80
> and by sending 1000 http2 requests/sec from JMeter Client to an embedded
> tomcat application we did not observe any memory issue , but on sending
> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the
> application's heap space of 20 GB is occupied in 2 minutes and after 2 full
> GCs the memory clears and comes down to 4GB (expected) .
>

I am not sure whether you follow the other discussions at users@.
In another email thread we discuss load testing Tomcat HTTP2 and we are
able to make around 12K reqs/s with another load testing tool -
https://github.com/tsenart/vegeta
For me JMeter itself failed with OOM when increasing the number of the
virtual users above 2K.
There are several improvements in Tomcat master and 9.0.x in the HTTP2
area. Some of the changes are not yet downported to 9.0.x. We still test
them, trying to avoid introducing regressions in 9.0.x.


>
> Embedded tomcat Version:9.0.38
> Max Threads : 200
>

The number of threads should be less if you do only CPU calculations
without IO/network. If your app blocks on IO/network calls then you need
more spare threads.
With more threads there will be more context switches and less throughput.
That's why there is no one golden rule that applies to all applications.
200 is a good default that works for most of the applications. But you need
to test with different values to see which one gives the best performance
for your scenaria.


> All other properties are the tomcat defaults.
>
> Why is tomcat not able to process many connections ?
>

You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and
-XX:HeapDumpPath=. Once you have the .hprof file you can
examine it with Eclipse Memory Analyzer tool and see what is leaking.
I will try to reproduce this issue tomorrow with Vegeta.


> Why is the memory filled when the connections are increased, are there any
> parameters to tune connections ?
> Please let us know.
>
> Thanks and Regards
> Arshiya Shariff
>

Re: ByteBuffer pooling

[Martin Grigorov]

On Mon, Sep 28, 2020 at 6:11 PM Rémy Maucherat  wrote:

> On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > I've profiled the memory allocation during load testing HTTP2:
> > https://pasteboard.co/Jtblqfl.png
> >
> > As you can see there are a lot of ByteBuffer.allocate(int) calls.
> > org.apache.catalina.connector.Response#Response(int)
> > org.apache.catalina.connector.Request#inputBuffer
> > org.apache.coyote.http2.Http2AsyncParser#readFrame
> > org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer
> >
> > Has it been discussed in the past to use a pool of ByteBuffer instances ?
> > Netty provides such for its abstraction ByteBuf:
> > https://netty.io/wiki/using-as-a-generic-library.html
> >
> > Here is a simple implementation:
> >
> >
> https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44
> > It's give() method could be optimized to not return the buffer to the
> pool
> > if the pool size is bigger than N, so it doesn't use huge list of buffers
> > which are used just once.
> >
> > The big unknown to me is: where to return the buffers to the pool ?
> > For HTTP2 maybe this could be done after writing the buffer to the
> socket.
> > If the Request/Response is recycled then their Input/OutputBuffer are
> > reused and everything is OK. But if recycling is not in use then new
> > allocations are done for each new request.
> >
> > I see that org.apache.tomcat.util.net.WriteBuffer does some pooling
> > already.
> >
> > What do you think?
> >
>
> I removed a lot of pooling in the connector with zero impact, so I don't
> see the point.
>

ConcurrentDateFormat pools SimpleDateFormat instances and it does make a
difference:
https://medium.com/@martin.grigorov/compare-performance-of-javas-simpledateformat-against-datetimeformatter-31be58cadf1d
 !
I'd give it a try for the HTTP2 Stream related allocations and let you know
how it goes!


>
> Rémy
>
>
> >
> > Martin
> >
>

ByteBuffer pooling

[Martin Grigorov]

Hi,

I've profiled the memory allocation during load testing HTTP2:
https://pasteboard.co/Jtblqfl.png

As you can see there are a lot of ByteBuffer.allocate(int) calls.
org.apache.catalina.connector.Response#Response(int)
org.apache.catalina.connector.Request#inputBuffer
org.apache.coyote.http2.Http2AsyncParser#readFrame
org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer

Has it been discussed in the past to use a pool of ByteBuffer instances ?
Netty provides such for its abstraction ByteBuf:
https://netty.io/wiki/using-as-a-generic-library.html

Here is a simple implementation:
https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44
It's give() method could be optimized to not return the buffer to the pool
if the pool size is bigger than N, so it doesn't use huge list of buffers
which are used just once.

The big unknown to me is: where to return the buffers to the pool ?
For HTTP2 maybe this could be done after writing the buffer to the socket.
If the Request/Response is recycled then their Input/OutputBuffer are
reused and everything is OK. But if recycling is not in use then new
allocations are done for each new request.

I see that org.apache.tomcat.util.net.WriteBuffer does some pooling already.

What do you think?

Martin

Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

[Martin Grigorov]

On Fri, Sep 25, 2020 at 1:18 PM Arshiya Shariff
 wrote:

> Thanks Martin .
>
> Our expectation Is that , on receiving a RST_STREAM with CANCEL or
> NO_ERROR from the client after 1.5 secs for a particular stream , we don’t
> want the connection to be closed  with a GOAWAY:NO_ERROR  (while trying to
> send response after 2secs asynchronously over a stream that is closed) as
> the other streams during the same time are processing data fine .
>
> Additional info for GOAWAY: Connection [], Stream[],An error occurred
> during processing that was fatal to the connection .
>

I guess some other error happened and that led to the GOAWAY. But I cannot
be sure.
Any errors in the logs ?


>
> Thanks and Regards
> Arshiya Shariff
>
> -----Original Message-
> From: Martin Grigorov 
> Sent: Tuesday, September 22, 2020 7:31 PM
> To: Tomcat Users List 
> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a
> stream where the client has already sent RST_STREAM:NO ERROR
>
> Hi,
>
> On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff <
> arshiya.shar...@ericsson.com.invalid> wrote:
>
> > Thank you so much Martin for the response.
> >  Yes , 9.0.38 testing is on going .
> >
> > As we don’t get this clear with the RFC , please help us with the
> > below two cases :
>
>   * If a client sends RST_STREAM with NO_ERROR, then while sending async
> > response is it expected behavior to close connection with GOAWAY ?
> >   * If a client sends RST_STREAM with CANCEL , then while sending
> > async response will tomcat send RST_STREAM or GOAWAY , from http2
> > protocol perspective ?
> >
>
> As per https://tools.ietf.org/html/rfc7540#section-6.4 and
> https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R"
> are processed then the stream is moved to CLOSED state.
> GOAWAY should be sent to the client only if the connection will be closed
> ( https://tools.ietf.org/html/rfc7540#section-6.8). The server should
> close the connection only if some serious problem has happened, e.g. an
> IOException.
>
> Tell us more about your use case. What do you want to do when 1.5secs pass
> ? What do you expect to happen ?
>
>
> >
> > Thanks and Regards
> > Arshiya Sharif
> >
> > -Original Message-
> > From: Martin Grigorov 
> > Sent: Tuesday, September 22, 2020 1:18 AM
> > To: Tomcat Users List 
> > Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a
> > stream where the client has already sent RST_STREAM:NO ERROR
> >
> > Hi,
> >
> > On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff <
> > arshiya.shar...@ericsson.com.invalid> wrote:
> >
> > > Hi All,
> > >
> > > The client has configured a response timeout of 1.5 seconds. In a
> > > case when our application tries to respond over a http2 stream
> > > asynchronously after 2 seconds where the client has already sent
> > > RST_STREAM with NO ERROR in 1.5 seconds
> >
> >
> > Why does the client send NO_ERROR to the server ? I think it should
> > send a CANCEL instead.
> > https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for
> > "Graceful shutdown of the server".
> >
> >
> > > (due to no response) , then tomcat sends GOAWAY and closes the HTTP2
> > > connection . Is this behavior of GOAWAY and connection closure
> expected ?
> > > We have planned to upgrade to Embedded tomcat version 9.0.38 . These
> > > are the behaviors we see in production with version 9.0.22 ,  so we
> > > need some help with analyzing / validating  the existing behavior
> > > before
> > the upgrade .
> > > Please let us know.
> > >
> >
> > Friendly advice:
> > Please setup 9.0.38 locally and test on it.
> > 9.0.22 is way too old. It is up to you to use it for your production
> > but for reporting bugs it is recommended to use the latest available
> version.
> > I, personally, prefer to spend my spare time with my family and
> > friends than to debug old versions just because the user doesn't
> > bother to test on a newer version.
> >
> >
> > >
> > > Thanks and Regards
> > > Arshiya Shariff
> > >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Martin Grigorov]

On Thu, Sep 24, 2020 at 2:11 PM Martin Grigorov 
wrote:

> Hi,
>
> On Thu, Sep 24, 2020 at 1:02 PM Nils Breunese  wrote:
>
>> Hello,
>>
>> I recently learned that when a server that supports path parameters [0] —
>> like Tomcat (I found Jetty also does) — is run behind a reverse proxy that
>> does path-based access control checks and does not support path parameters,
>> your combined setup could be vulnerable.
>>
>> Consider this setup:
>>
>> 1. A Tomcat application without access restrictions
>> 2. An reverse proxy that only allows requests to /v1/* on the Tomcat
>> application. I’ve used Envoy on Kubernetes, configured via Istio’s
>> Role-Based Access Control (RBAC), but also observed this issue with other
>> proxies.
>>
>> Now I send a request for /v1/..;color=red/internal/secret to the proxy.
>>
>> - Envoy allows the request based on the /v1/* rule, because it does not
>> support path parameters, because they are not part of any recent standard
>> (RFC 2396 dropped them in 1998 [1])
>> - Tomcat parses ;color=red as a path parameter, resolves the rest of the
>> path to /v1/../internal/secret and consequently serves /internal/secret
>> - I’m pretty sure that whoever configured that RBAC policy did not want
>> this to be possible
>>
>> Note that:
>> - The client can request any URL on the Tomcat server via this path
>> traversal attack.
>> - This also works with ‘empty’ path parameters, e.g.
>> /v1/..;/internal/secret
>> - The URL doesn’t necessarily need to contain ‘..’ for naughtiness to be
>> possible. When access to /public/secret would be restricted by the proxy,
>> but other /public/* paths would be allowed, then an attacker would still be
>> able to get the contents of /public/secret by requesting /public;/secret
>>
>> The fact that different servers handle paths with path parameters
>> differently seems to be known [2], but especially setups in which reverse
>> proxies don’t support path parameters, but the servers behind them do,
>> seems risky.
>>
>> Would it be reasonable to create a ticket with a feature request for a
>> flag to disable path parameter support in Tomcat? Or an even more secure
>> suggestion: make path parameter support an opt-in feature, because it was
>> dropped from the URI standard in 1998 and can make setups like I described
>> vulnerable?
>>
>
> I remember a similar discussion from a few months ago.
> Which version of Tomcat do you use ? Please try with the latest in case
> you use an older one.
>

I've just checked my mail archives.
Someone else had the same/similar problem and the conclusion was that
according to the Servlet specification this is the proper way to process
the request - the request url should be normalized. If you need to protect
some paths then you should do whatever is necessary in your application.
Please use secur...@tomcat.apache.org for reporting (possible) security
problems in the future! Thanks!

Martin


>
>
>>
>> Thanks, Nils.
>>
>> [0] RFC 1738 https://tools.ietf.org/html/rfc1738 <
>> https://tools.ietf.org/html/rfc1738> (1994) and RFC 1808
>> https://tools.ietf.org/html/rfc1808 <https://tools.ietf.org/html/rfc1808>
>> (1995).
>> [1] RFC 2396 G.4. Modifications from RFC 1808
>> https://tools.ietf.org/html/rfc2396 <https://tools.ietf.org/html/rfc2396>
>> (1998). Path parameters are also absent in RFC 3986
>> https://tools.ietf.org/html/rfc3986 <https://tools.ietf.org/html/rfc3986>
>> (2005).
>> [2]
>> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf
>> <
>> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf>
>> slides 41 and further
>
>

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Martin Grigorov]

Hi,

On Thu, Sep 24, 2020 at 1:02 PM Nils Breunese  wrote:

> Hello,
>
> I recently learned that when a server that supports path parameters [0] —
> like Tomcat (I found Jetty also does) — is run behind a reverse proxy that
> does path-based access control checks and does not support path parameters,
> your combined setup could be vulnerable.
>
> Consider this setup:
>
> 1. A Tomcat application without access restrictions
> 2. An reverse proxy that only allows requests to /v1/* on the Tomcat
> application. I’ve used Envoy on Kubernetes, configured via Istio’s
> Role-Based Access Control (RBAC), but also observed this issue with other
> proxies.
>
> Now I send a request for /v1/..;color=red/internal/secret to the proxy.
>
> - Envoy allows the request based on the /v1/* rule, because it does not
> support path parameters, because they are not part of any recent standard
> (RFC 2396 dropped them in 1998 [1])
> - Tomcat parses ;color=red as a path parameter, resolves the rest of the
> path to /v1/../internal/secret and consequently serves /internal/secret
> - I’m pretty sure that whoever configured that RBAC policy did not want
> this to be possible
>
> Note that:
> - The client can request any URL on the Tomcat server via this path
> traversal attack.
> - This also works with ‘empty’ path parameters, e.g.
> /v1/..;/internal/secret
> - The URL doesn’t necessarily need to contain ‘..’ for naughtiness to be
> possible. When access to /public/secret would be restricted by the proxy,
> but other /public/* paths would be allowed, then an attacker would still be
> able to get the contents of /public/secret by requesting /public;/secret
>
> The fact that different servers handle paths with path parameters
> differently seems to be known [2], but especially setups in which reverse
> proxies don’t support path parameters, but the servers behind them do,
> seems risky.
>
> Would it be reasonable to create a ticket with a feature request for a
> flag to disable path parameter support in Tomcat? Or an even more secure
> suggestion: make path parameter support an opt-in feature, because it was
> dropped from the URI standard in 1998 and can make setups like I described
> vulnerable?
>

I remember a similar discussion from a few months ago.
Which version of Tomcat do you use ? Please try with the latest in case you
use an older one.


>
> Thanks, Nils.
>
> [0] RFC 1738 https://tools.ietf.org/html/rfc1738 <
> https://tools.ietf.org/html/rfc1738> (1994) and RFC 1808
> https://tools.ietf.org/html/rfc1808 
> (1995).
> [1] RFC 2396 G.4. Modifications from RFC 1808
> https://tools.ietf.org/html/rfc2396 
> (1998). Path parameters are also absent in RFC 3986
> https://tools.ietf.org/html/rfc3986 
> (2005).
> [2]
> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf
> <
> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf>
> slides 41 and further

Re: Issue in unpacking files

[Martin Grigorov]

The response was inline:

The mailing list server does not allow embedded images.
Please use some image pastebin service.
Or if this is plain text then better just copy/paste it.

On Wed, Sep 23, 2020 at 3:35 PM Mohan T  wrote:

> Hi,
>
> I am not able to see any response in the mail received.
>
> Please treat the file name deployed as ems.war and not ems.wzr.
>
> Thanks
>
> Mohan
>
>
> -Original Message-
> From: Martin Grigorov 
> Sent: Wednesday, September 23, 2020 6:00 PM
> To: Tomcat Users List 
> Subject: Re: Issue in unpacking files
>
> Warning : This is an External Email. Check reliability of sender &  email
> id mentioned in ;
> use caution with links and attachments – IMG Team
>
>
>
>
>
>
> Hi,
>
> On Wed, Sep 23, 2020 at 3:03 PM Mohan T  wrote:
>
> > Dear All,
> >
> >
> >
> > We have two environments where in one the .war file is getting
> > extracted and on the other it is not. See the below for your
> > reference. The file being deployed is ems.wzr
> >
> >
> >
> > We are using apache-tomcat-8.5.35.  Any pointers would be of great help.
> >
> >
> >
> > Catalinabase0  - Here it has been correctly deployed -
> >
> >
> >
> >
> >
> > Catalinabase1 – Here it has not been deployed correctly -
> >
> >
> >
> >
>
> The mailing list server does not allow embedded images.
> Please use some image pastebin service.
> Or if this is plain text then better just copy/paste it.
>
>
> >
> >
> >
> > DISCLAIMER: This communication contains information which is
> > confidential and the copyright of Ramco Systems Ltd, its subsidiaries
> > or a third party (“Ramco”). This email may also contain legally
> privileged information.
> > Confidentiality and legal privilege attached to this communication are
> > not waived or lost by reason of mistaken delivery to you.This email is
> > intended to be read or used by the addressee only. If you are not the
> > intended recipient, any use, distribution, disclosure or copying of
> > this email is strictly prohibited without the express written approval
> > of Ramco. Please delete and destroy all copies and email Ramco at
> > le...@ramco.com immediately. Any views expressed in this communication
> > are those of the individual sender, except where the sender
> > specifically states them to be the views of Ramco. Except as required
> > by law, Ramco does not represent, warrant and/or guarantee that the
> > integrity of this communication has been maintained nor that the
> > communication is free of errors, virus, interception or interference.
> > If you do not wish to receive such communications, please forward this
> > communication to market...@ramco.com and express your wish not to
> receive such communications henceforth.
> >
> DISCLAIMER: This communication contains information which is confidential
> and the copyright of Ramco Systems Ltd, its subsidiaries or a third party
> (“Ramco”). This email may also contain legally privileged information.
> Confidentiality and legal privilege attached to this communication are not
> waived or lost by reason of mistaken delivery to you.This email is intended
> to be read or used by the addressee only. If you are not the intended
> recipient, any use, distribution, disclosure or copying of this email is
> strictly prohibited without the express written approval of Ramco. Please
> delete and destroy all copies and email Ramco at le...@ramco.com
> immediately. Any views expressed in this communication are those of the
> individual sender, except where the sender specifically states them to be
> the views of Ramco. Except as required by law, Ramco does not represent,
> warrant and/or guarantee that the integrity of this communication has been
> maintained nor that the communication is free of errors, virus,
> interception or interference. If you do not wish to receive such
> communications, please forward this communication to market...@ramco.com
> and express your wish not to receive such communications henceforth.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Issue in unpacking files

[Martin Grigorov]

Hi,

On Wed, Sep 23, 2020 at 3:03 PM Mohan T  wrote:

> Dear All,
>
>
>
> We have two environments where in one the .war file is getting extracted
> and on the other it is not. See the below for your reference. The file
> being deployed is ems.wzr
>
>
>
> We are using apache-tomcat-8.5.35.  Any pointers would be of great help.
>
>
>
> Catalinabase0  - Here it has been correctly deployed -
>
>
>
>
>
> Catalinabase1 – Here it has not been deployed correctly -
>
>
>
>

The mailing list server does not allow embedded images.
Please use some image pastebin service.
Or if this is plain text then better just copy/paste it.


>
>
>
> DISCLAIMER: This communication contains information which is confidential
> and the copyright of Ramco Systems Ltd, its subsidiaries or a third party
> (“Ramco”). This email may also contain legally privileged information.
> Confidentiality and legal privilege attached to this communication are not
> waived or lost by reason of mistaken delivery to you.This email is intended
> to be read or used by the addressee only. If you are not the intended
> recipient, any use, distribution, disclosure or copying of this email is
> strictly prohibited without the express written approval of Ramco. Please
> delete and destroy all copies and email Ramco at le...@ramco.com
> immediately. Any views expressed in this communication are those of the
> individual sender, except where the sender specifically states them to be
> the views of Ramco. Except as required by law, Ramco does not represent,
> warrant and/or guarantee that the integrity of this communication has been
> maintained nor that the communication is free of errors, virus,
> interception or interference. If you do not wish to receive such
> communications, please forward this communication to market...@ramco.com
> and express your wish not to receive such communications henceforth.
>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi,



I've debugged further with Wireshark:

1) Golang: https://pasteboard.co/Jsnvx1z.png  (SETTINGS, HEADERS+DATA)
2) Nodejs: https://pasteboard.co/JsnwRNV.png (SETTINGS, HEADERS+PARTIAL
DATA, DATA)
3) Rust: https://pasteboard.co/JsnxvKGU.png  (SETTINGS, HEADERS+PARTIAL
DATA+DATA)
4) Netty: https://pasteboard.co/JsnxYJp.png (SETTINGS+HEADERS+DATA)
5) Tomcat: https://pasteboard.co/JsnyjpA.png (SETTINGS, HEADERS, DATA)

>From the images above you can see that only Tomcat sends two separate
packets for HEADERS and DATA frames. All others put several frames into one
packet.
Netty puts the SETTINGS, the HEADERS and the DATA into one TCP packet. I
guess this is the reason for the high throughput
Node.js writes the HEADERS and PARTIAL DATA in one packet and the actual
DATA in a separate one
Rust also uses PARTIAL DATA but puts the actual one in the same TCP packet
Tomcat's way is spec compliant but most probably is also a reason for the
lower throughput.

More findings:
I've debugged Netty code and it never receives RST_STREAM with CANCEL error
at
https://github.com/martin-g/http2-server-perf-tests/blob/2f628c407e2ea29be99e756a61e555d88addaa01/java/netty-http2-test/src/main/java/com/example/netty/http2/Http2ServerResponseHandler.java#L30
I've looked at Golang net code and found that it will send RST_STREAM with
CANCEL if there is a problem reading the body:
https://github.com/tsenart/vegeta/issues/540#issuecomment-696757695
It is not clear to me whether this is about the request body or response
body. But since the unit test uses Transport.RoundTrip() I think it is
about the response body.
So, it looks like Tomcat sends a DATA frame which is not liked by Golang
net package code and Golang sends this CANCEL error that causes the
problems.
The DATA frames in Wireshark look good to me, but I've made them by using
Firefox as a client to be able to decrypt TLS in Wireshark (SSLKEYLOGFILE)

Martin

Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

[Martin Grigorov]

Hi,

On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff
 wrote:

> Thank you so much Martin for the response.
>  Yes , 9.0.38 testing is on going .
>
> As we don’t get this clear with the RFC , please help us with the below
> two cases :

  * If a client sends RST_STREAM with NO_ERROR, then while sending async
> response is it expected behavior to close connection with GOAWAY ?
>   * If a client sends RST_STREAM with CANCEL , then while sending async
> response will tomcat send RST_STREAM or GOAWAY , from http2 protocol
> perspective ?
>

As per https://tools.ietf.org/html/rfc7540#section-6.4 and
https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R" are
processed then the stream is moved to CLOSED state.
GOAWAY should be sent to the client only if the connection will be closed (
https://tools.ietf.org/html/rfc7540#section-6.8). The server should close
the connection only if some serious problem has happened, e.g. an
IOException.

Tell us more about your use case. What do you want to do when 1.5secs pass
? What do you expect to happen ?


>
> Thanks and Regards
> Arshiya Sharif
>
> -Original Message-
> From: Martin Grigorov 
> Sent: Tuesday, September 22, 2020 1:18 AM
> To: Tomcat Users List 
> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a
> stream where the client has already sent RST_STREAM:NO ERROR
>
> Hi,
>
> On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff <
> arshiya.shar...@ericsson.com.invalid> wrote:
>
> > Hi All,
> >
> > The client has configured a response timeout of 1.5 seconds. In a case
> > when our application tries to respond over a http2 stream
> > asynchronously after 2 seconds where the client has already sent
> > RST_STREAM with NO ERROR in 1.5 seconds
>
>
> Why does the client send NO_ERROR to the server ? I think it should send a
> CANCEL instead.
> https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for "Graceful
> shutdown of the server".
>
>
> > (due to no response) , then tomcat sends GOAWAY and closes the HTTP2
> > connection . Is this behavior of GOAWAY and connection closure expected ?
> > We have planned to upgrade to Embedded tomcat version 9.0.38 . These
> > are the behaviors we see in production with version 9.0.22 ,  so we
> > need some help with analyzing / validating  the existing behavior before
> the upgrade .
> > Please let us know.
> >
>
> Friendly advice:
> Please setup 9.0.38 locally and test on it.
> 9.0.22 is way too old. It is up to you to use it for your production but
> for reporting bugs it is recommended to use the latest available version.
> I, personally, prefer to spend my spare time with my family and friends
> than to debug old versions just because the user doesn't bother to test on
> a newer version.
>
>
> >
> > Thanks and Regards
> > Arshiya Shariff
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 4:46 PM Rémy Maucherat  wrote:

> On Mon, Sep 21, 2020 at 2:49 PM Martin Grigorov 
> wrote:
>
> > Hi Remy,
> >
> > On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
> >
> > 
> >
> >
> > > > 2020-09-21 14:25:04.850 DEBUG 232086 ---
> [https-jsse-nio-18080-exec-8]
> > > > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> > > > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> > > > :java.nio.channels.SocketChannel[connected
> > > > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> > > > 2020-09-21 14:25:04.850 DEBUG 232086 ---
> [https-jsse-nio-18080-exec-6]
> > > > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> > > >
> > > > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> > > > at org.apache.tomcat.util.net
> > > > .SecureNioChannel.read(SecureNioChannel.java:766)
> > > > at org.apache.tomcat.util.net
> > > >
> > >
> >
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> > > >
> > >
> > > When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> > > what's going on. Otherwise you don't really know what proportion of TLS
> > or
> > > HTTP/2 impacts performance more [and then you have to test more things
> > like
> > > OpenSSL, be careful that the ciphers used are the same and equally well
> > > optimized in each impl, etc].
> > >
> > > Then once you feel happy about h2c, you can see how things go with TLS.
> > >
> >
> > Chris also suggested that TLS might be the problem for the low throughput
> > but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> > reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear
> now
> > - it is due to the extra RST packets being sent.
> >
> > Vegeta does support h2c! So I can give it a try!
> >
> > How one configures Tomcat to support h2c ? Just remove  ?
> > I've just tried this but I am not sure how to verify that it works.
> > The browsers do not support h2c. curl and httpie neither.
> > Quick search suggested me https://github.com/fstab/h2c "h2c connect
> > localhost:18080" failed with "Failed to connect to localhost:18080: tls:
> > oversized record received with length 20527"
> >
>
> Yes, it depends on the client. h2load works just fine without TLS for
> example, and it's actually the reason why I added h2c support in Tomcat.
>
> As for the configuration, it's a normal non TLS HTTP/1.1 config with the
> UpgradeProtocol element added.
>

The way to use h2c with curl is: curl -i --http2-prior-knowledge
http://localhost:18080/...

(with my change in reset() to avoid the connection closes):

vegeta ... -http2=f -h2c ... gives 13 K reqs/s
vegeta ... -http2 ... gives 8.2 K reqs/s

so, it seems TLS reduces the throughput with 40%

Martin


>
> Rémy
>
>
> >
> >
> >
> > >
> > > Rémy
> > >
> > >
> > >
> > >
> >
>

Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

[Martin Grigorov]

Hi,

On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff
 wrote:

> Hi All,
>
> The client has configured a response timeout of 1.5 seconds. In a case
> when our application tries to respond over a http2 stream asynchronously
> after 2 seconds where the client has already sent RST_STREAM with NO ERROR
> in 1.5 seconds


Why does the client send NO_ERROR to the server ? I think it should send a
CANCEL instead.
https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for "Graceful
shutdown of the server".


> (due to no response) , then tomcat sends GOAWAY and closes the HTTP2
> connection . Is this behavior of GOAWAY and connection closure expected ?
> We have planned to upgrade to Embedded tomcat version 9.0.38 . These are
> the behaviors we see in production with version 9.0.22 ,  so we need some
> help with analyzing / validating  the existing behavior before the upgrade .
> Please let us know.
>

Friendly advice:
Please setup 9.0.38 locally and test on it.
9.0.22 is way too old. It is up to you to use it for your production but
for reporting bugs it is recommended to use the latest available version.
I, personally, prefer to spend my spare time with my family and friends
than to debug old versions just because the user doesn't bother to test on
a newer version.


>
> Thanks and Regards
> Arshiya Shariff
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 9:31 PM Martin Grigorov 
wrote:

>
>
> On Mon, Sep 21, 2020 at 5:52 PM Mark Thomas  wrote:
>
>> On 21/09/2020 13:48, Martin Grigorov wrote:
>> > Hi Remy,
>> >
>> > On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
>> >
>> > 
>> >
>> >
>> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
>> >>> o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
>> >>> socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
>> >>> :java.nio.channels.SocketChannel[connected
>> >>> local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
>> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
>> >>> o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
>> >>>
>> >>> java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
>> >>> at org.apache.tomcat.util.net
>> >>> .SecureNioChannel.read(SecureNioChannel.java:766)
>> >>> at org.apache.tomcat.util.net
>> >>>
>> >>
>> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> >>>
>> >>
>> >> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
>> >> what's going on. Otherwise you don't really know what proportion of
>> TLS or
>> >> HTTP/2 impacts performance more [and then you have to test more things
>> like
>> >> OpenSSL, be careful that the ciphers used are the same and equally well
>> >> optimized in each impl, etc].
>> >>
>> >> Then once you feel happy about h2c, you can see how things go with TLS.
>> >>
>> >
>> > Chris also suggested that TLS might be the problem for the low
>> throughput
>> > but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
>> > reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear
>> now
>> > - it is due to the extra RST packets being sent.
>> >
>> > Vegeta does support h2c! So I can give it a try!
>>
>> I've just done that. I am seeing the client sending RST_STREAM frames
>> AFTER the request/response has been processed cleanly.
>>
>> The RST_STREAM frames are being sent with the CANCEL error. i.e. the
>> client is no longer interested in reading the response body.
>>
>> This doesn't make much sense as there is little more than two or three
>> milliseconds between the start of the request and the RST_STREAM. It
>> isn't happening due to a timeout. It is also always sent AFTER the
>> server has sent the complete response (the end of stream flag is set)
>> and sometimes after the next request has been sent.
>>
>> However you look at it, this looks like a bug in Vegeta. That doesn't
>>
>
> I will contact the Vegeta community!
>

https://github.com/tsenart/vegeta/issues/540


>
>
>> exclude, of course, the possibility of improving how Tomcat handles
>> traffic like this.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 5:52 PM Mark Thomas  wrote:

> On 21/09/2020 13:48, Martin Grigorov wrote:
> > Hi Remy,
> >
> > On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
> >
> > 
> >
> >
> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
> >>> o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> >>> socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> >>> :java.nio.channels.SocketChannel[connected
> >>> local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
> >>> o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> >>>
> >>> java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> >>> at org.apache.tomcat.util.net
> >>> .SecureNioChannel.read(SecureNioChannel.java:766)
> >>> at org.apache.tomcat.util.net
> >>>
> >>
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> >>>
> >>
> >> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> >> what's going on. Otherwise you don't really know what proportion of TLS
> or
> >> HTTP/2 impacts performance more [and then you have to test more things
> like
> >> OpenSSL, be careful that the ciphers used are the same and equally well
> >> optimized in each impl, etc].
> >>
> >> Then once you feel happy about h2c, you can see how things go with TLS.
> >>
> >
> > Chris also suggested that TLS might be the problem for the low throughput
> > but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> > reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear
> now
> > - it is due to the extra RST packets being sent.
> >
> > Vegeta does support h2c! So I can give it a try!
>
> I've just done that. I am seeing the client sending RST_STREAM frames
> AFTER the request/response has been processed cleanly.
>
> The RST_STREAM frames are being sent with the CANCEL error. i.e. the
> client is no longer interested in reading the response body.
>
> This doesn't make much sense as there is little more than two or three
> milliseconds between the start of the request and the RST_STREAM. It
> isn't happening due to a timeout. It is also always sent AFTER the
> server has sent the complete response (the end of stream flag is set)
> and sometimes after the next request has been sent.
>
> However you look at it, this looks like a bug in Vegeta. That doesn't
>

I will contact the Vegeta community!


> exclude, of course, the possibility of improving how Tomcat handles
> traffic like this.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 3:48 PM Martin Grigorov 
wrote:

> Hi Remy,
>
> On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
>
> 
>
>
>> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
>> > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
>> > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
>> > :java.nio.channels.SocketChannel[connected
>> > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
>> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
>> > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
>> >
>> > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
>> > at org.apache.tomcat.util.net
>> > .SecureNioChannel.read(SecureNioChannel.java:766)
>> > at org.apache.tomcat.util.net
>> >
>> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> >
>>
>> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
>> what's going on. Otherwise you don't really know what proportion of TLS or
>> HTTP/2 impacts performance more [and then you have to test more things
>> like
>> OpenSSL, be careful that the ciphers used are the same and equally well
>> optimized in each impl, etc].
>>
>> Then once you feel happy about h2c, you can see how things go with TLS.
>>
>
> Chris also suggested that TLS might be the problem for the low throughput
> but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear now
> - it is due to the extra RST packets being sent.
>
> Vegeta does support h2c! So I can give it a try!
>
> How one configures Tomcat to support h2c ? Just remove  ?
> I've just tried this but I am not sure how to verify that it works.
> The browsers do not support h2c. curl and httpie neither.
> Quick search suggested me https://github.com/fstab/h2c "h2c connect
> localhost:18080" failed with "Failed to connect to localhost:18080: tls:
> oversized record received with length 20527"
>

fstab/h2c is a normal http2 client. Confusingly named one though.


>
>
>>
>> Rémy
>>
>>
>>
>>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Remy,

On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:




> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
> > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> > :java.nio.channels.SocketChannel[connected
> > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
> > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> >
> > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> > at org.apache.tomcat.util.net
> > .SecureNioChannel.read(SecureNioChannel.java:766)
> > at org.apache.tomcat.util.net
> >
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> >
>
> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> what's going on. Otherwise you don't really know what proportion of TLS or
> HTTP/2 impacts performance more [and then you have to test more things like
> OpenSSL, be careful that the ciphers used are the same and equally well
> optimized in each impl, etc].
>
> Then once you feel happy about h2c, you can see how things go with TLS.
>

Chris also suggested that TLS might be the problem for the low throughput
but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear now
- it is due to the extra RST packets being sent.

Vegeta does support h2c! So I can give it a try!

How one configures Tomcat to support h2c ? Just remove  ?
I've just tried this but I am not sure how to verify that it works.
The browsers do not support h2c. curl and httpie neither.
Quick search suggested me https://github.com/fstab/h2c "h2c connect
localhost:18080" failed with "Failed to connect to localhost:18080: tls:
oversized record received with length 20527"



>
> Rémy
>
>
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 12:08 PM Martin Grigorov 
wrote:

>
>
> On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:
>
>> On 21/09/2020 08:18, Martin Grigorov wrote:
>> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
>> >
>> >> On 18/09/2020 14:07, Martin Grigorov wrote:
>> >>
>> >> 
>> >>
>> >>> What is the difference
>> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
>> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>> >>
>> >> Compare the parameters used to construct the enums.
>> >>
>> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't
>> see
>> >>> anything related to two types of CLOSED state.
>> >>
>> >> Section 5.1. Definition of the closed state (page 20) explains the
>> >> difference between the two.
>> >>
>> >
>> > Still I do not understand what RX and TX stand for. But this is not
>> > important.
>>
>> TX - transmit
>>
>> RX - receive
>>
>
> Thanks!
>
>
>>
>>
>> > The following patch fixes the issue for me/Vegeta:
>> >
>> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
>> > implements InternalHttpUpgradeH
>> >
>> >  @Override
>> >  public void reset(int streamId, long errorCode) throws
>> Http2Exception
>> >  {
>> > -Stream stream = getStream(streamId, true);
>> > -boolean active = stream.isActive();
>> > -stream.checkState(FrameType.RST);
>> > -stream.receiveReset(errorCode);
>> > -if (active) {
>> > -activeRemoteStreamCount.decrementAndGet();
>> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
>> errorCode;
>> > +Stream stream = getStream(streamId, unknownIsError);
>> > +if (stream != null) {
>> > +boolean active = stream.isActive();
>> > +stream.checkState(FrameType.RST);
>> > +stream.receiveReset(errorCode);
>> > +if (active) {
>> > +activeRemoteStreamCount.decrementAndGet();
>> > +}
>> >  }
>> >  }
>> >
>> > I.e. do not throw an error if the remote peer is trying to cancel an
>> > already closed stream.
>>
>> RFC 7540 allows the connection to be closed with a protocol error if the
>> reset is received "a significant time after sending END_STREAM".
>>
>> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
>> being pruned (oldest first) every 10 new streams.
>>
>
> The spec talks about time but Tomcat uses the number of streams.
> I understand that under normal usage this could be enough "time" but under
> heavy load this number is reached quickly and the time is short.
>
> I've tried with these values for
> maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU cores),
> 16, 100 (the default), 1024 and 8196. The throughput is around 10K only
> until 100. After that it drops (as you said earlier).
>
>
>>
>> It isn't clear at this point why the client is sending the RST_STREAM.
>>
>
> I don't know either. But Vegeta is using the Golang standard libraries, so
> any Golang application should behave the same way.
> Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
> do not complain about it.
>
>
>> This normally indicates an error condition. From the description of the
>> load test I don't see why the client would be sending a RST_STREAM.
>>
>> If the RST_STREAM is valid and it isn't received a significant time
>> after the END_STREAM then Tomcat needs to handle this.
>>
>> If the RST_STREAM is valid but received a significant amount of time
>> after the END_STREAM that would be a client error would be a client error.
>>
>> Of course, significant isn't defined in the specification.
>>
>> Any special handling for RST_STREAM also needs to be applied to
>> WINDOW_UPDATE. It also needs to differentiate between a frame received
>> for a past closed stream and a frame received for an stream that never
>> existed.
>>
>> I think the key question here is why is the client sending the
>> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
>> not doing something it should to cause this? If so, we need to fix the
>&

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 12:08 PM Martin Grigorov 
wrote:

>
>
> On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:
>
>> On 21/09/2020 08:18, Martin Grigorov wrote:
>> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
>> >
>> >> On 18/09/2020 14:07, Martin Grigorov wrote:
>> >>
>> >> 
>> >>
>> >>> What is the difference
>> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
>> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>> >>
>> >> Compare the parameters used to construct the enums.
>> >>
>> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't
>> see
>> >>> anything related to two types of CLOSED state.
>> >>
>> >> Section 5.1. Definition of the closed state (page 20) explains the
>> >> difference between the two.
>> >>
>> >
>> > Still I do not understand what RX and TX stand for. But this is not
>> > important.
>>
>> TX - transmit
>>
>> RX - receive
>>
>
> Thanks!
>
>
>>
>>
>> > The following patch fixes the issue for me/Vegeta:
>> >
>> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
>> > implements InternalHttpUpgradeH
>> >
>> >  @Override
>> >  public void reset(int streamId, long errorCode) throws
>> Http2Exception
>> >  {
>> > -Stream stream = getStream(streamId, true);
>> > -boolean active = stream.isActive();
>> > -stream.checkState(FrameType.RST);
>> > -stream.receiveReset(errorCode);
>> > -if (active) {
>> > -activeRemoteStreamCount.decrementAndGet();
>> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
>> errorCode;
>> > +Stream stream = getStream(streamId, unknownIsError);
>> > +if (stream != null) {
>> > +boolean active = stream.isActive();
>> > +stream.checkState(FrameType.RST);
>> > +stream.receiveReset(errorCode);
>> > +if (active) {
>> > +activeRemoteStreamCount.decrementAndGet();
>> > +}
>> >  }
>> >  }
>> >
>> > I.e. do not throw an error if the remote peer is trying to cancel an
>> > already closed stream.
>>
>> RFC 7540 allows the connection to be closed with a protocol error if the
>> reset is received "a significant time after sending END_STREAM".
>>
>> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
>> being pruned (oldest first) every 10 new streams.
>>
>
> The spec talks about time but Tomcat uses the number of streams.
> I understand that under normal usage this could be enough "time" but under
> heavy load this number is reached quickly and the time is short.
>
> I've tried with these values for
> maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU cores),
> 16, 100 (the default), 1024 and 8196. The throughput is around 10K only
> until 100. After that it drops (as you said earlier).
>
>
>>
>> It isn't clear at this point why the client is sending the RST_STREAM.
>>
>
> I don't know either. But Vegeta is using the Golang standard libraries, so
> any Golang application should behave the same way.
> Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
> do not complain about it.
>
>
>> This normally indicates an error condition. From the description of the
>> load test I don't see why the client would be sending a RST_STREAM.
>>
>> If the RST_STREAM is valid and it isn't received a significant time
>> after the END_STREAM then Tomcat needs to handle this.
>>
>> If the RST_STREAM is valid but received a significant amount of time
>> after the END_STREAM that would be a client error would be a client error.
>>
>> Of course, significant isn't defined in the specification.
>>
>> Any special handling for RST_STREAM also needs to be applied to
>> WINDOW_UPDATE. It also needs to differentiate between a frame received
>> for a past closed stream and a frame received for an stream that never
>> existed.
>>
>> I think the key question here is why is the client sending the
>> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
>> not doing something it should to cause this? If so, we need to fix the
>&g

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:

> On 21/09/2020 08:18, Martin Grigorov wrote:
> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
> >
> >> On 18/09/2020 14:07, Martin Grigorov wrote:
> >>
> >> 
> >>
> >>> What is the difference
> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
> >>
> >> Compare the parameters used to construct the enums.
> >>
> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't
> see
> >>> anything related to two types of CLOSED state.
> >>
> >> Section 5.1. Definition of the closed state (page 20) explains the
> >> difference between the two.
> >>
> >
> > Still I do not understand what RX and TX stand for. But this is not
> > important.
>
> TX - transmit
>
> RX - receive
>

Thanks!


>
>
> > The following patch fixes the issue for me/Vegeta:
> >
> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
> > implements InternalHttpUpgradeH
> >
> >  @Override
> >  public void reset(int streamId, long errorCode) throws
> Http2Exception
> >  {
> > -Stream stream = getStream(streamId, true);
> > -boolean active = stream.isActive();
> > -stream.checkState(FrameType.RST);
> > -stream.receiveReset(errorCode);
> > -if (active) {
> > -activeRemoteStreamCount.decrementAndGet();
> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
> errorCode;
> > +Stream stream = getStream(streamId, unknownIsError);
> > +if (stream != null) {
> > +boolean active = stream.isActive();
> > +stream.checkState(FrameType.RST);
> > +stream.receiveReset(errorCode);
> > +if (active) {
> > +activeRemoteStreamCount.decrementAndGet();
> > +}
> >  }
> >  }
> >
> > I.e. do not throw an error if the remote peer is trying to cancel an
> > already closed stream.
>
> RFC 7540 allows the connection to be closed with a protocol error if the
> reset is received "a significant time after sending END_STREAM".
>
> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
> being pruned (oldest first) every 10 new streams.
>

The spec talks about time but Tomcat uses the number of streams.
I understand that under normal usage this could be enough "time" but under
heavy load this number is reached quickly and the time is short.

I've tried with these values for
maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU cores),
16, 100 (the default), 1024 and 8196. The throughput is around 10K only
until 100. After that it drops (as you said earlier).


>
> It isn't clear at this point why the client is sending the RST_STREAM.
>

I don't know either. But Vegeta is using the Golang standard libraries, so
any Golang application should behave the same way.
Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
do not complain about it.


> This normally indicates an error condition. From the description of the
> load test I don't see why the client would be sending a RST_STREAM.
>
> If the RST_STREAM is valid and it isn't received a significant time
> after the END_STREAM then Tomcat needs to handle this.
>
> If the RST_STREAM is valid but received a significant amount of time
> after the END_STREAM that would be a client error would be a client error.
>
> Of course, significant isn't defined in the specification.
>
> Any special handling for RST_STREAM also needs to be applied to
> WINDOW_UPDATE. It also needs to differentiate between a frame received
> for a past closed stream and a frame received for an stream that never
> existed.
>
> I think the key question here is why is the client sending the
> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
> not doing something it should to cause this? If so, we need to fix the
> root cause rather than tackle the symptom.
>
> There are a couple of options for debugging this to see why the client
> is sending the RST_STREAM:
> - Enable debug logging for HTTP/2 in Tomcat. This is very verbose and
>   if the root cause is threading related typically changes the timing
>   enough to mask the issue.
>

Here are the logs of 20 seconds load:
https://gist.githubusercontent.com/martin-g/37dacc12149d5cecbfd4e16dc2248cfd/raw/76693cd1dd1f37ba126edb84ebd7631802aa91a6/tomcat-http2.log

This is with my pa

Re: Low throughput with HTTP2

[Martin Grigorov]

On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:

> On 18/09/2020 14:07, Martin Grigorov wrote:
>
> 
>
> > What is the difference
> > between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
> > and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>
> Compare the parameters used to construct the enums.
>
> > I read some parts of https://tools.ietf.org/html/rfc7540 but I don't see
> > anything related to two types of CLOSED state.
>
> Section 5.1. Definition of the closed state (page 20) explains the
> difference between the two.
>

Still I do not understand what RX and TX stand for. But this is not
important.

The following patch fixes the issue for me/Vegeta:

@@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
implements InternalHttpUpgradeH

 @Override
 public void reset(int streamId, long errorCode) throws Http2Exception
 {
-Stream stream = getStream(streamId, true);
-boolean active = stream.isActive();
-stream.checkState(FrameType.RST);
-stream.receiveReset(errorCode);
-if (active) {
-activeRemoteStreamCount.decrementAndGet();
+boolean unknownIsError = Http2Error.CANCEL.getCode() != errorCode;
+Stream stream = getStream(streamId, unknownIsError);
+if (stream != null) {
+boolean active = stream.isActive();
+stream.checkState(FrameType.RST);
+stream.receiveReset(errorCode);
+if (active) {
+activeRemoteStreamCount.decrementAndGet();
+}
 }
 }

I.e. do not throw an error if the remote peer is trying to cancel an
already closed stream.

With this change and Vegeta's -max-workers=100 I can get 12 K reqs/sec.

With more workers it starts failing with:

"status_codes": {
"0": 1000,
"200": 1
  },
  "errors": [
"Get \"https://localhost:18080/testbed/plaintext\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)",
"Get \"https://localhost:18080/testbed/plaintext\": context deadline
exceeded",
"Get \"https://localhost:18080/testbed/plaintext\": context deadline
exceeded (Client.Timeout exceeded while awaiting headers)"
  ]
i.e. out of 1001 requests only one succeeds and the others fail with
timeout. I will try to debug this one too and run a profiler too.



>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Fri, Sep 18, 2020 at 10:48 AM Martin Grigorov 
wrote:

> Hi,
>
> On Wed, Sep 16, 2020 at 10:24 AM Martin Grigorov 
> wrote:
>
>> Hi Remy,
>>
>> On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat  wrote:
>>
>>> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
>>> wrote:
>>>
>>> > Hi Mark,
>>> >
>>> > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
>>> >
>>> > > On 15/09/2020 12:46, Martin Grigorov wrote:
>>> > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <
>>> mgrigo...@apache.org>
>>> > > > wrote:
>>> > > >
>>> > > >> Hi,
>>> > > >>
>>> > > >> I am running some load tests on Tomcat and I've noticed that when
>>> > HTTP2
>>> > > is
>>> > > >> enabled the throughput drops considerably.
>>> > > >>
>>> > > >> Here are the steps to reproduce:
>>> > > >>
>>> > > >> 1) Enable HTTP2, e.g. by commenting out this connector:
>>> > > >>
>>> > > >>
>>> > >
>>> >
>>> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>>> > > >>
>>> > > >> 2) Download Vegeta load tool from:
>>> > > >> https://github.com/tsenart/vegeta/releases/
>>> > > >>
>>> > > >> 3) Run the load tests:
>>> > > >>
>>> > > >> 3.1) HTTP/1.1
>>> > > >> echo -e '{"method": "GET", "url": "
>>> http://localhost:8080/examples/"}'
>>> > |
>>> > > >> vegeta attack -format=json  -rate=0 -max-workers=1000
>>> -duration=10s |
>>> > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
>>> > > >> /tmp/http1.json | jq .
>>> > > >>
>>> > > >> 3.2) HTTP2
>>> > > >> echo -e '{"method": "GET", "url": "
>>> https://localhost:8443/examples/
>>> > "}'
>>> > > |
>>> > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000
>>> -insecure
>>> > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
>>> > > >> -type=json /tmp/http2.json | jq .
>>> > > >>
>>> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0
>>> means
>>> > > >> that Vegeta will try to send as many requests as possible with the
>>> > > >> configured number of workers.
>>> > > >> I use '-insecure' because I use self-signed certificate.
>>> > > >>
>>> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
>>> > > responses
>>> > > >> with code=200 .
>>> > > >> But for HTTP2 Tomcat starts returning such kind of errors:
>>> > > >>
>>> > > >>  "errors": [
>>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>>> > > GOAWAY
>>> > > >> and closed the connection; LastStreamID=9259,
>>> ErrCode=PROTOCOL_ERROR,
>>> > > >> debug=\"Stream [9,151] has been closed for some time\"",
>>> > > >> "http2: server sent GOAWAY and closed the connection;
>>> > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151]
>>> has
>>> > > been
>>> > > >> closed for some time\"",
>>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>>> > > GOAWAY
>>> > > >> and closed the connection; LastStreamID=239,
>>> ErrCode=PROTOCOL_ERROR,
>>> > > >> debug=\"Stream [49] has been closed for some time\""
>>> > > >>   ]
>>> > > >>
>>> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>>> > >
>>> > > That indicates that the client has sent a frame associated with a
>>> stream
>>> > > that the server closed previously and that that stream has been
>>>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi,

On Wed, Sep 16, 2020 at 10:24 AM Martin Grigorov 
wrote:

> Hi Remy,
>
> On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat  wrote:
>
>> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
>> wrote:
>>
>> > Hi Mark,
>> >
>> > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
>> >
>> > > On 15/09/2020 12:46, Martin Grigorov wrote:
>> > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <
>> mgrigo...@apache.org>
>> > > > wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> I am running some load tests on Tomcat and I've noticed that when
>> > HTTP2
>> > > is
>> > > >> enabled the throughput drops considerably.
>> > > >>
>> > > >> Here are the steps to reproduce:
>> > > >>
>> > > >> 1) Enable HTTP2, e.g. by commenting out this connector:
>> > > >>
>> > > >>
>> > >
>> >
>> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>> > > >>
>> > > >> 2) Download Vegeta load tool from:
>> > > >> https://github.com/tsenart/vegeta/releases/
>> > > >>
>> > > >> 3) Run the load tests:
>> > > >>
>> > > >> 3.1) HTTP/1.1
>> > > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/
>> "}'
>> > |
>> > > >> vegeta attack -format=json  -rate=0 -max-workers=1000
>> -duration=10s |
>> > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
>> > > >> /tmp/http1.json | jq .
>> > > >>
>> > > >> 3.2) HTTP2
>> > > >> echo -e '{"method": "GET", "url": "
>> https://localhost:8443/examples/
>> > "}'
>> > > |
>> > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000
>> -insecure
>> > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
>> > > >> -type=json /tmp/http2.json | jq .
>> > > >>
>> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0
>> means
>> > > >> that Vegeta will try to send as many requests as possible with the
>> > > >> configured number of workers.
>> > > >> I use '-insecure' because I use self-signed certificate.
>> > > >>
>> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
>> > > responses
>> > > >> with code=200 .
>> > > >> But for HTTP2 Tomcat starts returning such kind of errors:
>> > > >>
>> > > >>  "errors": [
>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>> > > GOAWAY
>> > > >> and closed the connection; LastStreamID=9259,
>> ErrCode=PROTOCOL_ERROR,
>> > > >> debug=\"Stream [9,151] has been closed for some time\"",
>> > > >> "http2: server sent GOAWAY and closed the connection;
>> > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151]
>> has
>> > > been
>> > > >> closed for some time\"",
>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>> > > GOAWAY
>> > > >> and closed the connection; LastStreamID=239,
>> ErrCode=PROTOCOL_ERROR,
>> > > >> debug=\"Stream [49] has been closed for some time\""
>> > > >>   ]
>> > > >>
>> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>> > >
>> > > That indicates that the client has sent a frame associated with a
>> stream
>> > > that the server closed previously and that that stream has been
>> removed
>> > > from the Map of known streams to make room for new ones. See
>> > > Http2UpgardeHandler.pruneClosedStreams()
>> > >
>> > > It looks like the client is making assumptions about server behaviour
>> > > that go beyond the requirements of RFC 7540, section 5.3.4.
>> > >
>> >
>> > This is pos

Re: tomcat warnings. [EXTERNAL]

[Martin Grigorov]

On Thu, Sep 17, 2020, 16:07 Beard, Shawn 
wrote:

> Yes its 9.0.31.0
>

The last 0 makes me think that you have installed Tomcat by using your OS
package manager.
Try with stock Tomcat downloaded from Apache servers.


> [mwuser@usilg01-tcd003 ~]$ ./version.sh
> Using CATALINA_BASE:   /path/to/catalina_base
> Using CATALINA_HOME:   /path/to/catalina_home
> Using CATALINA_TMPDIR: /path/to/catalina_base/temp
> Using JRE_HOME:/
> Using CLASSPATH:   /path/to/catalina_base
> /bin/bootstrap.jar:/usr/apache/tomcat/tomcat-current/bin/tomcat-juli.jar
> Server version: Apache Tomcat/9.0.31
> Server built:   Feb 5 2020 19:32:12 UTC
> Server number:  9.0.31.0
> OS Name:Linux
> OS Version: 3.10.0-957.21.3.el7.x86_64
> Architecture:   amd64
> JVM Version:1.8.0_212-b04
> JVM Vendor: Oracle Corporation
>
> I do have 3 connectors, one of them is the AJP connector, could the
> warning be coming from them?
>

Remove the problematic attributes from each connector one by one and see
when the warnings will disappear.


> Below is all 3 connectors:
>
> connectionTimeout="2"
>Server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>redirectPort="9443"
>
>  
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />
>
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>connectionTimeout="2"
>Server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>scheme="https"
>secure="true"
>SSLEnabled="true"
>sslEnabledProtocols="+TLSv1.2"
>keystoreFile="/path/to/tuststore/TomcatTrustStore.p12"
>truststoreFile="/path/to/keystore/TomcatTrustStore.p12"
>keystorePass="XXX"
>truststorePass="XX"
>
> ciphers="TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA"
>
>  
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />
>
> address="hostname"
>secretRequired="false"
>connectionTimeout="2"
>server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>redirectPort="9443"
>scheme="https"
>secure="true"
>
>  
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />
>
>
>
> Shawn Beard
> Sr. Systems Engineer
> BTS
> +1-515-564-2528
>
> -Original Message-
> From: Mark Thomas 
> Sent: Thursday, September 17, 2020 3:19 AM
> To: users@tomcat.apache.org
> Subject: Re: tomcat warnings. [EXTERNAL]
>
> ** CAUTION: External message
>
>
> On 16/09/2020 20:45, Beard, Shawn wrote:
> >  >protocol="HTTP/1.1"
> >connectionTimeout="2"
> >Server=" "
> >maxHttpHeaderSize="8192"
> >maxThreads="500"
> >minSpareThreads="30"
> >enableLookups="false"
> >disableUploadTimeout="true"
> >acceptCount="150"
> >redirectPort="9444"
> >
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
> >compression="on"
> >compressionMinSize="2048"
> >   

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Remy,

On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat  wrote:

> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
> wrote:
>
> > Hi Mark,
> >
> > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
> >
> > > On 15/09/2020 12:46, Martin Grigorov wrote:
> > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <
> mgrigo...@apache.org>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I am running some load tests on Tomcat and I've noticed that when
> > HTTP2
> > > is
> > > >> enabled the throughput drops considerably.
> > > >>
> > > >> Here are the steps to reproduce:
> > > >>
> > > >> 1) Enable HTTP2, e.g. by commenting out this connector:
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
> > > >>
> > > >> 2) Download Vegeta load tool from:
> > > >> https://github.com/tsenart/vegeta/releases/
> > > >>
> > > >> 3) Run the load tests:
> > > >>
> > > >> 3.1) HTTP/1.1
> > > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/
> "}'
> > |
> > > >> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s
> |
> > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> > > >> /tmp/http1.json | jq .
> > > >>
> > > >> 3.2) HTTP2
> > > >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/
> > "}'
> > > |
> > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000
> -insecure
> > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> > > >> -type=json /tmp/http2.json | jq .
> > > >>
> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0
> means
> > > >> that Vegeta will try to send as many requests as possible with the
> > > >> configured number of workers.
> > > >> I use '-insecure' because I use self-signed certificate.
> > > >>
> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
> > > responses
> > > >> with code=200 .
> > > >> But for HTTP2 Tomcat starts returning such kind of errors:
> > > >>
> > > >>  "errors": [
> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
> > > GOAWAY
> > > >> and closed the connection; LastStreamID=9259,
> ErrCode=PROTOCOL_ERROR,
> > > >> debug=\"Stream [9,151] has been closed for some time\"",
> > > >> "http2: server sent GOAWAY and closed the connection;
> > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151]
> has
> > > been
> > > >> closed for some time\"",
> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
> > > GOAWAY
> > > >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> > > >> debug=\"Stream [49] has been closed for some time\""
> > > >>   ]
> > > >>
> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
> > >
> > > That indicates that the client has sent a frame associated with a
> stream
> > > that the server closed previously and that that stream has been removed
> > > from the Map of known streams to make room for new ones. See
> > > Http2UpgardeHandler.pruneClosedStreams()
> > >
> > > It looks like the client is making assumptions about server behaviour
> > > that go beyond the requirements of RFC 7540, section 5.3.4.
> > >
> >
> > This is possible!
> > I've just tested with two more HTTP2 impls:
> >
> > 1) Node.js
> >
> > http2-server.js
> > ===
> > const http2 = require('http2');
> > const fs = require('fs');
> >
> > const server = http2.createSecureServer({
> > key: fs.readFileSync('/path/to/server.key'),
> > cert: fs.readFileSync('/path/to/server.crt')
> > });
> > server.on('error', (err) => console.error(

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Chris,

On Tue, Sep 15, 2020 at 5:09 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Martin,
>
> On 9/15/20 07:37, Martin Grigorov wrote:
> > I am running some load tests on Tomcat and I've noticed that when
> > HTTP2 is enabled the throughput drops considerably.
> >
> > Here are the steps to reproduce:
> >
> > 1) Enable HTTP2, e.g. by commenting out this connector:
> > https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f31
> 7c135d9d/conf/server.xml#L103-L112
> <https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112>
> >
> >  2) Download Vegeta load tool from:
> > https://github.com/tsenart/vegeta/releases/
> >
> > 3) Run the load tests:
> >
> > 3.1) HTTP/1.1 echo -e '{"method": "GET", "url":
> > "http://localhost:8080/examples/"}' | vegeta attack -format=json
> > -rate=0 -max-workers=1000 -duration=10s | vegeta encode >
> > /tmp/http1.json; and vegeta report -type=json /tmp/http1.json | jq
> > .
> >
> > 3.2) HTTP2 echo -e '{"method": "GET", "url":
> > "https://localhost:8443/examples/"}' | vegeta attack -format=json
> > -http2 -rate=0 -max-workers=1000 -insecure -duration=10s | vegeta
> > encode > /tmp/http2.json; and vegeta report -type=json
> > /tmp/http2.json | jq .
>
> You are using HTTP with 1.1 and HTTPS with h2. Maybe you are seeing
> CPU slowdown for the (probably double encryption) taking place on the
> same host?
>

I've tested against HTTPS without H2 - it makes 13K reqs/sec here, i.e. 2-3
K less than HTTP.
The above numbers are when I use the examples' index.html page as a target.
When I use

=
package load.servlet;

import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

@WebServlet(urlPatterns = PlainTextServlet.URL_PATTERN)
public class PlainTextServlet extends HttpServlet {

  static final String URL_PATTERN = "/servlet/plaintext";
  private static final String CONTENT_TYPE = "text/plain;charset=UTF-8";
  private static final byte[] CONTENT = "Hello
world!".getBytes(StandardCharsets.UTF_8);
  private static final int CONTENT_LENGTH = CONTENT.length;

  @Override
  protected void doGet(final HttpServletRequest req, final
HttpServletResponse resp) throws IOException {
resp.setContentType(CONTENT_TYPE);
resp.setContentLength(CONTENT_LENGTH);
resp.getOutputStream().write(CONTENT);
  }
}
=

I can get 33-35 K reqs/sec with HTTP



>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9gyyIACgkQHPApP6U8
> pFiNUxAAnUQVU73nNitIyn6zD9t4JfkLIv/AKTlds4/W/p6TRtIQgTX7nIJjGDfw
> BOKznCHmieMJon4rMZ3d8GFhmUP8CawQlJHpqABeITBzLZZ5x9fuOf22G6HJb3r+
> +k0qDoKzFTWlJuWLwaLZHy6fO9ugi4OPAW0G0efa2T6sTDZzImGnjmoZ5PWBExoz
> mXmYWnZeP7R+3QkAWUYArJh9yPEJyIb9nFX1YKZ1l5Erzrn0F9uEYFgWT/UkQoKM
> L65AMh/qEvzJhP2wHOLm4NfAiNO4OgTmo+nm4F/SIMGFNURPFi2sl/jTUHVAzEa4
> mAqlJqX1swimyjjsunlfhbU/bApvVFsYSPuSYcZmLN1lkmaQOAuWHnZdd4e9h+tt
> rhoKXipk8OairYzwQsPVnzCTHaiAhOXJ3MSE966YwlvhSMOoqDsN3y7ySrboresD
> iC0cDo+43/wR3IQlOJYFxcFX+tI2Y29ZjrX/IwnJXuVyU095YZWmRFC2JgRfzBtI
> toM2ofpqnSBaS22ZBTbqp+q1QxRZfC3r0vuvuiXK620QRcbk1Ya0+U17LOIEYnuY
> URY94kL80upiADQMIdryq4ubRAma2t0s5c6JuO/QqsXVjJfawlRGQA5arORgfE2J
> yDCscyyFCHitEGTglIJUXW/KfFPtraWnON3TSCm7dQ55EmInxpc=
> =76Zf
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Mark,

On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:

> On 15/09/2020 12:46, Martin Grigorov wrote:
> > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov 
> > wrote:
> >
> >> Hi,
> >>
> >> I am running some load tests on Tomcat and I've noticed that when HTTP2
> is
> >> enabled the throughput drops considerably.
> >>
> >> Here are the steps to reproduce:
> >>
> >> 1) Enable HTTP2, e.g. by commenting out this connector:
> >>
> >>
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
> >>
> >> 2) Download Vegeta load tool from:
> >> https://github.com/tsenart/vegeta/releases/
> >>
> >> 3) Run the load tests:
> >>
> >> 3.1) HTTP/1.1
> >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
> >> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
> >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> >> /tmp/http1.json | jq .
> >>
> >> 3.2) HTTP2
> >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}'
> |
> >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
> >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> >> -type=json /tmp/http2.json | jq .
> >>
> >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means
> >> that Vegeta will try to send as many requests as possible with the
> >> configured number of workers.
> >> I use '-insecure' because I use self-signed certificate.
> >>
> >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
> responses
> >> with code=200 .
> >> But for HTTP2 Tomcat starts returning such kind of errors:
> >>
> >>  "errors": [
> >> "Get \"https://localhost:8443/examples/\": http2: server sent
> GOAWAY
> >> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
> >> debug=\"Stream [9,151] has been closed for some time\"",
> >> "http2: server sent GOAWAY and closed the connection;
> >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has
> been
> >> closed for some time\"",
> >> "Get \"https://localhost:8443/examples/\": http2: server sent
> GOAWAY
> >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> >> debug=\"Stream [49] has been closed for some time\""
> >>   ]
> >>
> >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>
> That indicates that the client has sent a frame associated with a stream
> that the server closed previously and that that stream has been removed
> from the Map of known streams to make room for new ones. See
> Http2UpgardeHandler.pruneClosedStreams()
>
> It looks like the client is making assumptions about server behaviour
> that go beyond the requirements of RFC 7540, section 5.3.4.
>

This is possible!
I've just tested with two more HTTP2 impls:

1) Node.js

http2-server.js
===
const http2 = require('http2');
const fs = require('fs');

const server = http2.createSecureServer({
key: fs.readFileSync('/path/to/server.key'),
cert: fs.readFileSync('/path/to/server.crt')
});
server.on('error', (err) => console.error(err));

server.on('stream', (stream, headers) => {
// stream is a Duplex
stream.respond({
'content-type': 'text/plain; charset=utf-8',
':status': 200
});
stream.end('Hello world!');
});

server.listen(18080);
===

run with: node http2-server.js

Runs fine with -rate=0 and gives around 8K reqs/sec

2) Rust

Cargo.toml
===
[package]
name = "my-http2-server"
version = "0.0.1"
publish = false
authors = ["Martin Grigorov "]
license = "MIT/Apache-2.0"
description = "Load test HTTP/2 "
repository = "https://github.com/martin-g/http2-server-rust;
keywords = ["http2"]
edition = "2018"

[dependencies]
actix-web = { version = "3", features = ["openssl"] }
openssl = { version = "0.10", features = ["v110"] }
===

src/main.rs
===
use actix_web::{web, App, HttpRequest, HttpServer, Responder};
use openssl::ssl::{SslAcc

Re: Low throughput with HTTP2

[Martin Grigorov]

On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov 
wrote:

> Hi,
>
> I am running some load tests on Tomcat and I've noticed that when HTTP2 is
> enabled the throughput drops considerably.
>
> Here are the steps to reproduce:
>
> 1) Enable HTTP2, e.g. by commenting out this connector:
>
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>
> 2) Download Vegeta load tool from:
> https://github.com/tsenart/vegeta/releases/
>
> 3) Run the load tests:
>
> 3.1) HTTP/1.1
> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> /tmp/http1.json | jq .
>
> 3.2) HTTP2
> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}' |
> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> -type=json /tmp/http2.json | jq .
>
> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means
> that Vegeta will try to send as many requests as possible with the
> configured number of workers.
> I use '-insecure' because I use self-signed certificate.
>
> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only responses
> with code=200 .
> But for HTTP2 Tomcat starts returning such kind of errors:
>
>  "errors": [
> "Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
> debug=\"Stream [9,151] has been closed for some time\"",
> "http2: server sent GOAWAY and closed the connection;
> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has been
> closed for some time\"",
> "Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> debug=\"Stream [49] has been closed for some time\""
>   ]
>
> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>
>
> All the access logs look like:
>
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
>
> i.e. there are no error codes, just 200.
> Vegeta reports the error with status code = 0. I think this just means
> that it didn't get a proper HTTP response but just TCP error.
> There are no errors in catalina.out.
>
> Are there any settings I can tune to get better throughput with HTTP2 ?
>
> Tomcat 10.0.0-M8.
>

Forgot to mention that I've also tested with JMeter +
https://github.com/Blazemeter/jmeter-http2-plugin but there it fails with
OOM if I use more than 2000 virtual users. Increasing the memory still does
not give such good results as Vegeta for HTTP/1.1. Also JMeter uses
sequential model.

For comparison, I've also tested with a simple Golang based HTTP2 server:

http2-server.go:
==
package main

import (
"fmt"
"log"
"net/http"
"os"
)

func main() {

port := 8080
if port == "" {
  log.Fatal("Please specify the HTTP port as environment variable, e.g.
env PORT=8081 go run http-server.go")
}

tls_root := "/path/to/certs/"
srv := {Addr: ":" + port, Handler: http.HandlerFunc(handle)}
log.Fatal(srv.ListenAndServeTLS(tls_root + "server.crt", tls_root +
"server.key"))
}

func handle(w http.ResponseWriter, r *http.Request) {
//log.Printf("Got connection: %s", r.Proto) // prints HTTP/2.0
fmt.Fprintf(w, "Hello World")
}
==

Here Vegeta makes around 13K reqs/sec without error responses.

To run this app do: go run http2-server.go


> Regards,
> Martin
>

Low throughput with HTTP2

[Martin Grigorov]

Hi,

I am running some load tests on Tomcat and I've noticed that when HTTP2 is
enabled the throughput drops considerably.

Here are the steps to reproduce:

1) Enable HTTP2, e.g. by commenting out this connector:
https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112

2) Download Vegeta load tool from:
https://github.com/tsenart/vegeta/releases/

3) Run the load tests:

3.1) HTTP/1.1
echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
vegeta encode > /tmp/http1.json; and vegeta report -type=json
/tmp/http1.json | jq .

3.2) HTTP2
echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}' |
vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
-duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
-type=json /tmp/http2.json | jq .

As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means that
Vegeta will try to send as many requests as possible with the configured
number of workers.
I use '-insecure' because I use self-signed certificate.

On my machine I get around 14-15K reqs/sec for HTTP1.1 with only responses
with code=200 .
But for HTTP2 Tomcat starts returning such kind of errors:

 "errors": [
"Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
debug=\"Stream [9,151] has been closed for some time\"",
"http2: server sent GOAWAY and closed the connection;
LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has been
closed for some time\"",
"Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
debug=\"Stream [49] has been closed for some time\""
  ]

when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s


All the access logs look like:

127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126

i.e. there are no error codes, just 200.
Vegeta reports the error with status code = 0. I think this just means that
it didn't get a proper HTTP response but just TCP error.
There are no errors in catalina.out.

Are there any settings I can tune to get better throughput with HTTP2 ?

Tomcat 10.0.0-M8.

Regards,
Martin

Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

[Martin Grigorov]

hi,

On Tue, Sep 15, 2020 at 8:20 AM Pratik Shrestha  wrote:

> Hi Guys,
>
> Just wanted to know if anyone found an idea on fixing it or a workaround.
>

Did you find what is the expected behavior by Qualis ?


>
> Thanks
>
> Pratik.
>
> On Fri, Aug 28, 2020 at 10:46 AM Pratik Shrestha 
> wrote:
>
> > Hi Chris
> >
> >
> >
> >
> > *This wasn't the case for httpd for many years. I don't know what itdoes
> > these days, but it used to reply with a nice "400 Bad Request"error just
> > like Tomcat is doing. The difference is that httpd has richconfiguration
> > options to allow you to override that behavior. *
> >
> > Correct. By default HTTP also gives 400 Bad Request message. But we can
> > override this behavior by using ErrorDocument directive to redirect to
> > https URL. Currently in Tomcat this feature is not available. Tomcat has
> > RewriteValve but it does not work in this case.
> >
> > On Fri, Aug 28, 2020 at 12:30 AM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> >> -BEGIN PGP SIGNED MESSAGE-
> >> Hash: SHA256
> >>
> >> Merka,
> >>
> >> On 8/27/20 06:32, Phoenix, Merka wrote:
> >> > I think what the Qualys scan is trying to flag is that the server
> >> > (Tomcat) is listening for both secured and unsecured traffic on
> >> > the _same_ TCP port when the server should be listening for just
> >> > one of the two options (unsecured or secured), not both.
> >> This actually might be a semi-legitimate test. If the client says "Our
> >> policy is to only communicate using encrypted connections" and the
> >> test says "well, here's a non-encrypted connection right here!" then
> >> it makes some small amount of sense. In that case, it's all driven by
> >> policy, and the policy can say "we have a carve-out for TLS handshake
> >> failures" and then allow that particular test to pass.
> >>
> >> > The error message returned by the Tomcat service, while certainly
> >> > helpful to the remote client, is returning more information than
> >> > it should (from a security-viewpoint).
> >> Not really.
> >>
> >> > If the default behavior for Tomcat is to return this "helpful"
> >> > message for misconfigured clients, would it be reasonable for the
> >> > Connector to have an option (attribute) for turning off this
> >> > feature and simply reject with a TLS error any unsecured traffic on
> >> > the port? This would address the security concern without imposing
> >> > too much "bloat" to the Tomcat side.
> >> I think this might actually be better/easier than implementing a
> >> redirect. It's a simple flag that says "return nice errors on
> >> plaintext-over-TLS connection attempts" (or similar) and the only
> >> thing that changes is that we STOP trying to be nice to the client if
> >> the setting is set to "fail" versus "be-nice".
> >>
> >> > For most other web servers (Apache httpd, NGINX, etc.) that offer
> >> > https, the normal behavior is that when an http client tries to
> >> > speak http to server expecting https, the client sees some garbled
> >> > text (the server's TLS response to the connection attempt).
> >> This wasn't the case for httpd for many years. I don't know what it
> >> does these days, but it used to reply with a nice "400 Bad Request"
> >> error just like Tomcat is doing. The difference is that httpd has rich
> >> configuration options to allow you to override that behavior.
> >>
> >> - -chris
> >> -BEGIN PGP SIGNATURE-
> >> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> >>
> >> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7ZkACgkQHPApP6U8
> >> pFgOng/8DtMJkQc+MYLRm1iuCD7GZ2f2S7oQ59vFeCGAbmkiUjESvQ42QRGqIMy8
> >> 47giFc3ERm84DxyyyU7O/YDFxinOnCrC/v9A6RzpYKlZBSOq9Oy6732xTUAqGGIw
> >> +3QGPXvyjE2Vcg1iavW+7cUKN1Q9R1NGcDRRBpBL0KRQMA3NV9pxGU71/In8TQvi
> >> VZ51f7VTNXaJ2l+w6G23XBJkKQk3csFixevVzr4Xr56FLfqPxUc3m6QNu4BaHmgb
> >> c95hceV/N7tkR9yHdaRtahpZq0lhGqXbNXfqjf7kElSkRmeAZ3MSsdnFD4fBHThn
> >> xvz204xffSE71Z4W24W9gx23+Bg0y2EfPRo1CWC93rEvNRKMK6ILzLczTRA0w6QW
> >> 9zP1XC+VwC25LQOGFDgFukQVupPYiMoNSb6DRey5ZUhur6v25nevwbhM0QsAm/oO
> >> oZpreKaUMy+ZoixwGhaZ+UFiZRav7DRLSj85BjK9PqcP4VdPzFR9MarvMqLPxRoq
> >> YxL/jNet4L+29Z2tDkZv4gfGJqI7oWkfXUsBFBjj5JgXrqE94Q8PzAK87pLeU80y
> >> p3IL4krovHbu01j1fE3/aDotEvBu/wxWTCWze9+vL09a82PuTT2pihyCVqFuP9rS
> >> kP0DtVTfbaUMyD2dryjyw4q1NdLYht4y/HHkyU/3cPopCbxEopU=
> >> =4F4z
> >> -END PGP SIGNATURE-
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
>

Re: Tomcat Processing Timer Question

[Martin Grigorov]

Hi,

On Sat, Sep 12, 2020, 02:57 Eric Robinson  wrote:

> I'm not sure what you mean by measuring at the load balancer level. We're
> using the jasper logs and those only exist on the tomcat server itself. I
> must be misunderstanding your meaning.
>

He meant to use the LB's logs for the same.
What software do you use for load balancing?


> Get Outlook for Android
>
> 
> From: Christopher Schultz 
> Sent: Thursday, September 10, 2020 3:11:43 PM
> To: users@tomcat.apache.org 
> Subject: Re: Tomcat Processing Timer Question
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Eric,
>
> On 9/10/20 15:29, Eric Robinson wrote:
> > Chris --
> >
> >
> >> You should also look at worker-thread availability. When you see
> >>  these "high latency" (which is usually a term reserved for I/O
> >> characterization) events, do you have:>> 1. Available worker
> >> threads (from the executor thread pool) 2. Any other
> >> shared/limited resource (e.g. DB connection pool)
> >>
> >
> > Good thought. I should mention that the hosted application is
> > canned, and is the same for all tomcat instances, with only minor
> > variations in version between them. User workflow is also similar.
> > Over the years we've developed a good feel for expected
> > performance and resource utilization based on the user count per
> > instance. So when one instance exhibits anomalous performance, we
> > tend to go right to networking issues.
> >
> >> Also, are you seeing the otherwise unexpected slowness on each
> >> Tomcat node, or are you seeing it at the
> >> load-balancer/multiplexer level?
> >>
> >
> > We run multi-tenanted servers, with many instances of tomcat on
> > each server. We've never seen issues at the load-balancer.
>
> What I mean is, are you measuring the request at the Tomcat level, or at
> the load-balancer level? If you are watching at the lb, then your lb
> might pick a "busy" Tomcat and the request has to "wait in line" before
> processing even begins. If you sample at the Tomcat level, you'll see no
> discernible slowdown because the time "waiting in line" does not count.
>
> > Very occasionally, there might be a problem at the server level.
> > When that happens, all instances on that server may become
> > sluggish. What I'm talking about in this thread are cases where
> > only one instance on a server is showing slowness in its jasper
> > logs. Also, we typically do not see the same slowness when we test
> > the application locally from the same network. I've had my eye on
> > TCP retransmits as a possible culprit for a while, but I just
> > didn't know for sure if my understanding of the tomcat processing
> > timer is correct.
> I hope we've cleared that up for you, then.
>
> You might also want to read about "buffer bloat" if you aren't already
> familiar with that term.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9aiH4ACgkQHPApP6U8
> pFiFfBAAvuUbRXK+iDDy7lLsw6eplMFrXXDbkxzwtSNafvdGlDWmcPWwdazZwhQ+
> TJ0pzkUwf3/RBslu/oORJYelYKhpUJLodj0Y85ZtbuKBcU2JpKk1uueJ/aqnmVFK
> 9yep3ReYdggEXQ3JNb1VeI4ASdEhFWoFw8pc6DAcJZ4K2JaUtGKrtoWG8n+oEXos
> kmthl9Dm9ge3edLimd7TPTx11iODi6pX3ddJ+uRh7qmvXZp4wVyX8W+hkKiOhUQM
> hokUd8RruXQm6wut5m+JSO6eLHqkKUBiLspzlz1x/Y4cuaqAlC8Pl5y9NFTuLK3e
> gFJeDmBUthN2y5h9KNKW5r+Gf9bKpuv1+kw7CIaNoFv2JxCGTmfL3VKM+Bp/rh7J
> 1SbshsTW6ffo5hKRNJUJKvxry3uUvzrss0AYe338tJ1QA+sHuXHsN8ZVtY3b+51O
> HBOYf3pgIPsSd6zXkjaSRoOAhVc9G5sbJHx8ycQt+yAyVvXEUwrqeeRbsJeADk2s
> reaizm9WvO2kHSqP93ANNYe1QJ+rw9b5og0uoCE8x9eO+czRHbJ7LFF6/rvX+6Pn
> TIYB7AHyV8P3PHpHtBGIgaNfnvIYbqx/hzxJpLlpNEcS2zARfi1YCnuNtbiH0KU/
> AKkBx5FnZvwclCA3qK2oqBnSEcBUFz2yobq4wAy//qwgL2gEFNc=
> =mcpm
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
> Disclaimer : This email and any files transmitted with it are confidential
> and intended solely for intended recipients. If you are not the named
> addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the company
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments.
>

Re: How to Set Content Security Policy headers in Tomcat 8.5.x

[Martin Grigorov]

Hi,

On Wed, Sep 9, 2020 at 8:54 PM Nitin Kadam  wrote:

> Hi All,
>
> Need to set the *Content security policy* header for Tomcat Web server
> (8.5..x) which hosted on Windows server 2012, As per the internal security
> team same is not a complaint
> can you please help me setting CSP filters for my Tomcat application
> hosted on windows server.
>

You can use javax.servlet.Filter to add such custom headers.
See
https://github.com/apache/tomcat/blob/53c304ad1f65a09c921c40e03a115de438f6c68a/java/org/apache/catalina/filters/HttpHeaderSecurityFilter.java
for
inspiration.
More about Filters you can read in the web, e.g.
https://www.tutorialspoint.com/servlets/servlets-writing-filters.htm


>
> below the screenshot from securityheaders.com
>
> [image: image.png]
>
> --
> Regards
> Nitin Kadam
>
>

Re: HTTP2: Connections abruptly closed by sending GOAWAY

[Martin Grigorov]

Hi,

On Wed, Sep 9, 2020 at 3:31 PM Arshiya Shariff
 wrote:

> Thank you for the response Mark .
>
>
>
> Hi All,
>
>
>
> Can you please help us understand this behavior .
>
> The following is the sequence of events that is happening for a few
> streams .
>
> Initially when the stream is opened , it processes fine , but beyond some
> time tomcat does not process the headers, for which the client is sending
> RST_STREAM (ERROR: NO_ERROR) and then tomcat closes the stream with GOAWAY
> (An error occurred during processing that was fatal to the connection).
>
>
>
>
Your images didn't make it thru the mailing list.
https://markmail.org/message/c5pz2bmuaeodrlhr
Please use some image bin and give us the url.

> Please help us analyze under which cases this happens .
>
> Embedded tomcat:9.0.22
>
>
>
> Thanks and Regards
>
> Arshiya Shariff
>
>
>
>
>
> -Original Message-
> From: Mark Thomas 
> Sent: Monday, September 7, 2020 10:21 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: Connections abruptly closed by sending GOAWAY
>
>
>
> On 07/09/2020 09:29, Arshiya Shariff wrote:
>
> > Hi All,
>
> > Tomcat is closing connections abruptly by sending GOAWAY with reason
>
> > Connection [5309], Stream [57,359], An error occurred during processing
> that was fatal to the connection .
>
> >
>
> > Just trying to understand in what scenarios this happens. Can you please
> help us.
>
>
>
> An unhandled InterruptedIOException during I/O.
>
>
>
> I/O exception writing an intermediate 100 response for a request with an
> expectation.
>
>
>
> I/O exception trying to initiate a push request.
>
>
>
> I/O exception committing, flushing or closing a stream.
>
>
>
> So, generally, any I/O exception that indicates the connection between the
> client and the server is broken.
>
>
>
> Mark
>
>
>
> -
>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>

Re: Release date of latest Tomcat version - 9.0.38

[Martin Grigorov]

Hi,

On Wed, Sep 2, 2020, 17:46 Arshiya Shariff
 wrote:

> Hi ,
> Thank you for your response .
> Is there a way I can track when the version is released for Voting (or)
> will I get notified in the Users list ?
>

You need to subscribe to the dev@ mailing list.



> Thanks and Regards
> Arshiya Shariff
>
> -Original Message-
> From: Mark Thomas 
> Sent: Wednesday, September 2, 2020 12:03 AM
> To: users@tomcat.apache.org
> Subject: Re: Release date of latest Tomcat version - 9.0.38
>
> On 01/09/2020 14:01, Christopher Schultz wrote:
> > Arshiya,
> >
> > On 9/1/20 08:13, Arshiya Shariff wrote:
> >> Hi all,
> >
> >> The following reported issue - "HTTP/2 Stream.receivedData method
> >> throwing continuous NullPointerException in the logs" has been fixed
> >> in the latest tomcat.
> >> https://bz.apache.org/bugzilla/show_bug.cgi?id=64671
> >
> >> Can you please share us the release date of tomcat version 9.0.38 .
> >> We are waiting for the release dates so we can plan accordingly.
> > There are no promises about release schedule.
> >
> > Mark, however, has fairly consistently been rolling releases around
> > the beginning of each month. If you read the developers list, you'll
> > see there was a conversation started last week about the next batch of
> > releases.
>
> I'm expecting to tag the next round of releases in a few days. I am
> currently waiting for the Commons Daemon 1.2.3 release (it was delayed by
> an issue with the code signing service) and the Tomcat Native 1.2.25
> release (vote should hopefully complete shortly).
>
> If the Commons Daemon release slips then the Tomcat release will probably
> slip.
>
> > Once a release candidate has been announced (look for [VOTE] threads),
> > you can help by testing the release:
>
> Big +1
>
> >   1. Run the test suite (ant test)
> >   2. Run the release-candidate on your own application
> >
> > Your vote is not binding, but if you find a problem, we will likely
> > stop the release to fix it.
>
> Indeed. We've done that a few times. Much better to test the release
> during the vote (when we can fix it and re-roll the release) rather than
> once it is released (when you have to wait for the next release in ~1
> month). The number of times people have downloaded a new release and found
> an issue that they could have caught if they tested during the vote...
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat 9 : Unable to specify wildcard care name in Host

[Martin Grigorov]

On Tue, Aug 25, 2020 at 3:01 PM Tom Chiverton  wrote:

> I'd like to propose some changes to the docs, but can not locate their
> source to submit a request.
>

https://github.com/apache/tomcat/blob/master/webapps/docs/config/host.xml
This is for Tomcat 10.x (master branch).


>
> For instance, in https://tomcat.apache.org/tomcat-9.0-doc/config/host.html
>
> "If the name takes the form of |*.domainname| (e.g. |*.apache.org|) then
> it will be treated as a match for any host in that domain unless a host
> that has an exactly matching name is found."
>
> would be clearer as
>
> "The name can not contain a wildcard (e.g. |*.apache.org|), this is only
> valid in an Alias."
>
> And then in
> https://tomcat.apache.org/tomcat-9.0-doc/config/host.html#Host_Name_Aliases
>
> "Aliases may also use the wildcard form (|*.domainname|) permitted for
> the *name* attribute of a *Host*."
>
> to
>
> "Aliases may also use the wildcard form (|*.domainname|), unlike for the
> name attribute of a Host**."
>
> On 24/08/2020 13:14, Tom Chiverton wrote:
> > Am I mis-reading the docs ?
>
>
> __
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> __

Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

[Martin Grigorov]

Hi,

On Wed, Aug 26, 2020 at 7:53 AM Pratik Shrestha  wrote:

> Thanks for reply,
>
> Hi Peter - it complains on port 8443 which belongs to Tomcat.
>
> Hi Mark - Yes. making HTTP request on HTTPS is wrong. But this security
> vulnerability is given to us by Qualys scan. It tries to post plain HTTP
> request on HTTPS port and then gets error message "Bad Request. This 
> combination
> of host and port requires TLS." which is security loop hole for Qualys.
> This is behaviour of Apache HTTP server also. But in Apache though, we can
> get rid of this by using "ErrorDocument 400" directive. Do we have similar
> in Tomcat? I have already tried using
>
> 
>400
>/error.jsp
>  
>

This won't work because Tomcat stops the request earlier and doesn't pass
it to your application.
I haven't tried it but it may work with a custom Valve, extending
ErrorReportValve.


>
> Not sure, but my idea was to add redirect code on error.jsp page. But
> above never works. It never reaches error.jsp page. Just sticks in default
> error message page mentioned above.
>
> Btw..you can see the result from Qualys attached.
>

What is the desired behavior expected by Qualys ?
Because at the moment Tomcat returns a text/html error page and you try to
"fix" it by returning a custom text/html error page. I don't see how this
will change the Qualys report.


>
> Thanks again guys for getting back.
>
> Regards,
> Pratik
>
> On Tue, Aug 25, 2020 at 5:36 PM Mark Thomas  wrote:
>
>> On 25/08/2020 11:14, Pratik Shrestha wrote:
>> > Hi all,
>> >
>> > Tomcat version: 9.0.37
>> >
>> > Our website is running on Tomcat. We did Qualys vulnerability scan on
>> our
>> > site. Scan shows below vulnerability.
>> >
>> > Insecure transport
>> > Group: Information Disclosure
>> > CWE CWE-319
>> > OWASP A3 Sensitive Data Exposure
>> > WASC WASC-4 INSUFFICIENT TRANSPORT LAYER PROTECTION
>> >
>> > Please note
>> > 1. HTTP port is not enabled.
>> > 2. We have only opened HTTPS port 8443. But when we connect this HTTPS
>> port
>> > with HTTP (http://www.oursite.com:8443/), we get an error "Bad
>> Request. This
>> > combination of host and port requires TLS."
>> > 3. Due to the above error message, we get this vulnerability error from
>> > Qualys.
>> > 4. We have already enabled HSTS.
>> > 5. We have enabled Rewrite Valve also to direct all HTTP to HTTPS. But
>> it
>> > never works. It is like, Tomcat doesn't care about Rewrite or HSTS. It
>> just
>> > finds someone is accessing HTTPS port with HTTP protocol and then just
>> > throws error 400 'Bad Request'
>> > 6. Note that Tomcat version 7 used to send the error 'ERR_EMPTY_RESP'
>> which
>> > should still be okay.
>> >
>> > We already tried to find the fix for this issue on the web but in vain.
>> >
>> > Kindly help if anyone has found a way to fix it.
>>
>> Fix what?
>>
>> If you make an HTTP request to an HTTPS port, Tomcat provides a helpful
>> error message.
>>
>> I don't see any security issues here.
>>
>> (And before anyone claims the request sent in the clear is insecure I'll
>> point out that the request is sent in the clear irrespective of whether
>> Tomcat responds with an HTTP/1.1 clear text error message or a cryptic
>> TLS failure).
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Date of EOL and EOS for Tomcat8.5

[Martin Grigorov]

Hi,

On Wed, Aug 5, 2020 at 5:59 PM Rajat Gupta  wrote:

> Hi,
>
> Please let us know the date of End Of Service and End Of Life for Tomcat
> *8.5*
>

It is not known yet.
At the moment there is a date set only for 7.0.x:
https://tomcat.apache.org/tomcat-70-eol.html
Newer versions will live several years longer.


> Thank you!
>

Welcome!
You have missed to add few mailing lists in CC!


>
> Best regards,
> Rajat Gupta
>

Re: Best practices for profiling

[Martin Grigorov]

Hi,

On Sun, Aug 2, 2020 at 1:09 AM calder  wrote:

> On Sat, Aug 1, 2020, 15:16 Aryeh Friedman 
> wrote:
>
> > On Sat, Aug 1, 2020 at 3:28 PM calder  wrote:
> >
> > > On Sat, Aug 1, 2020 at 12:53 PM Aryeh Friedman <
> aryeh.fried...@gmail.com
> > >
> > > wrote:
> > > >
> > > > What are the best practices for (open source) profiling a JSP (no
> > taglibs
> > >
> > > There's no "one rule" for this
> > >
> > > > or any EE outside of JSP and JDBC) webapp running on a remote machine
> > all
> > > > the way down the method level (our main user is complaining about
> > > > performance and we have done all the easy fixes that can be done
> > without
> > > > profiling).   I have played with VisualVM and while it was helpful in
> > > > finding the grosser high level stuff it was not the best for
> > pinpointing
> > > > specific bottlenecks in the actual code.
> > >
> >
> >
> > > [1] We prefer to not profile / debug in a client's environment,
> > > because there may be "too much going on elsewhere".
> > >
> >
> > Ours is a custom web app (remote cardiac monitoring/record keeping) that
> > has only one production server and thus might make the choices narrower.
> > The primary problem is it runs just fine on our development machines
> (even
> > under stress tests) because we have more efficent setup then what their
> > hosting company is willing to provide (we already use VM's and have them
> > backed by local SSD's w/ traditional file systems but the hosting company
> > insists on doing networked object storage [which is impossible to model
> > outside of that environment]).   When not loaded outside of peak business
> > hours it also runs just fine on the hosting company.   Thus it comes down
> > to a problem only when it is under a production load and the client is
> not
> > willing to pay for a duplicate setup at the hosting firm (nor would it
> let
> > us see it under load).   Our main goal in profiling is to identify which
> > classes/methods that are the bottlenecks.   We are fairly sure it is
> > backend I/O driven and just need to see which of the I/O calls are the
> big
> > offenders so we can work out ways of avoiding making them.
> >
> > As to debugging except for confirming the bug in production and/or using
> > its logs to pinpoint the bug we do not do debugging in production.
> >
>
> You'll probably want to look at an APM type, such as AppDynamics
>

Before doing this I'd suggest you to try something simpler: ask them to
provide you with thread dumps while the system is under load.
I.e. whenever they feel the system is slow someone who has access to Tomcat
should take several thread dumps (e.g. 3-5 seconds between each dump). They
could use jstack, jcmd, kill -3, VisualVM ...
>From the stack traces you could figure out which methods are busiest and
what kind of network calls are being done

Re: Error in stopping application tomcat !!

[Martin Grigorov]

On Thu, Jul 23, 2020, 15:52 Kushagra Bindal 
wrote:

> Thanks Martin.
>
> But with the old version i.e. 8.5.24 it is working smoothly. So, what could
> be the problem? Or some specific property/configuration changes that need
> to be made around this?
>

You will have to consult with the changelogs for all the versions in
between.


> On Thu, Jul 23, 2020 at 6:00 PM Martin Grigorov 
> wrote:
>
> > On Thu, Jul 23, 2020 at 3:10 PM Kushagra Bindal <
> bindal.kusha...@gmail.com
> > >
> > wrote:
> >
> > > Hi Martin,
> > >
> > > Due to our environment I was not able to use pastebin service. I have
> > > taken different Thread dump during shutdown and attaching the same with
> > > this email.
> > >
> > > Please review the same and let me know, what is the probable root cause
> > of
> > > the problem and what could be the fix of the same.
> > >
> >
> > You have many AMQP (RabbitMQ) listener threads which are not daemons.
> > It seems your application does not notify Spring Framework that it needs
> to
> > destroy its application context or the beans with
> > type
> > org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer
> > are not notified to stop for some other
> > reason.
> >
> >
> > >
> > > On Thu, Jul 23, 2020 at 3:22 PM Martin Grigorov 
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> On Thu, Jul 23, 2020 at 6:35 AM Kushagra Bindal <
> > >> bindal.kusha...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Martin,
> > >> >
> > >> > These are the only two behaviors right now which I am getting on a
> > >> > regular basis.
> > >> > 1. During startup of the application and then shutdown
> > >> > 2. Running application and then shutdown.
> > >> >
> > >> > Please let me know if anything specific is further needed from my
> end
> > >> which
> > >> > I can provide to have a better clarity.
> > >> >
> > >> > I have shared the server.xml and command which we are using in
> > stopping
> > >> the
> > >> > tomcat.
> > >> >
> > >> >
> > >> > On Thu, Jul 23, 2020 at 2:49 AM Martin Grigorov <
> mgrigo...@apache.org
> > >
> > >> > wrote:
> > >> >
> > >> > > On Wed, Jul 22, 2020, 15:55 Kushagra Bindal <
> > >> bindal.kusha...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Christopher,
> > >> > > >
> > >> > > > Did you get a chance to look into this?
> > >> > > >
> > >> > > > Please help us in resolving this issue.
> > >> > > >
> > >> > > > On Sat, Jul 18, 2020 at 11:26 AM Kushagra Bindal <
> > >> > > > bindal.kusha...@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Chris,
> > >> > > > >
> > >> > > > > Additionally when trying to stop running application, we are
> > >> getting
> > >> > > > below
> > >> > > > > error.
> > >> > > > >
> > >> > > > > Sat Jul 18 05:49:40 UTC 2020
> > >> > > > > **
> > >> > > > > *  Stopping the Web Server
> > >> > > > > **
> > >> > > > > Sat Jul 18 05:49:40 UTC 2020
> > >> > > > > ./wfc: line 28: /usr/local/nginx/nginx: No such file or
> > directory
> > >> > > > > ./wfc: line 233: /usr/local/nginx/nginx: No such file or
> > directory
> > >> > > > >
> > >> > > > > Sat Jul 18 05:49:40 UTC 2020
> > >> > > > > *  Nginx has been stopped
> > >> > > > > **
> > >> > > > > *  Shutdown the wfc Server gracefully
> > >> > > > > **
> > >> > > > > #
> > >> >
> *
> > >> > > > > # Tomcat shutdown with wait time: 30
> > >> > > > > #
> > >> >
> ***

Re: Error in stopping application tomcat !!

[Martin Grigorov]

On Thu, Jul 23, 2020 at 3:10 PM Kushagra Bindal 
wrote:

> Hi Martin,
>
> Due to our environment I was not able to use pastebin service. I have
> taken different Thread dump during shutdown and attaching the same with
> this email.
>
> Please review the same and let me know, what is the probable root cause of
> the problem and what could be the fix of the same.
>

You have many AMQP (RabbitMQ) listener threads which are not daemons.
It seems your application does not notify Spring Framework that it needs to
destroy its application context or the beans with
type org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer
are not notified to stop for some other
reason.


>
> On Thu, Jul 23, 2020 at 3:22 PM Martin Grigorov 
> wrote:
>
>> Hi,
>>
>> On Thu, Jul 23, 2020 at 6:35 AM Kushagra Bindal <
>> bindal.kusha...@gmail.com>
>> wrote:
>>
>> > Hi Martin,
>> >
>> > These are the only two behaviors right now which I am getting on a
>> > regular basis.
>> > 1. During startup of the application and then shutdown
>> > 2. Running application and then shutdown.
>> >
>> > Please let me know if anything specific is further needed from my end
>> which
>> > I can provide to have a better clarity.
>> >
>> > I have shared the server.xml and command which we are using in stopping
>> the
>> > tomcat.
>> >
>> >
>> > On Thu, Jul 23, 2020 at 2:49 AM Martin Grigorov 
>> > wrote:
>> >
>> > > On Wed, Jul 22, 2020, 15:55 Kushagra Bindal <
>> bindal.kusha...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Christopher,
>> > > >
>> > > > Did you get a chance to look into this?
>> > > >
>> > > > Please help us in resolving this issue.
>> > > >
>> > > > On Sat, Jul 18, 2020 at 11:26 AM Kushagra Bindal <
>> > > > bindal.kusha...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Chris,
>> > > > >
>> > > > > Additionally when trying to stop running application, we are
>> getting
>> > > > below
>> > > > > error.
>> > > > >
>> > > > > Sat Jul 18 05:49:40 UTC 2020
>> > > > > **
>> > > > > *  Stopping the Web Server
>> > > > > **
>> > > > > Sat Jul 18 05:49:40 UTC 2020
>> > > > > ./wfc: line 28: /usr/local/nginx/nginx: No such file or directory
>> > > > > ./wfc: line 233: /usr/local/nginx/nginx: No such file or directory
>> > > > >
>> > > > > Sat Jul 18 05:49:40 UTC 2020
>> > > > > *  Nginx has been stopped
>> > > > > **
>> > > > > *  Shutdown the wfc Server gracefully
>> > > > > **
>> > > > > #
>> > *
>> > > > > # Tomcat shutdown with wait time: 30
>> > > > > #
>> > *
>> > > > > Using CATALINA_BASE:   /usr/local/xyz/tomcat
>> > > > > Using CATALINA_HOME:   /usr/local/xyz/tomcat
>> > > > > Using CATALINA_TMPDIR: /usr/local/xyz/tomcat/temp
>> > > > > Using JRE_HOME:/usr/local/xyz/jdk
>> > > > > Using CLASSPATH:
>> > > > >
>> > > >
>> > >
>> >
>> /usr/local/xyz/tomcat/bin/bootstrap.jar:/usr/local/xyz/tomcat/bin/tomcat-juli.jar
>> > > > > Using CATALINA_PID:/usr/local/xyz/tomcat/tomcat.pid
>> > > > > Tomcat did not stop in time.
>> > > > > To aid diagnostics a thread dump has been written to standard out.
>> > > >
>> > >
>> > > Anything interesting in this thread dump?
>> >
>>
>> Tomcat does not stop properly in time and a thread dump is written to the
>> stdout.
>> Please share the thread dump. Use some pastebin service!
>>
>>
>> > >
>> > >
>> > > > Killing Tomcat with the PID: 4280
>> > > > > The Tomcat process has been killed.
>> > > > > #
>> > *
>> > > > &

Re: Error in stopping application tomcat !!

[Martin Grigorov]

Hi,

On Thu, Jul 23, 2020 at 6:35 AM Kushagra Bindal 
wrote:

> Hi Martin,
>
> These are the only two behaviors right now which I am getting on a
> regular basis.
> 1. During startup of the application and then shutdown
> 2. Running application and then shutdown.
>
> Please let me know if anything specific is further needed from my end which
> I can provide to have a better clarity.
>
> I have shared the server.xml and command which we are using in stopping the
> tomcat.
>
>
> On Thu, Jul 23, 2020 at 2:49 AM Martin Grigorov 
> wrote:
>
> > On Wed, Jul 22, 2020, 15:55 Kushagra Bindal 
> > wrote:
> >
> > > Hi Christopher,
> > >
> > > Did you get a chance to look into this?
> > >
> > > Please help us in resolving this issue.
> > >
> > > On Sat, Jul 18, 2020 at 11:26 AM Kushagra Bindal <
> > > bindal.kusha...@gmail.com>
> > > wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > Additionally when trying to stop running application, we are getting
> > > below
> > > > error.
> > > >
> > > > Sat Jul 18 05:49:40 UTC 2020
> > > > **
> > > > *  Stopping the Web Server
> > > > **
> > > > Sat Jul 18 05:49:40 UTC 2020
> > > > ./wfc: line 28: /usr/local/nginx/nginx: No such file or directory
> > > > ./wfc: line 233: /usr/local/nginx/nginx: No such file or directory
> > > >
> > > > Sat Jul 18 05:49:40 UTC 2020
> > > > *  Nginx has been stopped
> > > > **
> > > > *  Shutdown the wfc Server gracefully
> > > > **
> > > > #
> *
> > > > # Tomcat shutdown with wait time: 30
> > > > #
> *
> > > > Using CATALINA_BASE:   /usr/local/xyz/tomcat
> > > > Using CATALINA_HOME:   /usr/local/xyz/tomcat
> > > > Using CATALINA_TMPDIR: /usr/local/xyz/tomcat/temp
> > > > Using JRE_HOME:/usr/local/xyz/jdk
> > > > Using CLASSPATH:
> > > >
> > >
> >
> /usr/local/xyz/tomcat/bin/bootstrap.jar:/usr/local/xyz/tomcat/bin/tomcat-juli.jar
> > > > Using CATALINA_PID:/usr/local/xyz/tomcat/tomcat.pid
> > > > Tomcat did not stop in time.
> > > > To aid diagnostics a thread dump has been written to standard out.
> > >
> >
> > Anything interesting in this thread dump?
>

Tomcat does not stop properly in time and a thread dump is written to the
stdout.
Please share the thread dump. Use some pastebin service!


> >
> >
> > > Killing Tomcat with the PID: 4280
> > > > The Tomcat process has been killed.
> > > > #
> *
> > > > # Tomcat shutdown result: 0
> > > > #
> *
> > > >
> > > >
> > > >
> > > > Whereas below error that I mentioned earlier is coming when
> application
> > > is
> > > > in INPROGRESS and not yet come into running state and when we stop
> the
> > > same
> > > > it is throwing below error.
> > > >
> > > > On Sat, Jul 18, 2020 at 10:17 AM Kushagra Bindal <
> > > > bindal.kusha...@gmail.com> wrote:
> > > >
> > > >>  Hi Chris,
> > > >>
> > > >> To stop tomcat we are using the below command.
> > > >>
> > > >> bin/shutdown.sh -$sleeptime -force
> > > >>
> > > >> Where in our case sleeptime is set to 30.
> > > >>
> > > >> Please find the attached server.xml which we are using in our
> > > application.
> > > >>
> > > >> Also I have copy-paste the complete content below in this email as
> > well.
> > > >>
> > > >>
> > >
> >
> 
> > > >>
> > > >> 
> > > >> 
> > > >>> > className="org.apache.catalina.startup.VersionLoggerListener"
> > > >> />
> > > >>className="org.apache.catalina.core.AprLifecycleListener"
> > > >> SSLEngine="on"

Re: Error in stopping application tomcat !!

[Martin Grigorov]

On Wed, Jul 22, 2020, 15:55 Kushagra Bindal 
wrote:

> Hi Christopher,
>
> Did you get a chance to look into this?
>
> Please help us in resolving this issue.
>
> On Sat, Jul 18, 2020 at 11:26 AM Kushagra Bindal <
> bindal.kusha...@gmail.com>
> wrote:
>
> > Hi Chris,
> >
> > Additionally when trying to stop running application, we are getting
> below
> > error.
> >
> > Sat Jul 18 05:49:40 UTC 2020
> > **
> > *  Stopping the Web Server
> > **
> > Sat Jul 18 05:49:40 UTC 2020
> > ./wfc: line 28: /usr/local/nginx/nginx: No such file or directory
> > ./wfc: line 233: /usr/local/nginx/nginx: No such file or directory
> >
> > Sat Jul 18 05:49:40 UTC 2020
> > *  Nginx has been stopped
> > **
> > *  Shutdown the wfc Server gracefully
> > **
> > # *
> > # Tomcat shutdown with wait time: 30
> > # *
> > Using CATALINA_BASE:   /usr/local/xyz/tomcat
> > Using CATALINA_HOME:   /usr/local/xyz/tomcat
> > Using CATALINA_TMPDIR: /usr/local/xyz/tomcat/temp
> > Using JRE_HOME:/usr/local/xyz/jdk
> > Using CLASSPATH:
> >
> /usr/local/xyz/tomcat/bin/bootstrap.jar:/usr/local/xyz/tomcat/bin/tomcat-juli.jar
> > Using CATALINA_PID:/usr/local/xyz/tomcat/tomcat.pid
> > Tomcat did not stop in time.
> > To aid diagnostics a thread dump has been written to standard out.
>

Anything interesting in this thread dump?


> Killing Tomcat with the PID: 4280
> > The Tomcat process has been killed.
> > # *
> > # Tomcat shutdown result: 0
> > # *
> >
> >
> >
> > Whereas below error that I mentioned earlier is coming when application
> is
> > in INPROGRESS and not yet come into running state and when we stop the
> same
> > it is throwing below error.
> >
> > On Sat, Jul 18, 2020 at 10:17 AM Kushagra Bindal <
> > bindal.kusha...@gmail.com> wrote:
> >
> >>  Hi Chris,
> >>
> >> To stop tomcat we are using the below command.
> >>
> >> bin/shutdown.sh -$sleeptime -force
> >>
> >> Where in our case sleeptime is set to 30.
> >>
> >> Please find the attached server.xml which we are using in our
> application.
> >>
> >> Also I have copy-paste the complete content below in this email as well.
> >>
> >>
> 
> >>
> >> 
> >> 
> >>className="org.apache.catalina.startup.VersionLoggerListener"
> >> />
> >>>> SSLEngine="on" />
> >>>> className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
> >>>> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
> >>>> className="org.apache.catalina.core.ThreadLocalLeakPreventionListener"
> />
> >>   
> >>  >>   type="org.apache.catalina.UserDatabase"
> >>   description="User database that can be updated and saved"
> >>
> >> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> >>   pathname="conf/tomcat-users.xml" />
> >>  >>   type="javax.sql.DataSource"
> >>   username="db_username_stub"
> >>   password="db_password_stub"
> >>   factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
> >>   url="READ_WRITE_URL_STUB"
> >>   driverClassName="com.edb.Driver"
> >>   initialSize="5"
> >>   maxWait="3"
> >>   maxActive="50"
> >>   maxIdle="20"
> >>   minIdle="5"
> >>   maxAge="720"
> >>   validationQuery="SELECT 1; COMMIT;"
> >>  initSQL="ALTER SESSION SET statement_timeout=360; ALTER SESSION SET
> >> idle_in_transaction_session_timeout=366; COMMIT;"
> >>   poolPreparedStatements="true"
> >>   testWhileIdle="false"
> >>   testOnBorrow="true"
> >>   testOnReturn="false"
> >>   validationInterval="12"
> >>   timeBetweenEvictionRunsMillis="15000"
> >>   removeAbandonedTimeout="300"
> >>   removeAbandoned="false"
> >>   logAbandoned="false"
> >>   minEvictableIdleTimeMillis="12"
> >>   jmxEnabled="true"
> >>
> >>
> jdbcInterceptors="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;
> >>
> >> org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer;
> >>
> >> org.apache.tomcat.jdbc.pool.interceptor.StatementCache(max=4000)" />
> >>  >>   type="javax.sql.DataSource"
> >>   username="db_username_stub"
> >>   password="db_password_stub"
> >>   factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
> >>   url="READ_ONLY_URL_STUB"
> >>   driverClassName="com.edb.Driver"
> >>   

Re: Memory leak in the PKCS11 how to fix the problem

[Martin Grigorov]

On Wed, Jul 22, 2020, 16:58 Ragavendhiran Bhiman (rabhiman)
 wrote:

> Sorry Martin,
>
> My answers are inline.
>

Take a heap dump and analyze it!


> Thanks & Regards,
> Raghav
>
> On 22/07/20, 7:20 PM, "Ragavendhiran Bhiman (rabhiman)"
>  wrote:
>
> Hello Martin,
>
> Thanks for your reply
>
>
> https://www.dropbox.com/sh/o6zra7pf2o1xpge/AAA1J7BaVdPDF7s3RjPmy0xBa?dl=0
>
> Here is the link I have shared the flame graph.
> Also kindly check my answers in red as well.
>
> Thanks & Regards,
> Raghav
>
>
> On 22/07/20, 2:33 PM, "Martin Grigorov"  wrote:
>
> Hi Ragavendhiran,
>
> On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
>  wrote:
>
> > Hello All,
> >
> >
> >
> > I am seeing the memory leaks from tomcat apache in the following
> SSL path
> > using PKCS11. Attached the flame graph of memory possible memory
> leaks in
> > this area.
> >
> > Please check the attached flame graph of the memory trace. On
> simply a
> > long run the memory keep on allocated in these back traces only
> causing the
> > memory leak, and the polling of the async profiler for more than
> 6hours
> > shows this clearly. Could you please help how to fix this
> problem?
> >
> > (open this svg graph in browser only)
> >
> >
> >
> > Note: If C_DestroyObject is not called because of finalizer
> accumulation
> > is also tested by inducing the gc using the jmap command still
> could see
> > the memory never gone down after the Full GC collection as well.
> Expecting
> > your advice on the same.
> >
>
> With AsyncProfiler '-e alloc' you can see what part of the code is
> responsible for making most of the memory allocations, but it
> doesn't tell
> you whether those objects leak or not. AsyncProfiler helps you to
> identify
> the top allocations and if you manage to reduce them then you will
> reduce
> the GC runs and the time they take.
> To debug memory leaks you need to take a heap dump, e.g. with 'jmap
> -dump:live,format=b,file=heap.bin ' and analyze it. I'd
> recommend you
> to use Eclipse MAT to do that.
>
>  "Yes with async profiler and just leaving the server without
> any sequence of action the memory started growing (RSS grows) when I
> profiled during that sequence there is no other memory allocation happening
> except this one.
> That’s why I am suspecting this flow clearly. Samples are only through
> this flow only".
>
> Also in your flame graph I see that Netty is responsible for
> 49.04% of
> the allocated objects and Tomcat for just 25.32%.
>
>
> >
> > Regards,
> >
> > Raghav
> >
> > Infrastructure engineer,
> >
> > Cisco ISE.
> >
> >
> >
> >
> >
> >
> -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>

Re: Memory leak in the PKCS11 how to fix the problem

[Martin Grigorov]

Hi Ragavendhiran,

On Sat, Jul 18, 2020 at 3:55 PM Ragavendhiran Bhiman (rabhiman)
 wrote:

> Hello All,
>
>
>
> I am seeing the memory leaks from tomcat apache in the following SSL path
> using PKCS11. Attached the flame graph of memory possible memory leaks in
> this area.
>
> Please check the attached flame graph of the memory trace. On simply a
> long run the memory keep on allocated in these back traces only causing the
> memory leak, and the polling of the async profiler for more than 6hours
> shows this clearly. Could you please help how to fix this problem?
>
> (open this svg graph in browser only)
>
>
>
> Note: If C_DestroyObject is not called because of finalizer accumulation
> is also tested by inducing the gc using the jmap command still could see
> the memory never gone down after the Full GC collection as well. Expecting
> your advice on the same.
>

With AsyncProfiler '-e alloc' you can see what part of the code is
responsible for making most of the memory allocations, but it doesn't tell
you whether those objects leak or not. AsyncProfiler helps you to identify
the top allocations and if you manage to reduce them then you will reduce
the GC runs and the time they take.
To debug memory leaks you need to take a heap dump, e.g. with 'jmap
-dump:live,format=b,file=heap.bin ' and analyze it. I'd recommend you
to use Eclipse MAT to do that.

Also in your flame graph I see that Netty is responsible for 49.04% of
the allocated objects and Tomcat for just 25.32%.


>
> Regards,
>
> Raghav
>
> Infrastructure engineer,
>
> Cisco ISE.
>
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Rewrite Valve Problem

[Martin Grigorov]

Hi Jerry,

On Tue, Jul 21, 2020 at 2:39 AM Jerry Malcolm 
wrote:

>
> On 7/20/2020 5:00 PM, Mark Thomas wrote:
> > On 20/07/2020 22:43, Jerry Malcolm wrote:
> >
> > 
> >
> >>> Do you have a ROOT web application deployed? If not, this could be
> >>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64593
> >> Mark, I do not have a root context.  So that very likely is the
> >> problem.  Not 100% sure why the thought is that there should always be a
> >> root context.  But I'm sure there are good reasons.  Easy enough to
> >> create one.  I'll let you know.
> > Generally, I'd expect to see a ROOT context so there is something there
> > to handle requests that don't map to any other context. It is easier to
> > deploy custom 404 error pages if you have a ROOT context for example.
> >
> > That said, it isn't necessary and we do treat stuff not working without
> > a ROOT context as a bug.
> >
> > All you need to fix BZ 64593 is a directory called ROOT in webapps.
>
> Wow.  4 little letters I've been using TC for over 15 years without
> a ROOT.  Probably because I never routed anything through mod_jk that
> didn't match a known context.   I have no problem if you want to treat
> not having a ROOT context as a bug,   But please give an error message
>

As explained earlier by Mark this was a bug and it is fixed for the next
release.
Using a dummy ROOT folder is a workaround until the next release.


> in startup and/or provide an error message if/when rewrite valve fails
> because there is no ROOT context defined.  Thank you for explaining
> what's happening.  There is not a chance I would have correlated my
> symptoms in 100 years to not having a ROOT context.  I really do
> appreciate the help.   All working now
>
> Jerry
>
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat relicts from previous installation - ubuntu

[Martin Grigorov]

On Thu, Jul 9, 2020, 19:13 Christoph Kukulies  wrote:

> Is it an Ubuntu peculiarity that a tomcat.service gets installed? I
> assume, I’m getting this automatically when doing an
> apt-get install tomcat8, right?
>

Yes. All distros do that.
You can download the tar.gz file from https://tomcat.apache.org if you
prefer to manage it yourself.


> —
> Christoph
>
> Am 09.07.2020 um 15:19 schrieb calder :
>
> On Thu, Jul 9, 2020, 06:59 Christoph Kukulies  wrote:.
>
>
> might try:
> user@machine:~ /> ps systemctl status 
> so in your example
> user@machine:~ /> ps systemctl status 1092
>
>
> systemctl status 1092
>
>
>
> worked (without the „ps“) :)
>
>
> sorry about that ... somehow fat-fingered that in : /
>
>
>

Re: Tomcat relicts from previous installation - ubuntu

[Martin Grigorov]

Hi Christoph,

On Thu, Jul 9, 2020 at 12:48 PM Christoph Kukulies 
wrote:

> I’m resurrecting tomcat on a server (Ubuntu 18.04.4) on which I had been
> running tomcat7 some time ago.
> At the moment, when the system has started up, I’m seeing a process:
>
> root@kuku:~# ps ax | grep -i tomcat
>  1092 ?Sl 0:08 /usr/lib/jvm/java-1.11.0-openjdk-i386/bin/java
> -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M
> -Xmx1024M -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath
> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar
> -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat
> -Djava.io.tmpdir=/usr/share/tomcat/temp
> org.apache.catalina.startup.Bootstrap start
>  2079 pts/0S+ 0:00 grep --color=auto -i tomcat
> root@kuku:~#
>
>
>
> Since I have no tomcat service script in /etc/init.d I have no idea at the
> moment, who or which script is starting this java process.
>

Most probably it is systemd.
Try with: systemctl status tomcat

If this is the case you can stop it with: systemctl stop tomcat
and disable it so it does not start on the next reboot with: systemctl
disable tomcat


>
> Any clues?
>
> Christoph
>
>
>
>
>

Re: [ANN] New committer: Raymond Augé

[Martin Grigorov]

Welcome Raymond!

On Thu, Jul 2, 2020, 17:40 Mark Thomas  wrote:

> On behalf of the Tomcat committers I am pleased to announce that
> Raymond Augé (rotty3000) has been voted in as a new Tomcat committer.
>
> Please join me in welcoming him.
>
> Kind regards,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Junk characters in SOAP request after upgrade to tomcat 9.0.31

[Martin Grigorov]

Hi,

On Fri, Jun 12, 2020 at 8:17 AM Naveen Kumar  wrote:

> Hi All,
>
> Can someone please help on this ? It is in production and affecting many
> customers. Upgrading tomcat is a big task.
> Any pointers will be really appreciated.
>

You say "If I upgrade tomcat to 9.0.35, the error disappears." - this is
the right solution - upgrade to the latest release. We make releases
because there are bug fixes and improvements.

If you just want to know which particular commit caused the regression you
experience in 9.0.31 then you have two options:
1) read the changelogs and try to figure out
2)  'git bisect' Tomcat sources:
git clone https://github.com/apache/tomcat
git bisect start
git bisect good tag/9.0.24
git bisect bad tag/9.0.31
ant clean
ant
deploy your app and test


>
> Thanks
> Naveen
>
> On Wed, Jun 10, 2020 at 4:24 PM Naveen Kumar  wrote:
>
> > Hi Luis,
> >
> > Thanks for your suggestion.
> > But I am wondering what has changed in 9.0.31. Because my webapp works
> > perfectly fine in 9.0.24 and 9.0.35.
> >
> > Thanks
> > Naveen
> >
> > On Wed, Jun 10, 2020 at 2:52 PM Luis Rodríguez Fernández <
> > uo67...@gmail.com> wrote:
> >
> >> Hello Naveen,
> >>
> >> Recently we have had a similar issue migrating a webapp from another
> >> application server to tomcat. We solved it specifying
> >> UTF-8 in the
> >> web.xml descriptor.
> >>
> >> You can read here [1] the long story :)
> >>
> >> Hope it helps,
> >>
> >> Luis
> >>
> >> [1]
> https://cwiki.apache.org/confluence/display/TOMCAT/Character+Encoding
> >>
> >>
> >> El mié., 10 jun. 2020 a las 11:08, Naveen Kumar ()
> >> escribió:
> >>
> >> > Hi All,
> >> >
> >> > I have a webapp A which has few SOAP services and I consume those
> >> services
> >> > from webapp B.
> >> > I started getting below error since I upgraded the tomcat to 9.0.31
> >> (from
> >> > 9.0.24):
> >> > com.sun.xml.ws.transport.http.HttpAdapter.invokeAsync Couldn't create
> >> SOAP
> >> > message due to exception: XML reader error:
> >> > javax.xml.stream.XMLStreamException: java.io.EOFException: Unexpected
> >> EOF
> >> >
> >> > Then I wrote a filter at webapp A to intercept the request and I could
> >> see
> >> > that some junk characters are added in the SOAP request.
> >> >
> >> > If I upgrade tomcat to 9.0.35, the error disappears.
> >> >
> >> > Problematic request:
> >> > LoggingFilter.doFilter - The servlet  request soap mapping  body is:à
> >>  8Ï
> >> > S(http://schemas.xmlsoap.org/soap/envelope/ð??? Envelope??? Body8Ï
> ns13
> >> >
> >> > Correct request:
> >> > The servlet  request soap mapping  body is: >> > encoding='UTF-8'?> >> >
> >> > Does anyone know what could be the possible reason for this?
> >> >
> >> > Thanks in advance.
> >> > - Naveen
> >> >
> >>
> >>
> >> --
> >>
> >> "Ever tried. Ever failed. No matter. Try Again. Fail again. Fail
> better."
> >>
> >> - Samuel Beckett
> >>
> >
>

Re: Should Tomcat 10 enable response compression by default?

[Martin Grigorov]

On Tue, Jun 9, 2020 at 11:26 PM Manuel Dominguez Sarmiento 
wrote:

> I would not change this default. GZIP (or other kinds) of response
> compression are better addressed as servlet filters. Having the Tomcat
> feature is good, but IMHO it should only be enabled by those who need it.
>
> At least in our case we have our own code to deal with this, considering
> proxying, CDN, buggy browsers, etc.
>

I think the same - it should be opt-in feature.

My 2c.

Martin


>
> *Manuel Dominguez Sarmiento*
>
> On 09/06/2020 17:20, Mark Thomas wrote:
> > Hi all,
> >
> > An enhancement has been opened to enable response compression by default:
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=64431
> >
> > In short, the proposal is to change the default for the Connector's
> > compression attribute from "off" to "on".
> >
> > This would be for Tomcat 10 onwards only.
> >
> > The following would be unchanged:
> > - compressibleMimeType
> > - compressionMinSize
> > - noCompressionStrongETag
> >
> > It would be helpful to know what the range of views of the user
> > community are on this proposal.
> >
> > So, thoughts?
> >
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
>