Re: Possible AbstractProtocol.waitingProcessors leak in Tomcat 9.0.75

[Jakub Remenec]

Hi,

I've experienced the same issue as described on Apache Tomcat 10.1.13.
After downgrading to 10.1.5 it started to work correctly. I also inspected
the heapdump of the application with memory problems and found out that
there were many org.apache.tomcat.websocket.WsSession present in
OUTPUT_CLOSED state. When I tried locally, I found out, that when I open
few websocket connections from Chrome and then go to Offline mode the
WsSessions remain in OUTPUT_CLOSED state. New connections afterwards have
state OPEN. In heapdump from production I saw around 4600 WsSessions but
only 40 were open. The rest was in the output closed state.
WsSessions are accessible through org.apache.coyote.AbstractProtocol ->
waitingProcessors Set. In heapdump and it was clearly visible that 49% of
heap was taken by the waitingProcessors set. When tomcat was downgraded to
10.1.5, I saw that WsSessions got cleared after going to offline mode.

Additional info - I've set the session timeout to 10minutes. The app uses
Java 17 with Spring Boot 3.1.x stack. It does not use any external STOMP
broker relay.

Regards,
Jakub.

On 2023/08/20 22:44:46 Mark Thomas wrote:
> On 20/08/2023 05:21, Mark Thomas wrote:
> > On 18/08/2023 11:28, Rubén Pérez wrote:
>
> 
>
> >> I started experiencing exactly the same issue when updating from Spring
> >> 6.0.7 to 6.0.9, therefore updating tomcat from 10.1.5 to 10.1.8. The
> >> Memory
> >> leak is very clearly visible in my monitoring tools. A further heap
dump
> >> reveals like many times more entries in waitingProcessors map than real
> >> active connections, and we end up with like 8 retained GB in memory
> >> full of
> >> those entries.
> >>
> >> I believe I have found a way to reproduce the issue locally. Open a
> >> websocket session from a client in Chrome, go to dev-tools and switch
the
> >> tab to offline mode, wait > 50secs, go and switch it back to No
> >> Throttling.
> >> Sometimes I get an error back to the client like:
> >>
> >> a["ERROR\nmessage:AMQ229014\\c Did not receive data from
> >> /192.168.0.1\\c12720
> >> within the 5ms connection TTL. The connection will now be
> >> closed.\ncontent-length:0\n\n\u"]
> >>
> >> And other times I get instead something like c[1002, ""] from Artemis
> >> followed by an "Invalid frame header" error from Chrome (websockets
> >> view in
> >> dev-tools).
> >>
> >> Only when it is the latter case, looks to be leaking things in that
map.
> >> Maybe it is a casualty or not, but that is what I have observed at
> >> least 2
> >> times.
> >>
> >> After the error appeared, I waited long enough for FE to reconnect the
> >> session, and then I just quitted Chrome.
> >
> > Thanks for the steps to reproduce. That is helpful. I'll let you know
> > how I get on.
>
> Unfortunately, I didn't get very far. Based on the log messages it looks
> very much like those are application generated rather than Tomcat
generated.
>
> At this point I am wondering if this is an application or a Tomcat
> issue. I'm going to need a sample application (ideally as cut down as
> possible) that demonstrates the issue to make progress on this.
>
> Another option is debugging this yourself to figure out what has
> changed. I can provide some pointers if this is of interest. Giv en you
> can repeat the issue reaosnable reliably, tracking down the commit that
> trigger the change isn't too hard.
>
> >> Again, after forcefully downgrading Tomcat 10.1.8 to 10.1.5 while
> >> preserving the same Spring version, the issue is gone (confirmed in
> >> production), in fact I have never managed to get an "Invalid frame
> >> header"
> >> in Chrome again with Tomcat 10.1.5 (in like 10 attempts). Before I got
it
> >> in 2 out of 4 attempts.
> >
> > Could you do some further testing and see if you can narrow down exactly

> > which version (10.1.6, 10.1.7 or 10.1.8) the issue first appears in?
> >
> > It would also be helpful to confirm if the issue is still present in
> > 10.1.12.
>
> Answers to the above would still be helpful.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

RE: Re: Possible AbstractProtocol.waitingProcessors leak in Tomcat 9.0.75

[Jakub Remenec]

Hi,

I've experienced the same issue as described on Apache Tomcat 10.1.13.
After downgrading to 10.1.5 it started to work correctly. I also inspected
the heapdump of the application with memory problems and found out that
there were many org.apache.tomcat.websocket.WsSession present in
OUTPUT_CLOSED state. When I tried locally, I found out, that when I open
few websocket connections from Chrome and then go to Offline mode the
WsSessions remain in OUTPUT_CLOSED state. New connections afterwards have
state OPEN. In heapdump from production I saw around 4600 WsSessions but
only 40 were open. The rest was in the output closed state.
WsSessions are accessible through org.apache.coyote.AbstractProtocol ->
waitingProcessors Set. In heapdump and it was clearly visible that 49% of
heap was taken by the waitingProcessors set. When tomcat was downgraded to
10.1.5, I saw that WsSessions got cleared after going to offline mode.

Additional info - I've set the session timeout to 10minutes. The app uses
Java 17 with Spring Boot 3.1.x stack. It does not use any external STOMP
broker relay.

Regards,
Jakub.

Looking for examples...

[Mcalexander, Jon J.]

Good evening,

I am trying to find decent examples for Proerty Replacements in 
Catalina.properties. I have an instance that is giving me the following Warning 
and it bugs me:

Dec 05, 2023 5:48:51 PM org.apache.tomcat.util.digester.Digester 
replaceSystemProperties
WARNING: System property [common.loader] failed to update and remains 
["${catalina.base}/lib","${catalina.base}/lib/*.jar","${catalina.home}/lib","${catalina.home}/lib/*.jar"]


I've looked at the docs, and all I'm really finding is the following:

Property replacements
Property
Description
org.apache.tomcat.util.digester. PROPERTY_SOURCE

Set this to a comma separated list of fully qualified name of classes that 
implement org.apache.tomcat.util.IntrospectionUtils.PropertySource. Required to 
have a public constructor with no arguments.

Use this to add a property source, that will be invoked when 
${parameter:-default-value} denoted parameters (with optional default values) 
are found in the XML files that Tomcat parses.

Property replacement from the specified property source on the JVM system 
properties can also be done using the REPLACE_SYSTEM_PROPERTIES system property.

org.apache.tomcat.util.digester.ServiceBindingPropertySource can be used to 
replace parameters from any Kubernetes service bindings that follows the 
servicebinding.io spec

org.apache.tomcat.util.digester.EnvironmentPropertySource can be used to 
replace parameters from the process' environment variables, e.g. injected 
ConfigMaps or Secret objects in container based systems like OpenShift or 
Kubernetes.

org.apache.tomcat.util.digester.SystemPropertySource does replacement with 
system properties. It is always enabled, but can also be specified as part of 
the property value.
org.apache.tomcat.util.digester. REPLACE_SYSTEM_PROPERTIES

Set this boolean system property to true to cause property replacement from the 
digester property source on the JVM system properties.

Any good examples of how to use it properly would be great.

Thank you for your time.

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.

Re: [EXTERNAL EMAIL] Re: JSP include not working

[Niranjan Rao]

I'm one step further. Cloned the tomcat repo, switched to appropriate 
tag/branch and looked at the code.  Comments indicated need to have full 
path and my paths did not start with "/". Added the "/" on couple of 
pages and no more include error.



As of now, even though I don't get the include error, contents of the 
include file are not getting included. Trying to figure out why. At 
least now status code has changed from 500 to 200. Not working yet but 
progress never the less.



Regards,


Niranjan

On 12/5/23 11:09, Rob Sargent wrote:
On 12/5/23 12: 01, Niranjan Rao wrote: > Greetings, > > I'm missing 
something obvious and hoping that someone can point my > nose in right 
direction. > > > We have a application WAR file that works fine on 
tomcat 7. 0. 78. 

ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
ZjQcmQRYFpfptBannerEnd
On 12/5/23 12:01, Niranjan Rao wrote:
> Greetings,
>
> I'm missing something obvious and hoping that someone can point my 
> nose in right direction.

>
>
> We have a application WAR file that works fine on tomcat 7.0.78. We're 
> trying to migrate this application to 9.0.82. When trying to hit the 
> pages, I'm getting error JSP file not found at WEB-INF/> location>. The app works fine in tomcat version 7. We have other 
> applications which use similar pattern and are working fine on tomcat 9.

>
>
> I've checked and checked and confirmed file exists at desired 
> location. Only significant difference between migrated apps and this 
> app I noticed was web.xml points version 4.0 where it works and 3.0 
> where it does not. Changing version to 4.0 in broken app did not help.

>
>
> The applications use Spring and serve JSP through spring controllers. 
> Problematic application seems to be initializing properly based on 
> application logs, just that it's not able to include JSP. Any log 
> files do not show any exception trace, but access log file logs the 
> error with 500 status code.

>

Case sensitive file names, perhaps.

>
> Am I missing something obvious?
>
> Regards,
>
> Niranjan
>


-
To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
For additional commands, e-mail:users-h...@tomcat.apache.org


--
*Niranjan Rao | Sr.Architect
*/he/his/
*Paymentus
**Paymentus Corporation***/The Real-Time Bill Payment Company^TM //^
/
860 Hillview Court Suite 220 Milpitas CA 95035
www.paymentus.com 

RE: Thread Pool Question

[John.E.Gregg]

You have to refer to it in your connector:

https://tomcat.apache.org/tomcat-10.0-doc/config/http.html


> -Original Message-
> From: William Crowell 
> Sent: Tuesday, December 5, 2023 1:39 PM
> To: Tomcat Users List 
> Subject: Re: Thread Pool Question
> 
> I should clarify the ask here...
> 
> I have some long running JDBC queries against Oracle, and I do not want to
> tie up Tomcat's web thread pool with them.  I would only have between 1-10
> threads in this pool.
> 
> Regards,
> 
> William Crowell
> 
> 
> 
> 
> This e-mail may contain information that is privileged or confidential. If you
> are not the intended recipient, please delete the e-mail and any attachments
> and notify us immediately.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Thread Pool Question

[William Crowell]

I should clarify the ask here…

I have some long running JDBC queries against Oracle, and I do not want to tie 
up Tomcat’s web thread pool with them.  I would only have between 1-10 threads 
in this pool.

Regards,

William Crowell




This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Thread Pool Question

[William Crowell]

If I create a separate thread pool in Tomcat’s server.xml like this:



Then how do I get a thread to assign any work to it?

Regards,

William Crowell


This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

Re: [EXTERNAL EMAIL] Re: JSP include not working

[Niranjan Rao]

Thank you Rob, but doubtful. We're a Ubuntu shop and always mindful of 
case sensitivity.



On 12/5/23 11:09, Rob Sargent wrote:
On 12/5/23 12: 01, Niranjan Rao wrote: > Greetings, > > I'm missing 
something obvious and hoping that someone can point my > nose in right 
direction. > > > We have a application WAR file that works fine on 
tomcat 7. 0. 78. 

ZjQcmQRYFpfptBannerStart
This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
ZjQcmQRYFpfptBannerEnd
On 12/5/23 12:01, Niranjan Rao wrote:
> Greetings,
>
> I'm missing something obvious and hoping that someone can point my 
> nose in right direction.

>
>
> We have a application WAR file that works fine on tomcat 7.0.78. We're 
> trying to migrate this application to 9.0.82. When trying to hit the 
> pages, I'm getting error JSP file not found at WEB-INF/> location>. The app works fine in tomcat version 7. We have other 
> applications which use similar pattern and are working fine on tomcat 9.

>
>
> I've checked and checked and confirmed file exists at desired 
> location. Only significant difference between migrated apps and this 
> app I noticed was web.xml points version 4.0 where it works and 3.0 
> where it does not. Changing version to 4.0 in broken app did not help.

>
>
> The applications use Spring and serve JSP through spring controllers. 
> Problematic application seems to be initializing properly based on 
> application logs, just that it's not able to include JSP. Any log 
> files do not show any exception trace, but access log file logs the 
> error with 500 status code.

>

Case sensitive file names, perhaps.

>
> Am I missing something obvious?
>
> Regards,
>
> Niranjan
>


-
To unsubscribe, e-mail:users-unsubscr...@tomcat.apache.org
For additional commands, e-mail:users-h...@tomcat.apache.org


--
*Niranjan Rao | Sr.Architect
*/he/his/
*Paymentus
**Paymentus Corporation***/The Real-Time Bill Payment Company^TM //^
/
860 Hillview Court Suite 220 Milpitas CA 95035
www.paymentus.com 

Re: JSP include not working

[Rob Sargent]

On 12/5/23 12:01, Niranjan Rao wrote:

Greetings,

I'm missing something obvious and hoping that someone can point my 
nose in right direction.



We have a application WAR file that works fine on tomcat 7.0.78. We're 
trying to migrate this application to 9.0.82. When trying to hit the 
pages, I'm getting error JSP file not found at WEB-INF/location>. The app works fine in tomcat version 7. We have other 
applications which use similar pattern and are working fine on tomcat 9.



I've checked and checked and confirmed file exists at desired 
location. Only significant difference between migrated apps and this 
app I noticed was web.xml points version 4.0 where it works and 3.0 
where it does not. Changing version to 4.0 in broken app did not help.



The applications use Spring and serve JSP through spring controllers. 
Problematic application seems to be initializing properly based on 
application logs, just that it's not able to include JSP. Any log 
files do not show any exception trace, but access log file logs the 
error with 500 status code.




Case sensitive file names, perhaps.



Am I missing something obvious?

Regards,

Niranjan




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

JSP include not working

[Niranjan Rao]

Greetings,

I'm missing something obvious and hoping that someone can point my nose 
in right direction.



We have a application WAR file that works fine on tomcat 7.0.78. We're 
trying to migrate this application to 9.0.82. When trying to hit the 
pages, I'm getting error JSP file not found at WEB-INF/. 
The app works fine in tomcat version 7. We have other applications which 
use similar pattern and are working fine on tomcat 9.



I've checked and checked and confirmed file exists at desired location. 
Only significant difference between migrated apps and this app I noticed 
was web.xml points version 4.0 where it works and 3.0 where it does not. 
Changing version to 4.0 in broken app did not help.



The applications use Spring and serve JSP through spring controllers. 
Problematic application seems to be initializing properly based on 
application logs, just that it's not able to include JSP. Any log files 
do not show any exception trace, but access log file logs the error with 
500 status code.



Am I missing something obvious?

Regards,

Niranjan

--
*Niranjan Rao | Sr.Architect
*/he/his/
*Paymentus
**Paymentus Corporation***/The Real-Time Bill Payment Company^TM //^
/
860 Hillview Court Suite 220 Milpitas CA 95035
www.paymentus.com 

RE: Tomcat Build Issue

[Mcalexander, Jon J.]

Now that is a definite possibility

Dream * Excel * Explore * Inspire
Jon McAlexander
Senior Infrastructure Engineer
Asst. Vice President
He/His

Middleware Product Engineering
Enterprise CIO | EAS | Middleware | Infrastructure Solutions

8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508

jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are 
not the addressee or authorized to receive this for the addressee, you must not 
use, copy, disclose, or take any action based on this message or any 
information herein. If you have received this message in error, please advise 
the sender immediately by reply e-mail and delete this message. Thank you for 
your cooperation.

> -Original Message-
> From: Noelette Stout 
> Sent: Tuesday, December 5, 2023 10:36 AM
> To: Tomcat Users List 
> Subject: Re: Tomcat Build Issue
> 
> I see this kind of thing in my tomcat logs when the security team runs their
> Nessus scanner against everything.
> 
> On Tue, Dec 5, 2023 at 2:45 AM Burle, Saicharan
>  wrote:
> 
> > Hi All,
> >
> > I am trying to build a tomcat instance in a net new server and getting
> > the below error while starting. Although instance has come up but I am
> > unable to debug the below error. Can someone please assist in this regard?
> >
> > [https-jsse-nio-11511-exec-4]
> > org.apache.coyote.http11.Http11Processor.service Error parsing HTTP
> > request header
> > Note: further occurrences of HTTP request parsing errors will be
> > logged at DEBUG level.
> > java.lang.IllegalArgumentException: Invalid character found in
> > method name
> >
> [000x020x010x01`+0x020x010x030x040x08cn=Admin0x800x1c0x7d1a9b5edae
> 2b4de4ff533738b...].
> > HTTP method names must be tokens
> > at
> >
> org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11Input
> Buffer.java:407)
> > at
> >
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:263
> )
> > at
> >
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.ja
> va:63)
> > at
> >
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractPr
> otocol.java:928)
> > at org.apache.tomcat.util.net
> > .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
> > at org.apache.tomcat.util.net
> > .SocketProcessorBase.run(SocketProcessorBase.java:52)
> > at
> >
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolE
> xecutor.java:1191)
> > at
> >
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPool
> Executor.java:659)
> > at
> >
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThr
> ead.java:61)
> > at java.lang.Thread.run(Thread.java:750)
> >
> >
> > Apache Tomcat Version : 9.0.82
> > JDK: jdk1.8.0_391/ jdk-11.0.21(Same issue with this version as well)
> >
> > Regards,
> > Saicharan Burle
> > Lead Infrastructure Engineer
> > Chief Technology Office | Foundational Hosting Platform | Middleware
> > Product Engineering G=ETI DPS Middleware Prod Eng
> >
> > Block B1A Divyasree Orion | Hyderabad - 500081 | India MAC O2806-010 |
> > Cell +91 9966964321
> >
> > MPE Service Desk:
> > https://urldefense.com/v3/__https://devops-servicedesk.wellsfargo.net/
> >
> servicedesk/customer/portal/14/group/385?groupId=385__;!!F9svGWnIaVP
> GS
> > wU!vjsFtRAPNxJFFWH4Z4x5gfKqlwgErDYwGG3YJC0CzHyyHpk-
> KHtdaWjFVzlppQYoL4C
> > h5T7AMVUZCviGAJWAXB4sMA$
> > Enterprise Middleware Services:
> > https://wim-
> wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE
> > le=Middleware+Product+Engineering+Team+Home
> > Normal Working hours: 4:00 AM to 1:00 PM EST Upcoming PTO:
> > Be Green, Leave it on Screen!!!
> > [
> > https://urldefense.com/v3/__http://f.connect.wellsfargoemail.com/i/38/
> > 2082566297/EML-224071_4726581_6304406_logo-stagecoach-
> eml_511x80.png__
> >
> ;!!F9svGWnIaVPGSwU!vjsFtRAPNxJFFWH4Z4x5gfKqlwgErDYwGG3YJC0CzHyy
> Hpk-KHt
> > daWjFVzlppQYoL4Ch5T7AMVUZCviGAJUVQ99VhQ$
> > ]
> >
> >
> 
> --
> Noelette Stout
> ITS Enterprise Applications - Senior Application Administrator Idaho State
> University
> E-mail: stounoel "at" isu "dot" edu
> Desk: 208-282-2554

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat Build Issue

[Noelette Stout]

I see this kind of thing in my tomcat logs when the security team runs
their Nessus scanner against everything.

On Tue, Dec 5, 2023 at 2:45 AM Burle, Saicharan
 wrote:

> Hi All,
>
> I am trying to build a tomcat instance in a net new server and getting the
> below error while starting. Although instance has come up but I am unable
> to debug the below error. Can someone please assist in this regard?
>
> [https-jsse-nio-11511-exec-4]
> org.apache.coyote.http11.Http11Processor.service Error parsing HTTP request
> header
> Note: further occurrences of HTTP request parsing errors will be logged at
> DEBUG level.
> java.lang.IllegalArgumentException: Invalid character found in
> method name
> [000x020x010x01`+0x020x010x030x040x08cn=Admin0x800x1c0x7d1a9b5edae2b4de4ff533738b...].
> HTTP method names must be tokens
> at
> org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:407)
> at
> org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:263)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
> at org.apache.tomcat.util.net
> .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
> at org.apache.tomcat.util.net
> .SocketProcessorBase.run(SocketProcessorBase.java:52)
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
> at
> org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
> at
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:750)
>
>
> Apache Tomcat Version : 9.0.82
> JDK: jdk1.8.0_391/ jdk-11.0.21(Same issue with this version as well)
>
> Regards,
> Saicharan Burle
> Lead Infrastructure Engineer
> Chief Technology Office | Foundational Hosting Platform | Middleware
> Product Engineering
> G=ETI DPS Middleware Prod Eng
>
> Block B1A Divyasree Orion | Hyderabad - 500081 | India
> MAC O2806-010 | Cell +91 9966964321
>
> MPE Service Desk:
> https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385
> Enterprise Middleware Services:
> https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE=Middleware+Product+Engineering+Team+Home
> Normal Working hours: 4:00 AM to 1:00 PM EST
> Upcoming PTO:
> Be Green, Leave it on Screen!!!
> [
> http://f.connect.wellsfargoemail.com/i/38/2082566297/EML-224071_4726581_6304406_logo-stagecoach-eml_511x80.png
> ]
>
>

-- 
Noelette Stout
ITS Enterprise Applications - Senior Application Administrator
Idaho State University
E-mail: stounoel "at" isu "dot" edu
Desk: 208-282-2554

Re: Tomcat Build Issue

[Mark Thomas]

On 05/12/2023 15:15, Burle, Saicharan wrote:

Hi Mark/Chris,

We are getting this error without even deploying any application.


Then start looking at your network to see what is sending this invalid 
data to Tomcat.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat Build Issue

[Burle, Saicharan]

Hi Mark/Chris,

We are getting this error without even deploying any application.

Regards,
Saicharan Burle
Lead Infrastructure Engineer
Chief Technology Office | Foundational Hosting Platform | Middleware Product 
Engineering
G=ETI DPS Middleware Prod Eng

Block B1A Divyasree Orion | Hyderabad - 500081 | India
MAC O2806-010 | Cell +91 9966964321

MPE Service Desk: 
https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385
Enterprise Middleware Services: 
https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE=Middleware+Product+Engineering+Team+Home
Normal Working hours: 4:00 AM to 1:00 PM EST
Upcoming PTO:
Be Green, Leave it on Screen!!!





-Original Message-
From: Christopher Schultz  
Sent: Tuesday, December 5, 2023 7:22 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat Build Issue

Mark,

On 12/5/23 07:15, Mark Thomas wrote:
> On 05/12/2023 09:45, Burle, Saicharan wrote:
>> Hi All,
>>
>> I am trying to build a tomcat instance in a net new server and 
>> getting the below error while starting. Although instance has come up 
>> but I am unable to debug the below error. Can someone please assist 
>> in this regard?
>>
>> [https-jsse-nio-11511-exec-4]
>> org.apache.coyote.http11.Http11Processor.service Error parsing HTTP 
>> request header
>> Note: further occurrences of HTTP request parsing errors will be 
>> logged at DEBUG level.
>>  java.lang.IllegalArgumentException: Invalid character found 
>> in method name 
>> [000x020x010x01`+0x020x010x030x040x08cn=Admin0x800x1c0x7d1a9b5edae2b4
>> de4ff533738b...]. HTTP method names must be tokens
> 
> The client isn't sending a valid HTTP request. What it is sending 
> isn't even close.
> 
> The presence of "cn=Admin" in what is sent makes me wonder if the 
> client is trying to send HTTPS to an HTTP connection.

Probably not, looking at the thread name:

> [https-jsse-nio-11511-exec-4]
> org.apache.coyote.http11.Http11Processor.service Error parsing HTTP 
> request header
Maybe doubly-wrapped HTTPS? Like connecting making an HTTPS connection to an 
stunnel instance in client-mode?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.

[Bodavula, Sai Vamsi Mohan Krishna (TR Technology)]

As you suggested.,
I have added my annotations in Catalina_opts ., and I am not able to see any 
Java Process with the JMX, neither the port is being listened.
It came back to previous stage where Annotations are not on the JVM .,
Came back to previous stage, where connection refused error.,  as follows.

Could not initialize instance: tomcat-10.75.12.136-49151:
java.util.concurrent.ExecutionException: java.io.IOException: Failed to 
retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root 
exception is java.rmi.ConnectException: Connection refused to host: 
10.75.12.136; nested exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.datadog.jmxfetch.App.processRecoveryResults(App.java:953)
at org.datadog.jmxfetch.App$5.invoke(App.java:927)
at 
org.datadog.jmxfetch.tasks.TaskProcessor.processTasks(TaskProcessor.java:63)
at org.datadog.jmxfetch.App.init(App.java:917)
at org.datadog.jmxfetch.App.start(App.java:413)
at org.datadog.jmxfetch.App.run(App.java:173)
at org.datadog.jmxfetch.JmxFetch.main(JmxFetch.java:57)
at org.datadog.jmxfetch.App.main(App.java:91)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: 
javax.naming.ServiceUnavailableException [Root exception is 
java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested 
exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at 
java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
at 
java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at org.datadog.jmxfetch.Connection.createConnection(Connection.java:63)
at 
org.datadog.jmxfetch.RemoteConnection.(RemoteConnection.java:101)
at 
org.datadog.jmxfetch.ConnectionFactory.createConnection(ConnectionFactory.java:38)
at org.datadog.jmxfetch.Instance.getConnection(Instance.java:405)
at org.datadog.jmxfetch.Instance.init(Instance.java:418)
at 
org.datadog.jmxfetch.InstanceInitializingTask.call(InstanceInitializingTask.java:15)
at 
org.datadog.jmxfetch.InstanceInitializingTask.call(InstanceInitializingTask.java:3)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.naming.ServiceUnavailableException [Root exception is 
java.rmi.ConnectException: Connection refused to host: 10.75.12.136; nested 
exception is:
java.net.ConnectException: Connection refused (Connection refused)]
at 
jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)
at 
java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)
at 
java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
at 
java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
at 
java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
at 
java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
... 12 more
Caused by: java.rmi.ConnectException: Connection refused to host: 10.75.12.136; 
nested exception is:
java.net.ConnectException: Connection refused (Connection refused)
at 
java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
at 
java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209)
at 
java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at 
java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at 
jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)
... 17 more

Here we were not able to find any process , to confirm that atleast port is 
being listened to.
Thanks & Regards,
--

SAI VAMSI .B
Senior DevOps Engineer


From: Christopher Schultz 
Sent: Tuesday, December 5, 2023 19:19
To: users@tomcat.apache.org 
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.

Sai Vamsa,

On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
> Thanks for the response..
>
> as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead 
> of Java_Opts and Java_tool_options.,
>
> 

Re: Weird CSRF prevention behavior

[Christopher Schultz]

All,

Ping. Any ideas?

-chris

On 12/1/23 15:26, Christopher Schultz wrote:

All,

I'm experimenting with the CsrfPreventionFilter in Tomcat 8.5. I've had 
issues with it in the past so I haven't actually enabled it in any of my 
applications, but I'm sufficiently motivated at this point to get it done.


My "application" is actually split up into two applications, each 
running in a separate JVM but in the same URL space. I have a 
reverse-proxy that figures this all out and it's been working for years. 
I don't see why this wouldn't work.


I have enabled CSRF prevention in Application A which is the "primary 
application" and the secondary application (Application B) is capable of 
mimicking/proxying the csrf token back to Application A.


Application B has a feature where we present a web form to the user. 
It's fairly simple (paraphrasing):



   


When I submit this form, I get an HTTP 403 response. Our application 
doesn't send 403 responses. When I remove the CsrfPreventionFilter from 
the configuration) by commenting-out the  in 
WEB-INF/web.xml, I do not get the 403 response and the form submission 
is successful. I'm sure that the CSRF token is *NOT* in the POST 
request: the browser shows me what is sent and it's not there. I have 
hacked the form and added the token, submitted it, and it /works/.


But this is an HTTP POST and should be ignored by the filter.

So I figure I'll enable logging and see what's happening. There isn't 
much logging in CsrfPreventionFilter, so I add this line to the 
beginning of the skipNonceCheck method:


     log.trace("skipNonceCheck(" + request.getMethod() + " " + 
request.getRequestURI() + ")");


I build-from-source and launch my custom-build Tomcat with my 
application in it. No logging. Oh, right... logging.properties. So I add 
this to my conf/logging.properties file:


org.apache.catalina.filters.CsrfPreventionFilter.level = FINEST

To be sure there's no funny business, I use "catalina.sh run" and wait 
for the console log to settle down. I make a few requests. No logs. Hmm. 
Oh, the ConsoleAppender is set to FINE and not FINEST.


java.util.logging.ConsoleHandler.level = FINEST

Done. CTRL-C, catalina.sh run. Make some requests.

Nothing. Okay maybe the Filter is just ignoring these for some 
reason. So I add this line to the beginning of doFilter, before anything 
else happens:


     log.trace("doFilter(" + request + ")");

Re-build. CTRL-C. catalina.sh run. Make some requests.

Nothing.

The Filter is absolutely running. If I reload a page, the csrf tokens on 
all the links are changing. What's going on?


You'd think a Tomcat committer could figure out how to make logging work.

My application is using log4j2, but that library is only used by the 
application and the JAR file is in WEB-INF/lib/. I wouldn't expect that 
it would interfere with server-level logging.


Any ideas? About EITHER issue? If anyone can help with logging, maybe I 
can figure out what's happening in the Filter. If you have any 
suggestions about the Filter, I'm al ears. HTTP POST should not be 
prohibited unless I'm reading both the code and the CSRF specs incorrectly.


Thanks,
-chris


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat Build Issue

[Christopher Schultz]

Mark,

On 12/5/23 07:15, Mark Thomas wrote:

On 05/12/2023 09:45, Burle, Saicharan wrote:

Hi All,

I am trying to build a tomcat instance in a net new server and getting 
the below error while starting. Although instance has come up but I am 
unable to debug the below error. Can someone please assist in this 
regard?


[https-jsse-nio-11511-exec-4] 
org.apache.coyote.http11.Http11Processor.service Error parsing HTTP 
request header
Note: further occurrences of HTTP request parsing errors will be 
logged at DEBUG level.
 java.lang.IllegalArgumentException: Invalid character found 
in method name 
[000x020x010x01`+0x020x010x030x040x08cn=Admin0x800x1c0x7d1a9b5edae2b4de4ff533738b...]. HTTP method names must be tokens


The client isn't sending a valid HTTP request. What it is sending isn't 
even close.


The presence of "cn=Admin" in what is sent makes me wonder if the client 
is trying to send HTTPS to an HTTP connection.


Probably not, looking at the thread name:


[https-jsse-nio-11511-exec-4]
org.apache.coyote.http11.Http11Processor.service Error parsing HTTP
request header
Maybe doubly-wrapped HTTPS? Like connecting making an HTTPS connection 
to an stunnel instance in client-mode?


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.

[Christopher Schultz]

Sai Vamsa,

On 12/5/23 01:25, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:

Thanks for the response..

as you suggested., I tried adding my annotations in CATALINA_OPTS., Instead of 
Java_Opts and Java_tool_options.,

But its not even working out., Java process is not getting created., and port 
is not bound to any process.
Let me know if I am missing anything.


Okay. Can you please summarize the current situation? I think you've 
made a few changes, etc. so it would help to know the current state.


Stop Tomcat and any other related services, delete all appropriate log 
files, etc. and start Tomcat (or attempt to). Then post any output that 
is relevant, and if you get things like "address in use" then please 
post the details about which process id owns that port and the full 
command-line that launched that process.


Thanks,
-chris



From: Christopher Schultz 
Sent: Monday, December 4, 2023 23:22
To: users@tomcat.apache.org 
Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.

Sai Vamsi,

On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:

Firstly thanks for adding a point me in asking me to check, if the annotations 
are reflecting in the Java process, which opened me a door to add the concerned 
annotations in correct place., by adding in java_tool_options in stead of 
Java_opts.


You will probably want to use CATALINA_OPTS instead of any of the other
ones. JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat.
You certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS
any time it invokes a JVM. For example, running bind/digest.sh doesn't
need to have the JMX subsystem starting-up and trying to grab a port.

JAVA_TOOL_OPTS is an environment variable used by JVM-launching
processes, like jps for example...


yeah they are reflecting and creating a Java Process.
but I am facing a problem here., while i am checking JSP, Thats :  the port i 
am using here to enable JMX is been opening a process with the mentioned port 
and at the same time shows port is being used.:

root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M -XX:+UseStringDeduplication 
-XX:+UseContainerSupport -Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.local.only=false 
-Dcom.sun.management.jmxremote.port=49151 
-Djava.rmi.server.hostname=tomcat.default.svc.cluster.local 
-javaagent:/datadog-lib/dd-java-agent.jar 
-XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh 
-XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port 
already in use: 49151; nested exception is:
  java.net.BindException: Address already in use (Bind failed)


Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind
to your port which is aready bound by your Tomcat process.

The solution is to use only CATALINA_OPTS to set these options.

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat Build Issue

[Mark Thomas]

On 05/12/2023 09:45, Burle, Saicharan wrote:

Hi All,

I am trying to build a tomcat instance in a net new server and getting the 
below error while starting. Although instance has come up but I am unable to 
debug the below error. Can someone please assist in this regard?

[https-jsse-nio-11511-exec-4] org.apache.coyote.http11.Http11Processor.service 
Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at 
DEBUG level.
 java.lang.IllegalArgumentException: Invalid character found in method 
name 
[000x020x010x01`+0x020x010x030x040x08cn=Admin0x800x1c0x7d1a9b5edae2b4de4ff533738b...].
 HTTP method names must be tokens


The client isn't sending a valid HTTP request. What it is sending isn't 
even close.


The presence of "cn=Admin" in what is sent makes me wonder if the client 
is trying to send HTTPS to an HTTP connection.


Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.

[Bodavula, Sai Vamsi Mohan Krishna (TR Technology)]

Hey Thomas,
Thanks for the response.

Yeah., I am trying to add Heap size parameters to the JVM using the 
annotations.,

DO you smell anything wrong in this!?


Thanks & Regards,
--

SAI VAMSI .B
Senior DevOps Engineer


From: Thomas Meyer 
Sent: Tuesday, December 5, 2023 00:03
To: Tomcat Users List ; Christopher Schultz 
; users@tomcat.apache.org 

Subject: Re: [EXT] Re: Datadog _ JMX Integration facing connection issues.

Hi,

Also newer jvm do have XX:+UseContainerSupport set as default, also when using 
XX:+UseContainerSupport setting -Xms and -Xmx do not really make sense at all, 
you either want the JVM to deduce the max heap size from the memory cgroup or 
not

Mfg
Thomas

Am 4. Dezember 2023 18:52:13 MEZ schrieb Christopher Schultz 
:
>Sai Vamsi,
>
>On 12/4/23 03:53, Bodavula, Sai Vamsi Mohan Krishna (TR Technology) wrote:
>> Firstly thanks for adding a point me in asking me to check, if the 
>> annotations are reflecting in the Java process, which opened me a door to 
>> add the concerned annotations in correct place., by adding in 
>> java_tool_options in stead of Java_opts.
>
>You will probably want to use CATALINA_OPTS instead of any of the other ones. 
>JAVA_TOOL_OPTS isn't an environment variable regognized by Tomcat. You 
>certainly don't want to use JAVA_OPTS, because Tomcat uses JAVA_OPTS any time 
>it invokes a JVM. For example, running bind/digest.sh doesn't need to have the 
>JMX subsystem starting-up and trying to grab a port.
>
>JAVA_TOOL_OPTS is an environment variable used by JVM-launching processes, 
>like jps for example...
>
>> yeah they are reflecting and creating a Java Process.
>> but I am facing a problem here., while i am checking JSP, Thats :  the port 
>> i am using here to enable JMX is been opening a process with the mentioned 
>> port and at the same time shows port is being used.:
>>
>> root@lab1workflow4scalsvc2zus1-deployment-577d856494-ftb22:/# jps
>> Picked up JAVA_TOOL_OPTIONS: -Xms2048M -Xmx10240M 
>> -XX:+UseStringDeduplication -XX:+UseContainerSupport 
>> -Dcom.sun.management.jmxremote 
>> -Dcom.sun.management.jmxremote.authenticate=false 
>> -Dcom.sun.management.jmxremote.ssl=false 
>> -Dcom.sun.management.jmxremote.local.only=false 
>> -Dcom.sun.management.jmxremote.port=49151 
>> -Djava.rmi.server.hostname=tomcat.default.svc.cluster.local 
>> -javaagent:/datadog-lib/dd-java-agent.jar 
>> -XX:OnError=/datadog-lib/continuousprofiler/tmp/dd_crash_uploader.sh 
>> -XX:ErrorFile=/datadog-lib/continuousprofiler/tmp/hs_err_pid_%p.log
>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port 
>> already in use: 49151; nested exception is:
>>  java.net.BindException: Address already in use (Bind failed)
>
>Yes: you have set JAVA_TOOL_OPTS and then run jps. jps is trying to bind to 
>your port which is aready bound by your Tomcat process.
>
>The solution is to use only CATALINA_OPTS to set these options.
>
>-chris
>
>-
>To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>For additional commands, e-mail: users-h...@tomcat.apache.org
>

--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

Tomcat Build Issue

[Burle, Saicharan]

Hi All,

I am trying to build a tomcat instance in a net new server and getting the 
below error while starting. Although instance has come up but I am unable to 
debug the below error. Can someone please assist in this regard?

[https-jsse-nio-11511-exec-4] org.apache.coyote.http11.Http11Processor.service 
Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at 
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method 
name 
[000x020x010x01`+0x020x010x030x040x08cn=Admin0x800x1c0x7d1a9b5edae2b4de4ff533738b...].
 HTTP method names must be tokens
at 
org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:407)
at 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:263)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:750)


Apache Tomcat Version : 9.0.82
JDK: jdk1.8.0_391/ jdk-11.0.21(Same issue with this version as well)

Regards,
Saicharan Burle
Lead Infrastructure Engineer
Chief Technology Office | Foundational Hosting Platform | Middleware Product 
Engineering
G=ETI DPS Middleware Prod Eng

Block B1A Divyasree Orion | Hyderabad - 500081 | India
MAC O2806-010 | Cell +91 9966964321

MPE Service Desk: 
https://devops-servicedesk.wellsfargo.net/servicedesk/customer/portal/14/group/385?groupId=385
Enterprise Middleware Services: 
https://wim-wiki.wellsfargo.com/pages/viewpage.action?spaceKey=MPE=Middleware+Product+Engineering+Team+Home
Normal Working hours: 4:00 AM to 1:00 PM EST
Upcoming PTO:
Be Green, Leave it on Screen!!!
[http://f.connect.wellsfargoemail.com/i/38/2082566297/EML-224071_4726581_6304406_logo-stagecoach-eml_511x80.png]

Re: setenv.sh tomcat8 changelog

[Giuseppe Sacco]

Hello Christoph,

Il giorno lun, 04/12/2023 alle 16.10 +0100, Christoph Kukulies ha scritto:
[...]
> Where do I find "setenv.sh" in my tomcat9 (Ubuntu) distribution? 

I think all Debian derivatives use /etc/default/tomcat8 file for setting
those variables.

Bye,
Giuseppe

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org