Should allowHostHeaderMismatch be case sensitive

[David Cleary]

Just want to check if this is by design. The above property default was changed 
to better secure the default configuration. We started having some tests fail 
due to this.

In our scenario ( as shown below ), the Host header value in the HTTP request 
is case-sensitive difference compared to the Request Line, and it's crucial 
that Tomcat, our web server, does not block or reject requests based on 
variations in the letter case within this header.

Request Line: GET http://HZN-OE-A079:8080 HTTP/1.1
Host header: hzn-oe-a079:8080

Just want to confirm that this property, now with a default of false, is 
supposed to reject requests based on the case of the host name.

Thanks
David Cleary
Progress

Disabling cipher warning

[David Cleary]

2023-08-29T15:31:57.840-04:00 WARN  [main] o.a.t.u.n.j.JSSEUtil - Some of the 
specified [ciphers] are not supported by the SSL engine and have been skipped: 
[Dozens of OpenSSL ciphers]

We use OpenSSL and moving to Tomcat 10.1.13 has caused an overload of useless 
information to appear when starting Tomcat. Any way to turn this off? Looking 
at the source code, it doesn't look like it.

boolean warnOnSkip = !sslHostConfig.getCiphers().equals(SSLHostConfig.DEFAU.

Thanks
Dave

RE: Testing Tomcat pre-releases

[David Cleary]

> -Original Message-
> From: Christopher Schultz 
> Sent: Thursday, April 28, 2022 12:16 PM
> To: Tomcat Users List 
> Subject: Re: Testing Tomcat pre-releases
> 
> David,
> 
> (Replying to the Tomcat users@ list)
> 
> On 4/28/22 08:45, David Cleary wrote:
> > Hi Chris. We have spoken over the years at various Apachecons. In one
> > of your presentations, you talked about smoke testing Tomcat pre-releases.
> > We just got bitten by a regression in 9.0.62, and the team that is
> > responsible for updating it is interested in the details on doing this.
> > Can you give me details on where we would pick up pre-release builds
> > and what mailing list we should monitor and report any issues.
> 
> Sure.
> 
> Briefly, the release process goes like this[1]:
> 
> 1. Announce intent to do a release on dev@ mailing list; call for any 
> last-minute
> commits or conversations. This often doesn't happen because we have a
> release-cadence that follows a rough schedule of prep-and-release around the
> beginning of each month.
> 
> 2. Tag the release + prepare a release candidate build. This is a formal 
> process
> which results in a vote.
> 
> 3. Declare a vote on Tomcat x.y.z for a [VOTE] thread posted to the dev@
> mailing list. Here is your opportunity to give feedback on the release.
> (See below). Information about where to get the release candidates is 
> available
> in that [VOTE] message.
> 
> 4. Assuming the [VOTE] passes, the release is promoted from "candidate"
> to "official release", distributed to mirrors, and announced.
> 
> So, how can you participate in #3 above?
> 
> Well, the release candidate includes all the binary artifacts from a regular
> release, so you can use it just as you would usually use a "real" release. 
> You can
> also build it from source as you always could, etc.
> 
> The "Getting Started Hacking Tomcat" presentation[2] contains some
> information about how to build from source, run the unit-tests, etc. if you'd 
> like
> some guidance.
> 
> If you find a bug and are able to contribute a test-case for us to include in 
> our
> test process, that would be great: it will prevent the bug from coming-back 
> as a
> regression in the future.
> 
> Simply reply to the [VOTE] thread with any concerns you may have, or, if
> everything is great, we'd love to have your "+1 to release" vote as well.
> Technically speaking, non-PMC-members don't have a binding vote, but I have
> never seen a vote move-forward in spite of legitimate negative community
> feedback. If something is wrong with the release, we'll cancel it, the fix 
> issue,
> and repeat the process with a new release candidate (and version number).
> 
> Let us know if you have any questions.
> 
> -chris
> 

Thanks Chris. Is this where we can expect the pre-releases to show up?

https://repository.apache.org/content/groups/staging/org/apache/tomcat/tomcat/

One issue I've run into is that our Gradle builds use the Windows 32 and 64 bit 
zip files since we ship with the commons-daemon executable. Don't really know 
where or Artifactory gets those from. Are those available in staging somewhere 
before release?

Thanks
Dave

RE: Alternatives for AJP

[David Cleary]

https://nvd.nist.gov/vuln/detail/CVE-2020-1938

-Original Message-
From: stephane passignat  
Sent: Sunday, April 12, 2020 4:00 AM
To: Tomcat Users List 
Subject: Re: Alternatives for AJP

Hi
Which vulnerability are you mentioning ?
Thanks

⁣Envoyé par BlueMail ​

Le 10 avr. 2020 à 17:45, à 17:45, David Cleary  a écrit:
>Some of our customers are currently using the AJP connector. Given the 
>vulnerability and breaking change to address it, now may be a good time 
>to prompt them look at alternatives. One requirement is HTTPS support.
>What are the alternatives when hosting Tomcat behind Apache httpd, 
>nginx, or IIS? I do remember a presentation I thought was pretty good 
>at Apachecon in Miami on connectors a few years ago. Has there been 
>anything new that has come out since then? Are there any 
>recommendations on what is best to replace AJP13?
>
>Thanks
>Dave

Alternatives for AJP

[David Cleary]

Some of our customers are currently using the AJP connector. Given the 
vulnerability and breaking change to address it, now may be a good time to 
prompt them look at alternatives. One requirement is HTTPS support. What are 
the alternatives when hosting Tomcat behind Apache httpd, nginx, or IIS? I do 
remember a presentation I thought was pretty good at Apachecon in Miami on 
connectors a few years ago. Has there been anything new that has come out since 
then? Are there any recommendations on what is best to replace AJP13?

Thanks
Dave

Compiling TCNative on Windows

[David Cleary]

One of our customers got binged on a security audit because some Windows 
binaries weren't compiled with the security features listed below. TCNative is 
on that list. We only include it for our Windows distributions since it is 
available in binary form. One side effect of ASLR is that pointers can live in 
the entire address space. So, some code can have issues if they assume that 
pointers will only be in the lower 32 bit address space. Has any tried this or 
know of a reason that compiling with the switches will cause issues with 
TCNative? My choices are to bring it in house and build ourselves, or drop it 
from our distribution.

ASLR - Address Space Layout Randomization

Linker option: /DYNAMICBASE[:NO]

https://docs.microsoft.com/en-us/cpp/build/reference/dynamicbase-use-address-space-layout-randomization?view=vs-2019

DEP - Data Execution Prevention

Linker option: /NXCOMPAT[:NO]

https://docs.microsoft.com/en-us/cpp/build/reference/nxcompat-compatible-with-data-execution-prevention?view=vs-2019

SafeSEH - Safe Exception Handlers

Linker option: /SAFESEH[:NO]

https://docs.microsoft.com/en-us/cpp/build/reference/safeseh-image-has-safe-exception-handlers?view=vs-2019

Thanks
Dave

Any suggestions on Java 11 JVM options for Tomcat?

[David Cleary]

We made the decision to move to Java 11 for an upcoming release that will be 
our Long Term Servicing branch. We have noticed a non-trivial increase in 
memory usage compared to Java 8. On most platforms, it isn't something 
critical. On Solaris, with the Oracle JVM, we see about double our process 
memory footprint, and this memory is outside the heap. It also seems to be 
related to SSL/security as it is much worse where we heavily use 
security/encryption. While investigating this issue, we were looking at new JVM 
options over what we currently use on Java 8. At this point, we are think the 
Solaris (11.04) JVM may be buggy, but in general, has anyone been using Java 11 
with Tomcat and do they have some JVM option settings that work well for them?

Thanks
Dave

RE: Setting samesite attribute on JSESSIONID

[David Cleary]

On 10/10/19 14:08, David Cleary wrote:
> Have a customer asking about this. I see Tomcat supports it here.
> https://tomcat.apache.org/tomcat-9.0-doc/config/cookie-processor.html
> .apache.org
>
>
>
We currently use defaults, so I'm looking for an XML fragment and
> the file it goes in to add the samesite attribute to the JSESSIONID. 
> I'm assuming they want it globally for all webapps.

>What have you tried already?

To paraphrase Maxwell Smart, "Missed it by that much". Our shipping version is 
at 9.0.20, so the warnings about unknown attribute are expected.

Thanks
Dave


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Setting samesite attribute on JSESSIONID

[David Cleary]

Have a customer asking about this. I see Tomcat supports it here. 
https://tomcat.apache.org/tomcat-9.0-doc/config/cookie-processor.html

We currently use defaults, so I'm looking for an XML fragment and the file it 
goes in to add the samesite attribute to the JSESSIONID. I'm assuming they want 
it globally for all webapps.

Thanks
Dave

tomcat-users.xml dynamic changes

[David Cleary]

Mark,

I see this feature in the new Tomcat 8.5 release.

- Add the ability for a UserDatabase to monitor the backing XML file
for changes and reload the source file if a change in the last
modified time is detected. This is enabled by default meaning that
changes to $CATALINA_BASE/conf/tomcat-users.xml will now take
effect a short time after the file is saved.

I did some investigation and want to confirm this feature has been in 9.0, so 
that 9.0.20 already operates that way.

Thanks
Dave

Tomcat hanging on shutdown post 9.0.13

[David Cleary]

We currently ship our AppServer on Tomcat 9.0.13. We like to update to a more 
recent version when we ship a new release. However, Tomcat refuses to stop, on 
both Windows and Linux, when we go past 9.0.13. I know that it has something to 
do with our main web application as Tomcat does shut down when it isn't 
deployed. We also have Replaced Tomcat's Juli logger with Logback. For some 
reason, the 9.0.21 version of this library has not been released.

I'm looking for suggestions on tracking this down. At shutdown, we see a number 
of logback threads as well as some additional threads we create to watch 
resources through possible memory leak exceptions. Anyways, I am looking for 
suggestions on tracking this down. I'm able to attach to the process with 
Eclipse, but it hasn't helped point me at what the cause is.

Thanks
Dave

RE: ecj-4.6.3 no longer required?

[David Cleary]

User error. Not sure how I thought it was removed.

Thanks
Dave

-Original Message-
From: Jason Wee  
Sent: Tuesday, March 26, 2019 3:35 PM
To: Tomcat Users List 
Subject: Re: ecj-4.6.3 no longer required?

which 8.5.x u were referring? i just checked 8.5.39 has ecj in lib directory.

On Wed, Mar 27, 2019 at 1:58 AM David Cleary  wrote:
>
> I'm current updating our server that is based on Tomcat 8.5.x and found that 
> ecj-4.6.3.jar is no longer in the distribution. The changelog does not note 
> that it has been removed. I just want to confirm that I should remove this 
> library as part of the Tomcat update.
>
> Thanks
> Dave

ecj-4.6.3 no longer required?

[David Cleary]

I'm current updating our server that is based on Tomcat 8.5.x and found that 
ecj-4.6.3.jar is no longer in the distribution. The changelog does not note 
that it has been removed. I just want to confirm that I should remove this 
library as part of the Tomcat update.

Thanks
Dave

RE: Apache Tomcat Deploying war file every time on AIX even though there are no changes in war file.

[David Cleary]

Mark,

Turns out it is a JDK bug.

http://www-01.ibm.com/support/docview.wss?uid=swg1IV99686
https://bugs.openjdk.java.net/browse/JDK-8177809

While it has been fixed, we currently are required to support AIX 6.1, which 
doesn't have an updated JDK because IBM no longer supports that version of AIX.

Dave

-Original Message-
From: Mark Thomas  
Sent: Monday, November 12, 2018 4:30 AM
To: users@tomcat.apache.org
Subject: Re: Apache Tomcat Deploying war file every time on AIX even though 
there are no changes in war file.

On 12/11/2018 05:36, Nagaraju Varaparla wrote:
> Mark,
>Thanks.
> I have checked the last modified time of the META-INF/war-tracker file and 
> the war file.
> I see both are same. And still tomcat re-deploying the war file.
> 
> May be the time comparison failing?

Maybe. But it is hard to see how.

Can you repeat this in a test environment? If so, I'd recreate this while 
running Tomcat with a remote debugger and put a break point in
org.apache.catalina.startup.ExpandWar.expand(Host,URL,String) to see exactly 
what is going on. Unfortunately, there isn't any additional logging available.

Mark


> 
> Thanks,
> Nagaraju.V
> 
> 
> 
> From: Mark Thomas 
> Sent: Sunday, November 11, 2018 2:06 AM
> To: Tomcat Users List 
> Subject: Re: Apache Tomcat Deploying war file every time on AIX even though 
> there are no changes in war file.
> 
> On 10/11/2018 16:35, Nagaraju Varaparla wrote:
>> Hi,
>> greetings!!
>>
>> Issue Description:
>> Apache Tomcat Deploying war file every time on AIX even though there are no 
>> changes in war file.
>> This causing all my applications are getting un-registered.
>>
>> Environment.
>> Tomcat Version: apache-tomcat-8.5.34
>> JAVA Version : IBM JAVA 
>> http://8.5.0.25
>> .25> .5.0.25> OS : IBM AIX 6.1 (AIX VERSION 6100-09-09-1717)
>>
>> Steps to reproduce:
>> 1. Set JAVA PATH to IBM JDK http://1.8.5.25.
>> 2. Download apache-tomcat-8.5.34.
>> 3. Set CATALINA_HOME as tomcat folder(apache-tomcat-8.5.34).
>> 4. we can check the version of tomcat and java with catalina.sh 
>> version 5. Create sample war file and copy to webapps folder of tomcat.
>> 6. start the tomcat (catalina.sh start) 7. Wait till tomcat explode 
>> the war file into webapps folder.(till we see the modified time of the 
>> sample folder).
>> 8. stop the tomcat (catalina.sh stop) 9. again start the tomcat 
>> (catalina.sh start)
>>
>> This will re-deploy the application. Ideally this should not happen as the 
>> war file is already deployed and not changed.
>> This is deleting the existing application and re-deploying.
>> I can see this in the log file.
>> An expanded directory [apache-tomcat-8.5.34/webapps/sample] was found 
>> with a last modified time that did not match the associated WAR. It will be 
>> deleted. > it is getting delete here Please let us know if more 
>> information is needed.
> 
> When Tomcat unpacks a WAR file it adds a file called:
> META-INF/war-tracker
> 
> in the expanded directory. The last modified time of this file is set 
> to be the same as the WAR file. When Tomcat (re-)starts, if those 
> timestamps are found to be different, Tomcat assumes the WAR was 
> updated while Tomcat was shut down and redeploys the WAR.
> 
> You need to check those timestamps and investigate what, if anything, 
> might be modifying them.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: 
> users-unsubscr...@tomcat.apache.org ache.org> For additional commands, e-mail: 
> users-h...@tomcat.apache.org
> 



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: 21 second pause that randomly happens

[David Cleary]

Thanks for the feedback. I was confident all along this wasn't a Tomcat issue, 
and I believe we identified the culprit as the firewall. I was looking for 
logging that would show the socket behavior (accept, close, etc.) to show that 
as far as Tomcat was concerned, the delay was happening before it got there. 
Anyways, we were able to do some testing without the particular firewall they 
were using and the results were encouraging. Will be a few days of testing 
before they try it on their production system to confirm it, but so far, so 
good.

Thanks
Dave

-Original Message-
From: charlie arehart [mailto:charlie_li...@carehart.org] 
Sent: Thursday, July 19, 2018 12:24 PM
To: 'Tomcat Users List' 
Subject: RE: 21 second pause that randomly happens

David? Are you still needing help with this? Your answers to what I ask may 
help us get you to a solution.

/charlie

-Original Message-
From: charlie arehart  
Sent: Tuesday, July 17, 2018 01:14 PM
To: 'Tomcat Users List' 
Subject: RE: 21 second pause that randomly happens

Going back to David's original note, you wrote, " We have a customer who is 
experiencing a random, 21 second pause when using out Tomcat based application 
server. We believe this may be during a TCP connect and timeout. Logging 
indicates the pause happens before the request makes it to our back end."

Can you clarify what you mean by the "back end" here? 

Since you say it's a pause "when using" Tomcat, I can't tell if you're saying 
Tomcat IS the backend, or instead if the request hangs WHILE running in Tomcat 
but doesn't reach some other "backend" that the Tomcat-based app should then 
talk to. You did refer in a later note to a "backend agent log", so I'm 
inclined to think it's the latter.

In that case, it seems you are wondering (in one note) whether the request 
might be hung up in the connector (getting TO Tomcat). 

But do you know for sure whether the request is actually running IN Tomcat? You 
could use any of various JVM or Tomcat monitoring tools to know a) if the 
request reaches Tomcat, b) if it is hanging while running there, and then c) 
just WHAT the request is hanging and waiting for (which could well be that 
"other backend" the app needs to talk to). 

But before elaborating on approaches, I wanted to ask what you may or may not 
know in regard to the above, first. :-)

/charlie 

Re: 21 second pause that randomly happens

[David Cleary]

On 16.07.2018 16:35, David Cleary wrote:
> 2018-07-16 15:55 GMT+03:00 David Cleary :
>>> We have a customer who is experiencing a random, 21 second pause when using 
>>> out
Tomcat
> based application server. We believe this may be during a TCP connect and 
> timeout. Logging
> indicates the pause happens before the request makes it to our back end.

> Logging where then ?

Sorry for any formatting issues. I have a digest subscription which doesn't 
lend well to interactivity.

Clients are running on Windows machines. Server is running on AWS and Linux. 
There is a cloud firewall in between (pfSense). Do not have the details if they 
are running the cloud version available on AWS.

Client logging shows this:

[18/05/16@12:12:48.822+1000] P-006760 T-002372 1 4GL REV    Trying 
Connection
[18/05/16@12:13:09.925+1000] P-006760 T-002372 1 4GL REV    Connect 
Complete 21102
[18/05/16@12:13:09.925+1000] P-006760 T-002372 1 4GL REV    WARNING: 
LONG CONNECTION
[18/05/16@12:13:09.925+1000] P-006760 T-002372 1 4GL REV    
A4DC513EA548E24508E1E90AA9EA61DD9386DDB475AD.clintons connected 21102

Localhost access log shows this

localhost_access_log.2018-05-16.txt:10.255.11.250 - - [16/May/2018:12:13:16 
+1000] "POST 
/apsv?CONNHDL=A4DC513EA548E24508E1E90AA9EA61DD9386DDB475AD.clintons HTT P/1.1" 
200 253 1

And our back end agent log shows this:

clintons.agent.log:[18/05/16@12:13:16.294+1000] P-019364 T-2819262208 2 AS-19 
AS Application Server connected with connection id: 
A4DC513EA548E24508E1E90AA9EA61 DD9386DDB475AD.clintons. (8358)
clintons.agent.log:[18/05/16@12:13:16.299+1000] P-019364 T-3688318720 2 AS-19 
AS Application Server disconnected with connection id: 
A4DC513EA548E24508E1E90AA9E A61DD9386DDB475AD.clintons. (8359)

Customer had some weird reconnection logic that was part of their application. 
After removing the code so the logical connection would be kept open, we saw 
this pause happen on a standard request. I do not know how long this logical 
connection was idle before running. I also do not know if Tomcat closed the 
underlying socket either due to resources or a keep-alive timeout. I was hoping 
logging could tell me when Tomcat binds to an incoming socket and releases it. 
I was hoping to show in the above example, as far as Tomcat is concerned, the 
21 second delay happened outside of the server. Scouring the source code and 
trying some experimentation, it does not appear there is logging available at 
the socket level.

  It mostly happens
> when we create an initial logical connection, but we have also seen it 
> elsewhere where
we
> believe the TCP Keep alive was expired and a new socket had to be 
> established. However,
I
> do not know this and am hoping there is some logging I can turn on in the NIO 
> connector
to
> collect more data. I tried turning on logging in the Endpoint class, but that 
> did not
provide
> anything useful.

> If the connection request does not even reach the Tomcat back-end, that is 
> also unlikely 
> to provide much information. (Not being facetious here, just stating a fact).
> Can you do a "netstat" command on your Tomcat server when this happens ?
> If yes, maybe some part of the output would provide some information from the 
> TCP level 
> (such as a high number of connections, to the Tomcat NIO port, in some 
> specific TCP state

e.g.)

Customer did some probing with Wireshark and said they were seeing a TCP 
retransmission (sorry, I do not have many more details). In investigating this, 
we discovered this info on TCP timeouts:

" There's probably a million reasons why the client may never receive a 
SYN-ACK. The one I've seen more often is packet loss, which in turn can have 
lots of reasons, for example a malfunctioning or misconfigured network switch.
However, you can immediately spot if your timeout/hang problems are caused by 
TCP retransmission because they happen to cause response times that are 
unusually frequently distributed around 3, 9 and 21 seconds (and on, of course).
In fact, the TCP retransmission timeout starts at 3 seconds, but if the client 
tries to resend after a timeout and still receives no answer, it doubles the 
wait to 6 s, so the total response time will be 9 seconds, assuming that the 
client now finally receives the SYN-ACK. Otherwise, 3 + 6 + 12 = 21, then 3 + 6 
+ 12 + 24 = 45 s and so on and so forth."

This is why we are focusing on the TCP layer.

>  There is a NAT firewall between the client and server, so I'm looking for
> some TCP level logging that could point me in the proper direction.
>
>> Tomcat version = ?
>
> Sorry. Tomcat 8.5.27.
>

> And on which kind of O.S. is this happening ?

> Also maybe another question : is this happening on a Tomcat server which is 
> dedicated to 
> that particular customer ? or is the Tomcat serve

Re: 21 second pause that randomly happens

[David Cleary]

2018-07-16 15:55 GMT+03:00 David Cleary :
>> We have a customer who is experiencing a random, 21 second pause when using 
>> out Tomcat
based application server. We believe this may be during a TCP connect and 
timeout. Logging
indicates the pause happens before the request makes it to our back end. It 
mostly happens
when we create an initial logical connection, but we have also seen it 
elsewhere where we
believe the TCP Keep alive was expired and a new socket had to be established. 
However, I
do not know this and am hoping there is some logging I can turn on in the NIO 
connector to
collect more data. I tried turning on logging in the Endpoint class, but that 
did not provide
anything useful. There is a NAT firewall between the client and server, so I'm 
looking for
some TCP level logging that could point me in the proper direction.

>Tomcat version = ?

Sorry. Tomcat 8.5.27.

Dave

-

21 second pause that randomly happens

[David Cleary]

We have a customer who is experiencing a random, 21 second pause when using out 
Tomcat based application server. We believe this may be during a TCP connect 
and timeout. Logging indicates the pause happens before the request makes it to 
our back end. It mostly happens when we create an initial logical connection, 
but we have also seen it elsewhere where we believe the TCP Keep alive was 
expired and a new socket had to be established. However, I do not know this and 
am hoping there is some logging I can turn on in the NIO connector to collect 
more data. I tried turning on logging in the Endpoint class, but that did not 
provide anything useful. There is a NAT firewall between the client and server, 
so I'm looking for some TCP level logging that could point me in the proper 
direction.

Thanks
David Cleary
Progress

TomcatCon at ApacheCon?

[David Cleary]

Mark,

Will there be a TomcatCon within ApacheCon this year?

Dave

Supporting on-line deployment through JMX

[David Cleary]

We have web applications that require tailoring when being deployed. If Tomcat 
is running, we can start the context after tailoring with a deploy call to 
Manager. That is fine for development, but for production, the Manager web app 
is usually not there. Is there an mbean I can call that doesn't require the 
Manager web app?

Thanks
David Cleary
Progress Software