Re: Tomcat9 not listening to ipv4 port 8080, only ipv6

[Shawn Heisey]

On 11/28/23 05:24, Christoph Kukulies wrote:

root@mail:/var/lib/tomcat9/logs# lsof -i :8080
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    58986 tomcat   37u  IPv6 571175      0t0  TCP *:http-alt (LISTEN)
root@mail:/var/lib/tomcat9/logs#


On my local desktop (running Ubuntu 22), it says IPv6 but IPv4 works 
just fine:


sheisey@sheisey-desktop:~$ sudo lsof -Pn -i :81
COMMANDPID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
apache2   1681 root4u  IPv6  30968  0t0  TCP *:81 (LISTEN)
apache2 173394 www-data4u  IPv6  30968  0t0  TCP *:81 (LISTEN)
apache2 173395 www-data4u  IPv6  30968  0t0  TCP *:81 (LISTEN)
sheisey@sheisey-desktop:~$ ip a show br0
3: br0:  mtu 1500 qdisc noqueue state 
UP group default qlen 1000

link/ether be:3f:67:fc:ce:6b brd ff:ff:ff:ff:ff:ff
inet 192.168.217.210/24 brd 192.168.217.255 scope global dynamic 
noprefixroute br0

   valid_lft 2396sec preferred_lft 2396sec
inet6 fe80::bc3f:67ff:fefc:ce6b/64 scope link
   valid_lft forever preferred_lft forever
sheisey@sheisey-desktop:~$ telnet 192.168.217.210 81
Trying 192.168.217.210...
Connected to 192.168.217.210.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

I would suspect something like firewall/selinux/apparmor.  On this 
ubuntu desktop, ufw is disabled, so the connection also works from other 
systems on the LAN.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: HSTS on 401 / error pages

[Shawn Heisey]

On 9/14/23 08:03, Thomas Hoffmann (Speed4Trade GmbH) wrote:

Sorry, I thought removing all content and subject is sufficient. Maybe the 
message-id header is used internally(?)


TL;DR: technical details about message threading.  Not about Tomcat.

This is what happens when you reply to an existing message for a new 
topic rather than starting a brand new message:


https://www.dropbox.com/scl/fi/6f6xqoj9ndznr1pwnluuk/bad-threading-tomcat-user.png?rlkey=q6385e4fqyd2ngp97qgj4bj3y=0

There are headers in the message that facilitate threading that you 
can't normally see.  These are the relevant headers in the message you 
replied to:


References: <9ebb0e5d-1794-92f6-9c9f-47a235a4e...@touchtonecorp.com>
 <057e2b5435244011898683f843170...@speed4trade.com>
 <5f704d72-03ea-457c-8d44-792ecda97...@elyograg.org>
 <80c5af8d-d267-581a-0877-687413cd6...@apache.org>
 <4011ac44-d7b8-3fa1-5676-34cbd9207...@christopherschultz.net>
 
In-Reply-To: 
Message-ID:
 

And these are the relevant headers in your reply:

Message-ID: <62e04634351c4c62ae75848a77dac...@speed4trade.com>
References: <9ebb0e5d-1794-92f6-9c9f-47a235a4e...@touchtonecorp.com>
 <057e2b5435244011898683f843170...@speed4trade.com>
 <5f704d72-03ea-457c-8d44-792ecda97...@elyograg.org>
 <80c5af8d-d267-581a-0877-687413cd6...@apache.org>
 <4011ac44-d7b8-3fa1-5676-34cbd9207...@christopherschultz.net>
 
 
In-Reply-To:
 

While some mail clients will create threads on the message subject, 
these headers are the strictly correct way to show threads.  We also see 
messages where people send a reply to a thread by writing a new message 
with the same subject.  Clients that do threading properly will not show 
those messages as part of the thread.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: Solution to "Invalid keystore format" (cross-posted to Tomcat Users List at Apache, and Java 400 List at Midrange)

[Shawn Heisey]

On 9/12/23 01:06, Thomas Hoffmann (Speed4Trade GmbH) wrote:

I moved away from using the proprietary java keystore format.
I switched to using Base64 PEM format. This is usually also the format you get 
from the certificate issuer.
No need to convert it into Java format any more and you can also open it with 
any text editor.


I have never been able to get a Java program to accept a certificate/key 
in PEM format.  The closest I've been able to come is creating a PKCS12 
file with openssl.  Annoying because all the other software I use 
accepts PEM with no problem, and as you have said, PEM is the format 
generally produced by a CA.


How did you get it to take a PEM cert?

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Problem with the redirect after j_security_check

[Shawn Heisey]

On 7/22/23 12:03, Mark Thomas wrote:
Your target URL is invalid. No user agent should be sending the  
fragment (#index) part of the URL. At best Tomcat will ignore it. Later 
versions may even reject it (I have a memory of that but don't have easy 
acces to the source code to check right now).


What I have observed about #:

If you are working in a browser, the # character and anything after it 
are never sent to the webserver.  That value is completely handled 
within the browser, typically used to either anchor to a specific point 
in the web page or provide data to javascript code running in the browser.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: AW: Having trouble with Tomcat crashes. Interesting memory numbers in Manager

[Shawn Heisey]

On 2/11/23 08:17, Thad Humphries wrote:

Finally I profiled our Java utility with VisualVM, all on
my Mac Mini, and quickly found a leak from java.util.zip. This was a
surprise because we were not using java.util.zip anywhere, nor could I find
any reference to java.util.zip when I looked at the code of the few open
source libraries we were using.


The jar format is a zipfile with a particular directory structure.

Which means that unless the application has no jars and exists as ONLY 
extracted .class files, java.util.zip is going to be used behind the scenes.


Interesting that it would create a continual memory leak, though.  I 
guess your evidence means that it is the jar file, not the files 
included in the jar, which are mmapped by Java.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: AW: Having trouble with Tomcat crashes. Interesting memory numbers in Manager

[Shawn Heisey]

On 2/9/23 12:54, Christopher Schultz wrote:
It would be unusual for the OS to reclaim any of that memory from the 
JVM process. Are you looking at OS heap usage, or "JVM heap" usage? From 
your description above, it's tough to tell. The tool is called WRKJVMJOB 
so presumably it knows what the heck a JVM is, so maybe you were getting 
the exact numbers I was recommending.


TL;DR: I have seen Java return memory to the OS.

On the Oracle JVM, I think that happens with G1GC when -Xms is less than 
-Xmx.  I think G1 is the only collector in Java 8 that does this, but I 
can't say that for sure.  I don't see anything in the thread about 
precise JVM options.


Someone mentioned the J9 JVM, and I don't know if that behaves the same 
as Oracle.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is it possible to add hsts header over http response ?

[Shawn Heisey]

On 1/12/23 01:34, Mark Thomas wrote:

On 12/01/2023 08:26, Hiran CHAUDHURI wrote:
In that case the Connector would need to be configured with 
secure="true" to work correctly/securely and the 
HttpHeaderSecurityFilter would add the HSTS header if configured to do so.


My personal opinion is that the header should be added by whatever is 
handling the TLS.


I don't have Tomcat in my current setups, but the piece handling TLS for 
me is haproxy.  In a lot of cases it will be Apache httpd.  My haproxy 
frontend config has this:


  http-after-response set-header Strict-Transport-Security 
"max-age=1600; includeSubDomains; preload;"


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 10 on Windows 11 error

[Shawn Heisey]

On 12/30/22 17:03, Carles Franquesa wrote:
Once downloaded, installed, and configured in the settings of the 
netbeans project, the error has changed, what always is very hopeful.


Now, the thing is that the compiler gives this new output:

image.png

Tried to modify the compiler options in the project properties,
image.png
but, obviously, is not the place.

Do you now where should I modify the compiler option Xbootclasspath?

And I feel sorry for not having checked exactly "URLEncoder.encode 
TomcatManagerImpl". Trust me when I say I have been googling so far as 
extenuation...


The mailing list ate your image attachments.  Nobody can see them.

You need to put the images on a file sharing site and provide a URL like 
you did for the first message on the thread.


The later message says you got it working, so I guess the images are no 
longer needed.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 10 on Windows 11 error

[Shawn Heisey]

On 12/30/22 09:07, Carles Franquesa wrote:

I am trying to run a web app made with Netbeans 16 using Apache-Tomcat
10.0.27 on Windows 11. It worked fine on Windows 10 with the same tomcat.

Everything goes fine when starting the server (it takes just 625ms!) but
after that, when I try to debug the project, the next error is thrown:

[image: enter image description here] 


Tomcat 10.0.27 only requires Java 8, but the netbeans call that is 
failing requires at least Java 9, and apparently is not correctly marked 
as such.


https://github.com/apache/netbeans/issues/5134#issuecomment-1364282218

Upgrade to Java 11 and it will probably work.  Java 17 probably will 
work too, but IMHO it's not a good idea to jump a lot of major versions 
past the minimum requirement.  Java 11 seems to do a good job with code 
meant for Java 8.


I don't know why it worked before with Windows 10.  Maybe you had an 
earlier version of netbeans for that.


FYI, the link above was the first hit on a google search for 
"URLEncoder.encode TomcatManagerImpl".


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Receiving HTTP (any version but 3 prefered) over UDP

[Shawn Heisey]

On 12/10/22 15:15, Aryeh Friedman wrote:

Is there any browser support for direct UDP sockets in any browser
besides Chrome? I know WebRTC and Websockets force TCP. I know
Chrome does support UDP but can find no evidence one way for the other
browsers.


I'm sure you know that if Chrome is doing it, you can be sure that the 
other browsers either already do or are planning to have it in a new 
release very soon.



As far I can tell there are no web servers and/or major hosting
companies (i.e. AWS, etc.) that support UIUC/HTTP3.  For that reason,
I am playing with different options for making a stripped down UDP
HTTP server (prefer 3 but forcing TLS seems to be a deal breaker for
anyone except name brand sites like Google -- E.g. my main client had
a total cow over the $500/yr fee for a Comodo multi-domain cert even
though due to being healthcare it is a legal requirement.)


I'm not in touch with which of the major web servers can do H3/QUIC.  
Almost all of my web servers are behind haproxy, to offload TLS and make 
decisions about which webserver will receive each request.  Except in 
cases where it just wasn't possible for one reason or another (like 
plex) all backend communication in haproxy is unencrypted.  At this 
time, haproxy only supports H3/QUIC on the front end, not the back end.  
Most of the backends are h2c, some only do http/1.1.  (side note ... I 
am still seeing http/1.0 requests in my logs, and even a few 0.9.)


If I ever find myself in a situation where I need to make a container 
like Tomcat reachable from the Internet, I will be using haproxy as the 
front end, not the common choice of Apache of Nginx.  The only reason I 
can see for using one of those webservers is to have a separate logfile 
for each domain.  The log that haproxy produces is good enough for me.


I have another haproxy installation in AWS on my mail server, configured 
the same way as the main install in my basement. Mostly for my webmail, 
but I do have a few other websites on that install.  It handles 
UDP/443.  This little website is running on my AWS instance:


https://http3test.elyograg.org/

If you visit that page and look at the cert, you'll see it's made by 
LetsEncrypt.  Renewal and deployment of that cert is completely 
automated, so the 90 day lifetime is not the administrative hassle that 
it could be.  I am planning to implement the same automated LE 
certificate deployment at $DAYJOB that I already have for my personal 
domains.  I work for a company that handles healthcare data, and nobody 
has raised a concern about HITRUST not allowing free certificates.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Receiving HTTP (any version but 3 prefered) over UDP

[Shawn Heisey]

On 12/9/2022 2:12 PM, Aryeh Friedman wrote:

For example the following code works against port 7 (UDP echo service)
but not against 8080 (my local tomcat):


I suspect that you are running into the fact that HTTP/3 *always* uses 
TLS.  It's baked into the protocol and NOT optional as with earlier HTTP 
versions.  As far as I know, HTTP/3 is the only version of HTTP that 
uses UDP transport.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to set Ciphers in Server.xml Tomcat 10.1.zz

[Shawn Heisey]

On 11/23/22 14:46, Chuck Caldarale wrote:

On Nov 23, 2022, at 22:41, Shawn Heisey  wrote:

I am betting that Java is just refusing to use those ciphers because they are 
known to be weak.  Hopefully an expert can tell me if I am giving incorrect 
information here.

The reported error was the failure to set the non-existent property due to the 
misplaced configuration line. Whether or not the JVM would honor the requested 
ciphers would happen later - but it could well be rejected by newer JVMs, as 
you noted.


Thanks for the clarification!  I admit that I have almost no experience 
with actually writing Java SSL code, so I wouldn't know whether a given 
class name is right or wrong. :)


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to set Ciphers in Server.xml Tomcat 10.1.zz

[Shawn Heisey]

On 11/23/22 14:12, Edwin Mwangi wrote:

I need help with the correct parameter for setting Ciphers in Apache Tomcat
10.1.2, in the previous version 9 i would use the parameter below

ciphers="TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA"

However when I set the same in Apache Tomcat 10.1.2 I get the following
error in the logfiles


Did you also use a newer Java version with the new Tomcat version?

I have just double-checked results on a Qualys Labs SSL test against a 
webserver that allows those ciphers.  Both of them are considered weak.  
A newer Java version might choose to not allow those ciphers at all and 
throw an error when they are attempted. The information below does not 
involve Java or Tomcat, but hopefully will still be useful:




Here is part of a report using testssl.sh against one of my webservers, 
which gets an A+ security grade:


Hexcode  Cipher Suite Name (OpenSSL)   KeyExch.   Encryption 
Bits Cipher Suite Name (IANA/RFC)

-
SSLv2
 -
SSLv3
 -
TLSv1
 -
TLSv1.1
 -
TLSv1.2 (server order)
 xcca8   ECDHE-RSA-CHACHA20-POLY1305   ECDH 521   ChaCha20 256  
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
 xc030   ECDHE-RSA-AES256-GCM-SHA384   ECDH 521   AESGCM 256  
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

TLSv1.3 (server order)
 x1303   TLS_CHACHA20_POLY1305_SHA256  ECDH 521   ChaCha20 256  
TLS_CHACHA20_POLY1305_SHA256
 x1302   TLS_AES_256_GCM_SHA384    ECDH 521   AESGCM 256  
TLS_AES_256_GCM_SHA384


No ciphers using RSA at all, and each active TLS version only accepts 
two ciphers.


I am betting that Java is just refusing to use those ciphers because 
they are known to be weak.  Hopefully an expert can tell me if I am 
giving incorrect information here.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [Tomcat9][Linux]listening all local addresses by default is not security best practice

[Shawn Heisey]

On 11/23/22 12:43, Robert Turner wrote:

My 2 cents:

I think that it would be a very strange change to make to a generic product
and a "sample" configuration file. If Tomcat was packaged in a
distribution, that might be a more reasonable suggestion. I don't think
Tomcat is insecure because of this; binding to addresses/ports is a key
part of configuration of any deployed system and the responsibility of the
person(s) deploying it to ensure correct basic configuration (which this is
part of).


MySQL has shipped with the out of the box config only listening on 
localhost for a while now, on Ubuntu at least.  Solr, the project I am 
most heavily involved in, made that change starting with version 9.0.0.  
We just released version 9.1.0.  This is becoming a default 
configuration for a LOT of software ... nobody can attack a new service 
or do something they're not authorized to do if they cannot reach it on 
the network.  If your physical security is good, software like this 
cannot be attacked until the admin chooses to open it up beyond localhost.


I think it's perfectly reasonable for Tomcat to do the same, but maybe 
in a new major version.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Compatibility, 32 bit ..

[Shawn Heisey]

On 10/24/22 13:16, Mark Thomas wrote:

On 24/10/2022 20:04, John Dale (DB2DOM) wrote:

Mark and Chris - do you guys have a favorite flavor of Linux that has
yielded good results?


I use Ubuntu for my various test environments and the servers I run at 
home. Stuff I know well (Tomcat, Java, etc) I tend to install manually 
rather than using the package manager.


+1000 for Ubuntu.

Many years ago I used and loved Debian.  I have found that Ubuntu 
retains all the things I love about Debian but has a much larger 
selection of packages available, due to Debian's strict adherence to 
only including software that meets DFSG requirements.  I mostly use 
Ubuntu packages, and in some cases official project repos for Ubuntu 
(gitlab, zabbix, dovecot, etc), but there a few things that I prefer to 
use official downloads or source to install.  Solr and haproxy are the 
top two in that category.


Every time I try to do something useful on an RHEL clone, I find that 
most of the things I love about my Ubuntu servers don't exist.  I can 
usually find a way to accomplish what I'm after, but it takes a lot more 
effort and frustration to get it working.  In general, .deb packages 
work better than .rpm packages, and include many convenience features 
that make them easier to use.


Getting back to the original post:
has not been available in 32 bit flavors for a number of major 
releases.  But Debian still supports it.  A link to a network install 
ISO, which will require Internet access:


https://cdimage.debian.org/debian-cd/current/i386/iso-cd/debian-11.5.0-i386-netinst.iso

If the Internet requirement is a problem, you can find larger DVD images 
that can install a full system without Internet.


SUCCESS!


I built a VM and installed that Debian ISO on it.  I did not install a 
GUI.  Once I got it booted up, I was able to do "apt update && apt 
install -y openjdk-11-jdk" (as root) with no trouble.  If you just need 
the JRE, you can install openjdk-11-jre instead.  Once Java was 
installed, I downloaded the latest tomcat 10, extracted it, and started 
it.  I was then able to access port 8080 in a browser and see the 
"successfully installed Tomcat" page.



Note that 32 bit Java is limited to a 2GB heap.  Limitations like this 
are one of the primary reasons that 32 bit support is rapidly 
disappearing from the software world.  Chances are of course that your 
32 bit system won't really have an issue with that limitation.


The Linux kernel dropped support for 386 and 486 CPUs some time ago.  So 
I hope your 32 bit system has at least a Pentium CPU.  I am not positive 
that i586 support is still around in Debian, but i686 is there for sure.



On the 32 bit VM:

root@debian32bit:/opt/apache-tomcat-10.0.27# uname -a
Linux debian32bit 5.10.0-19-686-pae #1 SMP Debian 5.10.149-2 
(2022-10-21) i686 GNU/Linux


elyograg@debian32bit:~$ java --version
openjdk 11.0.16 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Debian-1deb11u1)
OpenJDK Server VM (build 11.0.16+8-post-Debian-1deb11u1, mixed mode, 
sharing)



On a 64-bit system:

elyograg@smeagol:~$ uname -a
Linux smeagol 5.17.0-1020-oem #21-Ubuntu SMP PREEMPT Fri Oct 14 09:33:24 
UTC 2022 x86_64 x86_64 x86_64 GNU/Linux


elyograg@smeagol:~$ java -version
openjdk version "11.0.16" 2022-07-19
OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.16+8-post-Ubuntu-0ubuntu122.04, 
mixed mode, sharing)


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: SSLLabs scan shows TLSv1.0 and TLSv1.1 even though I have sslProtocol="TLSv1.2"

[Shawn Heisey]

On 8/9/22 16:13, James H. H. Lampert wrote:


On customer box #1, I have:
protocol="org.apache.coyote.http11.Http11Protocol" address=""
   maxThreads="400" SSLEnabled="true" scheme="https" 
secure="true"
   keystoreFile="/tomcat/wttomcat.ks" 
keyAlias=""


ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384" 


   clientAuth="false" sslProtocol="TLSv1.2" />

and an SSLLabs scan shows it accepting only TLSv1.2, as it should.

But on customer box #2, I have:

   maxThreads="150" SSLEnabled="true" scheme="https" 
secure="true"
   keystoreFile="/tomcat/wttomcat.ks" 
keyAlias=""

   clientAuth="false" sslProtocol="TLSv1.2" />

and an SSLLabs scan shows it accepting TLSv1.0, TLSv1.1, and TLSv1.2.

What could be wrong here? I vaguely recall seeing something like this 
before. 


I am actually quite clueless about how to configure TLS in Tomcat ... 
but I can see that you clearly have a difference between the two 
configs.  The first one has a ciphers="stuff" section and the second one 
doesn't.  If I had to guess, and I confess I do not know TLS well enough 
to say anything definitively, I believe the ciphers you have listed in 
that config will not work with older TLS versions.


An SSL Labs report for my website (which is using haproxy for TLS, none 
of the backends are Tomcat) shows that it allows three ciphers for TLS 
1.3 and three ciphers for TLS 1.2.


1.3: TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, 
TLS_AES_128_GCM_SHA256
1.2: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, 
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256


I have more ciphers configured than this, but I am pretty sure that 
these are the only ones in my configured list that work with the newer 
TLS versions.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Enable HTTP Strict Transport Security (HSTS) in Tomcat 9.0.x

[Shawn Heisey]

On 4/29/22 12:14, Kaushal Shriyan wrote:
Thanks Peter for the link and it worked like a charm. I am running the 
tomcat version 9.0.56 on CentOS Linux release 7.9.2009 (Core). I have 
enabled the TLSv1.3 protocol as per the below block but when I ran the 
scan https://www.ssllabs.com/ssltest/analyze.html, It says *TLS 1.3 -> 
No* as per the below scan results.


               SSLEnabled="true"  scheme="https" 
ciphers="ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" 
keystoreFile="ssl/hsbcconsent.jks" keystorePass="tomcat" 
clientAuth="false" disableSessionTickets="true" 
honorCipherOrder="true" *SSLProtocol="TLSv1.2+TLSv1.3"*

               redirectPort="8443" />


I can think of two possible reasons for a problem like this.

1. Your cipher list isn't compatible with TLS 1.3.
2. You're not running a new enough Java version. (8u261 b12 minimum)

Based on what I have been able to figure out, I think it's probably your 
cipher list.  If you are using the standard Java TLS and not the tomcat 
native library that uses openssl, then your cipher list is unlikely to 
work -- those look like openssl cipher names, and Java uses different names.


I think this cipher list might get you TLS 1.2 and 1.3 support with Java:

TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256

To get that list, I converted the cipher list I use in haproxy, which 
uses openssl for tls, using the info found here:


https://stackoverflow.com/a/32654075/2665648

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: What is "h2c"? What is CVE-2021-25329? Re: Most recent security-related update to 8.5

[Shawn Heisey]

On 7/1/2021 6:10 PM, James H. H. Lampert wrote:

On 7/1/21 4:55 PM, Shawn Heisey wrote:

In that case, you don't need h2c, and probably don't want it.


O. . . . k.

That makes sense, so far, but how is it even enabled? Is there some way
I could have h2c enabled, with the situation I described (no HTTP at 
all, not even as a redirect), and not *know* I have it enabled?


I am a lurker on this list.  Although I used Tomcat quite a bit in a 
past job, it's not something I configure these days.  All my past Tomcat 
experience was before HTTP/2 became widespread, so I have no idea how to 
configure it.


Google has a number of hits that look useful if you search for "tomcat 
http/2".  I don't imagine it's difficult to do.


I do know that for full http/2 functionality it's best if you have a 
Java version newer than Java 8.  Java 11 is probably a good choice. 
Note that if you use Oracle Java, they have changed their licensing, and 
most people actually are required to pay to use it.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: What is "h2c"? What is CVE-2021-25329? Re: Most recent security-related update to 8.5

[Shawn Heisey]

On 7/1/2021 3:24 PM, James H. H. Lampert wrote:

On 6/21/21 9:42 AM, Christopher Schultz wrote:
If you are using h2c, you'll definitely want to 8.5.63 or later, as 
there is a critical fix there.


My understanding, based on what I looked up a week and a half ago, is 
that we're not using h2c, but at the same time, don't think I fully 
understand what "h2c" is.


h2c means HTTP/2 cleartext -- that is, without the TLS encryption that 
HTTPS provides.  If Tomcat is reached directly by clients and the 
traffic doesn't go through a load balancer or proxy, then generally you 
don't want to enable h2c, you just want to enable h2.  Technically h2c 
isn't in line with the goals of HTTP/2 ... encryption is assumed.  But 
there are situations where it's what you really do want.


With a load balancer or proxy in the mix, things get a little more 
complicated.  I personally would want the backend connection as fast as 
possible, and all the encryption handled by the front end -- the proxy 
or load balancer.  So my backend connections are h2c, not h2.


But I have run into web applications that only work right if the back 
end connection is encrypted.  Up until about a week ago, one such 
application for me was WordPress.  Then I figured out the right config 
to make WordPress always assume https even if the connections coming 
into the web server (Apache httpd in this case) were not encrypted ... 
so I immediately got rid of the double encryption by using h2c on the 
back end.


A bit of trivia that doesn't affect these answers, but some might want 
to know:  I use haproxy in front of my web services.  It's lightning 
fast, does awesome TLS, and is extremely configurable.


I will note, however, that the Tomcat servers in question are *not* 
configured to listen on any ports other than HTTPS (either 443, 8443, or 
something else in that vein) and the shutdown port.


In that case, you don't need h2c, and probably don't want it.

Also, I've got somebody complaining about CVE-2021-25329. I'm not sure I 
understand what CVE-2021-25329 is, or what the underlying CVE-2020-9484 
is.


I can't tell what those vulnerabilities are about, but I don't think 
they have anything to do with h2c.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: 500 instances of tomcat on the same server

[Shawn Heisey]

On 6/25/2021 8:58 PM, Eric Robinson wrote:

We can run 75 to 125 instances of tomcat on a single Linux server with 12 cores 
and 128GB RAM. It works great. CPU is around 25%, our JVMs are not throwing 
OOMEs, iowait is minimal, and network traffic is about 30Mbps. We're happy with 
the results.

Now we're upping the ante. We have a 48-core server with 1TB RAM, and we're 
planning to run 600+ tomcat instances on it simultaneously. What caveats or 
pitfalls should we watch out for? Are there any hard limits that would prevent 
this from working as expected?


I'm a lurker here.  I have some experience with Tomcat, but most of my 
experience is with other Apache projects.


I'm hoping that my question mirrors what the experienced folks around 
here are thinking:


For something like this ... why are you running so many multiple 
instances?  Why not run one instance, or a few of them, and have each 
one handle many many webapps?  I bet you'll find that the overall memory 
requirements go way down, because there will be far fewer instances of 
Java running.


Maybe you've got good reasons for the architecture you have chosen ... 
but it seems like a complete waste of resources to me.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Out of memory exception

[Shawn Heisey]

On 2/18/2021 12:11 PM, Niranjan Rao wrote:
Thank you the response. This is not a web application, but a standalone 
java program. Hence I said it's not a tomcat question, but a generic JVM 
question. I have been researching about this a lot and based on many 
mails on this list, lot of people here know about internal behavior of 
JVM and specs lot better than I do.


Apologies for getting that wrong.

Is it a custom app or something that you downloaded and installed?  Talk 
to whoever wrote it.  They will hopefully know what information is 
needed to troubleshoot further.


Is Java 15 required for the application to function?  If you can 
successfully use Java 11 or even Java 8, you'll be dealing with a far 
more stable platform.  Major show-stopper bugs in Java are rare, but 
they do happen.  I will warn you that although I do recommend 
downgrading Java for stability purposes, I do not hold out a lot of hope 
that it will solve this problem.


Which garbage collector are you using?  I would recommend one of the 
really stable collectors, like G1.  I wrote this wiki page a long time 
ago that includes garbage collection information for Solr ... I think it 
would apply well to any application where latency is important than 
throughput:


https://cwiki.apache.org/confluence/display/SOLR/ShawnHeisey

Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Out of memory exception

[Shawn Heisey]

On 2/18/2021 11:36 AM, Niranjan Rao wrote:
First apologies for non tomcat question. I have seen that there is 
enough expertise here to provide hints and hints are what I am looking 
for to solve the problem and question is generic enough. I have tried 
researching problem to best of my abilities.


I believe you're right to think this isn't a tomcat question.  There are 
a lot of things it could be.  Tomcat is a *possible* source, though I 
think the chance of that is low.  Without a LOT of info that I would 
probably be useless at interpreting or asking for, it's impossible to 
say for sure.


With problems like this, it is normally the application running inside 
Tomcat that has a problem, not Tomcat itself.  You're likely to get a 
lot more useful information if you go to the people responsible for 
those applications.


We have a java program that regularly throws 
"java.lang.OutOfMemoryError: Java heap space" exception. Puzzling point 
is it happens only on one VM. We have a set of two VMs/boxes spawned 
from same AWS image. Machine class/region is exactly same and since they 
are from same image, they should be mostly identical except stuff like 
host name, ip address etc.


Number of tasks performed by VMs are comparable and not a significant 
difference. Yet, one VM never runs of out of memory and other one does. 
Sometimes it's as soon as half an hour after restarting the process 
while on the other box process is running for days and no issues.


"Comparable" isn't "identical".

Are they running the same apps?  Which apps are involved?  Is the one 
that's throwing OOME handling substantially similar requests when 
compared to one that doesn't?  Is the request rate nearly the same, or 
is the problematic one handling a lot more?  Another applicable 
question, also off topic for this mailing list:  Are the apps in both 
cases configured identically?


I took memory dumps from both VMs and they look similar. Program is 
started with -Xmx1g flag and we have taken regular memory dumps. In many 
cases eclipse MAT reports total memory usage was less than 100MB when 
program crashed with out of memory exception.


That's extremely odd, unless the application requested a REALLY big 
chunk of memory such that the 100MB existing plus the new allocation 
would be larger than the max heap size of 1GB.


Do you have enough free memory that you could increase the max heap to 
2GB or beyond and see what happens?


Has anyone seen anything similar to this? Identical bits of code 
behaving differently? What else should I be looking for?


Earlier you said "comparable" and now you're saying "identical".  So I 
have to ask ... which is it?  Remember that differences in 
configurations, types of requests, and request load can lead to very 
different requirements, even if the apps running inside Tomcat are the same.


Most of my experience in the Java world comes from Solr.  Apache Solr is 
a servlet application, and ships with Jetty.  Tomcat is not usually 
involved.  I joined this mailing list because I was responsible for 
Tomcat servers running apps developed in-house, and every once in a 
while, I needed to ask something tomcat-specific.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: error 0 issue

[Shawn Heisey]

On 6/26/2019 11:18 AM, Mark Thomas wrote:

On 26/06/2019 18:10, Kumar R wrote:

Hi Team,
Is it possible to go for higher version of JDK(64 bit) and Tomcat(64bit) on
32 bit window 2003 architecture.


No.


A tiny bit more detail:

64-bit software requires a 64-bit operating system.  A 64-bit operating 
system requires a 64-bit CPU.


You're running 32-bit Windows, so you can't run any 64-bit software, 
even if the hardware is actually 64-bit.


32-bit programs on Windows, including Java, are limited to addressing 
2GB of memory.  Which typically means that if you make the heap larger 
than about 1.5 GB on a 32-bit system, Java's probably going to either 
crash or not work correctly.  This can be a serious limitation with 
modern software.


Intel hasn't mass-produced 32-bit server-class CPUs since 2005, and 
desktop-class CPUs since 2006.  Other CPU manufacturers like AMD 
probably did something similar around the same timeframe.  Which means 
that if your hardware is at all recent, it probably has a 64-bit CPU.


Thanks,
Shawn

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Anyway to set more max ram to Tomcat 32bit?

[Shawn Heisey]

On 1/4/2019 10:44 AM, John Dale wrote:

Interesting note: "likely includes memory that Java requires beyond
the Java heap size itself"

Can you expound on this?


In the majority of cases, most of the memory that Java allocates from 
the system is its heap.


But there are other memory structures required.  One of those is the 
space where Java loads things like classes.  Java is a very complicated 
piece of software that would require a number of in-memory structures 
for various operations.  The one that comes to mind most readily is 
garbage collection.


The OP said the max heap they could get allocated was 1600MB. Which 
suggests that Java is requiring a few hundred megabytes to be allocated 
for its own operation, fitting inside the 2GB max.


If more memory is needed in the heap, the only option will be to switch 
to 64-bit Java.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Anyway to set more max ram to Tomcat 32bit?

[Shawn Heisey]

On 1/3/2019 6:40 PM, ark...@tutanota.com wrote:

The strange part is I was able to set a max memory in the tomcat9w.exe 
configuration tool to about 1600 Megabytes and that was the hitting of the wall 
on that setup (the original physical one that I p2v'd to a VM), I can set less 
ram, but I cannot set it more than 1600 otherwise when I try to start the 
tomcat service in services.msc it will just crap outs.


Tomcat itself is not 32 bit or 64 bit.  It is *Java* that determines this.

If you have a 32-bit Java, then that Java cannot address more than 2GB 
of memory, and this likely includes memory that Java requires beyond the 
Java heap size itself.  This is a limitation that can only be overcome 
by running a 64-bit Java.  Naturally this requires a 64-bit operating 
system, which requires a 64-bit CPU.


For the server side, readily available Intel processors have been all 
64-bit since 2005.  For the client side, readily available Intel 
processors have been all 64-bit since 2006.  If you've purchased a 
computer in the last ten years, it is virtually guaranteed to have 
64-bit hardware.


I think it was either Java 9 or Java 10 where Oracle stopped making 
32-bit versions easy to find.


Run a 64-bit Java.  You won't have any memory limits.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [slightly OT] Re: Tomcat 9 does not work with Java 11

[Shawn Heisey]

On 11/27/2018 3:08 AM, Andi Meister wrote:

What I did now:
- removed Tomcat services by service.bat
- uninstalled all Tomcats (7 and 9)
- uninstalled all Java (was only Version 11)
- server reboot
- Installed Java 11 (File: jdk-11.0.1_windows-x64_bin.exe)


That filename tells me you're installing the JDK from Oracle.

I have a side note for you:  The license for Java from Oracle has 
changed dramatically in version 11 compared to previous versions.


If your use of Oracle Java will be for commercial or production, you 
have to pay Oracle a license fee for that machine.  The only way you can 
use their Java for free is if you're developing your application.  Once 
you deploy it to production running with Oracle Java 11, they expect 
payment.


You might think I'm trying to pull a fast one here.  I'm not.  Right 
there in the license mentioned on the Java download page, under the 
"Further, You may not" section, is this bullet point: "use the Programs 
for any data processing or any commercial, production, or internal 
business purposes other than developing, testing, prototyping, and 
demonstrating your Application;"


https://www.oracle.com/technetwork/java/javase/terms/license/javase-license.html

So if you want to use Java 11 for production, get OpenJDK.It is still free.


[2018-11-27 10:56:50] [error] [ 3060] Failed creating Java C:\Program
Files\Java\jdk-11.0.1\bin\server\jvm.dll


I downloaded Java 11 from Oracle's website, same executable filename you 
mentioned, and installed it.  The exact DLL path mentioned in this log 
entry is there.  If that DLL is there when you open the mentioned 
directory ... this problem makes me wonder if maybe the Tomcat service 
is running as a user that does not have permission to access that 
directory location, so it cannot find the parts of Java that it needs.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat redirects to http instead of https

[Shawn Heisey]

On 11/19/2018 6:16 AM, Dino Edwards wrote:

I'm not using Named-Based Virtual Hosts
Yes, you are. :)

I didn't think I was. How do you figure?


The first line in the config you shared was "".

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat embedded with Apache Solr

[Shawn Heisey]

On 10/18/2018 8:55 AM, Christopher Schultz wrote:

Actually, my goal was to convince the Solr team that switching from
Jetty to Tomcat was (a) possible and (b) possibly attractive.


Over on lucene-dev, I had said that I removed jetty from solr's ivy 
config and found only two classes with errors in eclipse.


Turns out this was because I hadn't removed jetty from the *lucene* ivy 
config, so most of the jetty jars were actually still referenced in the 
eclipse build path.


When I remove jetty from ALL ivy configs, there are 335 compile errors, 
across many more classes.  Some of those are on the Lucene side, where I 
have less concern.  The part of Lucene that utilizes Jetty is not used 
in Solr.


As I expected, a lot of the errors are in test code, but some of them 
are in code that's not for tests.


If you really want to see us switch to Tomcat, we'll need help fixing 
those errors ... switching the code over to generic APIs (servlet and 
native Java) where possible, and to Tomcat where necessary.  I would not 
expect the Tomcat community to actually do the work -- just provide 
expert guidance.  Although if anyone was interested in volunteering, I 
wouldn't turn away the help!


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat embedded with Apache Solr

[Shawn Heisey]

On 10/15/2018 2:15 AM, Jäkel, Guido wrote:

I have no experience with embedded tomcat, but it should be also straight 
forward. Said that, I can't imagine the advantage of such an approach against 
the currently used, which just start the Web Application Server (Jetty, Tomcat 
or whatever) with the preinstalled Solr application.


Probably the top reason for embedding the network layer (currently 
Jetty) is to have all of the global config (both for the 
network/container layer and for the Solr node itself) live in one place.


Right now, when running in cloud mode, a node could register itself with 
the cluster using completely invalid information -- wrong host/address, 
wrong port, wrong context path (normally /solr -- changing it causes big 
problems with 5.0 and later).  Changing to embedded would allow us to 
ensure that the info used in node registration is the same info used to 
initialize the node.  It would be overall much cleaner, and we would be 
able to finally say that the end user is starting Solr -- not Jetty or 
any other container.


Here's the info about why things are as they are currently, with a 
little bit of info about where we want things to end up:


https://wiki.apache.org/solr/WhyNoWar

Christopher already shared the issue I filed for the effort in setting 
Solr up as a standalone application.  It is an umbrella issue with one 
child issue currently:


https://issues.apache.org/jira/browse/SOLR-6733

Another point in Jetty's favor:  We are in the process of switching from 
Apache httpclient to Jetty httpclient, for http/2 support.  The new 
Apache client with http/2 support is only recently into beta status.  It 
was in alpha state for a REALLY long time, and we expect the beta state 
to last for another several months, possibly a year or longer.  The 
jetty client with http/2 support has been available in a stable version 
almost as long as the http/2 RFC has been published, and has had all 
that time with heavy usage for the showstopper bugs to be worked out.  I 
personally think it's tragic that we're switching away from Apache for 
our httpclient, but I'm in favor of the switch for the reasons 
outlined.  Compatibility with the server is expected to be better if 
both are from the same development team.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: WebSockets and JSPs

[Shawn Heisey]

On 10/15/2018 9:16 PM, Jerry Malcolm wrote:
I have several webapps that do a significant amount of recursive loads 
of snippits of HTML utilizing XHR/http/ajax requests. These apps are 
all debugged and in production.   The server has no problem whatsoever 
in keeping up with the multiple concurrent requests.  However, the 
open-connection limits on the browsers that block more than 6 
concurrent connections to a server is killing me in performance.


I know pretty much nothing about the things you asked about after this 
paragraph.  But I do know about one thing that might help you -- 
HTTP/2.  It can execute many HTTP requests simultaneously over a single 
TCP connection.


Here's a demo:

https://http2.akamai.com/demo

I really have no idea how to enable HTTP/2 in Tomcat, but I know that it 
can be done if you have a new enough version. Browsers with HTTP/2 
support should automatically multiplex requests to the same server over 
fewer connections.


Another possibility is to put your web services behind a load balancer 
that supports HTTP/2 on the front end.  Haproxy is one that I am aware 
of with that support.


HTTP/2 promises to obsolete many of the development and software tricks 
used in the past for improving web performance. Designing applications 
for performance has typically involved combining code into fewer files 
instead of aspiring to a highly modular application.  With HTTP/2, you 
can split things up without killing performance.


If anything I have said above is wrong, I would appreciate the community 
informing me about my errors.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat manager gui hangs on web-app reload for one web-app not others

[Shawn Heisey]

On 9/20/2018 8:30 AM, Bill Harrelson wrote:
Looking back through my sent folder I realize that I have been 
replying directly to people that posted directly to me instead of the 
list.


I see from message headers that you're using Thunderbird.

In Options/Advanced, open the config editor and change the setting for 
"mail.override_list_reply_to" to false.  You can do this by 
double-clicking on the setting.  That will fix this so that when you 
reply to an Apache mailing list, your reply will go to the list.  Apache 
lists use the Reply-To header to indicate where replies should go.


See this bug for a heated discussion about the problem:

https://bugzilla.mozilla.org/show_bug.cgi?id=1392371

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: End of Support for Tomcat versions

[Shawn Heisey]

On 8/6/2018 7:47 AM, gaurav.kuma...@wipro.com wrote:
> Could you please let us know End of Support/Life for the below mentioned 
> Tomcat Version:
>
> Tomcat 7.0
> Tomcat 8.5
> Tomcat 9.0
>
> We want to use it with RHEL 7.4 OS . Please let me know if you need any 
> further info to answer the query.

Looks like all three of those are still supported.

http://tomcat.apache.org/whichversion.html



I found additional detail.  Probably more than you wanted:

8.0 has reached end of support and is less than two months from complete
end of life:

https://tomcat.apache.org/tomcat-80-eol.html

The Tomcat 8 download page says that end of life has not yet been
announced for 8.5, so 8.5 and 9.0 are still good.

https://tomcat.apache.org/download-80.cgi

The Tomcat 7 and Tomcat 9 download pages don't mention anything about
end of life:

https://tomcat.apache.org/download-70.cgi
https://tomcat.apache.org/download-90.cgi

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: problem in starting tomcat

[Shawn Heisey]

On 6/25/2018 9:20 PM, Prateek wrote:
> My configuration:
> OS:REDHAT 7.5 (64 bit)
> Tomcat: 8.5.31
> Jdk- jdk-11(Early-Access)

+1 to everything else you've been told on this thread.

More stuff inline below.

> When I am trying to start my server I got following error as:
> A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x7fd4f206e28a, pid=2412, tid=2412
> #
> # JRE version:  (11.0+18) (build )
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (11-ea+18, mixed mode, aot, 
> sharing, tiered, compressed oops, g1 gc, linux-amd64)
> # Problematic frame:
> # C  [libc.so.6+0x8128a]  strlen+0x2a

This is saying that the segfault occurred in libc code, specifically the
strlen function.  This suggests that either the JVM is using the libc
library incorrectly, or that the libc library on your system is bad.

> # An error report file with more information is saved as:
> # /localdisk/corefiles/hs_err_pid2412.log

Can you provide the JVM error file above (or the one from the most
recent crash)?

> Saved corefile core.2412

If I had any idea how to interpret coredumps, I would ask for that file,
but I don't.  Maybe Oracle can do something useful with it.

I did find a JDK bug showing a crash in libc, but that bug should be
fixed in Java 9 and later.

https://bugs.openjdk.java.net/browse/JDK-8186817

If the behavior is exactly the same with Java 10, then I would think it
means the system itself has a problem.  My first thought would be that
the kernel or system libraries like libc are broken, but it could also
be faulty hardware.  Can you get this working if you put Java 8 on THIS
system and use that?

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Alias name does not identify a key entry

[Shawn Heisey]

On 6/26/2018 11:42 AM, Mark Thomas wrote:
> On 26/06/18 18:32, Cybulski, Adam M wrote:
>> Can you aim me at a guide to this? The steps I've been following are just 
>> from whatever I've found online. Most of the articles seem pretty dated.
> http://tomcat.apache.org/presentations.html
>
> Look for the TLS generation presentation from the 2016 webinar series.

I don't see anything in that presentation about pkcs12.  I see PEM for
the APR connector and JKS for the java connectors.

Can Java programs like Tomcat use pkcs12 stores created by other tools? 
I know how to use openssl to create a pkcs12 file for software on
Windows (typically for IIS, which is used by Exchange).  I use a command
like this:

openssl pkcs12 -export -in www.example.com.pem -inkey www.example.com.key \
  -out examplecert.p12 -CAfile intermediate.pem

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache Tomcat 9.0.8 install check

[Shawn Heisey]

On 6/18/2018 6:15 AM, Shailendra Kumar Verma wrote:
> I am trying to find out through registry checking whether or not Apache 
> Tomcat 9.0.8 is already installed or not. If the below registry is not there, 
> then my program installs Apache Tomcat 9.0.8 installer otherwise it moves on 
> to other installation and completes. It's kind of prerequisite check program.
>
> HKEY_LOCAL_MACHINE\SOFTWARE\Apache Software Foundation\Tomcat\9.0\Tomcat9
>
> However, everytime program just proceeds to install Tomcat despite it is 
> already installed? Am I checking wrong registry?

Do you think it could be possible that the tomcat installer and your
checking program are different architectures?  A 32-bit Windows program
will see a different registry location for HKEY_LOCAL_MACHINE\SOFTWARE
than a 64-bit program will.  A 64-bit program can see the 32-bit
location as HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node.

https://en.wikipedia.org/wiki/WoW64
https://en.wikipedia.org/wiki/Windows_on_Windows

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Amazon EC2 Tomcat 7.0.85 not starting up due to some memory issue .Please mask if

[Shawn Heisey]

On 5/16/2018 11:13 AM, Kiran Badi wrote:
> Yes tomcat is not starting up. I am also suspecting that EC2 instance was > 
> probably compromised. Not sure as how but I see some rogue programs
were > running under tomcat user. I use putty with private keys to login
and those > keys are not in public view for sure. > > These program were
talking to some servers based out of China,Russia and > Germany with
tcp,http and stratrum-tcp protocol with jsonp as data exchange > formt.
I am not sure as how they got access to my ec2 instance and got >
themselves installed. > > I did some initial analysis on this one and
have put those files in my g > drive which I have made public. I suspect
either they have used tomcat to > gain access or they might have used
yum updates for getting access to ec2 > instance.
Because the evil software is/was running as the tomcat user, it is
likely that a vulnerability in Tomcat or a vulnerability in the
application(s) you're running in tomcat was the entry point.  Your logs
may provide clues, but it's also possible that information about exactly
how they broke in isn't available.

Information in the jwzckuz.cf file you provided indicates that this is a
crypto-mining program for the monero crypto-currency.  They're using
your system resources to mine currency for themselves.

The Java Hotspot warning you received during startup indicates that Java
was not able to allocate memory from the operating system.

The information in the hotspot error log (near the end, from
/proc/meminfo) says that this machine only has 1GB of total memory, and
that at the time of the crash, 899240KB of that was actively being
used.  There wasn't enough memory for Java to allocate what it was being
asked to allocate.

Depending on how much memory the programs added by the attacker are
using, killing them might allow Tomcat to start up.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to update new JDK version in apache-tomcat-7.0.84 windows service

[Shawn Heisey]

On 4/18/2018 10:18 AM, Somu Sundar Reddy.Y wrote:
> Hi,  Recently I installed Tomcat windows service using 
> apache-tomcat-7.0.84.exe  installation file and during installation, I was 
> using JDK 1.8 Update 162 in my computer.  With this installation file , 
> windows service is automatically created and am able to start and stop the 
> service without any issues.
>  Today I installed JDK 1.8 Update 172 and modified JAVA_HOME   environment 
> variable to refer newer JDK version and uninstalled the  previous JDK version 
> ( i.e JDK 1.8 update 162).
> After previous JDK uninstallation, I am unable to start the tomcat windows 
> service using new JDK  version and I can see that   windows service is still 
> pointing to older JDK version.
> Could you pl let me know how can I update new JDK version in order to start 
> Tomcat service?

I found this:

https://serverfault.com/a/226697

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: ClassNotFoundException: org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory starting in 7.0.86

[Shawn Heisey]

On 4/17/2018 10:25 AM, Adam Rauch wrote:
> According to the tomcat70 GitHub mirror, a recent change to
> Constants.java switched DBCP_DATASOURCE_FACTORY to
> "org.apache.tomcat.dbcp.dbcp2.BasicDataSourceFactory", which seems
> suspicious. See
> https://github.com/apache/tomcat70/commit/08b7ca0fae77063202d82e719eb35e4eda881bbf

I've recently been asking the list about pool configurations in Tomcat
7, and I've gotten an education about it as a result.

The info I've received says that Tomcat 7 uses dbcp 1.4.  Version 8
updated to dbcp2.

To me, that looks like an incorrect change for version 7.  I think the
line in the change with the default class name for the factory should be
reverted.

You should be able to work around the problem by defining a factory in
your pools set to "org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
so that the default is not used.  This addition should only be made on
configs for Tomcat 7.

That commit says it was prep work for BZ bug number 50019.  I have put a
note on that bug about this.

https://bz.apache.org/bugzilla/show_bug.cgi?id=50019

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Getting all JDBC configs from the context

[Shawn Heisey]

On 3/30/2018 11:38 AM, Christopher Schultz wrote:
> No. In Java, the "class" is defined by the ClassLoader (which loaded
> it) plus the fully-qualified class name. It's entirely possible in

The subtleties of classloader-related problems make my head hurt. :)

> If objects a and b were instantiated using different ClassLoaders
> (which independently loaded the class). This would only happen if you
> did this:
>
> 1. Copy tomcat-dbcp.jar into your app's WEB-INF/lib directory
> 2. Let Tomcat create the DataSource (loaded from its ClassLoader)
> 3. Instantiated a new DataSource yourself (loaded from your webapp's
> ClassLoader)
>
> You'd have to take care to build against Tomcat, but deploy without
> adding Tomcat JARs to your application.

I don't know how much I can influence the deployment process, but if I
see any issues, I will discuss it with development.  Issues will
probably be apparent pretty quickly, though.  I am generally aware that
it's a bad idea to have the same jar (especially if they're different
versions) loaded more than once.

Would I be right in thinking that subtle problems will either cause
compile errors or throw ClassCastException, with the latter being much
more likely?

> I would just search Google for "how to use java executor services" or
> something similar.

Thanks for the tip.

> It's more fleshed-out, but I still think it's more code than
> necessary. You also have the problem that if you want to shut-down
> your application, Tomcat will claim you have a memory leak because
> that thread will continue to run during the 5-minute delay.

I thought I had ensured that if the runflag goes false, the thread
should only run for a maximum of one second before it exits.

So I looked over the code again.  The runflag is checked on every loop
of the 'wait' code, which only sleeps one second.  The time required for
program flow (if, while, try/catch, calculations, etc) should be too
small to worry about.  The runflag is also checked on the outer loop.

Because I placed the wait at the beginning of the outer loop, the thread
will output logs just before it quits.  That's easy enough to fix.

> The point is, yes: you can do all of this and it will work. But if you
> use existing code, you can rely on code that is probably
> better-written and maintained (no offense intended) and focus your OWN
> code on what you NEED to get done, instead of mixing it all up with
> the plumbing of waiting, launching threads, checking for the "done"
> flag, etc.

If I were doing a complex multi-thread highly concurrent task, I would
probably be a lot more concerned about the plumbing I'm using, and
looking to let other libraries handle almost everything.  This idea's
simple.  One thread that spits a few log lines every five minutes.

I'm occasionally guilty of premature optimization when I come up with a
code design... spending hours or days thinking about how to best write
some bit of code that makes the whole idea tick.  But I *TRY* not to
engage in that activity.  Sometimes a paper plane can do the job
wonderfully, and the super spy jet that's invisible to radar and breaks
the sound barrier will take too much time and effort.  If you're an
aerospace engineer, you might have a different opinion. :)

I almost used a ten millisecond value for the Thread.sleep call in the
thread's inner loop.  But then I thought about the fact that it would
run that loop one hundred times more frequently, and decided that a
longer sleep value was probably a lot more efficient.  Nobody's going to
care if they have to wait a few hundred more milliseconds before they
get another series of logs.  Is that a case of premature optimization? :)

I do tend to agree that letting somebody else bash their head against
the difficult problems and come up with optimized solutions is a great
idea.  No reason to re-invent every wheel you need -- third-party
dependencies and well-written features built into the language or the
core libraries are a lifesaver.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Getting all JDBC configs from the context

[Shawn Heisey]

On 3/29/2018 10:00 AM, Christopher Schultz wrote:
> I don't bother with any of that garbage. I use Tomcat's Manager
> application and the JMXProxyServlet. It's an HTTP-to-JMX bridge, so
> your client just has to speak HTTP.

I'm not sure that the manager application is active on our install.  The
load balancer doesn't mention it, and I haven't seen it in the
Apache/AJP config either.  I think that in order for it to be enabled
and NOT be a potential security risk, access to it would need to be
strictly controlled.  I tried accessing /manager paths in various ways
(even direct to tomcat, bypassing haproxy and apache) and none of those
requests worked.  I didn't find any immediate evidence that it's enabled
under another context path either.  But I am the first to admit that I
am not very familiar with configuring Tomcat!

> You may also run into ClassLoader problems where BasicDataSource (as
> loaded by your ClassLoader) != BasicDataSource (as instantiated by
> Tomcat).

Are these problems mitigated by my usage of the fully qualified class
name in the object declaration and the cast?  That part of the code
actually came from one of our developers ... I probably would have
imported the class and used the bare class name.  Maybe I should use
getCanonicalName instead of getName on the DataSource object.  When I
can finally run the code for the first time, I'll have a better idea of
whether I need to make any changes.

> Have a look. It will probably be (slightly) more efficient than the
> code you wrote, and hey, you'll have less code to babysit. It also
> gives you trivial "shutdown" semantics...

I will do my own searching later, but do you happen to know of some good
documentation that covers the basics of implementing ExecutorService?

> ServletContextListener is the way to go. In fact, you should
> probably be using a ServletContextListener to *launch* this thread in
> the first place. So, in your init() method, create+launch the thread
> (or ExecutorService), and in the destroy() method, shut it all down.

You may have noticed the new reply where I included a paste for new
code, using ServletContextListener.  It's now *complete* code, instead
of a fragment.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Getting all JDBC configs from the context

[Shawn Heisey]

On 3/28/2018 1:18 PM, Christopher Schultz wrote:

Don't forget to terminate the thread (or ExecutorService) when the
application is shutting-down, of you'll have a ClassLoader (and a a
huge memory) leak.


Here's a new paste, that I think addresses the thread leak problem.

https://paste.apache.org/rqaX

I've created a listener class that handles startup and shutdown.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is the thread status of tomcat 8.0.44 normal?

[Shawn Heisey]

On 3/28/2018 6:28 PM, 이의준 wrote:

Test sequence and inquiry contents

1. In the local test, the same load (hp-jmeter) for tomcat 7, 8 5 minutes,

2. Thread dump generated after 5 minutes of load termination

3. Most Thread in Tomcat 7 is in TIME_WAITING state (normally OK)

4. Thread in Tomcat 8 mostly RUNNABLE state (estimated to be abnormal)

  Thread Final Stack - Below (reading something)

  at org.apache.tomcat.jni.Socket.recvbb (Native Method)


I'm not an expert by any means.  I have only an elementary understanding 
of what the combined information on a thread dump says about system health.


Presumably if the server is not busy, most of the threads will be idle, 
waiting for something to do.  Whether threads that are "idle" will show 
up as runnable or waiting will depend on the *exact* part of the code 
that the thread is running at the moment of the dump.


I would imagine that if a thread can be kept in a runnable state instead 
of a waiting state while it is doing "nothing" and waiting for work, 
that thread will probably respond faster when the work actually 
arrives.  If the thread is in a waiting state, Java probably has to 
explicitly tell the thread to wake up before it will begin processing.


Looking at your actual stacktraces, I see that the Tomcat 8 thread is 
sitting at "org.apache.tomcat.jni.Socket.recvbb(Native Method)".  Tomcat 
7 is sitting at "sun.misc.Unsafe.park(Native Method)", and indicates 
that it is waiting on a lock object -- it says "parking to wait for  
<0x83769178> (a 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)" 
right after the first line of the stacktrace.


Documentation about the "recvbb" method says that this is the method 
that receives a block of data from the network.  And the stacktrace says 
that it is a "native" method -- which I am reasonably certain means that 
it is compiled native code for the CPU that isn't actually part of 
Java.  If I'm correct about that, it means that the "do nothing" part of 
this code is completely outside Java.  As far as Java is concerned, that 
thread is running, not waiting, because the pause is not being handled 
by Java.


For Tomcat 7, the thread is running Java code only, and it is using 
Java's locking mechanisms to keep the thread in a "paused" state.  Java 
actually knows that the thread is waiting.  The Unsafe.park method IS 
native code, but it is native code that's actually part of Java, so Java 
is able to detect precisely what it's doing.


The native code is more efficient than the Java code.  When a block of 
data finally arrives from the network, the native code can process that 
data very quickly and return control to Java on a thread that is 
*already* in a runnable state.  The Tomcat 7 code must go through the 
process of releasing and acquiring a lock before that thread can resume 
executing.  The time difference on a single request would be 
undetectable by a person, but for systems with high utilization, that 
difference could affect how much hardware is required to handle a 
certain request load.


Native libraries are available for both Tomcat versions.

https://tomcat.apache.org/tomcat-7.0-doc/apr.html

If you were to install/activate the native library for Tomcat 7, I bet 
that version would have very similar thread dumps to Tomcat 8.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Getting all JDBC configs from the context

[Shawn Heisey]

On 3/28/2018 1:18 PM, Christopher Schultz wrote:
> > I would like to write a logging thread that can get ALL of the
> > datasource objects from the context, and for types that it knows,
> > cast them to the appropriate object to log the active/idle
> > connection counts.
>
> It might be easier to do this via JMX.

Perhaps.  But then I have to figure out how to get the program started
with the sysprops that enable remote JMX.  The system has no GUI, and we
don't have any X-Windows infrastructure, so running JDK tools locally on
the Linux server is challenging.  Not impossible, just requires me to
set up things that I don't already have, to do X forwarding over SSH. 
With the logging thread idea, we just need to update code, recompile,
and redeploy.

> Since you already have a compile-time dependency upon Tomcat, there's
> no need to do class-name matching. Instead, just use "instanceof" as
> before:

I did the code that way so that I can handle any type of DataSource
object, even if we change our pool configurations, just by adding
stanzas to the switch.  And the default case logs the class name if it's
not handled explicitly, so I know what to add or change.  I went looking
for a way to write a switch statement that utilizes instanceof, didn't
find anything.

> Instead of running a single thread that goes to sleep, maybe consider
> using a ScheduledExecutorService with a simpler runnable object that
> just logs the status a single time.

Sounds like a good idea.  That I have absolutely no idea how to
implement.  I used the hammer I know. :)

> Don't forget to terminate the thread (or ExecutorService) when the
> application is shutting-down, of you'll have a ClassLoader (and a a
> huge memory) leak.

I did think about that, but I wasn't sure how to detect a shutdown.  Is
there something available that handles it easily and reliably with only
a small code change?  Just now, I found the following page when I went
looking for how to detect that (independent of the JVM shutting down):

http://www.deadcoderising.com/execute-code-on-webapp-startup-and-shutdown-using-servletcontextlistener/

I think I could use this to create the thread on application startup as
well as making sure it shuts down cleanly.  It would mean that I can
just create a new class for one of our applications and edit a config
file, instead of monkeying with other people's code.

It's very encouraging to me that you picked up on the lack of shutdown
handling.  It means I've got more than a sliver of your attention!

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Getting all JDBC configs from the context

[Shawn Heisey]

This is what the code in our application looks like that gets a
datasource object from the context:

  Context initContext = new InitialContext();
  Context envContext  = (Context)initContext.lookup("java:/comp/env");
  DataSource ds = (DataSource)envContext.lookup("jdbc/REDACTED");

I would like to write a logging thread that can get ALL of the
datasource objects from the context, and for types that it knows, cast
them to the appropriate object to log the active/idle connection counts.

This paste is the idea I've thought of for initializing and creating
this logging thread:

https://paste.apache.org/QcbE

The "log" variable is an slf4j Logger object.

Does that code look like it'll work, if we run it once at system
startup?  It doesn't have any errors in my IDE.  Which is not always an
indication of success!

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/27/2018 11:03 AM, Phil Steitz wrote:
> Not exactly, if what you are using is the DBCP pool.  To see the

The factory in use right now is
"org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory".  Information
gathered previously in this thread told me that this is DBCP code,
repackaged into the tomcat package space to avoid conflicting with the
official commons-dbcp package.  If this is effectively unmodified DBCP,
then what you wrote below likely applies.

> details of what is going on, look at the removeAbandoned code in
> DBCP's AbandonedObjectPool.  It calls
> o.a.c.pool.GenericObjectPool#invalidateObject, which calls
> o.a.c.dbco.PoolableConnectionFactory#destroyObject to close the
> connection.  If an exception occurs, it is swallowed by
> removeAbandoned, but it dumps a stack trace.
>
> So connections should in fact be closed if they are detected as
> abandoned.  As I said on commons-user, in your setup, that won't
> happen unless borrows are attempted when there are 57+ open
> connections.  The removeAbandoned method is called *only* by
> borrowObject in DBCP 1.x, with this test:

It would not surprise me to learn that on the pool with maxActive=60,
the pool is managing less than 57 connections.  Until I can get some
logging in place on production to show me the acive and idle connection
counts, I do not know what's actually happening.

If I can successfully get a config using
"org.apache.tomcat.jdbc.pool.DataSourceFactory" operational (which seems
to be failing right now with "too many connections" on startup), is it
true that this pool does not require maxActive-3 connections before
abandoned removal kicks in?  The description for
abandonWhenPercentageFull in the tomcat documentation implies that this
is the case.

So now I have one person telling me that removeAbandoned DOES close
connections, and another saying that it does NOT close connections.  Is
there a conflict here, or are both of these statements correct for
different pool implementations?  Keep in mind that the current config
uses "org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory" and that I
would like to try a new config with
"org.apache.tomcat.jdbc.pool.DataSourceFactory" ... unless there is a
good reason to NOT use that factory.  Which, by the way, is the factory
that tomcat's documentation SAYS to use.

The new config with the new factory fails to start on our staging
server, but aside from seeing "too many connection" exceptions in the
catalina log, I have no concrete information about what happened.  I
didn't do the restart, and wasn't watching the system when it happened. 
Switching back to the old config fixed the tomcat startup.  What the
developer described to me shouldn't have been possible.  I need to do
the restart myself and watch the system.

I've come up with another question, related to this, but headed in a
different direction.  I'll write a new email to the list for that.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/26/2018 2:39 PM, André Warnier (tomcat) wrote:
> Just a question, more to satisfy my curiosity : when you have these
> hundreds of "pending" connections, in what state are they, TCP/IP-wise ?

Not sure where you got "pending".  I don't recall mentioning anything
like that.

The TCP state is ESTABLISHED.  Entries in the TCP table on port 3306
with the ESTABLISHED state is how we have Xymon keeping track of the
connection count.

Slightly redacted screenshot:

https://www.dropbox.com/s/4yj8y2hzfrnr2aj/xymon-port-monitoring-mysql.png?dl=0

It's not a low-level networking issue.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/26/2018 11:28 AM, Christopher Schultz wrote:
> The pool doesn't kill abandoned connections. It simply removes them
> from the pool. Otherwise, you're right: you'd have torches and
> pitchforks everywhere.

That is a key piece of information.  And it should have perhaps been
obvious from the description, but it wasn't.  I have been thinking this
whole time that when abandoned connection removal is triggered, the
connection was closed as well as removed from the pool.

If the removal is literally just "take the object out of the pool" then
it sounds like that removal isn't going to affect the application USING
the connection either -- so the customer probably doesn't have any
reason to sharpen their pitchfork.  If I'm right, the connection may get
removed from the pool, but as long as the query doesn't throw an
exception, JDBC is still going to plug along and return results just fine.

Is there any facility in the pool that can actually close an abandoned
connection?  Yes, I am aware that ultimately the responsibility for that
is ours, and represents bad programming.  Just hoping the pool can make
the problem invisible to users.

Just to ask, since we're clearing up misconceptions ... when the idle
eviction thread evicts a connection, is that connection closed?  At the
moment I don't think we have idle eviction configured, but it's
something I want to add.

> Also, set logAbandoned="true" and you'll get a helpful message every
> time a connection is considered abandoned and you'll find out if you
> have a connection leak (as opposed to simply a too-short "abandoned"
> setting).

I will see about getting this into the existing config.  And brace for
the flood of log messages. ;)  I have added it already to the new config
I'm working on.

> It's probably happening, just not meeting your expectations. Those
> abandoned connections will pretty much live forever, no longer being
> managed by the pool and yet still counting as being used by the
> server. Maybe lower your idle-timeout on the server to help with this.

Yep, I've already got the gears turning to get that lowered.

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/25/2018 3:15 AM, Olaf Kock wrote:
* Liferay comes (optionally) bundled with Tomcat to ease installation, 
however, the tomcat in there will be your own and is up to you to 
upgrade. Yes, new versions of Liferay will come with new versions of 
Tomcat, but new versions of Liferay won't be released because a new 
version of Tomcat is available. Running on an old Tomcat is your 
decision, not Liferay's. And the tomcat committers have done a great 
job providing drop-in-backwards-compatible newer versions. Of course, 
you'll need to validate, but the bundled version is no excuse to not 
upgrade Tomcat


Thank you for your response!

I know that the Tomcat version is basically ours to use or replace.  
Upgrading Tomcat may be perfectly safe for Liferay itself, but we cannot 
be sure that it is completely safe for the applications we've written 
using Liferay.


Qualifying an upgrade of a major component is a very slow and 
time-consuming process.  It's only been fairly recently that we have 
concluded the problem MIGHT be in Tomcat.  So we have not been pursuing 
an upgrade.  Before I start that process, I want to determine whether 
our problem is caused by configuration or a bug.  I always assume 
configuration first.  And if it is a bug, I always assume it's OUR 
software that has the bug, because open source projects are quite good 
at killing bugs early in a version's lifetime.


Part of the challenge is that this has only become my responsibility 
recently, so I'm not completely up to speed on how everything is done.  
We've had key personnel find their dream jobs and move on.


Many of our sites are still using Liferay 6.1.  The customers on those 
platforms are so risk-averse that they haven't allowed us to migrate 
them to 6.2 yet.  We want to get everything upgraded to the latest 
Liferay version, but the upgrade from 6.1 to 6.2 was very challenging.  
We expect an upgrade to 7.x to be more difficult.


I'm here because I need help with the configuration.  I can't seem to 
get a straight answer as to precisely how we SHOULD configure the pools, 
and whether the Tomcat version we are using is capable of handling the 
configuration.  I know that the configuration we have does not seem to 
actually be working.  I do not know whether that's because the 
configuration is wrong, or if there's another problem.  I am working on 
a new configuration that I want to try, but the system doesn't work when 
we try to use that config.  I'm waiting for permission to try the change 
myself so I can have concrete information about what's happening.


If somebody can give me evidence to assure me that upgrading Tomcat and 
using configuration X will solve the problem, then I will pursue that.  
I can't justify an upgrade because it MIGHT fix the problem.


* Liferay's default configuration (as configured through the setup 
wizard) configures the database with an internal connection, not using 
an appserver-configured pool. Please confirm that you manually 
configured Liferay for the appserver-provided pool.


We are using Liferay's configuration in portal-ext.properties for the 
Liferay database.  The pools configured in tomcat are used for our 
application, not Liferay itself.  The only complaint I have about 
Liferay's pool is that it seems to use more connections than I think 
should be necessary. But it IS re-using the pool (all connections are 
idle less than one minute), so it's not on my radar at the moment.  The 
connections in the tomcat pools are NOT being re-used like I expect them to.


Can I configure additional pools in portal-ext.properties for our 
application to use?  If so, is that recommended?  That will probably 
require code changes, but if it solves our difficulties, I'm not opposed 
to it.  We already need to touch all of that code anyway to adjust how 
we're closing JDBC resources.  Maybe we should find a new way to handle 
connection pooling.


* It might help looking at the pool/connection states through JMX, so 
that you can determine which pool is active on which size.


Does JMX work for pools other than the one for Liferay?  As soon as I 
can work out exactly where to add the remote JMX properties, I can give 
that a try.  If it's easy for you to tell me best practices for 
adjusting the java commandline for startup, please do.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/24/2018 5:04 PM, Mark Thomas wrote:

Regarding your configuration:


Generally, that looks OK but I'd strongly recommend that you use
"autoReconnect=false" in the URL. autoReconnect is known to be
problematic with connection pools.

The removeAbandonedTimeout looks low but if all the queries are expected
to be well under 30s then that should be OK.


Somehow I did not see this part of your email at all when I read it the 
first time.  I just noticed it.  My previous reply probably has you 
scratching your head a little bit.  I'm sorry about that!


The timeout of 30 seconds is EXTREMELY low.  And if it were being 
honored, we would have customers lining up outside the office with 
pitchforks.  The webapp has reporting ability for users with elevated 
privileges, and a lot of those reports take a minute or two to run, 
sometimes even longer.  So if the pool were killing connections after 30 
seconds, the reporting mechanism would be failing a LOT.  If you check 
the *planned* configuration, you'll see that I have increased this 
timeout to 3600.  I wanted to make it 900, but some of the developers 
are worried that 900 is too aggressive.


Because there are no pitchforks, I know that abandoned connection 
removal is NOT happening, even though it IS configured.  Plus, there's 
the fact that there are MANY connections with idle times of several 
hours.  So that's a big part of what I'm trying to figure out -- why the 
configuration we have isn't doing what it has configured.  Is there 
something incorrect, or something missing?  Or is it maybe a bug in the 
pool code that's been fixed sometime in the last five years?


The autoReconnect setting has likely been in use on our configurations 
for YEARS.  I will make sure we remove it in any config changes.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/24/2018 5:04 PM, Mark Thomas wrote:

There are two pools available.



org.apache.tomcat.jdbc.pool.DataSourceFactory is a different pool
developed in the Tomcat project (generally called JDBC pool).


OK, so that means that the currently active config is using dbcp.  The 
*new* config that I'm building is using the tomcat pool, with the 
factory specified in the Tomcat JDBC pool documentation.  I do have at 
least one of the tomcat-specific configuration options in that config 
I'm building.  See the initial message on the thread for examples of 
both configurations.


Which directs me back to the second question I asked in the initial 
message:  Why does the application on the staging server appear to open 
a HUGE number of connections on startup when configured with the tomcat 
pool (which overloads the server and makes startup fail), and doesn't do 
that when configured with dbcp?  The number of connections that I am 
told were opened (I wasn't involved with the restart and can't confirm 
what they said) is far more than should have been possible.  The configs 
are fairly similar, but I have a lot more options on the new config, 
designed to try and keep the connection count down.


I should figure out which version of the MySQL driver we have.  I know 
for sure that there have been bugs in older releases and I have had 
trouble when not using the latest.


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/24/2018 3:34 PM, Christopher Schultz wrote:

Before we go too far, you have said:

1. You have 5 prod servers
2. They have several pools defined
3. The above is an example of a defined pool

Just above, that configuration says maxActive=60. 5 * 60 = 300
connections. And that's just for one pool.

Perhaps you simply have too many total connections when you add-up all
the possible connections your servers can make to your db server?


I'm aware of that, and the pool sizes probably are too big, but based on 
what I see when I look during the problem, I don't think it's what I 
need to be worried about right now.  If everything were working the way 
I would expect, each server would only need maybe a dozen connections at 
any given moment for ALL the pools, because connections would be re-used 
and new ones would not be needed.


Here's a couple of SQL queries showing the current connection state on 
the main server right now (on Saturday, a very low traffic period):


https://paste.apache.org/yTNp

This is slightly redacted.  I did the redaction so that the columns 
weren't broken.  There are 414 connections right now, which you can see 
from the second query.  Only four of them are not in the Sleep state, 
which you can see from the first query. Two of those four are slave 
servers, one of them is the event scheduler daemon, and one of them is 
the query I'm running right at that moment.  Which means that NONE of 
the webservers was doing anything on the database when I ran those queries.


The two webservers running Liferay 6.2 are fourqueens and fitzgeralds.  
The three running Liferay 6.1 and the legacy application are fremont, 
frontier, and fiesta.  (Yes, they are casino names.  We inherited the 
system with that naming convention, and we have continued it.)


When we reach 600 connections, maybe 5 of the connections are actually 
doing something.  The rest are showing "Sleep", many with large idle 
times.  Which is why I think each server should have maybe a dozen total 
connections open.  Unless the code is opening connections and not 
closing them.  For that, I'm trying to get the developers to fix the 
code, and also configure abandoned connection removal, so the pool can 
clean up after bad development.


The databases with names that start with "lr" are liferay databases.  
Those connections are using c3p0.  There are more connections active 
than I think SHOULD be there, but they all have VERY low idle times, so 
they ARE getting re-used.  I have plans for that config separate from 
what we're discussing here.



This is the Tomcat documentation I'm using as a reference:

https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html

That's the wrong pool. You are using DBCP, and the docs are for
Tomcat's (non-DBCP) pool. The configurations are roughly the same for
simple configs. But make sure you are reading the right docs for the
right pool.


The factory we have now is 
"org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory".  The factory in 
the documentation, and what I put in the configuration I'm building, is 
"org.apache.tomcat.jdbc.pool.DataSourceFactory". These are both in the 
tomcat package space.  Yet you're telling me that I'm not using the 
Tomcat pool.  How is anybody supposed to come to that conclusion on 
their own?


If the attributes on that page cannot be used with 
"org.apache.tomcat.jdbc.pool.DataSourceFactory" ... then why are they 
included on a page that says to use that factory?


By the way, the 9.0 version of the docs still says virtually the same 
thing.  I found a historical document using Google for the 7.0.42 
version we're running, and it ALSO has virtually the same information on it.


Since I can't rely on the documentation, can somebody please give me a 
configuration that will do what I'm trying to do?  And explain each 
difference from the config that I built, or point me to documentation 
that's complete enough for me to figure it out on my own?


Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

On 3/24/2018 5:36 AM, Filippo Machi wrote:

Hello Shawn, about this question, are you sure that none of the webapps
running on those tomcats are connecting to the database without using the
pools configured in the context.xml? Creating other pools or performing
direct connections? That could explain while changing the configuration you
are not able to limit the number of connections neither to mitigate the
problem with removedAbandoned configuration.


That code isn't familiar to me.  I do have access to it, but there is a 
LOT of it.  I've looked over a little bit of it.


I cannot be ABSOLUTELY sure that this isn't happening, but I very much 
doubt that it is.  All of the developers know that there are connection 
pools provided in the servlet context, so I don't think they would try 
to handle it themselves.  I will ask, and hope they don't bite my head 
off TOO severely. :)


I have already risked death by telling the developers that they're 
closing JDBC resources incorrectly, which I suspect is the root cause of 
our troubles.  I will be pursuing a solution to this with them.


I will be attempting to lower the MySQL server-side wait_timeout from 
eight hours to one hour.  Which is one of those things that shouldn't be 
necessary, but might clean things up when the developer's don't.  I 
don't know if closing the connection on the server side will clean up 
all the objects in the application, or if there might be a resource 
leak.  If there is a leak, then we are already facing it, because 
connections ARE closed at eight hours idle by the server.  But I'd 
rather have that problem than the problem we currently have.


If there is anyone else with ideas, please reply to the original message!

Thanks,
Shawn


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Trying to chase down "too many connection" problems with DB

[Shawn Heisey]

This message is long.  Lots of details, a fair amount of history.

The primary Tomcat version we've got is 7.0.42. Specifically, it is the 
Tomcat that's included with Liferay 6.2. This is why we haven't 
attempted an upgrade even though the version we're running is five years 
old -- we don't want to rock the boat too hard and risk everything not 
working.  Liferay is battle-tested with that version of Tomcat.


Background:

Every now and then, we find that our MySQL master has reached the 
connection limit of 600 connections, and websites stop working, because 
they are trying to open new connections and failing.  When I look at the 
DB server, it shows a HUGE number of connections (300 to 500 out of the 
600 max) that have idle times between one hour and eight hours.  The 
MySQL server-side idle timeout is at the default of 28800 seconds -- 
eight hours.  The idle connections are coming from the webservers.


There are five main webservers, two of which run exclusively Liferay 
6.2, and three that are running older apps on Tomcat version that's 
probably ANCIENT, as well as Liferay 6.1 with Tomcat 7.0.23.  I can't 
seem to figure out what version the really ancient one is.  There are no 
version numbers that I can see in the files in the program directory.  
We also have a staging server, a dev server, and a few ancillary 
systems.  The staging and dev servers mainly use a dev/staging database 
server, but do need to connect to the main DB server for some things.


When we run out of connections, each of these five webservers has nearly 
100 (and sometimes MORE than 100) open connections to the database 
server.  And the majority of them have been idle for a LONG time.


A number of JNDI database resources are configured in conf/context.xml.  
So we have connection pooling.  But even though there are TONS of 
connections already established from Tomcat, and completely idle from 
the DB server's perspective, the application is STILL trying to open a 
NEW connection when somebody visits a page.  This is the precise issue 
that connection pooling SHOULD be preventing.


I've discussed this with the commons-user mailing list, from a DBCP 
perspective.  I started there because there is some different code that 
I wrote, which doesn't run in Tomcat, and uses DBCP2 natively.  I'm a 
lot more familiar with my code than the code running in Tomcat.


Based on my discussions with commons-user, I really think that the 
reason that the DB pools are trying to create more connections even 
though there's already a lot of them open is because the connections are 
actually abandoned.  I suspect they were never closed by the 
application, so the pool still has them as active, and thinks it can't 
use them.  Now that the discussion directly involves configuring pools 
using Tomcat's own DBCP implementation, they're not really able to help 
me any further.


If we are dealing with abandoned connections as I suspect, then I need 
to ask why abandoned connection removal isn't actually working, and how 
to configure it so that it DOES work.


We have figured out how to log the number of idle and active connections 
on the datasource, but it's going to take some time to get that logging 
into the production servers, so I don't have definitive proof that the 
connections are actually active.


Here's a redacted configuration that we have in place for one of the DB 
connection pools on the production servers:


    factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory" 
driverClassName="com.mysql.jdbc.Driver" type="javax.sql.DataSource" 
maxActive="60" maxIdle="10" maxWait="3" removeAbandoned="true" 
removeAbandonedTimeout="30" username="REDACTED" password="REDACTED" 
testOnBorrow="true" validationQuery="select 1" 
url="jdbc:mysql://REDACTED.REDACTED.com:3306/REDACTED?autoReconnect=truezeroDateTimeBehavior=round" 
/>


This is the Tomcat documentation I'm using as a reference:

https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html

Looking at Tomcat 7.0 jdbc documentation, the first thing that I notice 
is that the factory we have configured is not even mentioned in the 
documentation.


One person (who replied privately and not to commons-user) said that 
BOTH factory and type attribute values mentioned in the Tomcat 7.0 docs 
are completely wrong!  And another (who did reply via the list) said 
that the factory mentioned in the documentation is "the alternative 
pool" and that I should be using the one documented here instead.  But 
this page is REALLY lean on details of how to actually set up the resource:


https://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html#Database_Connection_Pool_(DBCP)_Configurations

=

So now, with the sordid past covered, I have three main questions:

1) I think this is the really burning question in my mind: Why is the 
server opening NEW connections when there are dozens of them already 
open?  Does this mean that (as I