Re: Tomcat 9.0.83 - SSL handshake stops working for Google API calls after a while

[Simon Matter]

Hi,

> Hi,
>
> I am looking for help with a strange issue we are experiencing when trying
> to use Google APIs from a web application that is deployed on Tomcat
> 9.0.83.
>
> After a few hours of the server being up and running, all calls to the
> Google APIs fail because of SSL handshake errors. Attaching the SSL logs
> for your reference.

Without knowing exactly how it would look like, are you 100% sure you're
not running out of entropy for some reason?

At least it doesn't hurt to have available entropy in monitoring some how.

Regards,
Simon

>
> I see some differences in the ClientHello message. When the handshake
> fails, all TLSv1.3 ciphers are ignored, there is no "session id" and
> TLSv1.2 is sent as the only supported version.
>
> The Tomcat connector configuration is as follows:
>  protocol="com.precisionsoftware.tomcat.Http11Nio2Protocol" proxyPort="443"
> SSLEnabled="true"
> connectionTimeout="6"
> maxThreads="300"
> minSpareThreads="50"
> acceptCount="250"
> maxKeepAliveRequests="1"
> maxPostSize="-1"
> relaxedQueryChars='[]|{}^"<>'
> enableLookups="true"
> disableUploadTimeout="true"
> URIEncoding="UTF-8"
> compression="force"
> scheme="https"
> secure="true"
> clientAuth="false"
> sslProtocol="TLS"
> sslEnabledProtocols="TLSv1.2+TLSv1.3"
> keyAlias="1"
> keystoreFile="../wildcard_odqad.pfx"
> keystorePass="thepassword"
>
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_128_CCM_SHA256,TLS_AES_128_CCM_8_SHA256"/>
>
> I updated Tomcat to use the most recent native library - 2.0.7 - but that
> did not help. Below an extract from the server log.
>
> 2024-04-11 02:12:47,507 INFO
>  [org.apache.catalina.core.AprLifecycleListener:134] (main) Loaded Apache
> Tomcat Native library [2.0.7] using APR version [1.7.4].
> 2024-04-11 02:12:47,507 INFO
>  [org.apache.catalina.core.AprLifecycleListener:134] (main) APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false], random
> [true], UDS [true].
> 2024-04-11 02:12:47,507 INFO
>  [org.apache.catalina.core.AprLifecycleListener:134] (main) APR/OpenSSL
> configuration: useAprConnector [false], useOpenSSL [true]
> 2024-04-11 02:12:47,514 INFO
>  [org.apache.catalina.core.AprLifecycleListener:370] (main) OpenSSL
> successfully initialized [OpenSSL 3.0.13 30 Jan 2024]
>
> I am not very familiar with the SSL handshake process and do not really
> understand what can make it stop working.
>
> Thanks,
> Marcos
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Core Dump File Generation

[Simon Matter]

Hi,

> Hi,
>
> I am talking about java core dump file which is generating on tomcat/bin
> path and the OS is RHEL 6.

What's the exact version of Java running?

Regards,
Simon

>
> Thanks & Regards,
> Mohit Chaudhary
>
>
> -Original Message-
> From: Simon Matter 
> Sent: Wednesday, February 28, 2024 3:03 PM
> To: Tomcat Users List 
> Subject: Re: Core Dump File Generation
>
> [You don't often get email from simon.mat...@invoca.ch. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Hi Mohit,
>
>> Hi All,
>>
>> We are facing issues on tomcat. Core dump file generating very
>> frequent twice to thrice in a month and core file size would be 13GB
>> to 15GB every time .Whenever this issue is happening tomcat services
>> stopped
>
> I'm a bit confused, are you talking about a UNIX style core file here or
> some kind of dump from Java?
>
> If it's a UNIX style core file then the culprit may be Java and not Tomcat
> - because Java should never ever dump a core file if it's running without
> errors.
>
> Regards,
> Simon
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Core Dump File Generation

[Simon Matter]

Hi Mohit,

> Hi All,
>
> We are facing issues on tomcat. Core dump file generating very frequent
> twice to thrice in a month and core file size would be 13GB to 15GB every
> time .Whenever this issue is happening tomcat services stopped

I'm a bit confused, are you talking about a UNIX style core file here or
some kind of dump from Java?

If it's a UNIX style core file then the culprit may be Java and not Tomcat
- because Java should never ever dump a core file if it's running without
errors.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: EOL - Tomcat versions

[Simon Matter]

> Top posting since my comments are not 100% relevant to the issue in
> the thread (i.e. related but not in detail).
>
> It would be nice if Tomcat published EOL's since there are
> applications (like HIPAA webapps [I do remote cardiac monitoring])
> that are automatically declared to be insecure if the underlying
> platform has any EOL'ed components (this why just upgraded from 9.0.35
> to 9.0.85) and in some cases (like HIPAA) have goverment imposed fines
> if there is a breach due to using EOL'ed components.   Thus there is a
> need for known/published EOL dates in such apps.

Isn't it so that for every major version, like 9.0, all but the latest
should be considered EOL? Like for now, 9.0.85 is supported and 9.0.84 and
older should be considered EOL.

Simon

>
> On Fri, Jan 19, 2024 at 6:58 PM Mark Thomas  wrote:
>>
>> On 19/01/2024 19:06, Francisco Dellanio Leite Alencar wrote:
>> > @Mark Thomas,
>> >
>> > Is it possible to consider that the minimum support time of Apache
>> Tomcat 9.0.X is until 2027 (10 years since Released)?
>>
>> I'd say 2027 is a reasonable estimate of the likely EOL date for 9.0.x
>> but I'm not going to provide any guarantees on that.
>>
>> The Tomcat community has committed to providing at least 12 months
>> notice of EOL of any major version.
>>
>> More detail in the thread listed below against 9.0.x.
>>
>> If long term support is your concern then I'd consider looking at Tomcat
>> 10.1.x. It does require Java 11 (Tomcat 9.0.x requires Java 8) but it
>> will get you an additional ~3 years support.
>>
>> I will take the opportunity to point out that what you get with Tomcat
>> is already pretty good.
>>
>> - major versions support for ~10 years including new features, bug
>>fixes and security fixes
>>
>> - monthly releases throughout that ~10 year period (with the odd gap)
>>
>> - all reproducible bugs reported fixed in the next release (this is the
>>one where Tomcat really stands out)
>>
>> - you can actually talk to the folks the maintain the code
>>
>>
>> If you really need 9.0.x and really need guarantees on dates then there
>> are commercial organizations that will sell you that service. Just make
>> sure you pick one that has the skills and in-depth Tomcat knowledge
>> necessary to deliver that support.
>>
>> Mark
>>
>>
>>
>> >
>> > Thanks.
>> >
>> >
>> >
>> > On 2024/01/08 08:42:28 Mark Thomas wrote:
>> >>
>> >>
>> >> On 08/01/2024 06:47, i...@flyingfischer.ch wrote:
>> >>> https://endoflife.date/tomcat
>> >>>
>> >>> Am 08.01.24 um 07:39 schrieb Deshmukh, Kedar:
>>  Hello,
>> 
>>  Could you please throw some light on Tomcat versions and its EOL
>> plan?
>> >>
>> >> See https://tomcat.apache.org/whichversion.html
>> >>
>>  1.  8.5.X
>> >>
>> >> EOL 31 March 2024
>> >>
>>  2.  9.0.X
>> >>
>> >> No plans.
>> >> See https://lists.apache.org/thread/qlzpscgoqct9wspkj5qjkm34s66jswj0
>> >>
>>  3.  10.0.X
>> >>
>> >> Already EOL as of 31 October 2022
>> >>
>>  4.  10.1.X
>> >>
>> >> No plans.
>> >> See https://lists.apache.org/thread/qlzpscgoqct9wspkj5qjkm34s66jswj0
>> >>
>> >> Mark
>> >>
>> >> -
>> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> >> For additional commands, e-mail: users-h...@tomcat.apache.org
>> >>
>> >>
>> > -
>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> > For additional commands, e-mail: users-h...@tomcat.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>
> --
> Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: EOL - Tomcat versions

[Simon Matter]

> On 19/01/2024 19:06, Francisco Dellanio Leite Alencar wrote:
>> @Mark Thomas,
>>
>> Is it possible to consider that the minimum support time of Apache
>> Tomcat 9.0.X is until 2027 (10 years since Released)?
>
> I'd say 2027 is a reasonable estimate of the likely EOL date for 9.0.x
> but I'm not going to provide any guarantees on that.
>
> The Tomcat community has committed to providing at least 12 months
> notice of EOL of any major version.
>
> More detail in the thread listed below against 9.0.x.
>
> If long term support is your concern then I'd consider looking at Tomcat
> 10.1.x. It does require Java 11 (Tomcat 9.0.x requires Java 8) but it
> will get you an additional ~3 years support.
>
> I will take the opportunity to point out that what you get with Tomcat
> is already pretty good.
>
> - major versions support for ~10 years including new features, bug
>fixes and security fixes
>
> - monthly releases throughout that ~10 year period (with the odd gap)
>
> - all reproducible bugs reported fixed in the next release (this is the
>one where Tomcat really stands out)
>
> - you can actually talk to the folks the maintain the code
>

I'd like to thank the Tomcat community for all what they're doing. I know
a lot of projects but Tomcat is really at the top of the list for all the
things pointed out above!

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Regarding Tomcat is creating the zombie processes

[Simon Matter]

Hi,

I'm not Mark but still try to provide my help

> Hi Mark,
>
> Thanks for the response. For mainly related to our Thingworx IOT-based
> application, we are using the Tomcat 9.0.62 server. So for that, we are
> getting zombie or defunct processes.
>
> "Please provide the steps you used to recreate this issue in a clean
> installation of a standalone Tomcat instance." -> So for this,  We have
> executed our installer, which itself installs the Tomcat server. So, there
> are no manual steps that we ask the user to execute.

What you were asked is to provide full details about your installer.
That's most likely where we could see why you have this issue.

>
> As per the previous response, "A default Tomcat install does not create
> parent and child processes, so zombie processes cannot occur." -> So we
> have raised a support case with Tomcat to just understand the reason why
> defunct processes are created.

I'm not sure you can "raise a support case with Tomcat" as you do with
commercial companies where you pay for support contracts. At least I was
not aware Apache Tomcat provides such contracts, but I may have missed it.

>
> Is anybody already raised case similar to defunct or related to zombie
> processes. If yes, can you please share which resolution you have provided
> to them to prevent creation of those.

Your best bet may be to openly provide more detailed info so we, as a
community, can help you identifying why you see zombie processes.

Regards,
Simon

>
> Thanks,
> Omkar V.
>
> -Original Message-
> From: Mark Thomas 
> Sent: Friday, January 5, 2024 6:00 PM
> To: users@tomcat.apache.org
> Subject: Re: Regarding Tomcat is creating the zombie processes
>
> [You don't often get email from ma...@apache.org. Learn why this is
> important at https://aka.ms/LearnAboutSenderIdentification ]
>
> You will need to provide more details.
>
> A default Tomcat install does not create parent and child processes so
> zombie processes cannot occur.
>
> I'll also note that zombie process do not consume system resources (apart
> from a process ID).
>
> Please provide the steps you used to recreate this issue in a clean
> installation of a standalone Tomcat instance.
>
> Mark
>
>
> On 05/01/2024 09:48, Vaidya, Omkar wrote:
>> Adding information -
>> Tomcat Version - 9.0.62
>> Platform - Linux Platform
>>
>>
>> From: Vaidya, Omkar 
>> Sent: Friday, January 5, 2024 3:15 PM
>> To: users@tomcat.apache.org
>> Cc: Shriwardhankar, Varun 
>> Subject: Regarding Tomcat is creating the zombie processes
>>
>> Hi Team,
>>
>> This is regarding like we have one customer issue where on Linux
>> platform, we have configured our IOT-application (Thingworx), which is
>> using Tomcat as a server.
>> So we are able to identify that even when we remove our application,
>> Tomcat is creating a zombie (defunct) process, which is creating 200+
>> processes under the process table, which ultimately occupy all the OS
>> resources and the application goes in a hung state. This issue is also
>> reproducible on the Standalone Tomcat server also.
>> There are two scenarios mentioned below -
>> 1.If this is relatable to Tomcat can you please suggest any
>> article or documentation so that we can stop zombie process creation, if
>> this is a known issue or there is only way to clear zombie (defunct)
>> process from Processes table of linux.
>> 2.Also, let us know if this is a Operating System specific
>> issue like as it is reproducible only on Linux, So Linux os is the thing
>> that creates the zombie (defunct) processes?
>> we are eagerly waiting for the response.
>>
>> Thanks,
>> Omkar Vaidya.
>>
>>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Posting questions

[Simon Matter]

> Refer to attached screenshot.

Attached screenshot? This is a public mailing list so your best option is
to provide information in posted text format, screenshots and other images
most likely won't make it through in a usable way :)

Regards,
Simon

>
> -Original Message-
> From: Jalaj Asher 
> Sent: Friday, January 5, 2024 8:02 PM
> To: Tomcat Users List 
> Subject: RE: Posting questions
>
> [You don't often get email from jalaj.as...@eclinicalworks.com.invalid.
> Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Omkar,
> 2 questions
> 1. when you say processes what processes are you seeing being invoked and
> does it stop at 200 processes. May be a screen shot might help 2. does the
> tomcat have read write privilege on all its folders ? If not does giving
> those rights help ?
>
>
>
> -Original Message-
> From: Vaidya, Omkar 
> Sent: Friday, January 5, 2024 4:56 AM
> To: users@tomcat.apache.org
> Subject: Posting questions
>
> Attention! - This email has originated from an External Source outside of
> eClinicalWorks. Always use caution when opening attachments, clicking
> links, or when responding to this email. If you feel this is a phishing
> scam, please use the Phish Alert Report button in Outlook.
>
>
> Hi Team,
>
>
>
> Tomcat Version - 9.0.62
>
> Platform - Linux Platform
>
>
>
>
>
> This is regarding like we have one customer issue where on Linux platform,
> we have configured our IOT-application (Thingworx), which is using Tomcat
> as a server.
>
> So we are able to identify that even when we remove our application,
> Tomcat is creating a zombie (defunct) process, which is creating 200+
> processes under the process table, which ultimately occupy all the OS
> resources and the application goes in a hung state. This issue is also
> reproducible on the Standalone Tomcat server also.
>
> There are two scenarios mentioned below -
>
> 1.If this is relatable to Tomcat can you please suggest any
> article or documentation so that we can stop zombie process creation, if
> this is a known issue or there is only way to clear zombie (defunct)
> process from Processes table of linux.
>
> 2.Also, let us know if this is a Operating System specific
> issue like as it is reproducible only on Linux, So Linux os is the thing
> that creates the zombie (defunct) processes?
>
> we are eagerly waiting for the response.
>
>
>
> Thanks,
>
> Omkar Vaidya.
>
>
> CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains
> confidential information belonging to the sender that is legally
> privileged and proprietary and may be subject to protection under the law,
> including the Health Insurance Portability and Accountability Act (HIPAA).
> If you are not the intended recipient of this e-mail, you are prohibited
> from sharing, copying, or otherwise using or disclosing its contents. If
> you have received this e-mail in error, please notify the sender
> immediately by reply e-mail and permanently delete this e-mail and any
> attachments without reading, forwarding or saving them. Thank you.
>
> CONFIDENTIALITY NOTICE TO RECIPIENT: This transmission contains
> confidential information belonging to the sender that is legally
> privileged and proprietary and may be subject to protection under the law,
> including the Health Insurance Portability and Accountability Act (HIPAA).
> If you are not the intended recipient of this e-mail, you are prohibited
> from sharing, copying, or otherwise using or disclosing its contents. If
> you have received this e-mail in error, please notify the sender
> immediately by reply e-mail and permanently delete this e-mail and any
> attachments without reading, forwarding or saving them. Thank you.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Regarding Tomcat is creating the zombie processes

[Simon Matter]

> You will need to provide more details.
>
> A default Tomcat install does not create parent and child processes so
> zombie processes cannot occur.
>
> I'll also note that zombie process do not consume system resources
> (apart from a process ID).
>
> Please provide the steps you used to recreate this issue in a clean
> installation of a standalone Tomcat instance.
>
> Mark
>

As an easy start you could provide us with the Tomcat related process tree
and detailed description of how the lifecycle of Tomcat is managed.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk logging issue

[Simon Matter]

> Hi Rainer/Simon - I've just had another look at this. With no
> application running (IOW, all Java processes killed), I see this
> behaviour:

Sorry, I was confused because I thought we're talking about files from
Tomcat and not from Apache/mod_jk.

Regards,
Simon

>
>   # systemctl start apache2
>
> This create a number of apache2 processes (generally 7). 2 new mod_jk
> files are created, corresponding to the apache2 process with the lowest
> PID
>
>   # systemctl stop apache2
>
> This does not remove any files (but see below)
>
>   # systemctl restart apache2
>
> This has the same effect as a 'start' followed by a 'stop'. A 'reload',
> as expected, doesn't change the PIDs and has no effect on file creation
> or deletion.
>
> During testing, I did see one occasion on which the current mod_jk files
> were deleted. I though this might be a timeout issue, since the restart
> was carried out after 7 minutes, which was longer  that normal. So, I
> carried out 5 more tests, with the restart after 1, 2, 4, 8, and 11
> minutes, and in all these cases the old files were retained and not
> deleted.
>
> Maybe there's a race condition, or something distribution-specific, in
> the code which registers the cleanup?  I can't do much for a couple of
> days, at least, but I'll have a look when I get a minute.
>
>
> On 19/12/2023 19:03, Rainer Jung wrote:
>> Hi there,
>>
>> Am 19.12.23 um 18:05 schrieb EML:
>>> Hi - I'm running mod_jk with an Apache front-end, and I'm having an
>>> issue with the JkShmFile files.
>>>
>>> Every time Apache restarts mod_jk creates two new files
>>> (jk-runtime-status.PID and jk-runtime-status.PID.lock). These are
>>> never cleaned up; the log directory simply fills up with these files.
>>> This happens whether or not I explicitly set JkShmFile in the Apache
>>> conf.
>>
>> That should no happen. There is a cleanup routine registered, which
>> should delete the files during shutdown. And that's the behavior that
>> I normally observe.
>>
>>> Is there some way I can persuade mod_jk to use a single file pair,
>>> without the PID suffix, or to delete the previous file pair on a
>>> restart? I'm not doing any load sharing.
>>
>> If you must remove the PID, you can patch the code and build it
>> yourself. The ID is added in file common/jk_shm.c in the following line:
>>
>>     sprintf(jk_shmem.filename, "%s.%" JK_PID_T_FMT, fname,
>> getpid());
>>
>> You could replace it by:
>>
>>     sprintf(jk_shmem.filename, "%s", fname);
>>
>> If you compile the code yourself, please us the latest version 1.2.49.
>>
>> As I mentioned, a normal shutdown should already remove the files. A
>> reload should not change the pid and thereby the files. A restart in
>> the sense of stop-then-start should also remove the old files.
>>
>>> I'm on Ubuntu 22.04, Apache 2.4.52. The mod_jk version is possibly
>>> 1.2.48-1.
>>>
>>> Thanks.
>>
>> Regards,
>>
>> Rainer
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: mod_jk logging issue

[Simon Matter]

Hi,

> Hi - I'm running mod_jk with an Apache front-end, and I'm having an
> issue with the JkShmFile files.
>
> Every time Apache restarts mod_jk creates two new files
> (jk-runtime-status.PID and jk-runtime-status.PID.lock). These are never
> cleaned up; the log directory simply fills up with these files. This
> happens whether or not I explicitly set JkShmFile in the Apache conf.
>
> Is there some way I can persuade mod_jk to use a single file pair,
> without the PID suffix, or to delete the previous file pair on a
> restart? I'm not doing any load sharing.
>
> I'm on Ubuntu 22.04, Apache 2.4.52. The mod_jk version is possibly
> 1.2.48-1.

Can it be that your application crashes on shutdown? This is usually why
cleanup is not working properly.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: I can't find how to stop TOMCAT during INITIALIZATION phase

[Simon Matter]

Hi,

>
> Our question is:
> 1. It is possible to stop tomcat during initialization phase?
> 2. If yes how and if not are any plans to implement it in future versions?
>
> It seems to me that my solutions for now are:
> 1. sending SIGKILL signal to tomcat (this is very risky to me because
> stopping like this in the middle of something may corrupt data - but this
> situation is any way possible so I have to handle it)
> 2. wait for tomcat initilization procedure to finish and then trigger the
> shutdown since we can do something in our wrapper scripts
> Do you see any other possible solutions?
>

Did you try to terminate the process with SIGUSR1 instead of SIGKILL? I
usually terminate Java processes with SIGUSR1 if SIGTERM is not handled in
time and it still seems to do some cleanup while shutting down.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat9 not listening to ipv4 port 8080, only ipv6

[Simon Matter]

Hi,

> Christoph,
>
> On 11/28/23 09:53, Christoph Kukulies wrote:
>> That was my connector:
>>
>>   >                 connectionTimeout="2"
>>                 redirectPort="8443" />
>>
>> I triednetstat -tulpn as well and it could be seen there was  no
>> listener under ip4 and port 8080.
>
> If you use the "address" attribute, you can pick the interface you will
> listen to:
>
> "
> [address]
>
> For servers with more than one IP address, this attribute specifies
> which address will be used for listening on the specified port. By
> default, the connector will listen all local addresses. Unless the JVM
> is configured otherwise using system properties, the Java based
> connectors (NIO, NIO2) will listen on both IPv4 and IPv6 addresses when
> configured with either 0.0.0.0 or ::. The APR/native connector will only
> listen on IPv4 addresses if configured with 0.0.0.0 and will listen on
> IPv6 addresses (and optionally IPv4 addresses depending on the setting
> of ipv6v6only) if configured with ::.
> " [1]
>
> You have not specified an "address", so you get the default which should
> be "all local addresses". You only showed your lsof output, so I
> couldn't see which interface you had been bound to.
>

Also, it's a question what the interface config looks like *exactly* at
the time when Tomcat was starting up. On systems running systemd it's easy
to get into troubles with the way systemd parallelizes the system startup.
Systemd service units often need additional tuning to result in reliable
startup order.

Regards,
Simon

>
> [1]
> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#Standard_Implementation
>
>>> Am 28.11.2023 um 15:15 schrieb Christopher Schultz
>>> mailto:ch...@christopherschultz.net>>:
>>>
>>> Christoph,
>>>
>>> On 11/28/23 08:26, Christoph Kukulies wrote:
 not that I kew of (changes in JVM arguments). I will try your
 suggestion:
 -Djava.net.preferIPv4Stack=true
 and thanks, it helped:
 I put it into /etc/defaults/tomcat9 (under Ubuntu 22.04)
 JAVA_OPTS="-Djava.awt.headless=true -Djava.net.preferIPv4Stack=true"
 and now I have:
 root@mail:/etc/default# lsof -i :8080
 COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
 java    59579 tomcat   37u  IPv4 579485      0t0  TCP *:http-alt
 (LISTEN)
 root@mail:/etc/default#
>>>
>>> So... is that what you wanted?
>>>
>>> What does your  configuration look like?
>>>
>>> Try using netstat instead of lsof. It will show you the network
>>> interface being used as well as the port number and IP stack type.
>>>
>>> -chris
>>>
> Am 28.11.2023 um 13:58 schrieb Suvendu Sekhar Mondal
>   >:
>
> Hello Christoph,
>
> On Tue, Nov 28, 2023, 5:55 PM Christoph Kukulies
> mailto:k...@kukulies.org.invalid>>
> wrote:
>
>> I'm pulling my hairs on a suddenly occured - possibly -
>> misconfiguration.
>> But I can't find it out:
>>
>> catalina.2023-11-28.log:
>>
>>
>> 28-Nov-2023 13:15:43.742 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Server
>> version name:
>>  Apache Tomcat/9.0.58 (Ubuntu)
>> 28-Nov-2023 13:15:43.743 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Server built:
>>  Jan 6 1970 15:09:28 UTC
>> 28-Nov-2023 13:15:43.744 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Server version
>> number: 9.0.58.0
>> 28-Nov-2023 13:15:43.744 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log OS Name:
>>  Linux
>> 28-Nov-2023 13:15:43.744 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log OS Version:
>>  5.15.0-89-generic
>> 28-Nov-2023 13:15:43.745 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Architecture:
>>  amd64
>> 28-Nov-2023 13:15:43.745 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Java Home:
>>  /usr/lib/jvm/java-11-openjdk-amd64
>> 28-Nov-2023 13:15:43.745 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log JVM Version:
>>  11.0.20.1+1-post-Ubuntu-0ubuntu122.04
>> 28-Nov-2023 13:15:43.745 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
>>  Ubuntu
>> 28-Nov-2023 13:15:43.746 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:
>>  /var/lib/tomcat9
>> 28-Nov-2023 13:15:43.746 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:
>>  /usr/share/tomcat9
>> 28-Nov-2023 13:15:43.758 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: --add-opens=java.base/java.lang=ALL-UNNAMED
>> 28-Nov-2023 13:15:43.759 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: 

Re: Wondering about tomcat-users.xml could not be found

[Simon Matter]

Hi,

> I'm running tomcat9 under Ubuntu 22.04 with an haproxy 2.8 in front of it.
>
> I'm wondering about the following in the logs:
>
> Nov 15 16:19:23 mail tomcat9[832]: Reloading memory user database
> [UserDatabase] from updated source
> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
> Nov 15 16:19:23 mail tomcat9[832]: The specified user database
> [conf/tomcat-users.xml] could not be found
> Nov 15 16:19:33 mail tomcat9[832]: Reloading memory user database
> [UserDatabase] from updated source
> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
> Nov 15 16:19:33 mail tomcat9[832]: The specified user database
> [conf/tomcat-users.xml] could not be found
> Nov 15 16:19:43 mail tomcat9[832]: Reloading memory user database
> [UserDatabase] from updated source
> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
> Nov 15 16:19:43 mail tomcat9[832]: The specified user database
> [conf/tomcat-users.xml] could not be found
> Nov 15 16:19:53 mail tomcat9[832]: Reloading memory user database
> [UserDatabase] from updated source
> [file:/var/lib/tomcat9/conf/tomcat-users.xml]
> Nov 15 16:19:53 mail tomcat9[832]: The specified user database
> [conf/tomcat-users.xml] could not be found
>
>
>
> File /var/lib/tomcat9/conf/tomcat-users.xml is definitely there.
>
> It occurs every 10 seconds.
>
> Don't know who is causing this and why. Permissions? Ownership wrong?
>
> -rw-r- 1 root root   2756 Jan 15  2022 tomcat-users.xml

Is your Tomcat running as root? I hope not, but if it's running as user
tomcat or some other unprivileged user, it won't be able to read your
tomcat-users.xml as long as the user is not member of group root.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [EXTERNAL] - Need help tomcat

[Simon Matter]

> Yes I have deleted them and again I have sent a email with screenshot.
> Please check that.
>
> Regards,
> Deepak

Hi,

please note that attachments are not delivered on this mailing list.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

[Simon Matter]

> Jerry,
>
> On 6/15/23 00:41, Jerry Malcolm wrote:
>>
>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
 Rob,

 On 6/13/2023 11:34 AM, Rob Sargent wrote:
> In /etc/rc.local I have:
>>
>> --
>> sleep 120s
>> systemctl start tomcat9
>>
>> -
>>
>> I put the sleep in back a couple of years ago that for some reason
>> 'fixed' this same random, intermittent crypto file exception.
>>
>>
>
>>> One observation even though the error doesn't show up in the log
>>> until my first db call, I think the real error has to be occurring
>>> during tomcat boot.  If the failure occurs, I continue to get the same
>>> "Can't read cryptographic policy directory: unlimited" even hours
>>> later after boot every time I send another request to TC.  If I then
>>> reboot TC and this time I don't get the failure, that unlimited folder
>>> magically becomes available.  There is something happening in TC boot
>>> with some sort of crypto initialization that either succeeds or
>>> fails.  And the exception when trying to get connections long after TC
>>> boot is just hitting whatever problem occurred during boot.  In other
>>> words, delaying my calls, even for hours, has no effect on accessing
>>> that folder if the problem is present.  Rebooting TC gives it another
>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>> that could lock up that folder?
>>>
>>>
>> I never really found a definitive fix for this.  But after all of the
>> research and logging and everything, I pretty much concluded that
>> there's some kind of timing/race condition between Tomcat boot and Java
>> crypto initialization.  Given that, I figured it was highly unlikely
>> that two different Java JVM implementations would have the same precise
>> implementation code to trigger that race condition.  So as a last ditch
>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>> JVM.  It's circumstantial evidence at best, and running a production
>> environment on fixes that just went away goes against my grain. But if
>> the problem goes away, maybe it won't come back. At this time, when
>> using Corretto JVM, I have not encountered the Crypto directory error.
>> It's been running on all server instances now for almost 24 hours with
>> no problems.  So I guess my suggestion to anyone else that might
>> encounter something like this, trying using a different JVM from a
>> different provider.  That appears at this time to have worked for me.
>
> Yeah, that's certainly a very unsatisfying result, but ... you can't
> argue with results. I'd love to hear if you end up with any further
> information.
>
> -chris

I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?

Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Conclusion - Re: Crypto Randomly Not Getting Initialized

[Simon Matter]

>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
 In /etc/rc.local I have:
>
> --
> sleep 120s
> systemctl start tomcat9
>
> -
>
> I put the sleep in back a couple of years ago that for some reason
> 'fixed' this same random, intermittent crypto file exception.
>
>

>> One observation even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot.  If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC.  If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available.  There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails.  And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot.  In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present.  Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this.  But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization.  Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition.  So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM.  It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error. 
> It's been running on all server instances now for almost 24 hours with
> no problems.  So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider.  That appears at this time to have worked for me.
>

Maybe you could report this as a bug in the affected JVM? If it's really
coming from the JVM then it would be nice to have it fixed.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crypto Randomly Not Getting Initialized

[Simon Matter]

Hi,

> Jerry,
>
> On 6/13/23 11:42, Jerry Malcolm wrote:
>> Simon,
>>
>> On 6/13/2023 2:20 AM, Simon Matter wrote:
>>> Hi,
>>>
>>>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
>>>> Linux2 in a production environment.  A couple of years ago, we started
>>>> getting weird errors that the "Crypto Mechanism" failed to initialize.
>>>> Through a lot of trial and error, and reasons I don't quite remember,
>>>> we
>>>> put a 2-min delay in rc.local before starting Tomcat, and the problem
>>>> went away.  I'm not a Linux nor a crypto guru.  But we traced it to
>>>> some
>>>> crypto file that we assumed was not available until later in the Linux
>>>> boot sequence.  Anyway, the 2 minute delay made it go away, for over
>>>> two
>>>> years.  Then all of a sudden in the last day or so, it's back with a
>>>> vengeance.  It fails with the same crypto error from 2 years ago in
>>>> about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and
>>>> no change.
>>>>
>>>> I need help.  Our whole production environment is unstable now since
>>>> every time an ASG brings a new instance online, I've got a 50-50
>>>> chance
>>>> that tomcat is going to die (and the health check doesn't catch it,
>>>> but
>>>> that's a different issue).
>>>>
>>>> There are no errors in the Tomcat boot sequence logs.  But the first
>>>> time and every subsequent time I try to get a connection from the
>>>> DataSource pool, I get the stack dump shown below.
>>>>
>>>> I figure it has to be a timing/race condition.  But I have no clue
>>>> what
>>>> to do to fix it.  I'm baffled that it worked for two years, and now
>>>> fails every other time I start an instance.  And every instance is
>>>> running copies of the exact same Amazon Machine Image.  The same EC2
>>>> will come up clean 50% of the time the next time it boots.
>>> Could it be that your hosts are running out of entropy on boot?
>>>
>>> Maybe there were RNG related changes in the kernel or rng-tools?
>>>
>>> Maybe monitor available entropy in
>>> /proc/sys/kernel/random/entropy_avail,
>>> it should not go below 100 or so.
>>>
>>> Regards,
>>> Simon
>>>
>> I haven't done any Linux or other system updates in several weeks. I'll
>> look into the entropy possibility.  Would running out of entropy cause
>> an exception stating that a crypto directory doesn't exist?  I don't
>> know much about Java entropy.  Any ideas what would cause entropy to be
>> good on one boot and bad on the next boot?  Thx.
>
> This isn't about entropy. Focus on the actual error message that the
> system can't find the unlimited policy file for some reason.
>
> -chris

Sorry, I didn't read the errors too carefully :)

I don't know Amazon Linux2 but one question, does it have the directory
/etc/crypto-policies? Apart from this, does systemd somehow fiddling with
crypto stuff while booting? Since systemd parallelizes the whole boot
process many things can (and sometimes do) break and it's difficult to
find it out.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Crypto Randomly Not Getting Initialized

[Simon Matter]

Hi,

> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> Linux2 in a production environment.  A couple of years ago, we started
> getting weird errors that the "Crypto Mechanism" failed to initialize. 
> Through a lot of trial and error, and reasons I don't quite remember, we
> put a 2-min delay in rc.local before starting Tomcat, and the problem
> went away.  I'm not a Linux nor a crypto guru.  But we traced it to some
> crypto file that we assumed was not available until later in the Linux
> boot sequence.  Anyway, the 2 minute delay made it go away, for over two
> years.  Then all of a sudden in the last day or so, it's back with a
> vengeance.  It fails with the same crypto error from 2 years ago in
> about 50% of the EC2 boot ups.  I tried bumping the wait to 3 min, and
> no change.
>
> I need help.  Our whole production environment is unstable now since
> every time an ASG brings a new instance online, I've got a 50-50 chance
> that tomcat is going to die (and the health check doesn't catch it, but
> that's a different issue).
>
> There are no errors in the Tomcat boot sequence logs.  But the first
> time and every subsequent time I try to get a connection from the
> DataSource pool, I get the stack dump shown below.
>
> I figure it has to be a timing/race condition.  But I have no clue what
> to do to fix it.  I'm baffled that it worked for two years, and now
> fails every other time I start an instance.  And every instance is
> running copies of the exact same Amazon Machine Image.  The same EC2
> will come up clean 50% of the time the next time it boots.

Could it be that your hosts are running out of entropy on boot?

Maybe there were RNG related changes in the kernel or rng-tools?

Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
it should not go below 100 or so.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: Unable to start application

[Simon Matter]

Hi,

> On 18/03/2023 10:43, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>> Hello,
>>
>>> -Ursprüngliche Nachricht-
>>> Von: Kevin Huntly 
>>> Gesendet: Samstag, 18. März 2023 11:10
>>> An: Tomcat Users List 
>>> Betreff: Re: Unable to start application
>>>
>>> Here are the logs -
>>> https://drive.google.com/file/d/1jBsNaW_bQJ4KcDSvucJ5QWo642He6bgb/view
>>> ?usp=sharing
>>>
>>> The JDBC driver is located under /opt/mysql/, and I added that path to
>>> catalina.properties under the common loader. I did try to move it into
>>> ${catalina.home}/lib, this did not change anything.
>>> 
>>
>>
>> This message looks strange:
>> 18-Mar-2023 06:06:13.305 WARNING [main]
>> org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
>> JAR file
>> [/opt/Apache/Tomcat/apache-tomcat-9.0.73/lib/mysql-connector-j-8.0.32.jar],
>> exists: [true], canRead: [false]
>>
>> It seems that it cant load the jdbc driver from that path.
>> Could you download the jar again from the mysql website and replace it?
>> Can you open/unpack the jar without errors?
>
> More likely a permissions problem. That warning is generated before
> Tomcat tries loading the file. It means a call to java.io.File.canRead()
> returned false.
>
> Mark

Since this is on RHEL, it could also be an SELinux problem where access to
the JAR is denied.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Systemd file and umask for tomcat

[Simon Matter]

> Hello,
> today I was struggling with the umask on ubuntu and tomcat.
>
> The normal way to do it in systemd is:
>
> [Service]
> UMask=0022
>
> Unfortunately, this didn’t work and it took me a while to figure out, that
> Catalina.sh
> is overwriting the umask with 0027 if no env-variable was set.
> So for tomcat, I needed to set:
> Environment='UMASK=0022'
>
> Is there any reason, why tomcat has some unexpected logic which overrides
> the systemd settings?

I think Tomcat uses its own way to do things so that it can do it in all
platforms in the same way. Outside of Linux, nobody is using systemd
anyway :)

Therefore I have this in setenv.sh

# Umask for system reserved uid/gid
if [ -z "$UMASK" ]; then
  UMASK="0022"
fi

Regards,
Simon



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: AW: Having trouble with Tomcat crashes. Interesting memory numbers in Manager

[Simon Matter]

Hi,

> Hello James,
>
>> -Ursprüngliche Nachricht-
>> Von: James H. H. Lampert 
>> Gesendet: Montag, 6. Februar 2023 18:18
>> An: Tomcat Users List 
>> Betreff: Re: AW: Having trouble with Tomcat crashes. Interesting memory
>> numbers in Manager
>>
>> Thanks, Herr Hoffmann. Your questions were most helpful in determining
>> what information to gather and share. And thanks in advance to anybody
>> else who has any insights.
>>
>> First, I will note that the seemingly non-sequitur nursery-survivor
>> numbers
>> aren't just what we see during a crash; they're what we see when it's
>> running
>> normally.
>>
>> On 2/4/23 6:13 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>> > Could you describe "crash" in a bit more detail?
>>
>> Typically, the signed-on users start to get degraded response times,
>> before it
>> becomes completely unresponsive.
>>
>> > - does the tomcat / java process run but is unresponsive?
>>
>> Yes. Exactly. And shutting it down (and therefore freeing up the port
>> for a
>> restart) takes a fairly sizeable amount of time, and leaves a core dump
>> of
>> approximately 6G size, a Javacore dump of approximately 4M size, and a
>> JIT
>> dump of approximately 20M size.
>>
>> > - does the java process crash itself (then there should be a logfile
>> written)?
>> The job does not generally terminate itself, or even respond to a
>> shutdown
>> request; it has to be forcibly terminated (given that it's running on an
>> AS/400,
>> this would typically be either from WRKACTJOB, or from an ENDJOB
>> command, or from their GUI console equivalents).
>>
>> This may be relevant: even when it is not in this state, the Tomcat
>> server,
>> when being shut down, tends not to respond readily to shutdown
>> requests.
>>
>> > - Is there any OOM message in the logfiles?
>> Not out-of-memory, but there are chronic problems with contacting
>> outside
>> web services (many of them involving Oauth2), and with BIRT reporting.
>>
>> Around the time of the shutdown, I typically see stuff like:
>> Unhandled exception
>> Type=Segmentation error vmState=0x
>> J9Generic_Signal_Number=0004 Signal_Number=000b
>> Error_Value= Signal_Code=0032
>>
>> I am not sure whether this is going into catalina.out before or after
>> the job is
>> forcibly terminated.
>>
>> > - Is the process still alive but CPU at 100% ?
>> Yes.
>>
>> We just had a near-miss as I was typing this: CPU pushing up into the
>> high
>> 80s, and the JVM job for Tomcat eating up most of it, but it backed down
>> to
>> something more normal without my having to intervene, and without any
>> sign of anybody else intervening.
>>
>> One of my colleagues managed to get into manager during the near-miss,
>> and took a screen-shot. The "nursery-allocate" Used was at 400.97M
>> (34%),
>> "nursery-survivor" as I described last week, "tenured-LOA" Used was at
>> zero
>> used, and "tenured-SOA" was showing Initial 2918.40M, Total 3648.00M,
>> Maximum 4864.00M, and Used 1997.72M (41%).
>>
>> --
>> JHHL
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>
> The observations looks like java is running out of memory and the garbage
> collector can't keep up with making memory free again.
> Either the GC uses 100% or the application has some cpu intensive
> procedures. I would guess, that it’s the GC.
> One option would be to open a JMX port on tomcat and use VisualVM to
> connect to the java process and inspect the memory and GC usage.
> When the CPU is eating 100% CPU you might also consider generating a
> thread dump (kill -3) and check if there are any suspicious threads
> running.
>
> Also setting the java options HeapDumpOnOutOfMemoryError and HeapDumpPath
> might help, if the process stops because of OOM.
> If the GC can always free some bytes again, which the application is
> instantly eating again, an OOM might not occur.
>
> You can also add parameters to log some GC statistics, but I never used
> that : https://sematext.com/blog/java-garbage-collection-logs/
>
> Greetings,
> Thomas

I agree here, just one little thing to also keep in mind: I guess the Java
VM here is the IBM J9 flavor which can be a little bit different than the
SUN flavor. So configuration options may differ in details.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: tomcat is not coming Up

[Simon Matter]

> Hi team
>
>
> You mean to say, need to use native library version should be version
> 1.2.23??
>
>
> But we have not made any changes in this library is that Possible to  auto
> change in Version?

No, things usually don't change automagically on serious operating systems :)

But, in you old config you had '-Djava.library.path=/usr/local/apr/lib'
and maybe this is where the newer Tomcat native library is installed.
The new config seems to miss this path and it's likely that now Tomcat
uses the older library in one of the standard paths.

Regards,
Simon

>
> Thanks & Regards,
> _
> PrabuGanesan
> Consultant|MS-Nordics
> capgemini India Pvt. Ltd. | Bangalore 
> Contact: +91 8526554535
> Email: prabhu.c.gane...@capgemini.com
>
> www.capgemini.com
> People matter, results count.
> __
> Connect with Capgemini:
>
>  
> Please consider the environment and do not print this email unless
> absolutely necessary.
> Capgemini encourages environmental awareness.
>
> -Original Message-
> From: Simon Matter 
> Sent: 12 January 2023 16:24
> To: Tomcat Users List 
> Subject: RE: tomcat is not coming Up
>
> ***This mail has been sent by an external source***
>
>> Hi Team,
>>
>>
>> This is the only message we are receiving while starting the services.
>> Can some one help on this .
>>
>> 11-Jan-2023 18:21:58.101 INFO [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Command line
>> argument: -Djava.io.tmpdir=/srv/tomcats/tomcat1/temp
>> 11-Jan-2023 18:21:58.101 INFO [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent An older
>> version [1.2.17] of the APR based Apache Tomcat Native library is
>> installed, while Tomcat recommends a minimum version of [1.2.23]
>
> Maybe the problem is that you have Apache Tomcat Native library 1.2.17
> installed/referenced but you should likely have version 1.2.23?
>
> Regards,
> Simon
>
>> 11-Jan-2023 18:21:58.101 INFO [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
>> APR based Apache Tomcat Native library [1.2.17] using APR version
>> [1.4.8].
>> 11-Jan-2023 18:21:58.101 INFO [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
>> capabilities: IPv6 [true], sendfile [true], accept filters [false],
>> random [true].
>> 11-Jan-2023 18:21:58.101 INFO [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>> APR/OpenSSL
>> configuration: useAprConnector [false], useOpenSSL [true]
>> 11-Jan-2023 18:21:58.105 INFO [main]
>> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
>> successfully initialized [OpenSSL 1.0.2k-fips  26 Jan 2017]
>> LunaNamedSystemMutex: open() failed: Permission denied
>>
>> Thanks & Regards,
>> _[Email_CBE.gi
>> f]
>> PrabuGanesan
>> Consultant|MS-Nordics
>> capgemini India Pvt. Ltd. | Bangalore
>> Contact: +91 8526554535
>> Email:
>> prabhu.c.gane...@capgemini.com<mailto:prabhu.c.gane...@capgemini.com>
>>
>> www.capgemini.com<http://www.capgemini.com/>
>> People matter, results count.
>> __
>> Connect with Capgemini:
>> [cid:image003.gif@01D926A1.46D89710]<http://www.capgemini.com/insights
>> -and-resources/blogs>[cid:image004.gif@01D926A1.46D89710]<http://www.t
>> witter.com/capgemini>[cid:image005.gif@01D926A1.46D89710]<http://www.f
>> acebook.com/capgemini>[cid:image006.gif@01D926A1.46D89710]<http://www.
>> linkedin.com/company/capgemini>[cid:image007.gif@01D926A1.46D89710]> tp://www.slideshare.net/capgemini>[cid:image008.gif@01D926A1.46D89710]
>> <http://www.youtube.com/capgeminimedia>
>>
>> Please consider the environment and do not print this email unless
>> absolutely necessary.
>> Capgemini encourages environmental awareness.
>>
>> From: Hiran CHAUDHURI 
>> Sent: 11 January 2023 21:36
>> To: Tomcat Users List 
>> Subject: RE: tomcat is not coming Up
>>
>> This mail has been sent from an external source
>>
>> CONFIDENTIAL & RESTRICTED
>>
>> If on the old command line the apr libs were referenced but on the new
>> one it is not, I believe that could make the process not find the
>> libraries to communicate.
>> Enough of a reason to terminate with an error, but maybe it is on
>> stderr and nowhere in the logs.
>

RE: tomcat is not coming Up

[Simon Matter]

> Hi Team,
>
>
> This is the only message we are receiving while starting the services. Can
> some one help on this .
>
> 11-Jan-2023 18:21:58.101 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Command line
> argument: -Djava.io.tmpdir=/srv/tomcats/tomcat1/temp
> 11-Jan-2023 18:21:58.101 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent An older
> version [1.2.17] of the APR based Apache Tomcat Native library is
> installed, while Tomcat recommends a minimum version of [1.2.23]

Maybe the problem is that you have Apache Tomcat Native library 1.2.17
installed/referenced but you should likely have version 1.2.23?

Regards,
Simon

> 11-Jan-2023 18:21:58.101 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.17] using APR version [1.4.8].
> 11-Jan-2023 18:21:58.101 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false], random
> [true].
> 11-Jan-2023 18:21:58.101 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL
> configuration: useAprConnector [false], useOpenSSL [true]
> 11-Jan-2023 18:21:58.105 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.0.2k-fips  26 Jan 2017]
> LunaNamedSystemMutex: open() failed: Permission denied
>
> Thanks & Regards,
> _[Email_CBE.gif]
> PrabuGanesan
> Consultant|MS-Nordics
> capgemini India Pvt. Ltd. | Bangalore
> Contact: +91 8526554535
> Email:
> prabhu.c.gane...@capgemini.com
>
> www.capgemini.com
> People matter, results count.
> __
> Connect with Capgemini:
> [cid:image003.gif@01D926A1.46D89710][cid:image004.gif@01D926A1.46D89710][cid:image005.gif@01D926A1.46D89710][cid:image006.gif@01D926A1.46D89710][cid:image007.gif@01D926A1.46D89710][cid:image008.gif@01D926A1.46D89710]
>
> Please consider the environment and do not print this email unless
> absolutely necessary.
> Capgemini encourages environmental awareness.
>
> From: Hiran CHAUDHURI 
> Sent: 11 January 2023 21:36
> To: Tomcat Users List 
> Subject: RE: tomcat is not coming Up
>
> This mail has been sent from an external source
>
> CONFIDENTIAL & RESTRICTED
>
> If on the old command line the apr libs were referenced but on the new one
> it is not, I believe that could make the process not find the libraries to
> communicate.
> Enough of a reason to terminate with an error, but maybe it is on stderr
> and nowhere in the logs.
>
>
>   *   Check where the jvm's stderr/stdout goes to (could be cron, system,
> syslog, a docker container, ...). If it has an error message, share it
> or act on it.
>   *   Check what caused the changes that you are noticing
>   *   Check what else it may have impacted (e.g. JVM version, keystores,
> startup scripts, ...)
>
> From: Ganesan, Prabu
> mailto:prabhu.c.gane...@capgemini.com.INVALID>>
> Sent: Wednesday, January 11, 2023 16:25
> To: Tomcat Users List
> mailto:users@tomcat.apache.org>>
> Subject: tomcat is not coming Up
>
> EXTERNAL Email. Be careful with links and attachments! If in doubt, click
> the Report Mail button.
>
> Hi team
>
> Our Production Servers are down, Not Coming up When we are trying to start
> the services
>
> In logs we couldn't see Any Errors only I can see Info.
>
>
> Though I can see some changes in this. But we have not made any changes
>
>
>
>
> Old : INFO [main] org.apache.catalina.startup.VersionLoggerListener.log
> Command line argument: -Djava.library.path=/usr/local/apr/lib
>
>
>
> New : INFO [main] org.apache.catalina.startup.VersionLoggerListener.log
> Command line argument: -Xmx3072m
>
>
>
> Old : INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.23] using APR version [1.6.3]
>
>
>
> New : INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.17] using APR version [1.4.8]
>
>
>
>
>
>
> Could you please confirm. Is this could be the reason to not start up?
>
> We have not done any changes from our end.
>
>
> Please help us what more required I ll Provide logs
>
> Thanks & Regards,
> _[Email_CBE.gif]
> PrabuGanesan
> Consultant|MS-Nordics
> capgemini India Pvt. Ltd. | Bangalore
> Contact: +91 8526554535
> Email:
> prabhu.c.gane...@capgemini.com
>
> 

Re: Problems with requests without trailing slash Tomcat 9.0.65

[Simon Matter]

> On 12/01/2023 05:08, Fedor Makarov wrote:
>>
>> lundase and vvsguiden webapps used on different domains
>
> My recommendation would be to configure Tomcat for virtual hosting as
> well. [1]
>
> For example, configure the following hosts in the local hosts file:
>
> lundase-local
> vvsguiden-local
>
> Then the httpd redirects become:
>
> RewriteRule ^(.*)$ http://lundase-local:9090/$1 [P]
>
> and
>
> RewriteRule ^(.*)$ http://vvsguiden-local:9090/$1 [P]
   ^^^
That's a typo and meant to be 9091?

>
> respectively.
>
> And in Tomcat you have:
>
> 
>  
>  
> 
>
> With appropriate ROOT.war files (and any additional web applications)
> deployed in the appropriate webapps-... directory.
>
> If you don't want to go the virtual hosting route, you could have
> multiple Tomcat instances on different ports or multiple 
> elements in one instance e.g.
>
> 
>...
>
>  
>  
>...
>  
>
>
>  
>  
>...
>  
>
> 
>
> Although I do think the virtual hosting approach is the simplest - and
> it maps neatly to the httpd configuration.
>
> Mark
>
> [1] https://tomcat.apache.org/tomcat-9.0-doc/virtual-hosting-howto.html
>
>
>>
>> 
>> <-->ServerName new.vvsguiden-dev.gridnine.com
>> <-->ServerAlias localhost
>> <-->ServerAdmin webmaster@localhost
>> <-->DocumentRoot /var/www/html
>>
>> <-->RewriteEngine on
>>
>> <-->ProxyPreserveHost on
>>
>>          
>>              Order allow,deny
>>              Allow from all
>>              Require all granted
>>          
>>
>>          # static content
>>          RewriteRule ^/fb/(.*)$ /site-binary-data/fb/$1 [L]
>>
>> <-->RewriteRule     ^/vvsguidenapi/(.*)$                            
>>        http://localhost:9090/vvsguidenapi/$1 [NC,P,L]
>> <-->RewriteRule     ^(.*)$                                        
>> http://localhost:9090/vvsguiden/$1 [NC,P,L]
>> <-->
>>
>> 
>> <-->ServerName m.new.lundagrossisten-dev.gridnine.com
>> <-->ServerAdmin webmaster@localhost
>>
>> <-->
>>> Вторник, 10 января 2023, 11:41 +03:00 от Mark Thomas
>>> :
>>>
>>> On 10/01/2023 04:37, Fedor Makarov wrote:

 Also I tried to write a filter to manually redirect, but tomcat
 intercepts the request before it gets into the filter. Can I disable
 this behavior for tomcat and do it manually?
>>>
>>> That isn't the way to solve this problem. The problem is in the reverse
>>> proxy configuration and that is where you need to fix it.
>>>
>>> You previously provided the following set of rules:
>>>
>>> RewriteCond %{REQUEST_URI} ^(/api/|/mapi/|/binary/|/rpc/invoker/)
>>>
>>> RewriteRule ^/rpc/invoker/(.*)$ http://localhost:9090/rpc/invoker/$1
>>> [NC,P,L]
>>> RewriteRule ^/api/(.*)$ http://localhost:9090/api/$1 [NC,P,L]
>>> RewriteRule ^/mapi/(.*)$ http://localhost:9090/mapi/$1 [NC,P,L]
>>> RewriteRule ^(.*)$ http://localhost:9090/lundase/$1 [NC,P,L]
>>>
>>> I'll note that the RewriteCond appears to be unnecessary.
>>>
>>> I'll further note that the NC flag appears to be unnecessary. As is the
>>> L flag (P implies L).
>>>
>>> We can help you fix these rules but when we tried you mentioned
>>> "vvsguiden" but that does not appear in the rewrite rules.
>>>
>>>
>>> mod_rewrite isn't necessarily the best way to configure a reverse proxy
>>> but if that is what you are familiar with, we can work with it. What we
>>> do need, regardless of how the reverse proxy is configured, is a
>>> complete list of the mappings you want to implement.
>>>
>>> To re-iterate Chris's earlier point, trying to change the context path
>>> for an application in a reverse proxy is asking for trouble. It is far
>>> better to reconfigure the contexts in Tomcat so the reverse proxy
>>> doesn't need to change the URI path.
>>>
>>> Based on the paths above, you need to redeploy lundase.war as ROOT.war
>>> and then the rewrite rules become:
>>>
>>> RewriteRule ^(.*)$ http://localhost:9090/$1 [P]
>>>
>>> We need more details for "vvsguiden" to figure out why the above won't
>>> work.
>>>
>>> Mark
>>>
>>>


> Понедельник, 9 января 2023, 11:43 +03:00 от Fedor Makarov <
> ary...@mail.ru.invalid >:
>
>
>
> We have to webapps lundase and vvsguiden therefore, the options you
> have suggested do not look applicable on debug I saw that RequestURI
> in request looks like lundase/lundase/...
>
>> Вторник, 27 декабря 2022, 22:06 +03:00 от Christopher Schultz <
>> ch...@christopherschultz.net >:
>>
>> Fedor,
>>
>> On 12/27/22 05:55, Fedor Makarov wrote:
>>>
>>> proxy for local environment we use the js conf:
>>> proxy: {
>>>      '/api/': {
>>>        target: 'http://localhost:8080/',
>>>        changeOrigin: false,
>>>      },
>>>      '/': {
>>>        target: 'http://localhost:8080/lundase',
>>>        changeOrigin: false
>>>      }
>>>    },
>>>
>>> for normal 

Re: tomcat is not coming Up

[Simon Matter]

Hi

> Hi team
>
> Our Production Servers are down, Not Coming up When we are trying to start
> the services
>
> In logs we couldn't see Any Errors only I can see Info.
>
>
> Though I can see some changes in this. But we have not made any changes
>
>
>
>
> Old : INFO [main] org.apache.catalina.startup.VersionLoggerListener.log
> Command line argument: -Djava.library.path=/usr/local/apr/lib
>
>
>
> New : INFO [main] org.apache.catalina.startup.VersionLoggerListener.log
> Command line argument: -Xmx3072m
>
>
>
> Old : INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.23] using APR version [1.6.3]
>
>
>
> New : INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
> based Apache Tomcat Native library [1.2.17] using APR version [1.4.8]

You really didn't provide a lot of info but one thing is strange: Tomcat
native and APR are now an older version than before. Are you sure this is
really on the same host? Or was the whole Tomcat environment moved to
another server (VM?) and now things don't work anymore?

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: DB2 database locks

[Simon Matter]

Hi,

> Hello Christopher,
>
> Thankyou !
>
> Seems we are not using the connection pooling from Tomcat side , below are
> the DB configuration parameters on context.xml file, do not see any
> connection pool details here.
>
>  name="jdbc/db2"
> auth="Container"
> type="javax.sql.DataSource"
> driverClassName="com.ibm.db2.jcc.DB2Driver"
> url="jdbc:db2://30.177.13.12:3700/TIREHQ"
> maxActive="200"
> maxIdle="30"
> maxWait="1"
> username="xx"
> password="***" />

Don't forget to change this password!

Regards,
Simon

> 
>> I have one more question , Sorry to bother again .
>
> Can the below parameters can aid us with this issue , as given on the
> below Technote , if yes how we need to determine the timings for these
> parameters.
>
> minEvictableIdleTimeMillis
> timeBetweenEvictionRunsMillis
> validationQuery
> removeAbandoned
> removeAbandonedTimeout
>
> https://support.hcltechsw.com/csm?id=kb_article_article=KB0098601
>
>
> Thanks & Regards,
>
> Priyanka Kumawat | Middleware Admin
> T +91.7879364483
> EMail - priyanka.kuma...@dxc.com
> DL - ams-leveraged-webadmin-offsh...@dxc.com
>
> DXC Technology
>
> -Original Message-
> From: Christopher Schultz 
> Sent: 21 October 2022 00:50
> To: Kumawat, Priyanka ; Tomcat Users List
> 
> Subject: Re: DB2 database locks
>
> Priyanka,
>
> On 10/20/22 13:15, Kumawat, Priyanka wrote:
>> Thankyou muck for the explanation for this !!! we have got from below
>> mail that it is likely to be an application coding issue and they
>> needs to fix or use commit etc for long running transactions .
>>
>> The one steps that you have given below to set the "query timeout" ,
>> is this one we can set up on Tomcat side?
> That depends upon how you are obtaining you database connections. If you
> are using a connection pool provided by Tomcat (either tomcat-pool or
> dbcp-pool) then you can configure according to the documentation for
> whichever component you are using (they have similar but different
> configuration styles).
>
> -chris
>
>
> DXC Technology Company -- This message is transmitted to you by or on
> behalf of DXC Technology Company or one of its affiliates. It is intended
> exclusively for the addressee. The substance of this message, along with
> any attachments, may contain proprietary, confidential or privileged
> information or information that is otherwise legally exempt from
> disclosure. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient of this message, you are
> not authorized to read, print, retain, copy or disseminate any part of
> this message. If you have received this message in error, please destroy
> and delete all copies and notify the sender by return e-mail. Regardless
> of content, this e-mail shall not operate to bind DXC Technology Company
> or any of its affiliates to any order or other contract unless pursuant to
> explicit written agreement or government initiative expressly permitting
> the use of e-mail for such purpose.
> �Т�ХF�V�7V'67&��R���âW6W'2�V�7V'67&�F��6B�6�R��Фf�"FF�F����6����G2�R���âW6W'2ֆV�F��6B�6�R��Р



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

[Simon Matter]

> Not sure about Tomcat, but what IBM Liberty does is:
>
> It "will" try to redeploy the war when it detects a file change - and it
> does fail naturally since the war isn't complete.
>
> BUT - it will keep trying since during the upload, the timestamp and file
> size automatically keeps changing - so at the end, it will succeed in
> deploying the whole war file.

I may be wrong but I thought .war files are zip files. Wouldn't it be
possible to just wait until the file has a consistent content and then
extract it?

Simon

>
> I wish they would have just monitored the file size for a configurable
> "given" time.  And lets say - if the file size or timestamp doesn't change
> for -say - 15 seconds, then go ahead and do the deployment, but as what
> was mentioned earlier, different OS(s) may handle this differently, but
> the JAVA NIO API watchevents point you in the right direction in watching
> a file/folder in a loop for a "create" or "modify" or "delete" event to
> occur and fire off.
>
>thanks,
>   jason
>
> - Original Message -
> From: "chris" 
> To: "users" 
> Sent: Sunday, February 20, 2022 9:22:17 AM
> Subject: Re:  is too quick to respond
>
> John,
>
> On 2/20/22 05:50, John Barrow wrote:
>> Neil,
>>
>> Thanks for your useful feedback. I am still feeling my way as you can
>> probably see from my earlier emails trying to setup a development
>> environment.
>>
>> I did actually think of this but didn't put it in scope for a couple of
>> reasons.
>>
>> Firstly, the Tomcat documentation for readloadable quotes
>>
>> "Set to true if you want Catalina to monitor classes in
>> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically
>> reload the web application if a change is detected. This feature is
>> very useful during application development, but it requires
>> significant runtime overhead and is not recommended for use on
>> deployed production applications. That's why the default setting for
>> this attribute is false. You can use the Manager web application,
>> however, to trigger reloads of deployed applications on demand."
>>
>> Therefore, I took it to mean that this flag was geared at development,
>> not production which is what I assume when you would deploy a .war
>> file. So Tomcat would be listening to specific changes in .classes and
>> .jar files that had just been compiled and these are normally small in
>> size. But then I suppose that a single .jar file may be so sized that
>> Tomcat could react while the file was still being written to the disk.
>
> The patch you are currently working on should fix this aspect of the
> overall problem you are trying to solve.
>
>> Secondly, I sort of assumed that since the feature was already in
>> place and handles changes to single files that this check for
>> completeness has already been implemented, but then as I can't get a
>> development environment to run, I don't have enough skills to drill
>> into the sources without it being interactive to help me explore and
>> learn.
>>
>> However, it makes sense that your recommendation is implemented,
>> although I was imagining setting the delay to (say) 500ms to ensure
>> that whatever IDE had time to complete the copying of all the files as
>> that is a small price to pay for automatic refresh. Also by resetting
>> the timer after each event it would have to be quite a large upload
>> for Tomcat to start reacting.
>>
>> Like you, I am not sure how to formally check that a file has
>> completed its copy to the destination. The most common suggestion I
>> hear is to try and change its name and then change it back again and
>> capture the exception which will be raised if the file is locked. I
>> wonder whether attempting to set an attribute (e.g.toggle read-only)
>> would have the same effect (i.e. only allow if file wasn't locked) and
>> be a little more elegant. I would have to try it.
>
> Don't do anything like that; it won't work on various environments. For
> example, Windows obtains exclusive file-locks for even sometimes
> read-only operations. But *NIX does /not/. So you may develop something
> that works on Windows but doesn't work at all anywhere else.
>
> You basically can't check to see if a file is "done uploading"" or
> whatever else may be happening. What you *can* do is check to see if any
> file in the list-of-files-to-be it *too recent* indicating that a
> compile/copy/upload/whatever may still be in progress.
>
>> I assume that Windows has a way of querying a file lock but not sure
>> (a) whether that is exposed via a Java API and (b) whether that would
>> apply to Unix as well (as I have only ever used Windows for
>> development).
>>
>>> How does Tomcat test if a file has been updated?
>
> It's just relative timestamps. Dive into the code Mark suggested and
> you'll find it.
>
>> Again, I don't know this yet (lack of IDE again), but I assumed that
>> it would be similar to the method I implemented in the attached source
>> code, i.e. Create a listener for 

Re: Do I Need Network NameSpaces to Solve This Tomcat+Connector/J Problem?

[Simon Matter]

Hi,

> We want to run a large number of tomcat instances on the same server
> without virtualization or containerization. Each instance is executed from
> its own folder tree and listens on its own unique TCP port. Each instance
> will run code that connects to a backend database server to send queries
> that are triggered by JSP calls from users. We've done this successfully
> with up to 120 instances of tomcat running on the same server while
> avoiding the overhead of virtualization and the complexity of containers.
> Based on our experience over the past decade, we know that we could
> potentially host 500 or more separate tomcat instances on the same server
> without running into performance problems. So now we want to make it 500
> parallel instances.
>
>
> Here's the problem. When tomcat initiates an outbound connection (for
> example, with Connector/J to query a backend database) it establishes a
> socket, and the socket has a client port. With thousands of users making
> requests that require the tomcat services to query back end databases, the
> OS can easily run out of available client ports to allocate to sockets. To
> avoid that problem, we can assign multiple IPs to the server and use the
> localSocketAddress property of Connector/J to group tomcats such that only
> a subset of them each use the same source IP. Then each group will have
> its own range of 64,000-ish client ports. I've tested this and it works.
>
>
>
> My question is, is there a better way?

I guess the database is not on the Tomcat host, otherwise you could
connect via unix domain socket to avoid the limitations of TCP port
numbers.

Otherwise I think you could run a db proxy where your Tomcat clients
connect locally via unix domain socket and the proxy relays the queries to
the db backend.

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: AW: Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi,

> Hello Simon,
>
> if you use the Registry to bind Objects / Stubs, you must also call
> "unbind" on shutdown:
> https://docs.oracle.com/javase/7/docs/api/java/rmi/registry/Registry.html
>
> I think the developer who implemented the RMI stub, should also now what
> to unbind.
>
> Greetings,
> Thomas

The issue has been solved now by adding two unbind() where they were
clearly missing. Tomcat shuts down cleanly now with no delay.

Thanks to all who helped!

Regards,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi Thomas,

> Hello,
>
> it looks like your application opens several ports for RMI communication.
> One class is mentioned in your first mail: ShopdbCacheSynchronizer

Yes, and it's true that when I'm looking at the shut down Tomcat instance
VM, I see several RMI threads lingering around.

>
> Maybe you can ask the developer guys to check whether these threads /
> ports are terminated / closed cleanly on shutdown event.
> Quite often developers don't care much about shutting down their stuff
> cleanly.

Good guess, that's exactly what I'm trying to do, finding out why we have
the shutdown problems so the "developer guys" can finally fix this issue.

>From how I understand it, we have an internal application server which
initiates RMI connections to the Tomcat instance sitting in the DMZ. My
question is, can we terminate the RMI connections on the Tomcat instance
only and shut down Tomcat, or do we have to close the RMI connections on
the internal appserver which initiated the connections?

Apart from RMI, is there anything in the thread dump which indicates an
issue in out Tomcat app?

Kind regards,
Simon

>
> Greetings,
> Thomas
>
> -Ursprüngliche Nachricht-
> Von: Simon Matter 
> Gesendet: Dienstag, 30. November 2021 15:40
> An: Tomcat Users List 
> Betreff: Re: Tomcat 9 doesn't shutdown cleanly
>
> Hi Chris,
>
> Thank you for the quick reply.
>
>> Simon,
>>
>> On 11/30/21 08:21, Simon Matter wrote:
>>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with
>>> OpenJDK
>>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>>
>>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly
>>> (most threads are gone) but send a thread dump to catalina.out and is
>>> later killer by an OS signal.
>>
>> This should only happen if you have CATALINA_PID defined. Are you
>> indeed defining that environment variable?
>>
>> catalina.sh has this code in it, which is probably what you are seeing:
>>
>>echo "To aid diagnostics a thread dump has been written to
>> standard out."
>>kill -3 `cat "$CATALINA_PID"`
>>
>
> That's true, I have CATALINA_PID defined when the call of "catalina.sh
> start" is done. So yes, the script will kill the java process if it
> doesn't terminate.
>
>>> When shutting down Tomcat via the shutdown listener on port 8005, it
>>> also shuts down mostly but without the thread dump in catalina.out.
>>> Sending SIGTERM later to the still running java process terminates
>>> it.
>>
>> Right: when you manually connect to the shutdown port and send
>> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send
>> a signal. You have to do that manually, too.
>
> On a test host, when I send "SHUTDOWN" to the shutdown listener, Tomcat
> shuts down and the Java VM terminates.
>
> Only on this host with the application, when I send "SHUTDOWN" to the
> shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.
>
>>
>>> So both methods somehow terminate Tomcat partly but not completely.
>>
>> You have one or two of the following issues:
>>
>> 1. You have a non-daemon thread running 2. You have an unusually long
>> shutdown process that just takes "too long"
>>
>> The good news is that the thread-dup can answer that question: what's
>> in the thread dump that is generated when you run "catalina.sh stop"?
>>
>>> When simply sending SIGTERM on the OS level, Tomcat shuts down
>>> cleanly and terminates without issues.
>>>
>>> One thing in common is that I always see these messages while
>>> shutting
>>> down:
>>>
>>> 30-Nov-2021 13:59:36.985 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>>> rgets Found RMI Target with stub class class
>>> [sun.rmi.registry.RegistryImpl_Stub] and value
>>> [RegistryImpl_Stub[UnicastRef [liveRef:
>>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
>>> Target has been forcibly removed to prevent a memory leak.
>>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTa
>>> rgets Found RMI Target with stub class class [com.sun.proxy.$Proxy51]
>>> and value
>>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastR
>>> ef
>>> [liveRef:
>>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d

Re: Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi Chris,

Thank you for the quick reply.

> Simon,
>
> On 11/30/21 08:21, Simon Matter wrote:
>> I'm running an application on Tomcat 9.0.55 on x86_64 Linux with OpenJDK
>> JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.
>>
>> When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly
>> (most
>> threads are gone) but send a thread dump to catalina.out and is later
>> killer by an OS signal.
>
> This should only happen if you have CATALINA_PID defined. Are you indeed
> defining that environment variable?
>
> catalina.sh has this code in it, which is probably what you are seeing:
>
>echo "To aid diagnostics a thread dump has been written to
> standard out."
>kill -3 `cat "$CATALINA_PID"`
>

That's true, I have CATALINA_PID defined when the call of "catalina.sh
start" is done. So yes, the script will kill the java process if it
doesn't terminate.

>> When shutting down Tomcat via the shutdown listener on port 8005, it
>> also
>> shuts down mostly but without the thread dump in catalina.out. Sending
>> SIGTERM later to the still running java process terminates it.
>
> Right: when you manually connect to the shutdown port and send
> "SHUTDOWN" (or whatever), it asks Tomcat to shut down but doesn't send a
> signal. You have to do that manually, too.

On a test host, when I send "SHUTDOWN" to the shutdown listener, Tomcat
shuts down and the Java VM terminates.

Only on this host with the application, when I send "SHUTDOWN" to the
shutdown listener, Tomcat shuts down but the Java VM doesn't terminate.

>
>> So both methods somehow terminate Tomcat partly but not completely.
>
> You have one or two of the following issues:
>
> 1. You have a non-daemon thread running
> 2. You have an unusually long shutdown process that just takes "too long"
>
> The good news is that the thread-dup can answer that question: what's in
> the thread dump that is generated when you run "catalina.sh stop"?
>
>> When simply sending SIGTERM on the OS level, Tomcat shuts down cleanly
>> and
>> terminates without issues.
>>
>> One thing in common is that I always see these messages while shutting
>> down:
>>
>> 30-Nov-2021 13:59:36.985 SEVERE [main]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
>> Found RMI Target with stub class class
>> [sun.rmi.registry.RegistryImpl_Stub] and value
>> [RegistryImpl_Stub[UnicastRef [liveRef:
>> [endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
>> Target has been forcibly removed to prevent a memory leak.
>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
>> Found RMI Target with stub class class [com.sun.proxy.$Proxy51] and
>> value
>> [Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastRef
>> [liveRef:
>> [endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7ffd,
>> -4005526521234186948]]. This RMI Target has been forcibly removed to
>> prevent a memory leak.
>> 30-Nov-2021 13:59:36.987 SEVERE [main]
>> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
>> Found RMI Target with stub class class
>> [sun.rmi.registry.RegistryImpl_Stub] and value
>> [RegistryImpl_Stub[UnicastRef [liveRef:
>> [endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
>> Target has been forcibly removed to prevent a memory leak.
>>
>> Why do the three methods to shutdown Tomcat differ and how can I make
>> 'catalina.sh stop' or the shutdown listener work the same way like
>> sending
>> the OS signal.
>
> What's the difference? You don't want the thread dump in your
> catalina.out? That's supposed to be helpful in debugging why your
> shutdown isn't clean. It sounds like you are saying "help me with my
> unclean shutdown; please get rid of the debugging information that is
> trying to help me!".
>
>> I've tried, beside a lot of other things, to set
>> skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems to
>> change nothing at all.
>
> Tomcat will never kill your non-daemon thread(s) from inside the VM.
> Since you are using a CATALINA_PID environment variable (and, therefore,
> a pid-file), the shutdown process is sending the TERM signal. It's also
> possibly sending a KILL signal, depending on whether or not the TERM
> worked and if -force was supplied on the command-line.

Unfortunately I am not an insider when it comes to Java, so I'm not sure
what 

Tomcat 9 doesn't shutdown cleanly

[Simon Matter]

Hi,

I'm running an application on Tomcat 9.0.55 on x86_64 Linux with OpenJDK
JRE-11.0.13+8 and have problems shutting down Tomcat in certain ways.

When I shutdown Tomcat via 'catalina.sh stop', it shuts down mostly (most
threads are gone) but send a thread dump to catalina.out and is later
killer by an OS signal.

When shutting down Tomcat via the shutdown listener on port 8005, it also
shuts down mostly but without the thread dump in catalina.out. Sending
SIGTERM later to the still running java process terminates it.

So both methods somehow terminate Tomcat partly but not completely.

When simply sending SIGTERM on the OS level, Tomcat shuts down cleanly and
terminates without issues.

One thing in common is that I always see these messages while shutting down:

30-Nov-2021 13:59:36.985 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:2071](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class [com.sun.proxy.$Proxy51] and value
[Proxy[ShopdbCacheSynchronizer,RemoteObjectInvocationHandler[UnicastRef
[liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[-6f4b2f9d:17d70eb1ef4:-7ffd,
-4005526521234186948]]. This RMI Target has been forcibly removed to
prevent a memory leak.
30-Nov-2021 13:59:36.987 SEVERE [main]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesRmiTargets
Found RMI Target with stub class class
[sun.rmi.registry.RegistryImpl_Stub] and value
[RegistryImpl_Stub[UnicastRef [liveRef:
[endpoint:[157.161.91.47:1096](local),objID:[0:0:0, 0]. This RMI
Target has been forcibly removed to prevent a memory leak.

Why do the three methods to shutdown Tomcat differ and how can I make
'catalina.sh stop' or the shutdown listener work the same way like sending
the OS signal.

I've tried, beside a lot of other things, to set
skipMemoryLeakChecksOnJvmShutdown="true" in context.xml but it seems to
change nothing at all.

Any help would be much appreciated.

Thanks,
Simon


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org