Maximum connections and thread notifications

[M. Manna]

Hello all,

Since yesterday all of our loadbalanced tomcat servers have been refusing
further connection intermittently. During our internal stress testing,
we’ve managed to reproduce  the issue where maximum number of
threads/connections to Catalina were reached for one server, but then the
rest were loadbalanced to others. Eventually, everyone starts failing
health checks the Citrix network health monitor on port 80 and 8443.
Because all of them were eventually running out.

We wanted to pinpoint the time of the first hit observers  with Catalina
logs to see when max connections are reached or maxThreads are used. Is
there any log level for Catalina/localhost logs which we can use to see
this message ? If so which level should we be using ?

Also, we are using 8.5.37 with our own TSL cipher suites etc. this started
happening only since last few days. We would need to find out whether any
application threads are doubling up and exhausting the APR connectors
capacity for max threads/connections by holding them indefinitely.

Any suggestion/help you can give here would be appreciated.

Thanks,

Re: how to enable OCSP for Tomcat w OpenSSL

[John Palmer]

Well, after much research and experimentation I got OCSP working with the
JSSE  flaovor, NIO2 connector (renamed the OCSP-enabled tc-native-1.dll so
it isn't used and JSSE is used instead).

2 things had to be set:
1: server.xml: add to the SSLHostConfig section (inside the Connector
section)
revocationEnabled="true"
certificateVerification="require"

2:  java.security file in the (java)\jre\lib\security folder:
uncomment the line:ocsp.enable=true
(you get a "can't connect securely to this page" in IE if you forget.)
(if there's a way to do this with the Java options used by the tomcat
service(eg -D(something)ocsp.enable="true", I'd appreciate someone telling
me).

by adding -Djava.security.debug="certpath ocsp"  to the Java options used
by the tomcat service (Windows)... I have logging showing the OCSP checking
etc
and wireshark shows me the OCSP calls (there MAY be some caching being done
by the java (or possibly Windows CAPI) code, not all the expected OSCP
requests seem to always be there).

by restoring the NON-OCSP-enabled tc-native-1.dll, I found that the same
settings allow the same java calls to work the same way

and by restoring the OCSP-enabled tc-native-1.dll... those still work.

Apparently there is some OTHER setting or configuration needed for the
OCSP-enabled tc-native-1.dll to work... and I haven't found it yet.

I'm trying to understand from the tc-native source what I might be missing,
but apparently I'm overlooking it.


helpful suggestions are welcomed.

On Wed, Apr 3, 2019 at 12:32 PM John Palmer  wrote:

> I appreciate your response
>
> > Setting `certificateVerification="require"` on your Connector
>
> I changed
>   `certificateVerification="REQUIRED"
> to
>   `certificateVerification="require"`
>
> still not seeing any OCSP calls in wireshark for this
>
> I did find out how to enable logging better  (by adding either of these to
> logging.properties):
> org.apache.tomcat.util.net.openssl.level=ALL
> org.apache.tomcat.util.net.level=ALL
>
> and I can see logs confirming  that the trust store is being used:
> OpenSSLContext.init Added client CA cert:...) ;
>
> with logging set to org.apache.tomcat.level=ALL
> I see confirmation that the certificateVerification is being parsed,
> apparently correctly.
>
> but I still don't see any evidence in the tomcat/catlina logs or in
> wireshark  that anything is happening to accomplish this.
>
>
>
> On Tue, Apr 2, 2019 at 3:47 PM Coty Sutherland 
> wrote:
>
>> Hi,
>>
>> On Mon, Apr 1, 2019 at 3:30 PM John Palmer  wrote:
>>
>> > What, if anything, needs to be configured to ENABLE (preferably REQUIRE)
>> > tomat to do CLIENT certificate revocation checking via OCSP in Tomcat
>> > 8.5.38 using Openssl ?
>>
>>
>> Setting `certificateVerification="require"` on your Connector and using a
>> client certificate that has an OCSP URI should be it. See
>>
>> https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html#Using_OCSP_Certificates
>> for more information on how to configure it.
>>
>>
>> >
>>
>>
>> > I'm sure I'm missing something simple and obvious (once pointed out) but
>> > I've been struggling with this all morning).
>> >
>> > 1) using Openssl (the tc-native-1.dll binary for Windows, compiled w
>> OCSP
>> > support - the X64 dll from
>> > tomcat-native-1.2.21-openssl-1.1.1a-ocsp-win32-bin.zip)
>> > (will this even work with NIO2 ? - I don't HAVE to use NIO2)
>> >
>>
>> It will work, but only if you're using the openssl implementation.
>>
>>
>> > (i'd prefer to have this working with OpenSSl for a couple of reasons).
>> > (extra points for a configuration to allow it to use Axways (formerly
>> > Tumbleweed) Desktop Validator for its OCSP-caching features).
>> >
>> > 2) using JSSE (java 8 (1.8.0_202)) with the NIO2 connector
>> > (I've tried adding -Dcom.sun.net.ssl.checkRevocation=true to the Java
>> > options for the tomat service).
>> >
>> >
>> > I can't see anything indicating OCSP checks in the logs for either.
>> >
>>
>> There isn't any OCSP code in Tomcat and tomcat-native doesn't log much of
>> anything when it's in use, so there's not much indication that it's
>> working
>> there.
>>
>>
>> >
>> > (when the tc-native-1.dll is present, the logs show it being used:
>> > INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>> > Loaded APR based Apache Tomcat Native library [1.2.21] using APR version
>> > [1.6.5].
>> > INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>> > APR capabilities: IPv6 [true], sendfile [true], accept filters [false],
>> > random [true].
>> > INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>> > APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
>> > INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL
>> > OpenSSL successfully initialized [OpenSSL 1.1.1a  20 Nov 2018]
>> > INFO [main]
>> > org.apache.coyote.http11.AbstractHttp11Protocol.configureUpgradeProtocol
>> > The 

Re: Monitoring resources comsumption at context level

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Fellipe,

On 4/4/19 16:51, Fellipe Theophilo wrote:
> Hi Cris, thank you for reply. So, I've talked to the developer team
> about this and they told me that they want to be able to know at
> OBJECT level or at CLASS level what is consuming more resources, to
> make easier to debug the code. For example, suppose that some JVM
> is having high resource comsumption, they want to know what CLASS,
> or what METHOD is the responsible for the issue. Is there any way
> to do it?

Your developer team should already know that the only way to do that
would be to take a heap dump or attach a debugger to a running JVM.
What they want is simply not possible via JMX.

Taking a heap dump can cause your JVM to stall for an indeterminate
amount of time.

Attaching a debugger can significantly degrade performance.

For realtime monitoring, you may want to build-in some application
self-monitoring features and then keep an eye on those from e.g. Zabbix.

- -chris

> Em ter, 2 de abr de 2019 às 20:01, Christopher Schultz < 
> ch...@christopherschultz.net> escreveu:
> 
> Fellipe,
> 
> On 4/2/19 17:04, Fellipe Theophilo wrote:
 Hi everyone, I'm trying to find a way to monitor metrics of 
 resources comsumption at context level. I've opened this
 thread: 
 https://stackoverflow.com/questions/55070370/monitoring-multiple-ja
va-
>
 
applications-at-once-with-one-zabbix-java-gateway
> 


> 
But no one knows a solution. By using jConsole and Zabbix I can
 check the value of some objects at context level, but none of
 them is any metric. So I tried the Spring Actuator, which
 expose many metrics through REST , returning a JSON. However,
 the call to 
 http://:8080//metrics give
 metrics information of the JVM as a whole, even specifying
 the context name. To prove this I ran an AB(Apache Benchmark)
 to do a stress test and I've saw that memory usage grew
 together across the two contexts I have for testing. So the
 conclusion is that the Spring Actuator is not exposing
 metrics at context level. Do anyone know if is there a way to
 get metrics at context level?
> 
> The reason that no tool provides per-context memory consumption 
> metrics is because it's not practical to actually measure that kind
> of thing.
> 
> You'd basically need to walk the object tree from GC roots (like
> the GC does whenever it collects garbage) after determining what
> the list of "GC roots" is for each context -- such as the
> WebappClassLoader.
> 
> The JVM treats memory as a shared resource. There is no memory 
> isolation between contexts within an application server.
> 
> If you really want to monitor your web applications separately,
> run them each in isolated JVMs.
> 
> -chris
>> 
>> -
>>
>> 
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>> 
>> 
> 
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlymhnYACgkQHPApP6U8
pFi1lg//bvCIPS2VAnJ8wT+YiUCldhk0IrOROWHngppYSX7hSLnhrlswu1srP0tO
fhOpkXBXBbsp2wkX6xGugWhTqfmeRFplSokcWhKGsAi9822JeTqEk2LkhWsUA88R
HGbKT1jk4Qw0OG+T94gJJPXhcQQYalNIE4+z68Jq/qA5HjTjDckM/7ORt2Tx049w
rowge/AtBpZ3/E5GEtmh2LwOBA4oEUUJm6u3abnPU+0TlhZd1uVCR2MttNTaK4iG
nYnn12bk+Ho03tXBxjdhd5fELt+VtpYidvlDA3wAqPwLA5xDKiRJJPKMvI0zgElv
ShWZpE+2pGknROChEl+RCHNO3sSeOru4Ri5N0IHBfMcns6KpbM7xODoP7IrwFZUp
zRD4+ffVan4qGGnLVsVZXyY5HA2K3Z51wz445lWK9cuSXJDDd4u+5vAW8cF7KIhH
mXDypdS3L1RNatK6rDcRhDvkRgNiCzdTP2/qaAp5FOEkvGEshyX0AyL3WBD6Fqpb
NcuR11jH80SKxeSUFatxjRMi/qx8zoO7WEHesfL1cvA3O1MBXYb/FO1d23+tK/f9
EXETpC90KQE9QjHXi6tZeKUBKkcU2YQyCOtARhlKAWgo/CoXekNbJ7wE8YAKhJp6
IjHDBcOOGHgaNexg+847Ds6+KqP1JjFmol6sjIVQQM8Xij1Y/jw=
=N9dt
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Monitoring resources comsumption at context level

[Fellipe Theophilo]

Hi Cris, thank you for reply. So, I've talked to the developer team about
this and they told me that they want to be able to know at OBJECT level or
at CLASS level what is consuming more resources, to make easier to debug
the code. For example, suppose that some JVM is having high resource
comsumption, they want to know what CLASS, or what METHOD is the
responsible for the issue. Is there any way to do it?
-
Fellipe


Em ter, 2 de abr de 2019 às 20:01, Christopher Schultz <
ch...@christopherschultz.net> escreveu:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Fellipe,
>
> On 4/2/19 17:04, Fellipe Theophilo wrote:
> > Hi everyone, I'm trying to find a way to monitor metrics of
> > resources comsumption at context level. I've opened this thread:
> > https://stackoverflow.com/questions/55070370/monitoring-multiple-java-
> applications-at-once-with-one-zabbix-java-gateway
> 
> >
> >  But no one knows a solution. By using jConsole and Zabbix I can
> > check the value of some objects at context level, but none of them
> > is any metric. So I tried the Spring Actuator, which expose many
> > metrics through REST , returning a JSON. However,  the call to
> > http://:8080//metrics give metrics
> > information of the JVM as a whole, even specifying the context
> > name. To prove this I ran an AB(Apache Benchmark) to do a stress
> > test and I've saw that memory usage grew together across the two
> > contexts I have for testing. So the conclusion is that the Spring
> > Actuator is not exposing metrics at context level. Do anyone know
> > if is there a way to get metrics at context level?
>
> The reason that no tool provides per-context memory consumption
> metrics is because it's not practical to actually measure that kind of
> thing.
>
> You'd basically need to walk the object tree from GC roots (like the
> GC does whenever it collects garbage) after determining what the list
> of "GC roots" is for each context -- such as the WebappClassLoader.
>
> The JVM treats memory as a shared resource. There is no memory
> isolation between contexts within an application server.
>
> If you really want to monitor your web applications separately, run
> them each in isolated JVMs.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlyj6cMACgkQHPApP6U8
> pFhcKBAAiAUUS9pchJB0JIjTrldg4VOAuLxyTOMQG7qiaffDq/eQCac7OAH4uoaf
> Z7gCV44QK+rIpRmKqSdHWEEccGZ0b0ywPnLM7GUUptynL2ZYB7qTH7/wjnzsEKPV
> pMdDp06d5eo5dAFyKiKSkT8bT2h6+wmbP/xrWkgL4DtyUPQzbPOIgRdjDPfihaxp
> 0l0xI8H/HYaqyNhxj0xDmEy7IbO2buNt5kUzrC2HmS+TC+rl4rlUDEUAISK9vz6a
> WTGaYnGCbbbDmAJc2NtRMSwCMrzt4Oz3yLJR1+HzFCidmKYasCVTOp3elBtctvzE
> BEUuKOgNIo9f1rte2HKj1nAoxrKOw4hlWfdqqxfIm3yBk3ThtGEjMAoa0enfKXnt
> kWObLMuL64c+XEmmqDMh4Q1AwpvzFeIQ1KgcxBGSGcgsmYZQQRY2LIE7TMI20Xp/
> b4Xv19+0HoI6VqHVL8pXt8BWznjM+ygJb2mtUuO4nFRGuCYSLw6s2IqrL59p4Pld
> DtsBa5U5eehtIrYOlHi9qndQP5G9BtbSHe4HI5U3mkqC5i2f3oywzHKcE5/roTc0
> 19zMzkJL8ISDcnZ7EBUgV5Qx13q+nEQeu9gZfjQjQCMTTIcmWnyrvWcRjIUUZwb7
> 2cCG80EfB5iny3fR0gtozj/SDmx5TOxU3MMnMunBQVZOoqf7uyc=
> =uO0P
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

AW: Tomcat Hackathon - Brussels Belgium - 4/5 May 2019

[Thomas Meyer]

Hi,

a PropertySource that uses environment variables as source would be nice.
I.e. an OpenShift/Kubernetes Secret mapped into environemnt variables that can 
be used in server.xml or context.xml!

With Kind regards
Thomas


Von: Mark Thomas
Gesendet: Donnerstag, 4. April 2019 16:29
An: Tomcat Users List
Cc: Tomcat Developers List
Betreff: Tomcat Hackathon - Brussels Belgium - 4/5 May 2019

All,

You are invited!

As part of the EU-FOSSA 2 project[1], there will be a Tomcat Hackathon 
in Brussels, Belgium on 4-5 May 2019.[2]

The outline of the schedule is:
- general update on the status of the project
- hacking
- wrap-up
with the majority of the time spent hacking.

We are currently collating potential tasks on the wiki [3].

The EU-FOSSA 2 project is providing accommodation (on the basis of 2 
people sharing - you can request a single room if you want to pay the 
difference) and might be able to help with transport costs.

Space is limited so we are asking anyone who would like to attend this 
hackathon and contribute to the development of Tomcat to send an e-mail 
to priv...@tomcat.apache.org with the following information:

- First name
- Last name
- Email address
- Phone number
- City of departure
- Area you would like to work on
   (Feel free to add ideas directly to the wiki as well)

Time is fairly tight so if you are interested please let us know ASAP.

We hope to see you in Brussels

Mark
on behalf of the Apache Tomcat PMC


[1] https://joinup.ec.europa.eu/collection/eu-fossa-2
[2] https://eufossahackathon.bemyapp.com/
[3] https://cwiki.apache.org/confluence/display/TOMCAT/EU+FOSSA+May+2019


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat Hackathon - Brussels Belgium - 4/5 May 2019

[Mark Thomas]

All,

You are invited!

As part of the EU-FOSSA 2 project[1], there will be a Tomcat Hackathon 
in Brussels, Belgium on 4-5 May 2019.[2]


The outline of the schedule is:
- general update on the status of the project
- hacking
- wrap-up
with the majority of the time spent hacking.

We are currently collating potential tasks on the wiki [3].

The EU-FOSSA 2 project is providing accommodation (on the basis of 2 
people sharing - you can request a single room if you want to pay the 
difference) and might be able to help with transport costs.


Space is limited so we are asking anyone who would like to attend this 
hackathon and contribute to the development of Tomcat to send an e-mail 
to priv...@tomcat.apache.org with the following information:


- First name
- Last name
- Email address
- Phone number
- City of departure
- Area you would like to work on
  (Feel free to add ideas directly to the wiki as well)

Time is fairly tight so if you are interested please let us know ASAP.

We hope to see you in Brussels

Mark
on behalf of the Apache Tomcat PMC


[1] https://joinup.ec.europa.eu/collection/eu-fossa-2
[2] https://eufossahackathon.bemyapp.com/
[3] https://cwiki.apache.org/confluence/display/TOMCAT/EU+FOSSA+May+2019


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

the config about

[舒凯凯]

this is my question:
when I get an error of ---Error parsing HTTP request header--
the reason of this error i already know. the header is too large. by setting 
maxHttpHeaderSize to a bigger size can solve this error, but this is not my 
point.
the total message of this error is :
Info: Error parsing HTTP request header
 Note: further occurrences of HTTP header parsing errors will be logged at 
DEBUG level.
java.lang.IllegalArgumentException: Request header is too large
at 
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:530)
at 
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:519)
at 
org.apache.coyote.http11.InternalInputBuffer.parseHeader(InternalInputBuffer.java:407)
at 
org.apache.coyote.http11.InternalInputBuffer.parseHeaders(InternalInputBuffer.java:281)
at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1066)
at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:317)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)


this bold font-Note: further occurrences of HTTP header parsing errors will be 
logged at DEBUG level. 
is control by system properties: 
org.apache.juli.logging.UserDataHelper.CONFIG=INFO_THEN_DEBUG 
and INFO_THEN_DEBUG is the default value. 
when i change the value INFO_THEN_DEBUG to DEBUG_ALL, the console will not show 
any log error again, but the error is here.

i know the reason is my default root log level is INFO config in my 
logging.properites. 
but in my opinion， DEBUG_ALL is lower than INFO_THEN_DEBUG, when i change to a 
lower value, i can't get any error log again.
i dont know whether this is a bug?  if not, i want to know why is this logic?  
thanks!




My environment:
win 10
eclipse
tomcat-embed-core:7.0.93
tomcat-embed-logging-juli:7.0.93
logging.properties:
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = FINEST
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

Re: Tomcat 8.5.x configuration file differences: permission denied

[Maxim Solodovnik]

you can use github for this :)

On Thu, 4 Apr 2019 at 16:19, George Angeletos  wrote:
>
> I am getting permission denied when trying to view configuration file
> changes between 8.5.x versions (
> https://tomcat.apache.org/migration-85.html#Upgrading_8.5.x)
>
> https://gitbox.apache.org/repos/asf?p=tomcat.git=blobdiff=conf%2Fcatalina.policy=8.5.37=8.5.39
>
> Cheers,
> George



-- 
WBR
Maxim aka solomax

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat 8.5.x configuration file differences: permission denied

[George Angeletos]

I am getting permission denied when trying to view configuration file
changes between 8.5.x versions (
https://tomcat.apache.org/migration-85.html#Upgrading_8.5.x)

https://gitbox.apache.org/repos/asf?p=tomcat.git=blobdiff=conf%2Fcatalina.policy=8.5.37=8.5.39

Cheers,
George