Re: NGINX + tomcat 8.0.35 (110: Connection timed out)
On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan wrote:
> Chris,
>
> I was load testing using the ec2 load balancer dns. I have increased the
> connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I am
> not seeing connection timeout in nginx logs now. No errors in kernel.log I
> am not seeing any errors in tomcat catalina.out.
> During regular operations when the request count is between 4 to 6k
> requests per minute the open files count for the tomcat process is between
> 200 to 350. Responses from tomcat are within 5 seconds.
> If the requests count goes beyond 6.5 k open files slowly move up to 2300
> to 3000 and the request responses from tomcat become slow.
>
> I am not concerned about high open files as I do not see any errors related
> to open files. Only side effect of open files going above 700 is the
> response from tomcat is slow. I checked if this is caused from elastic
> search, aws cloud watch shows elastic search response is within 5
> milliseconds.
>
> what might be the reason that when the open files goes beyond 600, it slows
> down the response time for tomcat. I tried with tomcat 9 and it's the same
> behavior
>
Do you know what kind of files are being opened ?
>
>
>
>
>
>
> On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Ayub,
> >
> > On 11/3/20 10:56, Ayub Khan wrote:
> > > *I'm curious about why you are using all of cloudflare and ALB and
> > > nginx.Seems like any one of those could provide what you are getting
> from
> > > all3 of them. *
> > >
> > > Cloudflare is doing just the DNS and nginx is doing ssl termination
> >
> > What do you mean "Cloudflare is doing just the DNS?"
> >
> > So what is ALB doing, then?
> >
> > > *What is the maximum number of simultaneous requests that one
> > nginxinstance
> > > will accept? What is the maximum number of simultaneous proxiedrequests
> > one
> > > nginx instance will make to a back-end Tomcat node? Howmany nginx nodes
> > do
> > > you have? How many Tomcat nodes? *
> > >
> > > We have 4 vms each having nginx and tomcat running on them and each
> > tomcat
> > > has nginx in front of them to proxy the requests. So it's one Nginx
> > > proxying to a dedicated tomcat on the same VM.
> >
> > Okay.
> >
> > > below is the tomcat connector configuration
> > >
> > > > > connectionTimeout="6" maxThreads="2000"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > URIEncoding="UTF-8"
> > > redirectPort="8443" />
> >
> > 60 seconds is a *long* time for a connection timeout.
> >
> > Do you actually need 2000 threads? That's a lot, though not insane. 2000
> > threads means you expect to handle 2000 concurrent (non-async,
> > non-Wewbsocket) requests. Do you need that (per node)? Are you expecting
> > 8000 concurrent requests? Does your load-balancer understand the
> > topography and current-load on any given node?
> >
> > > When I am doing a load test of 2000 concurrent users I see the open
> files
> > > increase to 10,320 and when I take thread dump I see the threads are
> in a
> > > waiting state.Slowly as the requests are completed I see the open files
> > > come down to normal levels.
> >
> > Are you performing your load-test against the CF/ALB/nginx/Tomcat stack,
> > or just hitting Tomcat (or nginx) directly?
> >
> > Are you using HTTP keepalive in your load-test (from the client to
> > whichever server is being contacted)?
> >
> > > The output of the below command is
> > > sudo cat /proc/sys/kernel/pid_max
> > > 131072
> > >
> > > I am testing this on a c4.8xlarge VM in AWS.
> > >
> > > below is the config I changed in nginx.conf file
> > >
> > > events {
> > > worker_connections 5;
> > > # multi_accept on;
> > > }
> >
> > This will allow 50k incoming connections, and Tomcat will accept an
> > unbounded number of connections (for NIO connector). So limiting your
> > threads to 2000 only means that the work of each request will be done in
> > groups of 2000.
> >
> > > worker_rlimit_nofile 3;
> >
> > I'm not sure how many connections are handled by a single nginx worker.
> > If you accept 50k connections and only allow 30k file handles, you may
> > have a problem if that's all being done by a single worker.
> >
> > > What would be the ideal config for tomcat and Nginx so this setup on
> > > c4.8xlarge vm could serve at least 5k or 10k requests simultaneously
> > > without causing the open files to spike to 10K.
> >
> > You will never be able to serve 10k simultaneous requests without having
> > 10k open files on the server. If you mean 10k requests across the whole
> > 4-node environment, then I'd expect 10k requests to open (roughly) 2500
> > open files on each server. And of course, you need all kinds of other
> > files open as well, from JAR files to DB connections or other network
> > connections.
> >
> > But each connection needs a file descriptor, full stop. If you need
Re: Timeout waiting to read data from client
On Thu, Nov 12, 2020 at 12:34 AM Jerry Malcolm wrote: > On 11/9/2020 11:05 AM, Jerry Malcolm wrote: > > > > On 11/9/2020 3:10 AM, Mark Thomas wrote: > >> On 08/11/2020 01:33, Jerry Malcolm wrote: > >>> On 11/7/2020 6:56 PM, Christopher Schultz wrote: > Jerry, > > On 11/6/20 19:49, Jerry Malcolm wrote: > > I have a relatively new environment with a standalone tomcat (8.5) > > running on an AWS Linux2 EC2. I'm not using HTTPD/AJP. Its a direct > > connection to port 443. (Well technically, I have firewallD in the > > flow in order to route the protected port 80 to port 8080 and 443 to > > 8443 for TC). > > > > I am doing some stress testing on the server and failing miserably. > > I am sending around 130 ajax calls in rapid succession using HTTP/2. > > These are all very simple small page (JSP) requests. Not a lot of > > processing required. The first ~75 requests process normally. Then > > everything hangs up. In the tomcat logs I'm getting a bunch of > > "Timeout waiting to read data from client" exceptions. And in the > > stacktrace for these exceptions, they are all occurring when I'm > > trying to access a parameter from the request. Looking at the > > request network timing in the browser console, I see a bunch of the > > requests returning in typical time of a few milliseconds. Then > > another large block of requests that all start returning around 4 > > seconds, then another block that wait until 8 seconds to return. > > I've tried firefox and chrome with the same results. > > > > I've been using httpd in front of TC for years. So this is the first > > time I'm running TC standalone. It is very likely I've got some > > parameters set horribly wrong. But I have no clue where to start. > > This is not a tiny EC2, and my internet connection is not showing any > > signs of problems. So I really don't think this is a > > performance-related problem. The problem is very consistent and > > reproducible with the same counts of success/failure calls. What > > could be causing the "Timeout waiting to read data from client" after > > 75 calls, and then cause blocks of calls to wait 4 seconds, 8 > > seconds, etc before responding? I really need to handle more > > simultaneous load that this is currently allowing. > > > > Thanks in advance for the education. > Are you using HTTP Keepalives on your connections? Are you actually > re-using those connections in your test? What is your keepalive > timeout on your . Actually, what is your whole > configuration? > > -chris > > >>> Hi Chris, here are my two connector definitions from server.xml: > >>> > >>> >>>port="8080" > >>>protocol="HTTP/1.1" > >>>connectionTimeout="2" > >>>redirectPort="443" /> > >>> > >>> >>>port="8443" > >>>maxThreads="150" > >>>connectionTimeout="2" > >>>SSLEnabled="true" > >>>scheme="https" > >>>secure="true" > >>>clientAuth="false" > >>>SSLCertificateFile="ssl/a.com/cert.pem" > >>>SSLCertificateChainFile="ssl/a.com/chain.pem" > >>>SSLCertificateKeyFile="ssl/a.com/privkey.pem"> > >>> >>> className="org.apache.coyote.http2.Http2Protocol" /> > >>> > >> How are you stress testing this? All on a single HTTP/2 connection or > >> multiple connections? With which tool? > >> > >> You might want to test HTTP/1.1 requests (with and without TLS) to see > >> if the problem is specific to HTTP/2 or TLS as that should help narrow > >> down the root cause. > >> > >> Mark > > > > Hi Mark, technically it's not a 'designed' stress test. It's real > > production code that just happens to stress the server more than > > usual. It's just a page that makes a bunch of ajax calls, and the > > responses to each of those issue a second ajax call. > > > > If you don't see anything obvious in my configuration, we will > > definitely pursue the http/1.1 options, etc. I just wanted to > > eliminate the chance of obvious 'pilot error' before digging deeper. > > > > Specifically, where is that error detected in the TC flow? In my logs > > it fails on getting request parameters. It sounds like the input > > reader for the request is getting blocked.But the first part of > > the request is getting in since it does route to the appropriate JSP. > > Just seems strange that the http/2 or ssl layers would let half of the > > request in and then block the rest of the request. The browser > > appears to be sending everything. And it fails the same using firefox > > or chrome. Any ideas? > > > > Thx > > > > > Update on this. One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM > after things locked up. I removed the http2 'upgrade protocol' line > from my connector, and everything works. So it's apparently something
Re: HTTP thread pool size remained high after database failover
Thank you Chris for your response. On Wed, Nov 11, 2020 at 10:02 PM Christopher Schultz wrote: > > Suvendu, > > On 11/11/20 06:40, Suvendu Sekhar Mondal wrote: > > Application Setup: > > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server > > 2016 > > As Michael pointed-out, these all need to be upgraded, but they are not > affecting your experience, here. > > > Observations: > > After database failover, we noticed that HTTP thread pool size jumped > > from 10 to 50. No. of busy threads jumped to 40 for a minute and then > > came down to normal range(1-5). But thread pool size remained 50 for > > the next one hour. When we collected the heap dump after one hour, > > thread pool size jumped again to ~90. Thread dump analysis is showing > > most of the HTTP threads were just waiting to get some work[1]. That > > looked surprising to me. Few of them are runnable and doing some work > > - which matches "busy thread" count. During all this timeframe load on > > the system remained the same. > > > > We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We > > do not have any keepAliveTimeout set in Tomcat. App uses session > > affinity. We were looking for some connector properties which deal > > with idle threads but found none[3]. > > You won't find them. > Okay. > > We are not seeing any "problem" in our app after DB failover but this > > is something also newer to us and we are trying to explain it. > > What are you trying to explain? > We are simply trying to understand this behavior as this is something new we noticed. > > Tomcat's memory usage increased by a few MB as new threads were > > spawned. My expectation was that if threads in the thread pool are not > > getting used and idle, after sometime thread pool size will reduce to > > "minSpareThreads" size which is 10 in our case. Is that expectation > > wrong? Can you please share your thoughts? > > This is a common misconception. If you do not configure an "executor" > then Tomcat will create one for you and it will NOT remove idle threads > from the thread pool. > > The solution is to manually-specify an Executor: > > minSpareThreads="Y" /> > > > > With this configuration, you should see your thread count stay between Y > and X, but when X are not necessary, your thread count should drop > towards Y as appropriate. > Thanks for the explanation. This is the kind of information I was looking for in Tomcat docs but was not able to locate it. We will try this approach and let you know the outcome. > Hope that helps, > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Something I still don't qutite understand, Re: Let's Encrypt with Tomcat behind httpd
Dr it really does not work On Thu, Nov 5, 2020, 12:07 PM James H. H. Lampert wrote: > On 8/24/20 9:57 AM, Christopher Schultz wrote: > > > So your RewriteCond[ition] is expected to always be true? Okay. Maybe > > remove it, then? BTW I think your rewrite will strip query strings and > > stuff like that. Maybe you just want RedirectPermanent instead of > > Rewrite(Cond|Rule)? > > Ladies and Gentlemen: > > This past Friday, the cached challenge result expired, and so this past > Monday, I ran another certbot test. > > With the rewrite in place for our "subdomain of interest," the cert > covering everything else served by the httpd server renewed without > incident, but the separate cert covering this subdomain failed completely. > > I commented out the rewrite, and ran the test again, and both renewed > without incident. > > I posted a redacted version of the complete VirtualHost blocks back on > August 24th. And after I'd run my tests this week, I've also posted it > to ServerFault, at > https://serverfault.com/q/1041047/498231 > > I'm intrigued by Mr. Schultz's suggestion of > > > Maybe you just want RedirectPermanent instead of > > Rewrite(Cond|Rule)? > > Would that make a difference? Or is it just a matter of altering the > RewriteCond clause to specifically ignore anything that looks like a > Let's Encrypt challenge? Or is there something I can put on the default > landing page for the subdomain, rather than in the VirtualHost, to cause > the redirection? > > As I recall (unless there's a way to force-expire the cached challenge > result on a certbot call), I have to wait until December to run another > test. > > -- > JHHL > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Something I still don't quite understand, Re: Let's Encrypt with Tomcat behind httpd
On 8/21/20 1:02 PM, logo wrote:
From my experience I have excluded .well-known from the redirect.
That appears to be the correct answer. I probably didn't see that line
back in August, or I probably would have replied by asking something
like, "Ok, and how do I do that?"
Be that as it may, Andrew Schulman came up with an answer on my
ServerFault thread (https://serverfault.com/a/1041882/498231) to the
effect of changing the rewrite block from:
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
to:
RewriteEngine on
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L,QSA]
While I'm not going to be certain until December, when the cached
challenge expires, it certainly seems to work: if I go to
http://sub.domain.com, it immediately redirects me to
https://sub.domain.com, and I get the Tomcat server, whereas if I try to
go to http://sub.domain.com/.well-known/acme-challenge/foo, it remains
http, and gives me the expected "Not Found" error.
--
JHHL
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Weirdest Tomcat Behavior Ever?
> -Original Message- > From: Mark Thomas > Sent: Monday, November 9, 2020 5:59 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > Eric, > > Time to prune the history and provide another summary I think. This > summary isn't complete. There is more information in the history of the > thread. I'm trying to focus on what seems to be the key information. > Hi Mark -- So sorry for going silent for a couple of days. Our organization is neck-deep in a huge compliance project. Combine that with this issue we're working on together, and it's Perfect Storm time around here. We have a big meeting with the client and vendor tomorrow about all this and I'm working like heck to prevent this important customer from jumping ship. Now back to it! > > Overview: > A small number of requests are receiving a completely empty (no headers, > no body) response. > Just a FIN packet and that's all. > Environment > Tomcat 7.0.72 > - BIO HTTP (issue also observed with NIO) > - Source unknown (probably ASF) > Java 1.8.0_221, Oracle > CentOS 7.5, Azure > Nginx reverse proxy > - Using HTTP/1.0 > - No keep-alive > - No compression > No (known) environment changes in the time period where this issue started > > Results from debug logging > - The request is read without error > - The connection close is initiated from the Tomcat/Java side > - The socket is closed before Tomcat tries to write the response > - The application is not triggering the close of the socket > - Tomcat is not triggering the close of the socket > - When Tomcat does try and write we see the following exception > java.net.SocketException: Bad file descriptor (Write failed) > > We have confirmed that the Java process is not hitting the limit for file > descriptors. > > The file descriptor must have been valid when the request was read from > the socket. > > The first debug log shows 2 other active connections from Nginx to Tomcat at > the point the connection is closed unexpectedly. > > The second debug log shows 1 other active connection from Nginx to Tomcat > at the point the connection is closed unexpectedly. > > The third debug log shows 1 other active connection from Nginx to Tomcat at > the point the connection is closed unexpectedly. > > The fourth debug log shows no other active connection from Nginx to > Tomcat at the point the connection is closed unexpectedly. > > > Analysis > > We know the connection close isn't coming from Tomcat or the application. > That leaves: > - the JVM > - the OS > - the virtualisation layer (since this is Azure I am assuming there is > one) > > We are approaching the limit of what we can debug via Tomcat (and my area > of expertise. The evidence so far is pointing to an issue lower down the > network stack (JVM, OS or virtualisation layer). > Can't disagree with you there. > I think the next, and possibly last, thing we can do from Tomcat is log some > information on the file descriptor associated with the socket. That is going > to > require some reflection to read JVM internals. > > Patch files here: > http://home.apache.org/~markt/dev/v7.0.72-custom-patch-v4/ > > Source code here: > https://github.com/markt-asf/tomcat/tree/debug-7.0.72 > I will apply these tonight. > The file descriptor usage count is guarded by a lock object so this patch adds > quite a few syncs. For the load you are seeing that shouldn't an issue but > there is a change it will impact performance. > Based on observation of load, I'm not too concerned about that. Maybe a little. I'll keep an eye on it. > The aim with this logging is to provide evidence of whether or not there is a > file descriptor handling problem in the JRE. My expectation is that with these > logs we will have reached the limit of what we can do with Tomcat but will be > able to point you in the right direction for further investigation. > I'll get this done right away. > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Timeout waiting to read data from client
On 11/9/2020 11:05 AM, Jerry Malcolm wrote: On 11/9/2020 3:10 AM, Mark Thomas wrote: On 08/11/2020 01:33, Jerry Malcolm wrote: On 11/7/2020 6:56 PM, Christopher Schultz wrote: Jerry, On 11/6/20 19:49, Jerry Malcolm wrote: I have a relatively new environment with a standalone tomcat (8.5) running on an AWS Linux2 EC2. I'm not using HTTPD/AJP. Its a direct connection to port 443. (Well technically, I have firewallD in the flow in order to route the protected port 80 to port 8080 and 443 to 8443 for TC). I am doing some stress testing on the server and failing miserably. I am sending around 130 ajax calls in rapid succession using HTTP/2. These are all very simple small page (JSP) requests. Not a lot of processing required. The first ~75 requests process normally. Then everything hangs up. In the tomcat logs I'm getting a bunch of "Timeout waiting to read data from client" exceptions. And in the stacktrace for these exceptions, they are all occurring when I'm trying to access a parameter from the request. Looking at the request network timing in the browser console, I see a bunch of the requests returning in typical time of a few milliseconds. Then another large block of requests that all start returning around 4 seconds, then another block that wait until 8 seconds to return. I've tried firefox and chrome with the same results. I've been using httpd in front of TC for years. So this is the first time I'm running TC standalone. It is very likely I've got some parameters set horribly wrong. But I have no clue where to start. This is not a tiny EC2, and my internet connection is not showing any signs of problems. So I really don't think this is a performance-related problem. The problem is very consistent and reproducible with the same counts of success/failure calls. What could be causing the "Timeout waiting to read data from client" after 75 calls, and then cause blocks of calls to wait 4 seconds, 8 seconds, etc before responding? I really need to handle more simultaneous load that this is currently allowing. Thanks in advance for the education. Are you using HTTP Keepalives on your connections? Are you actually re-using those connections in your test? What is your keepalive timeout on your . Actually, what is your whole configuration? -chris Hi Chris, here are my two connector definitions from server.xml: How are you stress testing this? All on a single HTTP/2 connection or multiple connections? With which tool? You might want to test HTTP/1.1 requests (with and without TLS) to see if the problem is specific to HTTP/2 or TLS as that should help narrow down the root cause. Mark Hi Mark, technically it's not a 'designed' stress test. It's real production code that just happens to stress the server more than usual. It's just a page that makes a bunch of ajax calls, and the responses to each of those issue a second ajax call. If you don't see anything obvious in my configuration, we will definitely pursue the http/1.1 options, etc. I just wanted to eliminate the chance of obvious 'pilot error' before digging deeper. Specifically, where is that error detected in the TC flow? In my logs it fails on getting request parameters. It sounds like the input reader for the request is getting blocked. But the first part of the request is getting in since it does route to the appropriate JSP. Just seems strange that the http/2 or ssl layers would let half of the request in and then block the rest of the request. The browser appears to be sending everything. And it fails the same using firefox or chrome. Any ideas? Thx Update on this. One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM after things locked up. I removed the http2 'upgrade protocol' line from my connector, and everything works. So it's apparently something wrong with my http2 setup. Ideas? (See my connector config above in this thread). Thx
Re: NGINX + tomcat 8.0.35 (110: Connection timed out)
Chris,
I was load testing using the ec2 load balancer dns. I have increased the
connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I am
not seeing connection timeout in nginx logs now. No errors in kernel.log I
am not seeing any errors in tomcat catalina.out.
During regular operations when the request count is between 4 to 6k
requests per minute the open files count for the tomcat process is between
200 to 350. Responses from tomcat are within 5 seconds.
If the requests count goes beyond 6.5 k open files slowly move up to 2300
to 3000 and the request responses from tomcat become slow.
I am not concerned about high open files as I do not see any errors related
to open files. Only side effect of open files going above 700 is the
response from tomcat is slow. I checked if this is caused from elastic
search, aws cloud watch shows elastic search response is within 5
milliseconds.
what might be the reason that when the open files goes beyond 600, it slows
down the response time for tomcat. I tried with tomcat 9 and it's the same
behavior
On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:
> Ayub,
>
> On 11/3/20 10:56, Ayub Khan wrote:
> > *I'm curious about why you are using all of cloudflare and ALB and
> > nginx.Seems like any one of those could provide what you are getting from
> > all3 of them. *
> >
> > Cloudflare is doing just the DNS and nginx is doing ssl termination
>
> What do you mean "Cloudflare is doing just the DNS?"
>
> So what is ALB doing, then?
>
> > *What is the maximum number of simultaneous requests that one
> nginxinstance
> > will accept? What is the maximum number of simultaneous proxiedrequests
> one
> > nginx instance will make to a back-end Tomcat node? Howmany nginx nodes
> do
> > you have? How many Tomcat nodes? *
> >
> > We have 4 vms each having nginx and tomcat running on them and each
> tomcat
> > has nginx in front of them to proxy the requests. So it's one Nginx
> > proxying to a dedicated tomcat on the same VM.
>
> Okay.
>
> > below is the tomcat connector configuration
> >
> > > connectionTimeout="6" maxThreads="2000"
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > URIEncoding="UTF-8"
> > redirectPort="8443" />
>
> 60 seconds is a *long* time for a connection timeout.
>
> Do you actually need 2000 threads? That's a lot, though not insane. 2000
> threads means you expect to handle 2000 concurrent (non-async,
> non-Wewbsocket) requests. Do you need that (per node)? Are you expecting
> 8000 concurrent requests? Does your load-balancer understand the
> topography and current-load on any given node?
>
> > When I am doing a load test of 2000 concurrent users I see the open files
> > increase to 10,320 and when I take thread dump I see the threads are in a
> > waiting state.Slowly as the requests are completed I see the open files
> > come down to normal levels.
>
> Are you performing your load-test against the CF/ALB/nginx/Tomcat stack,
> or just hitting Tomcat (or nginx) directly?
>
> Are you using HTTP keepalive in your load-test (from the client to
> whichever server is being contacted)?
>
> > The output of the below command is
> > sudo cat /proc/sys/kernel/pid_max
> > 131072
> >
> > I am testing this on a c4.8xlarge VM in AWS.
> >
> > below is the config I changed in nginx.conf file
> >
> > events {
> > worker_connections 5;
> > # multi_accept on;
> > }
>
> This will allow 50k incoming connections, and Tomcat will accept an
> unbounded number of connections (for NIO connector). So limiting your
> threads to 2000 only means that the work of each request will be done in
> groups of 2000.
>
> > worker_rlimit_nofile 3;
>
> I'm not sure how many connections are handled by a single nginx worker.
> If you accept 50k connections and only allow 30k file handles, you may
> have a problem if that's all being done by a single worker.
>
> > What would be the ideal config for tomcat and Nginx so this setup on
> > c4.8xlarge vm could serve at least 5k or 10k requests simultaneously
> > without causing the open files to spike to 10K.
>
> You will never be able to serve 10k simultaneous requests without having
> 10k open files on the server. If you mean 10k requests across the whole
> 4-node environment, then I'd expect 10k requests to open (roughly) 2500
> open files on each server. And of course, you need all kinds of other
> files open as well, from JAR files to DB connections or other network
> connections.
>
> But each connection needs a file descriptor, full stop. If you need to
> handle 10k connections, then you will need to make it possible to open
> 10k file handles /just for incoming network connections/ for that
> process. There is no way around it.
>
> Are you trying to hit a performance target or are you actively getting
> errors with a particular configuration? Your subject says "Connection
> Timed Out".
Re: Max heap size when using -XX:MaxRAMPercentage
Christopher, thanks for your reply. Answers inline. > > What I see in the profiler is Max Heap Size 310378496 B, which is only 0.28 > > GiB. > > This is probably the maximum size of the /used/ heap. Can you confirm that? Nope. Netbeans profiler says: Max Heap Size 310378496 B, Max Used Heap 261052168 B > Try running your application without *any* memory parameters and run it > through a reasonable exercise. Then look at everything with your > profiles *and also* make sure you look at the OS's report of the > allocated memory for the process. That will give you a sense for how > much "native memory" versus heap memory your application actually needs. > You may find that the native memory requirements are roughly the same > size as the Java heap requirements. That means you can't even reasonably > approach a 50% heap size because you still need to be about to run the > OS which takes a bit of memory. I'll try that now. I've seen people recommending up to 80% RAM though: https://dev.to/focusedlabs/the-no-nonsense-guide-to-jvm-14-memory-on-kubernetes-508m - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Max heap size when using -XX:MaxRAMPercentage
Martynas, On 11/11/20 10:20, Martynas Jusevičius wrote: I am attempting to make my webapps that run in Tomcat to take advantage of the Docker container-specific JVM options such as -XX:MaxRAMPercentage: https://www.eclipse.org/openj9/docs/xxinitialrampercentage/ TL;DR It allows specifying JVM heap as % of the container memory limit rather than using the default 25%. Running tomcat:9.0.39-jdk11, what I did is specify CATALINA_OPTS=-XX:MaxRAMPercentage=75 which should use 75% of the container memory limit which is 3.702GiB in this case. This is probably going to be a problem for you. The heap you are specifying is the Java heap only. There are all kinds of other memory spaces the JVM needs in order to run. You may find that 75% of the available RAM is so large that you can't do other things. So I login into the container and execute ps aux to see Tomcat's command arguments: /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:MaxRAMPercentage=75.0 -Duser.timezone=Europe/Copenhagen -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start If I replace "org.apache.catalina.startup.Bootstrap start" with "-XshowSettings:vm -version" and run, this is what I get: Max. Heap Size (Estimated): 2.78G Using VM: OpenJDK 64-Bit Server VM Where 2.78G is 75% of 3.7GiB -- so far so good, the MaxRAMPercentage setting seems to have worked here. But I'm still getting "Cannot allocate memory" exceptions that kill the container when importing large datasets. So I attached a profiler. What I see in the profiler is Max Heap Size 310378496 B, which is only 0.28 GiB. This is probably the maximum size of the /used/ heap. Can you confirm that? So how much max heap does my webapp have access to -- is it 2.78G or 0.28G? If it's 0.28, why? How do I enforce the 75% RAM setting for the webapp? It's probably the 2.78G. You are likely getting "Cannot allocate memory" errors because the JVM is trying to allocate non-heap memory. Try running your application without *any* memory parameters and run it through a reasonable exercise. Then look at everything with your profiles *and also* make sure you look at the OS's report of the allocated memory for the process. That will give you a sense for how much "native memory" versus heap memory your application actually needs. You may find that the native memory requirements are roughly the same size as the Java heap requirements. That means you can't even reasonably approach a 50% heap size because you still need to be about to run the OS which takes a bit of memory. Hope that helps, -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Suvendu, On 11/11/20 06:40, Suvendu Sekhar Mondal wrote: Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 As Michael pointed-out, these all need to be upgraded, but they are not affecting your experience, here. Observations: After database failover, we noticed that HTTP thread pool size jumped from 10 to 50. No. of busy threads jumped to 40 for a minute and then came down to normal range(1-5). But thread pool size remained 50 for the next one hour. When we collected the heap dump after one hour, thread pool size jumped again to ~90. Thread dump analysis is showing most of the HTTP threads were just waiting to get some work[1]. That looked surprising to me. Few of them are runnable and doing some work - which matches "busy thread" count. During all this timeframe load on the system remained the same. We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We do not have any keepAliveTimeout set in Tomcat. App uses session affinity. We were looking for some connector properties which deal with idle threads but found none[3]. You won't find them. We are not seeing any "problem" in our app after DB failover but this is something also newer to us and we are trying to explain it. What are you trying to explain? Tomcat's memory usage increased by a few MB as new threads were spawned. My expectation was that if threads in the thread pool are not getting used and idle, after sometime thread pool size will reduce to "minSpareThreads" size which is 10 in our case. Is that expectation wrong? Can you please share your thoughts? This is a common misconception. If you do not configure an "executor" then Tomcat will create one for you and it will NOT remove idle threads from the thread pool. The solution is to manually-specify an Executor: minSpareThreads="Y" /> With this configuration, you should see your thread count stay between Y and X, but when X are not necessary, your thread count should drop towards Y as appropriate. Hope that helps, -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Max heap size when using -XX:MaxRAMPercentage
Hi, I am attempting to make my webapps that run in Tomcat to take advantage of the Docker container-specific JVM options such as -XX:MaxRAMPercentage: https://www.eclipse.org/openj9/docs/xxinitialrampercentage/ TL;DR It allows specifying JVM heap as % of the container memory limit rather than using the default 25%. Running tomcat:9.0.39-jdk11, what I did is specify CATALINA_OPTS=-XX:MaxRAMPercentage=75 which should use 75% of the container memory limit which is 3.702GiB in this case. So I login into the container and execute ps aux to see Tomcat's command arguments: /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -XX:MaxRAMPercentage=75.0 -Duser.timezone=Europe/Copenhagen -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start If I replace "org.apache.catalina.startup.Bootstrap start" with "-XshowSettings:vm -version" and run, this is what I get: Max. Heap Size (Estimated): 2.78G Using VM: OpenJDK 64-Bit Server VM Where 2.78G is 75% of 3.7GiB -- so far so good, the MaxRAMPercentage setting seems to have worked here. But I'm still getting "Cannot allocate memory" exceptions that kill the container when importing large datasets. So I attached a profiler. What I see in the profiler is Max Heap Size 310378496 B, which is only 0.28 GiB. So how much max heap does my webapp have access to -- is it 2.78G or 0.28G? If it's 0.28, why? How do I enforce the 75% RAM setting for the webapp? Martynas atomgraph.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat - Sentry configuration problem
On Wed, Nov 11, 2020 at 3:16 PM Tamás Barta wrote: > Hi, > > > Why do you use 1.7.29 ? > > Because our company has a Sentry server supports this version of clients. > > > Do you specify this class name in logging.properties ? Is > this class in 1.7.29 ? > > Yes and yes. If I add sentry jars to classpath, sentry works well. > Exceptions are sent to the server and visible in web interface. > > > You just need to add it as a normal Maven dependency. > > Maven dependency is needed only If I would like to use Sentry in code. But > I don't. > > > If you need to really put it on the bootstrap. > > When I say bootstrap I mean Tomcat bootstrap: > https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html > Which is standard java classpath, I see. > > BTW if I add jars to java bootclasspath, I got the same error. > > Just a very interesting test: if I add an empty JAR (no class files in it) > file to setenv.sh (no Sentry jars at all) that I got the same error. > It seems that adding an empty jar to classpath case this classloading > problem :( > I've just tested this on 10.0.0-M9 Using CATALINA_BASE: /home/martin/devel/apache-tomcat-10.0.0-M9 Using CATALINA_HOME: /home/martin/devel/apache-tomcat-10.0.0-M9 Using CATALINA_TMPDIR: /home/martin/devel/apache-tomcat-10.0.0-M9/temp Using JRE_HOME:/home/martin/devel/jdk-15 Using CLASSPATH: :/home/martin/devel/apache-tomcat-10.0.0-M9/empty.jar:/home/martin/devel/apache-tomcat-10.0.0-M9/bin/bootstrap.jar:/home/martin/devel/apache-tomcat-10.0.0-M9/bin/tomcat-juli.jar Using CATALINA_OPTS: NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED $ cat bin/setenv.sh export CLASSPATH=$CLASSPATH:$CATALINA_BASE/empty.jar $ ll empty.jar -rw-r- 1 martin martin 22 ное 11 15:48 empty.jar $ jar --list -f empty.jar There is no problem with loading this empty jar and there are no class loading issues either. > > Regards, Tamas > > > > > > > On Wed, Nov 11, 2020 at 12:33 PM Martin Grigorov > wrote: > > > Hi, > > > > On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta > wrote: > > > > > Hi, > > > > > > The ways you wrote is not usable, because they cause the following > > problem: > > > > > > Handler error > > > java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > > at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > at > > > > > > > > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582) > > > at > > > > > > > > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524) > > > at > > > > > > > > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322) > > > at java.util.logging.LogManager$3.run(LogManager.java:399) > > > at java.util.logging.LogManager$3.run(LogManager.java:396) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at > > > > > > > > > java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396) > > > at java.util.logging.LogManager.access$800(LogManager.java:145) > > > at java.util.logging.LogManager$2.run(LogManager.java:345) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at > > > > > > > > > java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338) > > > at > > java.util.logging.LogManager.getLogManager(LogManager.java:378) > > > at java.util.logging.Logger.demandLogger(Logger.java:448) > > > at java.util.logging.Logger.getLogger(Logger.java:502) > > > at > > > org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67) > > > at > > > org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187) > > > at > > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117) > > > at > > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141) > > > at > org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196) > > > at > > > org.apache.catalina.startup.Bootstrap.(Bootstrap.java:52) > > > 10-Nov-2020 17:44:11.137 INFO [main] > > > org.apache.catalina.startup.VersionLoggerListener.log Server version: > > > Apache Tomcat/8.5.23 > > > 10-Nov-2020 17:44:11.139 INFO [main] > > > org.apache.catalina.startup.VersionLoggerListener.log Server built: > > > Sep 28 2017 10:30:11 UTC > > > ... > > > > > > The Sentry log handler must be in Boostrap jars, it must be loaded > before > > > web application and before jars in $CATALINA_BASE/lib. If I am right, > the > > > best way to do that is by setting it in setenv.sh. >
Re: Tomcat - Sentry configuration problem
Hi, > Why do you use 1.7.29 ? Because our company has a Sentry server supports this version of clients. > Do you specify this class name in logging.properties ? Is this class in 1.7.29 ? Yes and yes. If I add sentry jars to classpath, sentry works well. Exceptions are sent to the server and visible in web interface. > You just need to add it as a normal Maven dependency. Maven dependency is needed only If I would like to use Sentry in code. But I don't. > If you need to really put it on the bootstrap. When I say bootstrap I mean Tomcat bootstrap: https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html Which is standard java classpath, I see. BTW if I add jars to java bootclasspath, I got the same error. Just a very interesting test: if I add an empty JAR (no class files in it) file to setenv.sh (no Sentry jars at all) that I got the same error. It seems that adding an empty jar to classpath case this classloading problem :( Regards, Tamas On Wed, Nov 11, 2020 at 12:33 PM Martin Grigorov wrote: > Hi, > > On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta wrote: > > > Hi, > > > > The ways you wrote is not usable, because they cause the following > problem: > > > > Handler error > > java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > at > > > > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582) > > at > > > > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524) > > at > > > > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322) > > at java.util.logging.LogManager$3.run(LogManager.java:399) > > at java.util.logging.LogManager$3.run(LogManager.java:396) > > at java.security.AccessController.doPrivileged(Native Method) > > at > > > > > java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396) > > at java.util.logging.LogManager.access$800(LogManager.java:145) > > at java.util.logging.LogManager$2.run(LogManager.java:345) > > at java.security.AccessController.doPrivileged(Native Method) > > at > > > > > java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338) > > at > java.util.logging.LogManager.getLogManager(LogManager.java:378) > > at java.util.logging.Logger.demandLogger(Logger.java:448) > > at java.util.logging.Logger.getLogger(Logger.java:502) > > at > > org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67) > > at > > org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187) > > at > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117) > > at > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141) > > at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196) > > at > > org.apache.catalina.startup.Bootstrap.(Bootstrap.java:52) > > 10-Nov-2020 17:44:11.137 INFO [main] > > org.apache.catalina.startup.VersionLoggerListener.log Server version: > > Apache Tomcat/8.5.23 > > 10-Nov-2020 17:44:11.139 INFO [main] > > org.apache.catalina.startup.VersionLoggerListener.log Server built: > > Sep 28 2017 10:30:11 UTC > > ... > > > > The Sentry log handler must be in Boostrap jars, it must be loaded before > > web application and before jars in $CATALINA_BASE/lib. If I am right, the > > best way to do that is by setting it in setenv.sh. > > > > I haven't used Sentry before so I just looked it up. > https://docs.sentry.io/platforms/java/usage/ says that the current version > is 3.1.1. Why do you use 1.7.29 ? There is no io.sentry.jul.SentryHandler > class in 3.1.1. Do you specify this class name in logging.properties ? Is > this class in 1.7.29 ? > There is nothing about "bootstrap" on > https://docs.sentry.io/platforms/java/usage/ or at > https://docs.sentry.io/platforms/java/guides/spring-boot/. You just need > to > add it as a normal Maven dependency. > If you need to really put it on the bootstrap path then you need to use > -Xbootclasspath:... > By using CLASSPATH=$CLASSPATH:... you actually *append* jars to the > classpath, while you sound like you need to *prepend* to it. > > > > > > I can't imagine how could it be possible without setting sentry jar the > web > > application works well, there is no problem, but if I add > sentry-1.7.29.jar > > to classpath then ServletContainerInitializer won't be found by Catalina. > > This class is only in $CATALINA_BASE/lib/servlet-api.jar. > > > > Thanks, Tamás > > > > On Wed, Nov 11, 2020 at 9:46 AM Martin Grigorov > > wrote: > > > > > Hi, > > > > > > On Wed, Nov 11, 2020 at 10:36 AM
Re: HTTP thread pool size remained high after database failover
Hi Michael, On Wed, Nov 11, 2020 at 6:03 PM Michael Osipov wrote: > > Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal: > > Hello Everyone, > > > > During database failover resiliency testing we noticed something > > unusual which we still cannot explain. So, reaching out to the > > community for help. > > > > Application Setup: > > AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server > > 2016 > > Almost of them them are outdated, you should upgrade first. I know. We will be upgrading them in the next release. For this release we have to use aforementioned softwares. For the last three releases we used the same softwares but had not noticed this issue - maybe we were not looking that closely. Now, we are seeing this and can't let it go without explaining it. :) So, any thoughts on this? > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP thread pool size remained high after database failover
Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal: Hello Everyone, During database failover resiliency testing we noticed something unusual which we still cannot explain. So, reaching out to the community for help. Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 Almost of them them are outdated, you should upgrade first. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HTTP thread pool size remained high after database failover
Hello Everyone, During database failover resiliency testing we noticed something unusual which we still cannot explain. So, reaching out to the community for help. Application Setup: AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016 Observations: After database failover, we noticed that HTTP thread pool size jumped from 10 to 50. No. of busy threads jumped to 40 for a minute and then came down to normal range(1-5). But thread pool size remained 50 for the next one hour. When we collected the heap dump after one hour, thread pool size jumped again to ~90. Thread dump analysis is showing most of the HTTP threads were just waiting to get some work[1]. That looked surprising to me. Few of them are runnable and doing some work - which matches "busy thread" count. During all this timeframe load on the system remained the same. We have 30sec connectionTimeout. Also TTL is set to 300 in httpd. We do not have any keepAliveTimeout set in Tomcat. App uses session affinity. We were looking for some connector properties which deal with idle threads but found none[3]. We are not seeing any "problem" in our app after DB failover but this is something also newer to us and we are trying to explain it. Tomcat's memory usage increased by a few MB as new threads were spawned. My expectation was that if threads in the thread pool are not getting used and idle, after sometime thread pool size will reduce to "minSpareThreads" size which is 10 in our case. Is that expectation wrong? Can you please share your thoughts? [1] "http-apr-18110-exec-63" #2170 daemon prio=5 os_prio=0 tid=0x45f03000 nid=0x30d4 waiting on condition [0x5a4cf000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0006a5400178> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None [2] [3] https://tomcat.apache.org/tomcat-7.0-doc/config/http.html Thanks & Regards, Suvendu - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat - Sentry configuration problem
Hi, On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta wrote: > Hi, > > The ways you wrote is not usable, because they cause the following problem: > > Handler error > java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582) > at > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524) > at > > org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322) > at java.util.logging.LogManager$3.run(LogManager.java:399) > at java.util.logging.LogManager$3.run(LogManager.java:396) > at java.security.AccessController.doPrivileged(Native Method) > at > > java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396) > at java.util.logging.LogManager.access$800(LogManager.java:145) > at java.util.logging.LogManager$2.run(LogManager.java:345) > at java.security.AccessController.doPrivileged(Native Method) > at > > java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338) > at java.util.logging.LogManager.getLogManager(LogManager.java:378) > at java.util.logging.Logger.demandLogger(Logger.java:448) > at java.util.logging.Logger.getLogger(Logger.java:502) > at > org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67) > at > org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187) > at > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117) > at > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141) > at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196) > at > org.apache.catalina.startup.Bootstrap.(Bootstrap.java:52) > 10-Nov-2020 17:44:11.137 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Server version: > Apache Tomcat/8.5.23 > 10-Nov-2020 17:44:11.139 INFO [main] > org.apache.catalina.startup.VersionLoggerListener.log Server built: > Sep 28 2017 10:30:11 UTC > ... > > The Sentry log handler must be in Boostrap jars, it must be loaded before > web application and before jars in $CATALINA_BASE/lib. If I am right, the > best way to do that is by setting it in setenv.sh. > I haven't used Sentry before so I just looked it up. https://docs.sentry.io/platforms/java/usage/ says that the current version is 3.1.1. Why do you use 1.7.29 ? There is no io.sentry.jul.SentryHandler class in 3.1.1. Do you specify this class name in logging.properties ? Is this class in 1.7.29 ? There is nothing about "bootstrap" on https://docs.sentry.io/platforms/java/usage/ or at https://docs.sentry.io/platforms/java/guides/spring-boot/. You just need to add it as a normal Maven dependency. If you need to really put it on the bootstrap path then you need to use -Xbootclasspath:... By using CLASSPATH=$CLASSPATH:... you actually *append* jars to the classpath, while you sound like you need to *prepend* to it. > > I can't imagine how could it be possible without setting sentry jar the web > application works well, there is no problem, but if I add sentry-1.7.29.jar > to classpath then ServletContainerInitializer won't be found by Catalina. > This class is only in $CATALINA_BASE/lib/servlet-api.jar. > > Thanks, Tamás > > On Wed, Nov 11, 2020 at 9:46 AM Martin Grigorov > wrote: > > > Hi, > > > > On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta > wrote: > > > > > Hi, > > > > > > Now it is a Spring Boot web application project without embedded > tomcat. > > > There is no library in war with javax.servlet api classes. There error > is > > > the same: > > > > > > 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1] > > > org.apache.catalina.core.ContainerBase.addChildInternal > > > ContainerBase.addChild: start: > > > org.apache.catalina.LifecycleException: Failed to start component > > > > > > > > > [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]] > > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167) > > > at > > > > > > > > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) > > > at > > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) > > > at > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) > > > at > org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988) > > > at > > > > > > org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860) > > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > > at
Re: Tomcat - Sentry configuration problem
Hi, The ways you wrote is not usable, because they cause the following problem: Handler error java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582) at org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524) at org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322) at java.util.logging.LogManager$3.run(LogManager.java:399) at java.util.logging.LogManager$3.run(LogManager.java:396) at java.security.AccessController.doPrivileged(Native Method) at java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396) at java.util.logging.LogManager.access$800(LogManager.java:145) at java.util.logging.LogManager$2.run(LogManager.java:345) at java.security.AccessController.doPrivileged(Native Method) at java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338) at java.util.logging.LogManager.getLogManager(LogManager.java:378) at java.util.logging.Logger.demandLogger(Logger.java:448) at java.util.logging.Logger.getLogger(Logger.java:502) at org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67) at org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187) at org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117) at org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141) at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196) at org.apache.catalina.startup.Bootstrap.(Bootstrap.java:52) 10-Nov-2020 17:44:11.137 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.23 10-Nov-2020 17:44:11.139 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Sep 28 2017 10:30:11 UTC ... The Sentry log handler must be in Boostrap jars, it must be loaded before web application and before jars in $CATALINA_BASE/lib. If I am right, the best way to do that is by setting it in setenv.sh. I can't imagine how could it be possible without setting sentry jar the web application works well, there is no problem, but if I add sentry-1.7.29.jar to classpath then ServletContainerInitializer won't be found by Catalina. This class is only in $CATALINA_BASE/lib/servlet-api.jar. Thanks, Tamás On Wed, Nov 11, 2020 at 9:46 AM Martin Grigorov wrote: > Hi, > > On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta wrote: > > > Hi, > > > > Now it is a Spring Boot web application project without embedded tomcat. > > There is no library in war with javax.servlet api classes. There error is > > the same: > > > > 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1] > > org.apache.catalina.core.ContainerBase.addChildInternal > > ContainerBase.addChild: start: > > org.apache.catalina.LifecycleException: Failed to start component > > > > > [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]] > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167) > > at > > > > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) > > at > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) > > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) > > at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988) > > at > > > org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > Caused by: java.lang.NoClassDefFoundError: > > javax/servlet/ServletContainerInitializer > > at java.lang.ClassLoader.defineClass1(Native Method) > > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > > at java.security.AccessController.doPrivileged(Native Method) > > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > > at
Re: Tomcat - Sentry configuration problem
Hi, On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta wrote: > Hi, > > Now it is a Spring Boot web application project without embedded tomcat. > There is no library in war with javax.servlet api classes. There error is > the same: > > 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1] > org.apache.catalina.core.ContainerBase.addChildInternal > ContainerBase.addChild: start: > org.apache.catalina.LifecycleException: Failed to start component > > [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]] > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167) > at > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) > at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) > at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988) > at > org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NoClassDefFoundError: > javax/servlet/ServletContainerInitializer > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:763) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > at java.lang.ClassLoader.loadClass(ClassLoader.java:411) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1277) > at > > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:348) > at > > org.apache.catalina.startup.WebappServiceLoader.loadServices(WebappServiceLoader.java:188) > at > > org.apache.catalina.startup.WebappServiceLoader.load(WebappServiceLoader.java:159) > at > > org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1622) > at > > org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1135) > at > > org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775) > at > > org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299) > at > > org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94) > at > > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5105) > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > ... 10 more > Caused by: java.lang.ClassNotFoundException: > javax.servlet.ServletContainerInitializer > Definitely you mess up the classpath and there are none or more than one servlet-api.jar in the classpath. What happens if you put your sentry related jars in the .war file (/WEB-INF/lib/) ? I.e. add them as proper Maven dependencies Or if you put them in $CATALINA_BASE/lib ? Why did you go with editing setenv.sh in first place ? > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 38 more > > On Tue, Nov 10, 2020 at 11:28 PM Christopher Schultz < > ch...@christopherschultz.net> wrote: > > > Tamás, > > > > On 11/10/20 15:42, Tamás Barta wrote: > > > This is the content of setenv.sh: > > > > > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/sentry-1.7.29.jar > > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/slf4j-api-1.7.24.jar > > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/jackson-core-2.8.10.jar > > > > > > The web application without this sentry configuration works well for > > ages. > > > But if I add the above configuration, it won't. > > > > > > Yes, it is a maven project, now I checked and I see that there are > Tomcat > > > jar-s in it: > > > > > > tomcat-annotations-api-8.5.23.jar tomcat-embed-core-8.5.23.jar > > > tomcat-embed-el-8.5.23.jar
Re: Tomcat - Sentry configuration problem
Hi, Now it is a Spring Boot web application project without embedded tomcat. There is no library in war with javax.servlet api classes. There error is the same: 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1] org.apache.catalina.core.ContainerBase.addChildInternal ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: javax/servlet/ServletContainerInitializer at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1277) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.catalina.startup.WebappServiceLoader.loadServices(WebappServiceLoader.java:188) at org.apache.catalina.startup.WebappServiceLoader.load(WebappServiceLoader.java:159) at org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1622) at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1135) at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775) at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299) at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5105) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) ... 10 more Caused by: java.lang.ClassNotFoundException: javax.servlet.ServletContainerInitializer at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 38 more On Tue, Nov 10, 2020 at 11:28 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > Tamás, > > On 11/10/20 15:42, Tamás Barta wrote: > > This is the content of setenv.sh: > > > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/sentry-1.7.29.jar > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/slf4j-api-1.7.24.jar > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/jackson-core-2.8.10.jar > > > > The web application without this sentry configuration works well for > ages. > > But if I add the above configuration, it won't. > > > > Yes, it is a maven project, now I checked and I see that there are Tomcat > > jar-s in it: > > > > tomcat-annotations-api-8.5.23.jar tomcat-embed-core-8.5.23.jar > > tomcat-embed-el-8.5.23.jar tomcat-embed-websocket-8.5.23.jar > > tomcat-jdbc-8.5.23.jar tomcat-juli-8.5.23.jar > > > > Maybe the problem is that Spring Boot application contains embed Tomcat > > jars? This is very interesting that this extra Sentry jar in classpath > > cause this problem. > > Wait... you are using Spring Boot and you are also using setenv.sh? > > I'm very confused. > > I wouldn't expect Sentry and its dependencies to cause a problem. > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail:
