Re: Recognizing Certificate Updates
Hi Christopher, if I manage to write a code that I think would help others regarding Letsencrypt/SSL issues, I'll send it to you. In the meantime these instructions sent by Peter sounds good enough: curl -u " https://localhost:8443/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443=reloadSslHostConfigs “ Add a to tomcat-users.xml Beware not to open the Manager App to the public - just localhost. Thank you, Mladen On Tue, Dec 29, 2020 at 3:42 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > Mladen, > > On 12/29/20 03:46, Mladen Adamović wrote: > > On Tue, Dec 29, 2020 at 3:18 AM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >>> Honestly, I thought that reloadAfterNDays param to server.xml would be > >>> better, but admins didn't have an understanding on this topic. > >> > >> Don't be a jerk. We understand it. We are just saying that we want it > >> built in stages. If you want radical changes, you'll need to work on a > >> server without a decades-long history of being stable and reliable. > >> > > > > Well, one thing is certainly correct here - that Tomcat at least in 2016 > > wasn't working properly on my server, Numbeo.com. The problem was noticed > > in the past few months and the update to 9.0.47 solved the issue, so > indeed > > Tomcat doesn't have a stable and reliable history. I haven't complained > > about it although. > > > > Regarding me being the jerk, I haven't seen regarding reloadAfterNDays > > param that any project maintainer said something like: "I think that's a > > good idea. If you create that code, I'll review it". > > We said "write it as a Valve and we'll review it." Maybe not word for > word, but I've tried to be encouraging about you going in that > direction. Everyone else seems to be ignoring you thus far. If you > continue to be an ass, I'll ignore you, too. > > > So from my point of view, there wasn't understanding. > > > > It looked that Romain and you want a full ACME client without > dependencies > > so that Tomcat could run in containers with SSL, while it's a valid idea, > > it seems I wouldn't be the one building that. > > > > There are a few reasons, i.e. I have "newbie Tomcat devs problems", and > > I'm not so motivated to work on a feature that makes more sense for big > > corporations rather than a single small developers. > > > > To note even my question to explain to add Class javadoc > > for LifecycleMBeanBase stayed unanswered so far in dev list, to my > surprise. > > You posted that on December 27th at 02:45am in my time zone. I wasn't > exactly looking at email around then. Or at all on Sunday. OR really > much yesterday, the 28th. I'm on holiday, like a LOT of other people > right now. > > Something that may seem like an emergency to you just ... is not so in > the eyes of all the *unpaid volunteers* who work on this project. > > FTR, that's a base class for implementations of Lifecycle that also adds > useful methods for any class which needs to implement both Lifecycle and > also be an MBean. It it were to have class-level javadoc it would be > something like "utility methods for things that subclass this class". > So... not terribly helpful to someone who doesn't know what it was, > originally. But it's also difficult to explain in clas-level javadoc > *why* someone would want to extend that particular class. > > > Back in 2007, I was good enough so that Google picked me to develop the > > software for them, I left to start my own business two years later, but I > > have a history of not being a good team player, seeing the same things > > differently than other people, and also I don't have a history of > > contributing to open-source projects (unless started by myself), so > > perhaps, at the end of the day, I'm not the good fit for Tomcat dev. > > Anyway, as it looks now, I'll unsubscribe soon from the Tomcat dev email > > list as it looks to me that I didn't fit. > > You can certainly take your ball and go home, but then everyone loses, > right? > > If you are motivated to work on this, we are happy to help you. > > If you are instead motivated to insult everyone, complain that nobody is > paying attention to your pet project, and refuse to accept the help and > direction provided, then we aren't very interested. > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Recognizing Certificate Updates
On Tue, Dec 29, 2020 at 3:18 AM Christopher Schultz < ch...@christopherschultz.net> wrote: > > Honestly, I thought that reloadAfterNDays param to server.xml would be > > better, but admins didn't have an understanding on this topic. > > Don't be a jerk. We understand it. We are just saying that we want it > built in stages. If you want radical changes, you'll need to work on a > server without a decades-long history of being stable and reliable. > Well, one thing is certainly correct here - that Tomcat at least in 2016 wasn't working properly on my server, Numbeo.com. The problem was noticed in the past few months and the update to 9.0.47 solved the issue, so indeed Tomcat doesn't have a stable and reliable history. I haven't complained about it although. Regarding me being the jerk, I haven't seen regarding reloadAfterNDays param that any project maintainer said something like: "I think that's a good idea. If you create that code, I'll review it". So from my point of view, there wasn't understanding. It looked that Romain and you want a full ACME client without dependencies so that Tomcat could run in containers with SSL, while it's a valid idea, it seems I wouldn't be the one building that. There are a few reasons, i.e. I have "newbie Tomcat devs problems", and I'm not so motivated to work on a feature that makes more sense for big corporations rather than a single small developers. To note even my question to explain to add Class javadoc for LifecycleMBeanBase stayed unanswered so far in dev list, to my surprise. Back in 2007, I was good enough so that Google picked me to develop the software for them, I left to start my own business two years later, but I have a history of not being a good team player, seeing the same things differently than other people, and also I don't have a history of contributing to open-source projects (unless started by myself), so perhaps, at the end of the day, I'm not the good fit for Tomcat dev. Anyway, as it looks now, I'll unsubscribe soon from the Tomcat dev email list as it looks to me that I didn't fit. > > Thanks, > -chris > > > On Sat, Dec 26, 2020 at 6:49 PM Jerry Malcolm > > wrote: > > > >> We have a production environment where we rarely reboot Tomcat. > >> LetsEncrypt auto-updates the certificates every couple of months. But > >> the new certificates are not loaded into Tomcat. So when the original > >> expiration date of the certs arrives, users get "certificate expired" > >> even though new certs exist. A simple reboot to load the new certs > >> fixes it. But we want to avoid reboots. Are there any config > >> parameters that tell TC to check for cert updates and reload the new > >> certs? Thx > >> > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Recognizing Certificate Updates
On Sat, Dec 26, 2020 at 6:46 PM John Larsen wrote: > This is why we set up SSL through the web server instead of tomcat. > Apache webserver -> SSL -> Mod_jk <-> Tomcat > It might be easier to install but performance-wise it doesn't make sense. If you care about performances, I think you should make Tomcat only server (to avoid pipelining through sockets). >
Re: Recognizing Certificate Updates
If you set up tomcat manager up, you can reload certificate with something like Stop Connector – curl http://localhost:8080/manager/jmxproxy?invoke=Catalina %3Atype%3DConnector%2Cport%3D8443=stop Start Connector – curl http://localhost:8080/manager/jmxproxy?invoke=Catalina %3Atype%3DConnector%2Cport%3D8443=start (source: http://people.apache.org/~schultz/ApacheCon%20NA%202017/Let's%20Encrypt%20Apache%20Tomcat.pdf ) This is probably faster than reboot the whole tomcat, I haven't tried it. This looks imperfect as hell. Honestly, I thought that reloadAfterNDays param to server.xml would be better, but admins didn't have an understanding on this topic. On Sat, Dec 26, 2020 at 6:49 PM Jerry Malcolm wrote: > We have a production environment where we rarely reboot Tomcat. > LetsEncrypt auto-updates the certificates every couple of months. But > the new certificates are not loaded into Tomcat. So when the original > expiration date of the certs arrives, users get "certificate expired" > even though new certs exist. A simple reboot to load the new certs > fixes it. But we want to avoid reboots. Are there any config > parameters that tell TC to check for cert updates and reload the new > certs? Thx > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: native connector, server problems with "No data received", what could be causing it?
On Thu, Dec 17, 2020 at 7:57 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > jstack isn't working? Hmm. Make sure that your jstack and Java binary > are the same version. If you are a different user you might have to > elevate privileges and/or use "jstack -F". > I have tried all 8 variations with sudo, -F , -J-d64 parameters and finally it worked only with simply sudo -u runninguser jstack without -F and -J-d64 params > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: native connector, server problems with "No data received", what could be causing it?
I have just updated to the latest Tomcat 9.0.41 It seems that the problem doesn't appear anymore. If I get it reappeared, I'll post new details to the Tomcat dev mailing list. On Thu, Dec 17, 2020 at 8:56 AM Mladen Adamović wrote: > I have these problems again. To narrow it down I have done the following: > - tested with the latest JDK8 - problem exists > - it exists in both Apr connector and Nio2 connector > - did log JVM Garbage Collector details - it's not due to Garbage collector > > Christopher (or anyone else), if we had 1570 requests per minute where > this problem happened (approx. 26 per second), what do you think should our > Connector params be, it's currently: > > protocol="org.apache.coyote.http11.Http11Nio2Protocol" > SSLCertificateFile="/etc/letsencrypt/live/ > numbeo.com/cert.pem" > SSLCertificateKeyFile="/etc/letsencrypt/live/ > numbeo.com/privkey.pem" > SSLCertificateChainFile="/etc/letsencrypt/live/ > numbeo.com/chain.pem" > SSLVerifyClient="optional" > SSLProtocol="TLSv1+TLSv1.1+TLSv1.2" > connectionTimeout="2" acceptCount="3" > acceptorThreadCount="4" > compression="on" maxConnections="5" maxThreads="500" > > compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,image/svg+xml,image/svg,image/png,image/jpeg" > > useSendfile="false" > maxHttpHeaderSize="16392" SSLEnabled="true" > enableLookups="false" > scheme="https" secure="true" clientAuth="false" > useBodyEncodingForURI="true" > URIEncoding="UTF-8" > /> > > > > > > On Wed, Dec 16, 2020 at 7:32 PM Mladen Adamović > wrote: > >> On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz < >> ch...@christopherschultz.net> wrote: >> >>> I think your scripts will restart Tomcat even when it's not necessary. >>> >> >> Hm, is this >> https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates >> the way to reload the certificates or is there another suggestion? >> >> >>> The $? check before sending the email message looks like it should be >>> checking the result of the certbot command, but it's checking the result >>> of the chmod command instead. (Or maybe the result of the .sh script, >>> which will proably be 0.) >>> >> >> Correct, thank you this is fixed now. >> >> >>> I might have found that odd had you posted that in your original >>> message, but you did not. >>> >>> You need to show the full stack trace for that thread to make it >>> meaningful. Sockets are added to the poller all the time. It's not >>> unusual to see that happening. It they are getting *stuck*, that would >>> be bad, of course. >>> >> >> I did post full thread dump. >> >> >>> > Don't you find it weird that all threads are trying to get >>> synchronized >>> > on a Poller instance and no one is in this block or another >>> synchronized >>> > block/method? >>> >>> I would find it weird if no threads were making any progress. Lots of >>> threads adding sockets to the poller is not out of the ordinary. >>> >>> If you suspect a bug in Tomcat's socket handling, upgrading to the >>> latest 8.5.x release and re-trying would be the best move. There have >>> been many fixes since your 8.5.5 release which is now 4+ years old. >>> >> >> I have switched today from Apr connector to Nio2 connector and no >> problem yet. >> What I have found strange is that processor usage is lower when using >> Nio2, I have never tested it or tried it before, I have setup APR for >> performances reason back in 2016. But oddly it seems with Nio2 processor >> usage is lower. >> I have tried the last OpenJDK but jstack is not working for me, so >> switched back to the previous old JRM. >> Let's see what will happen... >> >> >> >> >>> -chris >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>>
Re: native connector, server problems with "No data received", what could be causing it?
I have these problems again. To narrow it down I have done the following: - tested with the latest JDK8 - problem exists - it exists in both Apr connector and Nio2 connector - did log JVM Garbage Collector details - it's not due to Garbage collector Christopher (or anyone else), if we had 1570 requests per minute where this problem happened (approx. 26 per second), what do you think should our Connector params be, it's currently: On Wed, Dec 16, 2020 at 7:32 PM Mladen Adamović wrote: > On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> I think your scripts will restart Tomcat even when it's not necessary. >> > > Hm, is this > https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates > the way to reload the certificates or is there another suggestion? > > >> The $? check before sending the email message looks like it should be >> checking the result of the certbot command, but it's checking the result >> of the chmod command instead. (Or maybe the result of the .sh script, >> which will proably be 0.) >> > > Correct, thank you this is fixed now. > > >> I might have found that odd had you posted that in your original >> message, but you did not. >> >> You need to show the full stack trace for that thread to make it >> meaningful. Sockets are added to the poller all the time. It's not >> unusual to see that happening. It they are getting *stuck*, that would >> be bad, of course. >> > > I did post full thread dump. > > >> > Don't you find it weird that all threads are trying to get synchronized >> > on a Poller instance and no one is in this block or another >> synchronized >> > block/method? >> >> I would find it weird if no threads were making any progress. Lots of >> threads adding sockets to the poller is not out of the ordinary. >> >> If you suspect a bug in Tomcat's socket handling, upgrading to the >> latest 8.5.x release and re-trying would be the best move. There have >> been many fixes since your 8.5.5 release which is now 4+ years old. >> > > I have switched today from Apr connector to Nio2 connector and no problem > yet. > What I have found strange is that processor usage is lower when using > Nio2, I have never tested it or tried it before, I have setup APR for > performances reason back in 2016. But oddly it seems with Nio2 processor > usage is lower. > I have tried the last OpenJDK but jstack is not working for me, so > switched back to the previous old JRM. > Let's see what will happen... > > > > >> -chris >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >>
Re: native connector, server problems with "No data received", what could be causing it?
On Wed, Dec 16, 2020 at 7:07 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > I think your scripts will restart Tomcat even when it's not necessary. > Hm, is this https://stackoverflow.com/questions/5816239/how-do-i-force-tomcat-to-reload-trusted-certificates the way to reload the certificates or is there another suggestion? > The $? check before sending the email message looks like it should be > checking the result of the certbot command, but it's checking the result > of the chmod command instead. (Or maybe the result of the .sh script, > which will proably be 0.) > Correct, thank you this is fixed now. > I might have found that odd had you posted that in your original > message, but you did not. > > You need to show the full stack trace for that thread to make it > meaningful. Sockets are added to the poller all the time. It's not > unusual to see that happening. It they are getting *stuck*, that would > be bad, of course. > I did post full thread dump. > > Don't you find it weird that all threads are trying to get synchronized > > on a Poller instance and no one is in this block or another synchronized > > block/method? > > I would find it weird if no threads were making any progress. Lots of > threads adding sockets to the poller is not out of the ordinary. > > If you suspect a bug in Tomcat's socket handling, upgrading to the > latest 8.5.x release and re-trying would be the best move. There have > been many fixes since your 8.5.5 release which is now 4+ years old. > I have switched today from Apr connector to Nio2 connector and no problem yet. What I have found strange is that processor usage is lower when using Nio2, I have never tested it or tried it before, I have setup APR for performances reason back in 2016. But oddly it seems with Nio2 processor usage is lower. I have tried the last OpenJDK but jstack is not working for me, so switched back to the previous old JRM. Let's see what will happen... > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: native connector, server problems with "No data received", what could be causing it?
On Wed, Dec 16, 2020 at 3:27 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:
> > We have a self-monitoring script which runs on server and when the server
> > is not working properly it does a log save and the service restart.
>
> How do you detect this state? Just make a request and if you get "No
> data received" from curl, you restart the server?
>
If there is an error code or the specific text doesn't appear on the
response we monitor the state and do /etc/init.d/tomcat restart.
The full script is:
#!/bin/bash
serverFailure=0
cd /root
rm /root/numbeo_test.out
#wget -t 1 -T 5 --no-proxy --no-cache --cache=off -q
'localhost:8080/cost-of-living/city_result.jsp?country=Ireland=Dublin'
-O /root/numbeo_test.out
#curl -L -m 2 -v -o /root/numbeo_test.out --trace curl.log
'localhost:8008/cost-of-living/in/Dublin'
curl -L -m 2 -v --insecure -o /root/numbeo_test.out --trace curl.log '
https://localhost:8181/cost-of-living/in/Dublin'
wgetOutput=$?
grep -q "entries in the past" /root/numbeo_test.out
if [ $? != 0 ]; then
cd /root
rm /root/numbeo_test.out
sleep 10s
#wget -t 2 -T 2 --no-proxy --no-cache --cache=off -q
'localhost:8080/cost-of-living/city_result.jsp?country=Ireland=Dublin'
-O /root/numbeo_test.out
#curl -L -m 2 --retry 1 -v -o /root/numbeo_test.out --trace curl.log
'localhost:8008/cost-of-living/in/Dublin'
curl -L -m 2 -v --insecure -o /root/numbeo_test.out --trace curl.log '
https://localhost:8181/cost-of-living/in/Dublin'
wgetOutput=$?
grep -q "entries in the past" /root/numbeo_test.out
if [ $? != 0 ]; then
#echo 'server is down!';
ps -eo pid,comm | while read pid command
do
if [[ "$command" = "java" ]]
then
echo $pid
DATE=`date +%Y-%m-%d`
echo ${wgetOutput} > ~/wget_${DATE}_${pid}.log
cp /root/numbeo_test.out >
~/numbeo_test_out_${DATE}_${pid}.log
jstack -J-d64 -F $pid > ~/jstack_${DATE}_${pid}.log
iostat > ~/iostat_${DATE}_${pid}.log
vmstat > ~/vmstat_${DATE}_${pid}.log
netstat -tnp > ~/netstat_${DATE}_${pid}.log
netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d:
-f1 | sort | uniq -c | sort -n > ~/netstat_anp_outline_${DATE}_${pid}.log
ps aux > ~/ps_aux_${DATE}_${pid}.log
tail -n 5000
~glassfish/apache-tomcat-8.5.5/logs/catalina.out >
~/catalina_out_${DATE}_${pid}.log
break
fi
done
echo 'too many server failures... going to rebootsoftly' >> ~/reboot.log ;
date | mail -s "Numbeo soft reset" mladen.adamo...@gmail.com
date >> ~/reboot.log
killall -9 java
/root/fix_letsencrypt_chmod.sh
#/etc/init.d/glassfish start
/etc/init.d/tomcat start
#reboot
fi
fi
I see you are using Let's Encrypt. How are you managing the rotating of
> the keys and certificates?
>
Crontab: 5 1 1 * * /root/renew_cert_numbeo.sh
root@condor1796 ~ # cat renew_cert_numbeo.sh
#!/bin/bash
mkdir -p /tmp/letsencrypt/public_html
certbot certonly -n --force-renewal --webroot --webroot-path
/tmp/letsencrypt/public_html -d numbeo.com -d www.numbeo.com \
-d es.numbeo.com -d pt.numbeo.com -d fr.numbeo.com -d
ru.numbeo.com -d ja.numbeo.com -d de.numbeo.com -d nl.numbeo.com \
-d it.numbeo.com -d zh.numbeo.com -d ar.numbeo.com -d
jobs.numbeo.com \
--agree-tos --email mladen.adamo...@gmail.com
/root/fix_letsencrypt_chmod.sh
if [ $? != 0 ]; then
date | mail -s "Lets encrypt renew certificate fails for numbeo.com"
mladen.adamo...@gmail.com
else
/etc/init.d/tomcat restart
fi
root@condor1796 ~ # cat fix_letsencrypt_chmod.sh
#!/bin/bash
chmod o+rx /etc/letsencrypt
chmod -R o+rx /etc/letsencrypt/*
root@condor1796 ~ #
> > *What would be the next steps how to identify the problem and perhaps
> > solve it?*
> What have you done so far?
>
aaah... reading the Tomcat source to try to understand the state of Threads.
I don't see anything that sticks out in your thread dump.
>
There are several threads which are trying to get monitor in
AprEndpoint$Poller.add and no thread seems to be blocking it. Don't you
find it weird:
root@condor1796 ~ # grep Poller jstack_2020-12-16_31415.log | grep "Apr"
- org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
-
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
- org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
-
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
- org.apache.tomcat.util.net.AprEndpoint$Poller.add(long, long, int)
@bci=102, line=1398 (Compiled frame)
-
org.apache.tomcat.util.net.AprEndpoint$Poller.access$500(org.apache.tomcat.util.net.AprEndpoint$Poller,
long, long, int) @bci=5, line=1157 (Compiled frame)
-
Re: Tomcat serving old TXT file even after deleting work directory and rebooted
Lame!
It turned out that I have
@WebServlet(name = "AdsTxt", urlPatterns = {"/ads.txt"})
LOL
On Fri, Apr 24, 2020 at 4:48 PM Mladen Adamović
wrote:
> I've changed ads.txt did upload ROOT.war file and it now contains new
> version which is 18892 bytes old, checked files in the filesystem:
>
> root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # find . | grep
> "ads.txt" | xargs ls -l
> -rw-r- 1 glassfish nogroup 18892 Apr 24 2020
> ./appBaseLivingCost/ROOT/ads.txt
> -rw-r- 1 glassfish nogroup 18892 Apr 24 2020
> ./appBaseNonWwwNumbeo/ROOT/ads.txt
> root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # unzip -v
> appBaseNonWwwNumbeo/ROOT.war | grep "ads.txt"
>18892 Stored18892 0% 2020-04-24 16:13 dae45795 ads.txt
>
> Fine, all files are 18892 bytes.
>
> From server.xml:
> unpackWARs="true" autoDeploy="true">
> numbeo.com
>
> I did remove work, kill Catalina and Started it again.
>
> It's still serving old file! Checked with curl and browser (with delete
> cache and reload).
> apache-tomcat-8.5.5
>
> I'm totally confused. What is happening?
>
>
>
>
Tomcat serving old TXT file even after deleting work directory and rebooted
I've changed ads.txt did upload ROOT.war file and it now contains new version which is 18892 bytes old, checked files in the filesystem: root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # find . | grep "ads.txt" | xargs ls -l -rw-r- 1 glassfish nogroup 18892 Apr 24 2020 ./appBaseLivingCost/ROOT/ads.txt -rw-r- 1 glassfish nogroup 18892 Apr 24 2020 ./appBaseNonWwwNumbeo/ROOT/ads.txt root@condor1796 /home/glassfish/apache-tomcat-8.5.5 # unzip -v appBaseNonWwwNumbeo/ROOT.war | grep "ads.txt" 18892 Stored18892 0% 2020-04-24 16:13 dae45795 ads.txt Fine, all files are 18892 bytes. >From server.xml: numbeo.com I did remove work, kill Catalina and Started it again. It's still serving old file! Checked with curl and browser (with delete cache and reload). apache-tomcat-8.5.5 I'm totally confused. What is happening?
Re: CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
Hi Chris, On Tue, Nov 26, 2019 at 7:51 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > You never actually said what your definition of "high CPU usage" was. > Are you looking at load-average? %-CPU usage from "top"? > I'm sorry if I wasn't clear enough, I meant high system load (which was according to top) and actually spending time in futex_wait kernel function most of the time. > Do you have the option of comparing your CPU usage when you use APR > versus NIO? I would expect your CPU usage to o *up* by switching from > APR to NIO due to two factors: > This is a production server and it would be a bad idea to compare APR vs NIO on it. > Do you have any scope to separate-out TLS termination from the > application server? That is, use a reverse-proxy for TLS termination > and proxy to another server that only handles the application logic? > You may find that TLS is just "expensive" in terms of CPU which, well, > is simply the truth. > Well, we could separate TLS from the application server by using adequate "proxy" in the future, but I don't see a reason why we would do it now. When we migrated from http to https CPU usage went up multiple times, so I agree that TLS is expensive. I've setup things years ago and I don't remember all the details why things are like that, but there are (probably) some reasons why things are like they are. Our setup is documented here: https://mladenadamovic.wordpress.com/2016/09/06/configure-tomcat-with-ssl-on-ubuntu-minimal/ It wasn't easy for me to configure https/tomcat/letsencrypt... > [1] https://en.wikipedia.org/wiki/Busy_waiting > > > On Tue, Nov 26, 2019 at 4:50 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Mladen, > > > > On 11/25/19 14:36, Mladen Adamović wrote: > >>>> On Mon, Nov 25, 2019 at 5:57 PM Christopher Schultz < > >>>> ch...@christopherschultz.net> wrote: > >>>> > >>>>>> We certainly want to be able to serve 1 hits per > >>>>>> second (!), while some connections might be stalled. > >>>>> > >>>>> What might stall a connection? The network, or the > >>>>> application (or database, etc.)? > >>>>> > >>>> > >>>> Underlying (synchronized) monitors could stall every thread, > >>>> the network, whatever. > >>>> > >>>> The network itself demands a large number of connection, > >>>> i.e. current situation at the server (displaying only remove > >>>> connections): > >>>> > >>>> root@condor1796 ~ # netstat -tnp | grep -v "127.0.0" | wc -l > >>>> 1220 > > > > Note this is every connection, bound port, and cleanup connection > > the kernel knows about ; not just established/active connections to > > your application specifically. > > > >>>> If we now have 1220, we definitely need at least 1 > >>>> active connections for Tomcat and I don't see that setting > >>>> this to 5 is a bad idea. > > > > Okay. I think you need a reverse proxy and more servers if you > > think 5 is going to be your peak load. > > > >>>>> For real DDOS protection, you need a provider who can > >>>>> handle lots of traffic and respond quickly by black-holing > >>>>> that kind of traffic as > >>>> > >>>> Depending on how large server farm they use (hypothetically). > >>>> We want to be able to survive some DDoS attacks. If we limit > >>>> the number of concurrent connections by IP address and the > >>>> number of connections per second, that's some DoS > >>>> protection. > > > > But honestly, this is better done at another layer of the network; > > not at the host-level. > > > >>>> Regarding network delays, out of currently 1220 active > >>>> remove connections, most of them are in TIME_WAIT state. > >>>> Lowering TIME_WAIT settings in Linux are not recommended. > > > > Hmm. Lots of TIME_WAIT connections isn't good. I actually don't > > know if they count "against" your 5 limit in the Java process. > > > > -chris > >> > >> - > >> > >> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > >> > > > -
Re: CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
I dig into Tomcat source code and from what I've seen this is what happens: Tomcat "worker thread", tries the polls the work and if it doesn't succeed it invokes sun.misc.Unsafe.park (that's waiting which can be interrupted by another thread), internally in Linux goes through kernel function FUTEX_WAIT (futex definition is " The futex() system call provides a method for waiting until a certain condition becomes true." However, these futex_wait functions sums to top CPU load, although I don't find it's actually load here, as per infamous Linux change of 29 Oct 1993 by Matthias Urlichs: http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html On Tue, Nov 26, 2019 at 4:50 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Mladen, > > On 11/25/19 14:36, Mladen Adamović wrote: > > On Mon, Nov 25, 2019 at 5:57 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >>> We certainly want to be able to serve 1 hits per second > >>> (!), while some connections might be stalled. > >> > >> What might stall a connection? The network, or the application > >> (or database, etc.)? > >> > > > > Underlying (synchronized) monitors could stall every thread, the > > network, whatever. > > > > The network itself demands a large number of connection, i.e. > > current situation at the server (displaying only remove > > connections): > > > > root@condor1796 ~ # netstat -tnp | grep -v "127.0.0" | wc -l 1220 > > Note this is every connection, bound port, and cleanup connection the > kernel knows about ; not just established/active connections to your > application specifically. > > > If we now have 1220, we definitely need at least 1 active > > connections for Tomcat and I don't see that setting this to 5 > > is a bad idea. > > Okay. I think you need a reverse proxy and more servers if you think > 5 is going to be your peak load. > > >> For real DDOS protection, you need a provider who can handle lots > >> of traffic and respond quickly by black-holing that kind of > >> traffic as > > > > Depending on how large server farm they use (hypothetically). We > > want to be able to survive some DDoS attacks. If we limit the > > number of concurrent connections by IP address and the number of > > connections per second, that's some DoS protection. > > But honestly, this is better done at another layer of the network; not > at the host-level. > > > Regarding network delays, out of currently 1220 active remove > > connections, most of them are in TIME_WAIT state. Lowering > > TIME_WAIT settings in Linux are not recommended. > > Hmm. Lots of TIME_WAIT connections isn't good. I actually don't know > if they count "against" your 5 limit in the Java process. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3dSaUACgkQHPApP6U8 > pFhStRAArIHBU4UT6cw5jS7ys6aRlYpaxw4lJ1lhRA9WB5U7/bG+qnZlai6052X7 > MPrfjP8ZlNMugVwhHjMnY3iijfWT2K6bkd8WILT3gcu/ZSqwz2tr9QYru40zG/Bu > FHHlmoUwfWkUrwphJUgwvp1VsIU3exdG28LDlnGjjp1JmgALd7/KeBmS98kpSyKR > Dot/7tlW98Y9DaPOnOnwkWO/MIZLEuekjBRRgZcYr6OpY+9s0hRP/RJ8uEpSfOgA > +ZCvqrjR3MR26gbap9o6zBsZzI+tjFjH9YteAHkxAOmzU+ztiCoIRj6SA4LJErgT > z53yqxpVRszbWmJod3P7sphHJ+r2dmvf0iOEV4qbkBAYF2vP8wsV3jY/7B68OfNh > 6sSC9CWTg7l0wYzxFLrSVQqIt7WV4BBX/4yH9fQ72jHs8Qd5uIJoDbD5GJ1HW32E > viGpzg9/dlXxsRisow7wdKOFC+wTtWeoyDasMZqgdf+SofSTK1qGF/sR0n866dM3 > I1Rz8E0cVZKADtDrjkUK4BMTExfX0rS2WdpwqWOykvTOA9wvW5IzMfokblMQ1XxG > ctnIJA4sRfFwFmnQVu7ew0Ryu3P3tLzaXE7CqfveOgqu/YLi/9gwbvmSB0x0UGsk > YHepLdZ+CwB1vo0fTn0kVKf+anVoAq3xOguPB69gnBZwmsK4v6g= > =Pk1p > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Error after upgrading to Tomcat 9.0.29
Try simply to increase -Xss parameter in setenv.sh or setenv.bat, see if it works, since there is no infinite loop (it seems) On Mon, Nov 25, 2019 at 2:15 PM Juri Berlanda wrote: > Hi all, > > I just tried to deploy my WebApplication (OpenWebBeans, MyFaces) to > Tomcat 9.0.29. While everything works fine in 9.0.27, on 9.0.29 as soon > as I access any page I get: > > 25-Nov-2019 14:01:34.842 SEVERE [http-nio-8080-exec-4] > org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() > for servlet [Faces Servlet] in context with path [/censored] threw > exception [null] with root cause > java.lang.StackOverflowError > > Since it is a StackOverflow, I'm not posting the Stacktrace here. > > Has anybody had a different error? Is there a known fix or a workaround > for this? > > I'm happy to help debugging and fixing the issue, if there is one in > Tomcat. Just let me know how I can help. > > Cheers, > > Juri > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
On Mon, Nov 25, 2019 at 5:57 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > > We certainly want to be able to serve 1 hits per second (!), > > while some connections might be stalled. > > What might stall a connection? The network, or the application (or > database, etc.)? > Underlying (synchronized) monitors could stall every thread, the network, whatever. The network itself demands a large number of connection, i.e. current situation at the server (displaying only remove connections): root@condor1796 ~ # netstat -tnp | grep -v "127.0.0" | wc -l 1220 If we now have 1220, we definitely need at least 1 active connections for Tomcat and I don't see that setting this to 5 is a bad idea. For real DDOS protection, you need a provider who can handle lots of > traffic and respond quickly by black-holing that kind of traffic as > Depending on how large server farm they use (hypothetically). We want to be able to survive some DDoS attacks. If we limit the number of concurrent connections by IP address and the number of connections per second, that's some DoS protection. Regarding network delays, out of currently 1220 active remove connections, most of them are in TIME_WAIT state. Lowering TIME_WAIT settings in Linux are not recommended. root@condor1796 ~ # netstat -tnp | grep -v "127.0.0" Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 209.126.119.66:8181 191.89.96.97:61704 TIME_WAIT - tcp0 0 209.126.119.66:8181 14.0.170.118:4812 TIME_WAIT - tcp0229 209.126.119.66:8181 176.139.70.78:6 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 38.106.245.167:61899 TIME_WAIT - tcp0 0 209.126.119.66:8181 88.73.133.202:54659 ESTABLISHED 965/java tcp0 0 209.126.119.66:8080 34.222.139.225:40213 TIME_WAIT - tcp0 0 209.126.119.66:8181 93.86.200.251:57639 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 31.22.232.7:41072 TIME_WAIT - tcp0 0 209.126.119.66:8181 79.115.158.8:22648 TIME_WAIT - tcp0 0 209.126.119.66:8181 2.15.118.242:52703 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 72.136.68.68:57286 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 106.79.198.220:47784 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 153.213.195.19:36460 SYN_RECV- tcp0 0 209.126.119.66:8181 121.45.206.143:37628 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 93.202.38.207:51898 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 103.213.201.206:65330 ESTABLISHED 965/java tcp1 32 209.126.119.66:8181 112.79.139.186:4CLOSING - tcp0 0 209.126.119.66:8181 87.246.46.246:43906 TIME_WAIT - tcp0 57834 209.126.119.66:8181 82.143.93.204:59333 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 38.113.185.5:59561 TIME_WAIT - tcp0 0 209.126.119.66:8181 95.28.153.13:60671 FIN_WAIT2 - tcp0 0 209.126.119.66:8181 179.181.74.233:62630 FIN_WAIT2 - tcp0 0 209.126.119.66:8181 196.207.173.208:44696 TIME_WAIT - tcp0 0 209.126.119.66:8181 77.27.51.176:60600 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 171.77.144.224:1538 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 104.225.166.237:39534 TIME_WAIT - tcp0 0 209.126.119.66:8181 94.197.121.66:18358 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 188.37.83.27:43803 TIME_WAIT - tcp0 0 209.126.119.66:8181 190.58.8.4:28855 FIN_WAIT2 - tcp0363 209.126.119.66:8181 2.152.177.0:51771 LAST_ACK- tcp0 0 209.126.119.66:8181 194.242.157.132:64282 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 87.174.23.207:61923 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 95.164.172.129:49668 FIN_WAIT2 - tcp0 0 209.126.119.66:22 139.199.22.148:37104 ESTABLISHED 3677/sshd: [accepte tcp0 0 209.126.119.66:8181 157.50.223.161:37081 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 24.114.90.19:51281 ESTABLISHED 965/java tcp0 0 209.126.119.66:8181 82.15.98.139:50097 TIME_WAIT - tcp0 0 209.126.119.66:8181 103.31.101.103:21888 TIME_WAIT - tcp0 0 209.126.119.66:8181 95.91.230.80:57201 TIME_WAIT - tcp0 0 209.126.119.66:8181 31.22.232.7:41074 TIME_WAIT - tcp0 0 209.126.119.66:8181 37.161.207.143:8697 TIME_WAIT - tcp0 0 209.126.119.66:8008 34.222.139.225:37918 TIME_WAIT - tcp0 0 209.126.119.66:8181 77.205.186.123:51171 TIME_WAIT - tcp
Re: CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
Hi Christopher, the answers is inline. On Mon, Nov 25, 2019 at 4:54 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > 50k connections is quite a lot. Is this a physical or virtual server? > Do you expect to have lots of long-lived connections that are mostly > idle (e.g. WebSocket)? Or do you just want to handle huge amounts of > actual load (i.e. lots of requests)? > It's a physical server with a relatively high load (100 requests per second when low), serving mostly text/html content. Due to default TCP internals, one connection can be long-lasting, only round trip time to confirm that the message is received could last 200ms. 5 connections are how many connections server can accept at the same time. We certainly want to be able to serve 1 hits per second (!), while some connections might be stalled. And to survive a DDoS attack which tries to keep connections stalled using server farms (if it ever happens). > > - -chris > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl3b+TMACgkQHPApP6U8 > pFh9KxAAmlC7daF8lseSvI9K9P+9C1yfwGXMxYIqXTDy2mzwaG1PdcAfu2FEupAc > Judf3qmictIRPGRE8i0/vCveYP9ITzZZY/FG1/G2OhQR/8KjB2EcCNxJgVHEWgv1 > aOdCarYx54dtSESb0IWzSp5h7LbUswjpytsAqzvXDtlLVsbD4PqNeK72cdJgX4qh > 1zlHrKuHEEIta0S/Gu1T4EBAOG7OWuyuwY6oVHOPjaByItp+xgmkP+suxnnIQJxE > 4qF8hwH7iLpyHJkZIfgM3ju6Hw4LI8Pu6TaVBmjI8+zToYRs1qBpxgiXYimFQGrT > /+hFQqExU+hGeHjoMgO0popfZDC8mqgNTRzAGphdUmykBGGTDQEE9MbiPqdvEfA/ > Bkpx/do2OeMGzT86TXM/dDlzs/C03PgiwVL0g4SO+zP1vTOuooj8lijA/q0EWjRo > shNia1XMbYK2bKo0ZsXXePNmFPDuSXdDfqikVqz1eWp+SvF42Xs7aDgzmEJx/uh7 > VtS3hna/WSJmKDYtKYdMCCWxlKkw6Qdej1egTBXLYw1sdBb3RW7cMJHp8TIa6D7C > qGSsgQX1FCe8TDHqZ+RAY9E7CcB9ifQkkkhk4zgyVVLFhmqoSbDKSLWr1M3wXzwt > WTtIRUWQQQyjjhkfidnFLNYHid1a62pcHd6drg9uUf7eEIPG1AY= > =gRLR > -END PGP SIGNATURE- >
Re: Official documentation for running an embedded Tomcat server
I wonder why somebody would want to run "embedded Tomcat" and what it actually means? I'm working professionally in software engineering since 2003 and I'm puzzled. >From looking into that article, it seems that Java app starts Tomcat, why somebody would want that? On Sun, Nov 24, 2019 at 12:27 AM Behrang Saeedzadeh wrote: > I will create a PR if I find the time to write one. > > On Sat, Nov 23, 2019 at 9:40 PM Mark Thomas wrote: > > > On 23/11/2019 08:17, Behrang Saeedzadeh wrote: > > > Hi all, > > > > > > Are there any official docs for how to run Tomcat as an embedded > server? > > > Searching returns some results such as > > > > > > https://devcenter.heroku.com/articles/create-a-java-web-application-using-embedded-tomcat > > > but I haven't seen any official docs. > > > > > > Are there any? > > > > > http://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/catalina/startup/Tomcat.html > > > > is the closest thing we have but I suspect parts of it are out of date. > > The unit tests are a good source of samples of how to use it. > > > > Care to contribute a how-to page for the docs? > > > > Mark > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > >
Re: CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
Hi Manna, I didn't copied full stack, it seems most threads are actually in TaskQueue.poll, see the full thread trace: Thread 1714: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame) - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame) * - org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame)* - org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame) - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Compiled frame) - java.lang.Thread.run() @bci=11, line=745 (Compiled frame) Thread 1707: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame) - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame) - org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame) - org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame) - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Compiled frame) - java.lang.Thread.run() @bci=11, line=745 (Compiled frame) etc. (many more like these) On Sun, Nov 24, 2019 at 12:13 PM M. Manna wrote: > I’m suspecting it’s the same issue that we found out for 8.5.45 where the > commit didn’t get reverted out in Poll.java > > Or maybe not. > > Thanks, > > On Sun, 24 Nov 2019 at 10:36, Mark Thomas wrote: > > > Tomcat version? > > > > Operating system? > > > > Java version? > > > > Mark > > > > > > On 24/11/2019 09:57, Mladen Adamović wrote: > > > I couldn't find the explanation of this in other threads, I've tried to > > > search the archive. > > > > > > I have a high load average on a server and the reason for that is > kernel > > > function futex_wait invoked by java thread. > > > > > > By doing jstack, I see a lot of BLOCKED threads like these: > > > Thread 1725: (state = BLOCKED) > > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; > > information > > > may be imprecise) > > > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > > long) > > > @bci=20, line=215 (Compiled frame) > > > - > > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) > > > @bci=78, line=2078 (Compiled frame) > > > - java.util.concurrent.LinkedBlockingQueue.poll(long, > > > java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame) > > > - org.apache.tomcat.util.threads.TaskQueue.poll(long, > > > java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame) > > > - org.apache.tomcat.util.threads.TaskQueue.poll(long, > > > java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame) > > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, > line=1066 > > > (Compiled frame) > > > - > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > > > @bci=26, line=1127 (Compiled frame) > > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, > line=617 > > > (Compiled frame) > > > - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() > > @bci=4, > > > line=61 (Compiled frame) > > > - ja
Re: CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
Tomcat 8.5.5 Java 1.8.0_101 OS: Ubuntu 4.4.0-38-generic On Sun, Nov 24, 2019 at 11:36 AM Mark Thomas wrote: > Tomcat version? > > Operating system? > > Java version? > > Mark > > > On 24/11/2019 09:57, Mladen Adamović wrote: > > I couldn't find the explanation of this in other threads, I've tried to > > search the archive. > > > > I have a high load average on a server and the reason for that is kernel > > function futex_wait invoked by java thread. > > > > By doing jstack, I see a lot of BLOCKED threads like these: > > Thread 1725: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; > information > > may be imprecise) > > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) > > @bci=20, line=215 (Compiled frame) > > - > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) > > @bci=78, line=2078 (Compiled frame) > > - java.util.concurrent.LinkedBlockingQueue.poll(long, > > java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame) > > - org.apache.tomcat.util.threads.TaskQueue.poll(long, > > java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame) > > - org.apache.tomcat.util.threads.TaskQueue.poll(long, > > java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 > > (Compiled frame) > > - > > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > > @bci=26, line=1127 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 > > (Compiled frame) > > - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() > @bci=4, > > line=61 (Compiled frame) > > - java.lang.Thread.run() @bci=11, line=745 (Compiled frame) > > > > > > > > What does this > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() actually > > does when BLOCKED? > > > > Should I perhaps lower some tomcat config values or just forget about > > measuring system resources if the server works? My tomcat is configured > as > > follows: > > > > > > my Tomcat is configured as follows: > >> protocol="org.apache.coyote.http11.Http11AprProtocol" > > SSLCertificateFile="/etc/letsencrypt/live/ > numbeo.com/cert.pem" > > SSLCertificateKeyFile="/etc/letsencrypt/live/ > > numbeo.com/privkey.pem" > > SSLCertificateChainFile="/etc/letsencrypt/live/ > > numbeo.com/chain.pem" > > SSLVerifyClient="optional" > SSLProtocol="TLSv1+TLSv1.1+TLSv1.2" > > connectionTimeout="2" acceptCount="3" > > acceptorThreadCount="2" > > compression="on" maxConnections="5" maxThreads="500" > > > > > compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/javascript,image/svg+xml,image/svg,image/png,image/jpeg" > > > > useSendfile="false" > > maxHttpHeaderSize="16392" SSLEnabled="true" > > enableLookups="false" > > scheme="https" secure="true" clientAuth="false" > > useBodyEncodingForURI="true" > > URIEncoding="UTF-8" > > /> > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
CPU high usage, the reason org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run
I couldn't find the explanation of this in other threads, I've tried to search the archive. I have a high load average on a server and the reason for that is kernel function futex_wait invoked by java thread. By doing jstack, I see a lot of BLOCKED threads like these: Thread 1725: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame) - java.util.concurrent.LinkedBlockingQueue.poll(long, java.util.concurrent.TimeUnit) @bci=62, line=467 (Compiled frame) - org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=85 (Compiled frame) - org.apache.tomcat.util.threads.TaskQueue.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=31 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Compiled frame) - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Compiled frame) - java.lang.Thread.run() @bci=11, line=745 (Compiled frame) What does this org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() actually does when BLOCKED? Should I perhaps lower some tomcat config values or just forget about measuring system resources if the server works? My tomcat is configured as follows: my Tomcat is configured as follows:
Re: Production Tomcat 8.5.5 suddenly started to give ClassNotFoundError
It turned out this problem was caused by a class constructor which started to give RuntimeException (it has some calculations based on DB data). That somehow caused ClassNotFoundException. I've solved the cause and had to reinstall tomcat (as after deleting "work" directory tomcat did not work properly). On Mon, Jun 18, 2018 at 7:46 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Mladen, > > On 6/16/18 5:37 AM, Mladen Adamović wrote: > > Hi all, > > > > I have a production Tomcat server and this morning it started to > > give strange ClassNotFoundError for stuff which was working for > > years without a problem. > > > > I did redeploy app (by copying into the dir and deleting their > > ROOT), but it didn't solve the issue, same ClassNotFoundException. > > > > On the development machine I built a new release, checked that it > > worked in my localhost (it worked, no ClassNotFoundException), and > > stopped Tomcat on the production server, moved files from work to > > work_1 directory (that's specified as deleting the cache), copied > > the new WAR files into the destination and deleted the previous > > files. > > > > It didn't still solve the issue, now I got even bigger problem JSPs > > which were working fine previously now they don't work? > > > > What could cause this problem? How to fix it (without setting again > > new production tomcat from scratch)? > > It sounds to me like a botched upgrade from a lower version (e.g. 7.0, > 8.0) to 8.5 if JSPs are no longer working. > > Compare the following files in your environment to what Tomcat ships > with out of the box: > > conf/context.xml > conf/web.xml > > There should be no changes between what you have on your disk and what > comes with Tomcat. > > If there are no changes, then look at your own application's > WEB-INF/web.xml to see if you ave somehow overridden the *.jsp > mapping. Finally, look in your application's META-INF/context.xml to > see if there is anything JSP-related in there. (There probably > shouldn't be anything in there, but it's better to check). > > - -chris > -BEGIN PGP SIGNATURE- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlsn8A8ACgkQHPApP6U8 > pFhSaQ//fq2mr5JbsF9xueie/HHaO6lr7fwtK1VAJJObxbjCYWjVY4oly79QB5VU > lJJUi34WcZ3oBUBMGrwxThyg1+ll6IwzVJ5mh5GF0HAfwuYwUsNHhXf871a/uNqG > nGjpqQhbAFOHikZzZLdvLcgHJpVjXc24lI1C/R49gRGAhMOnYI1L3UjaLRF1SCNS > ZQrVQjWfwDpZLD0IDcsQxA0e5tR+xV3vr1bV4iNsqC7sjXcVWjf/OGLngmAenedL > ehIV9P15YceY1DCZOl8xbzFAJ4tJQvtE/s0dxvkUTZ4YXaf2zE5W/nFyVdHBhJ8C > U0oZWkvnLQv2u28MlzqK4cpA85Cpko+Y7ZDWxOIoiGhSVhBvdW3LwI0apphwIqMX > S3BA471kRzDUBgNNLU288HMF+4D7kk6MuaTJyzXLUM1vdy6fH16Hl7yYWaispj31 > 29vfNR2RfLdLhJ7SJ2v9FskciMkwD5LYCBL8Pybi8xUDeLyy/IrrwaNq3OrIUTIy > oAjRYd8R6rym3VB0WBvI4NlZxL2vhU2bJnDIsUGPljOuxntL/P3o19hcoqQwB3nM > Z/l9Tk5wJpSJjhlljOds3wOYHzJQR8OFAxW0kShgh3Pl9o3c2hU4o3Ff6zYKYnom > B5aRSAgGxeU9n+2DATwEakVq29ykB6o+gCIy91LmcXlEwyPdlqA= > =4Wmr > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Production Tomcat 8.5.5 suddenly started to give ClassNotFoundError
Hi all, I have a production Tomcat server and this morning it started to give strange ClassNotFoundError for stuff which was working for years without a problem. I did redeploy app (by copying into the dir and deleting their ROOT), but it didn't solve the issue, same ClassNotFoundException. On the development machine I built a new release, checked that it worked in my localhost (it worked, no ClassNotFoundException), and stopped Tomcat on the production server, moved files from work to work_1 directory (that's specified as deleting the cache), copied the new WAR files into the destination and deleted the previous files. It didn't still solve the issue, now I got even bigger problem JSPs which were working fine previously now they don't work? What could cause this problem? How to fix it (without setting again new production tomcat from scratch)? Thanks
web.xml and @WebServlet priority when more rules match
Lets assume that web.xml has a rule
One
/something/e*
While class Two.class has an annotation:
@WebServlet(name = "Two", urlPatterns = {"/something/er*"})
>From the Servlet 3.0 specification
8.2.3. (point 4). `The web.xml of the web application has the highest
precedence when resolving conflicts between the web.xml, web-fragment.xml
and annotations.`
While 12.2 states: The container will recursively try to match the longest
path-prefix. This is done by stepping down the path tree a directory at a
time, using the ’/’ character as a path separator. The longest match
determines the servlet selected.
So for the request /something/error Tomcat will choose Two.class?
And more general, in the case there are both annotation and web.xml rule
with the same prefix length, it will choose a first occurrence in web.xml?
If no one knows the answers but knows where to look in the source code,
that could be beneficial as well.
Thanks
Tutorial: Configure Tomcat with HTTPS/SSL on Ubuntu 16.04 LTS (Xenial) using Letsencrypt
Hi all, I'm running 3 servers with Tomcat (migrated from Glassfish which is not maintained well imo). But documentation is kind of not the best for configuring HTTPS/SSL. I've written my own tutorial how to do that using Letsencrypt and Tomcat native: https://mladenadamovic.wordpress.com/2016/09/06/configure-tomcat-with-ssl-on-ubuntu-minimal/ As you can notice from the tutorial: - I'm running 8.5.5 from website (not prepackaged with comes with Ubuntu) - run Tomcat as non-root user and do ip tables forwarding because of that - implemented my own ACME support for Letsencrypt (it's easy) I'm using it for my own purposes, but if people have suggestions to improve it, I'd be happy to consider it. I hope it will be useful to other people as well. Kind Regards, Mladen
Re: java.lang.ClassNotFoundException: org.apache.catalina.filters.HttpHeaderSecurityFilter for app specific web.xml under Tomcat 8.0.9 (bundled with Netbeans)
Hi Mark, It seems that in Tomcat 8 it was added in Tomcat 8.0.23 (merged from Tomcat 7 I guess) - [image: Add:] 54618 <http://bz.apache.org/bugzilla/show_bug.cgi?id=54618>: Add a new HttpHeaderSecurityFilter that adds the Strict-Transport-Security, X-Frame-Options and X-Content-Type-Options HTTP headers to the response. (markt) And Netbeans embedded version I was using was 8.0.9, I guess that was the problem. On Thu, Sep 8, 2016 at 11:43 AM, Mark Thomas <ma...@apache.org> wrote: > On 08/09/2016 10:12, Mladen Adamović wrote: > > I want in some specific apps to enable HttpHeaderSecurityFilter (I might > > have some insecure applications at the same server). > > > > > But I've got the error message when running from Tomcat 8.0.9.0. This > > happened in a development environment, this Tomcat was installed with > > Netbeans 8.0.1. > > > > 08-Sep-2016 09:35:37.108 SEVERE [http-nio-8084-exec-7] > > org.apache.catalina.core.StandardContext.filterStart Exception starting > > filter httpHeaderSecurity > > java.lang.ClassNotFoundException: org.apache.catalina.filters. > > HttpHeaderSecurityFilter > > > > > What could be the reason Tomcat is displaying ClassNotFoundExpceiotn for > > org.apache.catalina.filters.HttpHeaderSecurityFilter, since this exists > > since Tomcat 7? > > Read this: > http://svn.us.apache.org/repos/asf/tomcat/tc7.0.x/ > trunk/webapps/docs/changelog.xml > > and this: > http://svn.us.apache.org/repos/asf/tomcat/tc8.0.x/ > trunk/webapps/docs/changelog.xml > > Search for "HttpHeaderSecurityFilter" and read all the matching > changelog entries. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
java.lang.ClassNotFoundException: org.apache.catalina.filters.HttpHeaderSecurityFilter for app specific web.xml under Tomcat 8.0.9 (bundled with Netbeans)
I want in some specific apps to enable HttpHeaderSecurityFilter (I might have some insecure applications at the same server). I've edited web.xml of one application (not the tomcat/conf/web.xml file to add this filter): httpHeaderSecurity org.apache.catalina.filters.HttpHeaderSecurityFilter true hstsMaxAgeSeconds 31536000 httpHeaderSecurity /* / REQUEST But I've got the error message when running from Tomcat 8.0.9.0. This happened in a development environment, this Tomcat was installed with Netbeans 8.0.1. 08-Sep-2016 09:35:37.108 SEVERE [http-nio-8084-exec-7] org.apache.catalina.core.StandardContext.filterStart Exception starting filter httpHeaderSecurity java.lang.ClassNotFoundException: org.apache.catalina.filters. HttpHeaderSecurityFilter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.catalina.core.DefaultInstanceManager.loadClass( DefaultInstanceManager.java:540) at org.apache.catalina.core.DefaultInstanceManager. loadClassMaybePrivileged(DefaultInstanceManager.java:531) at org.apache.catalina.core.DefaultInstanceManager.newInstance( DefaultInstanceManager.java:150) at org.apache.catalina.core.ApplicationFilterConfig.getFilter( ApplicationFilterConfig.java:258) at org.apache.catalina.core.ApplicationFilterConfig. (ApplicationFilterConfig.java:105) at org.apache.catalina.core.StandardContext.filterStart( StandardContext.java:4603) at org.apache.catalina.core.StandardContext.startInternal( StandardContext.java:5210) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal( ContainerBase.java:724) at org.apache.catalina.core.ContainerBase.addChild( ContainerBase.java:700) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714) at org.apache.catalina.startup.HostConfig.deployDescriptor( HostConfig.java:581) at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:455) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1496) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.tomcat.util.modeler.BaseModelMBean.invoke( BaseModelMBean.java:300) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke( DefaultMBeanServerInterceptor.java:819) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke( JmxMBeanServer.java:801) at org.apache.catalina.manager.ManagerServlet.check( ManagerServlet.java:1437) at org.apache.catalina.manager.ManagerServlet.deploy( ManagerServlet.java:884) at org.apache.catalina.manager.ManagerServlet.doGet( ManagerServlet.java:335) at javax.servlet.http.HttpServlet.service(HttpServlet.java:618) at javax.servlet.http.HttpServlet.service(HttpServlet.java:725) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter( ApplicationFilterChain.java:206) at org.apache.tomcat.websocket.server.WsFilter.doFilter( WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter( ApplicationFilterChain.java:206) at org.netbeans.modules.web.monitor.server.MonitorFilter. doFilter(MonitorFilter.java:393) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter( ApplicationFilterChain.java:206) at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter( SetCharacterEncodingFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter( ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke( StandardWrapperValve.java:219) at org.apache.catalina.core.StandardContextValve.invoke( StandardContextValve.java:106) at org.apache.catalina.authenticator.AuthenticatorBase.invoke( AuthenticatorBase.java:615) at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:136) at org.apache.catalina.valves.ErrorReportValve.invoke(
