Re: Tomcat 8.0.24 seems broken (empty POST body)
On 31/07/2015 04:16, David Landis wrote: > Just tried to upgrade my app from 8.0.20 to 8.0.24 and noticed I couldn't > login. The POST requests were correct on the client side but had an empty > body on the server side. > > So instead I tried upgrading to 8.0.23 and everything works perfectly fine > there. Looked at the 8.0.24 changelog and didn't see anything obvious. Really? Look again at the first entry in the changelog for 8.0.24. Mark > > Env: > CentOS 7 > apr connector > java 8 > > > Example headers: > > POST /app/Login HTTP/1.1 > Host: localhost:8443 > User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 > Firefox/31.0 > Accept: */* > Accept-Language: en-US,en;q=0.5 > Accept-Encoding: gzip, deflate > Content-Type: application/x-www-form-urlencoded; charset=UTF-8 > X-Requested-With: XMLHttpRequest > Referer: https://localhost:8443/app/ > Content-Length: 35 > Cookie: JSESSIONID=jaksjdfkjasdkljflkajsdf > Connection: keep-alive > Pragma: no-cache > Cache-Control: no-cache > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 8.0.24 seems broken (empty POST body)
Just tried to upgrade my app from 8.0.20 to 8.0.24 and noticed I couldn't login. The POST requests were correct on the client side but had an empty body on the server side. So instead I tried upgrading to 8.0.23 and everything works perfectly fine there. Looked at the 8.0.24 changelog and didn't see anything obvious. Env: CentOS 7 apr connector java 8 Example headers: POST /app/Login HTTP/1.1 Host: localhost:8443 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0 Accept: */* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Requested-With: XMLHttpRequest Referer: https://localhost:8443/app/ Content-Length: 35 Cookie: JSESSIONID=jaksjdfkjasdkljflkajsdf Connection: keep-alive Pragma: no-cache Cache-Control: no-cache
Re: "Too Many Connections" exceptions after moving to Tomcat 8
On 7/30/15 9:03 AM, Christopher Schultz wrote: > Jerry, > > On 7/29/15 3:25 PM, Jerry Malcolm wrote: > > Well, it appears that we are slowly getting to the bottom of this. > > But with every answer, I get a few more questions > > > First, I installed the latest TC8 on my laptop, copied my > > server.conf and conf/catalina folder to it and started it up just > > to see what errors I got. After changing out an obsolete listener, > > it came up. I found all of the parm exceptions in > > stderr. So that question is cleared up. Thanks for clarifying > > where to find that. > > If you have an obsolete Listener, you probably copied your server.xml > from Tomcat 7 to Tomcat 8 which, while being less of a disaster than > with previous version-pairs, is not good practice. > > Instead, start with the stock server.xml that comes with your Tomcat > version and modify it to suit. These days, you should pretty much only > have to configure the and elements, unless you > have a particularly exotic configuration. > > > The site is a wedding vendor advertiser site that spans two major > > cities. There is no user login. Simply a very huge online > > catalog. I'm certain it's deployed only once. Whether I need that > > many connections is a valid question. As far as I know, I haven't > > hit the limit in normal operation until now. Could possibly reduce > > the count if I collect statistics. > > Our user load is roughly 250 concurrently logged-in users per Tomcat > node, and we have maxTotal="20". I never get alarms about hitting that > maximum. Your requirements may be different. > > > I've been monitoring the production server logs all day watching to > > be sure connection pool doesn't dry up again. About an hour ago, > > there was a single huge dump in stdout of approx 2000 > > 'logAbandoned' exceptions. They showed connections from 1am right > > after my last bounce of the server thru 1:35pm. > > It looks like your startup process (likely loading and caching stuff > from the db on launch) is leaky. That can run-up your connection could > quite quickly. > > > The good news is with the stack trace on one of them I was able to > > see the bug causing the leak. > > Good. > > But why did it decide to wait over 12 hours accumulating abandoned > > connections before dumping them back in the pool? > > I was about to say the following, but markt says it might be a bug in > DBCP. The bug is in Commons Pool (POOL-300). It is not flushing its abandoned object log. That means abandoned traces won't appear (in the default System.out configuration) until some have been accumulated. > I'll say it anyway: > > DBCP 2 looks like it only checks for abandoned connections "on borrow" > so it might only log their abandonment when you see a flurry of > connection-checkouts occurring, not when the connections are returned > to the pool. DBCP 1 would complain pretty much immediately when the > timeout was reached and the connection hadn't been returned. When DBCP checks for abandoned connections depends in its configuration properties. There are two relevant properties: removeAbandonedOnBorrow and removeAbandonedOnMaintenance. > > > I realize from now knowing the code bug that the leak is a slow > > drip that is continually leaking on a regular basis. But since that > > last 12-hour accumulated dump, the abandoning has returned to > > silence. Since leaks are occurring regularly and would be timing > > out regularly, shouldn't I see a similar 'slow drip' of > > logAbandoned entries in stdout instead of a big dump every 12 > > hours? > > > It's going to take a day or two to fix the leak, test, and deploy. > > For testing, set maxTotal="1". You'll find your leaks *very quickly* > that way, because everything will come grinding to a halt when you try > to fetch that second connection from the pool. > > > If indeed abandoned connections are now correctly being returned to > > the pool, then I presume we are back to working the way it did on > > TC7. Still not sure why it started working now. But I guess once I > > get the leak fixed and if TC8 is now configured to handle abandoned > > connections, I'm good. Still would like to know about the > > mega-dump vs. trickle of abandoned connections being logged. > > You should be able to run in testing with an upgraded DBCP 2. You > might have to build it from trunk, though. I'm not sure if you are > okay with that, but it might help you with your testing. The thing to swap out is Commons Pool. There is a release VOTE in progress now for an RC including a fix for POOL-300. A workaround that should work is to get a reference to the BasicDataSource instance, say, bds and do bds.setAbandonedLogWriter(new PrintWriter(System.out, true)); before using the pool. Not sure if this will work correctly to get the output properly directed under tomcat; but it is worth a try. Phil > > Glad you are rooting-out some problems with your code. > > I highly recommend the use of findbugs: > ht
Re: Tomcat8 startup service exit (return value 143)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris, On 7/30/2015 1:03 PM, Christopher Schultz wrote: > Mark, > > On 7/30/15 12:59 PM, Mark Eggers wrote: >> Comments inline: > >> On 7/30/2015 9:16 AM, Christopher Schultz wrote: >>> Robert, > >>> On 7/30/15 2:28 AM, Robert Jacobs wrote: We have double checked the environment settings we set and the only thing we currently set is JAVA_HOME, which is set to /usr/java/latest . This points to 1.8.0_20-b26. > We also created a new virtual machine via virtualbox, also running RHEL 7.1 and tomcat 8.0.18. We experience the exact same issues as with our vmware virtual machine. Could it be that we are experiencing this because we are using virtual machines? It should not matter, right? Is it related to RHEL 7.1? > We noticed that Tomcat startup times vary in terms of 900ms up to 9000ms on this new virtual machine. > >> The 9ms (90 seconds) is one clue I think. > > >>> If you are using SSL, the problem is likely lack of entropy. >>> Virtual machines are often hamstrung by virtual devices not >>> providing enough noise to gather entropy. Make sure your VM has >>> access to the underlying host's entropy source. > >>> This can also happen when creating a SecureRandom instance for >>> generating session ids. Previously-posted log messages seem to >>> show that this is what is happening to you. If you do a lot of >>> restarts on a VM in a short period of tie, you can drain your >>> entropy source. > > >> This is the other clue I think. > We also noticed that Tomcat returns 143 when deploying first webapp (which is the Tomcat docs folder). > >>> When does the "143" get returned? Does the JVM exit and produce >>> that as the return code? Or is jsvc producing that return >>> code? Or is systemctl returning that code? It would be great to >>> know what tool is returning this "143". That's not a standard >>> OS exit code, at least not on any system I have access to. > >> Chris, I did a little looking around, and bash will report a >> signal + 128. So 143 is a SIGTERM. Some other process is killing >> off Tomcat. > > Good thought. I wonder if systemctl is running daemon.sh, not > getting a response in a certain amount of time, and them killing > itself or something equally foolish like that. > > I must admit I have no experience using daemon.sh or jsvc, so I > can't speak from experience. > >>> When you say "when deploying the first webapp" are you saying >>> that the JVM exits when the first web application is trying to >>> deploy, or something else? Is there a second web application? >>> Does that ever start loading? > >> Here's my current working theory. > >> As Chris has stated, lots of VM restarts mean that you're >> probably short on entropy. If you're short on entropy, you'll >> need to wait longer for Tomcat to start up. > >> By default, systemd has a 90 second timeout for starting a unit. >> If a unit doesn't signal a start-up within that time, systemd >> will shut down that service. > > With a SIGTERM? It appears to be so - according to the documentation on freedesktop.org. If the unit doesn't exit after a SIGTERM, then systemd issues a SIGKILL. This appears to be configurable - see man 5 systemd.kill for particulars . Like the start-up timeout, this can be configured per unit as well. > >> You can configure a per-unit start-up time. Do a man 5 >> systemd.service to see how. > >> It would be better if you could access the underlying entropy >> source as Chris has suggested. 90 seconds is a long time to >> wait. > > Sounds like a good working theory. > > -chris /mde/ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJVupUnAAoJEEFGbsYNeTwtXJMIAIGgRZs/gUoXFi2HmyVa2Cwm dH1zsdi1Y9EfAqg9JYAdKN2s9EZ28xUMJRPRntun8JWWrEQCN60vVIxTgxog28+S lXkMVrEPhGPLpgRKka513mO9eyI4SVcGWgmuqIK6Q0/2uobiad+cg24KmjAfytU1 xbQzYb5V8rnw1zqIJko4AQYGA2cuF08auwYTdtQdBVBR/afmpOw4DVwOyH3ltqCZ k667JD7N5hG/Kus+vTwr/qoXqunKVoAKXQnTRazYVhKkN0h3WgaRsYfq8qiW0VB6 Jwv6I8p0Suhg5/PujxAvcpXb+wuUTbM4ajnRjQ7shawSV9b/uuq8+b3S1geOX/8= =5I2B -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat8 startup service exit (return value 143)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mark, On 7/30/15 12:59 PM, Mark Eggers wrote: > Comments inline: > > On 7/30/2015 9:16 AM, Christopher Schultz wrote: >> Robert, > >> On 7/30/15 2:28 AM, Robert Jacobs wrote: >>> We have double checked the environment settings we set and the >>> only thing we currently set is JAVA_HOME, which is set to >>> /usr/java/latest . This points to 1.8.0_20-b26. > >>> We also created a new virtual machine via virtualbox, also >>> running RHEL 7.1 and tomcat 8.0.18. We experience the exact >>> same issues as with our vmware virtual machine. Could it be >>> that we are experiencing this because we are using virtual >>> machines? It should not matter, right? Is it related to RHEL >>> 7.1? > >>> We noticed that Tomcat startup times vary in terms of 900ms up >>> to 9000ms on this new virtual machine. > > The 9ms (90 seconds) is one clue I think. > > >> If you are using SSL, the problem is likely lack of entropy. >> Virtual machines are often hamstrung by virtual devices not >> providing enough noise to gather entropy. Make sure your VM has >> access to the underlying host's entropy source. > >> This can also happen when creating a SecureRandom instance for >> generating session ids. Previously-posted log messages seem to >> show that this is what is happening to you. If you do a lot of >> restarts on a VM in a short period of tie, you can drain your >> entropy source. > > > This is the other clue I think. > >>> We also noticed that Tomcat returns 143 when deploying first >>> webapp (which is the Tomcat docs folder). > >> When does the "143" get returned? Does the JVM exit and produce >> that as the return code? Or is jsvc producing that return code? >> Or is systemctl returning that code? It would be great to know >> what tool is returning this "143". That's not a standard OS exit >> code, at least not on any system I have access to. > > Chris, I did a little looking around, and bash will report a signal > + 128. So 143 is a SIGTERM. Some other process is killing off > Tomcat. Good thought. I wonder if systemctl is running daemon.sh, not getting a response in a certain amount of time, and them killing itself or something equally foolish like that. I must admit I have no experience using daemon.sh or jsvc, so I can't speak from experience. >> When you say "when deploying the first webapp" are you saying >> that the JVM exits when the first web application is trying to >> deploy, or something else? Is there a second web application? >> Does that ever start loading? > > Here's my current working theory. > > As Chris has stated, lots of VM restarts mean that you're probably > short on entropy. If you're short on entropy, you'll need to wait > longer for Tomcat to start up. > > By default, systemd has a 90 second timeout for starting a unit. If > a unit doesn't signal a start-up within that time, systemd will > shut down that service. With a SIGTERM? > You can configure a per-unit start-up time. Do a man 5 > systemd.service to see how. > > It would be better if you could access the underlying entropy > source as Chris has suggested. 90 seconds is a long time to wait. Sounds like a good working theory. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVuoL9AAoJEBzwKT+lPKRYBHUQAKkgO+aUAPqzkbQnowhAjTNf SJvRZKKa1MKBnyBwFL69dWncCgdzW0Ly3h8m24JgFlKIEtABw5fnbgrYl9cmZPCb NrZTjPdX/wnX/w3OPB1rVygiP/MK4RRLIMsJB1A6mCocxPEjsKhfSMMt84uA14dt MtoUM75+WBC5SexOhTY56XUm1+5Lpkf8j9Labrsw+GXQK/b6x/SMUTDdTE7q+L5q JmTtQha6FQTh0B5vnZr8aawNWmRbqGJxYOYxM7BL0PSzHi1CS+sa1SvSF+J0/5BG M7QN8WIFT4P+FjzXvveYJGeXNHWLu9A5pRmUWothyJQGFNVqVZyF98V6KrkjOO81 4wqezad2g/K6QfQdw99YT4ihLba8ZfYNZSqEus34/txb8+DJIrebHVl5yeHl4wv/ 5OHFETWfTKwQWgSFOYADb0v2f4gARNsVtCCa32TB1NaLdxZBrL0o9xzjul2D0ETr c3J+Po1nnSi1J0HKqByMndGzCieB35wCDbPAdmTpnfzXa2MgMuDCb08TsHj7HZ8N 5ae5pR6JlTosttzLAgcPYTeM71MaU6nmg4SIFjeqFY0C9/PYHu7iiHLxqGNtBk4q n8fD4HL4tQ5y47lXCmA/ehAJq7YQXnYJ3J3oazqO2p6a1Cm2ENpoPfTZZp4m/1v7 2s0Yg9yM/z10N26wI/W3 =rM46 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat8 startup service exit (return value 143)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Comments inline: On 7/30/2015 9:16 AM, Christopher Schultz wrote: > Robert, > > On 7/30/15 2:28 AM, Robert Jacobs wrote: >> We have double checked the environment settings we set and the >> only thing we currently set is JAVA_HOME, which is set to >> /usr/java/latest . This points to 1.8.0_20-b26. > >> We also created a new virtual machine via virtualbox, also >> running RHEL 7.1 and tomcat 8.0.18. We experience the exact same >> issues as with our vmware virtual machine. Could it be that we >> are experiencing this because we are using virtual machines? It >> should not matter, right? Is it related to RHEL 7.1? > >> We noticed that Tomcat startup times vary in terms of 900ms up >> to 9000ms on this new virtual machine. The 9ms (90 seconds) is one clue I think. > > If you are using SSL, the problem is likely lack of entropy. > Virtual machines are often hamstrung by virtual devices not > providing enough noise to gather entropy. Make sure your VM has > access to the underlying host's entropy source. > > This can also happen when creating a SecureRandom instance for > generating session ids. Previously-posted log messages seem to > show that this is what is happening to you. If you do a lot of > restarts on a VM in a short period of tie, you can drain your > entropy source. > This is the other clue I think. >> We also noticed that Tomcat returns 143 when deploying first >> webapp (which is the Tomcat docs folder). > > When does the "143" get returned? Does the JVM exit and produce > that as the return code? Or is jsvc producing that return code? Or > is systemctl returning that code? It would be great to know what > tool is returning this "143". That's not a standard OS exit code, > at least not on any system I have access to. Chris, I did a little looking around, and bash will report a signal + 128. So 143 is a SIGTERM. Some other process is killing off Tomcat. > > When you say "when deploying the first webapp" are you saying that > the JVM exits when the first web application is trying to deploy, > or something else? Is there a second web application? Does that > ever start loading? Here's my current working theory. As Chris has stated, lots of VM restarts mean that you're probably short on entropy. If you're short on entropy, you'll need to wait longer for Tomcat to start up. By default, systemd has a 90 second timeout for starting a unit. If a unit doesn't signal a start-up within that time, systemd will shut down that service. You can configure a per-unit start-up time. Do a man 5 systemd.service to see how. It would be better if you could access the underlying entropy source as Chris has suggested. 90 seconds is a long time to wait. . . . just my two cents /mde/ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQEcBAEBAgAGBQJVulgLAAoJEEFGbsYNeTwtILkH/0x/d4Z4j1AdifTz3ySpwAya J8Z0/qxCT4PJ9Xld5ibHkJQqDAjwuab32jqmI0ksvd2DmRI7YRRwz33Tf5U/ffDV 4oNbw30VGiOtyGCaEOSswrMnlvMQHfoi9N/P0dvA8LaEdIdm2yasGEc+mxe5X8EO MwfFZ0IkkMgB8xnF3YP2iPu93ayimDRxnbkUPR/ucwEsVgsfkrEw6OZwHItCl7Jc 8O1+ZiTjEMw8bfXHFCjmtyRxi294Hea4qw4E/avFimQPtMoIlvootcHbMximQDcf +BwCyztd9CCHCNRuBYDZxo8db4jnDDPGZKFxHLPSBY4vH0+OUSQhGFRYOhCQn0M= =jS2N -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: memory trace of the server with 2 instances of tomcat 7 running
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ethen, On 7/30/15 6:52 AM, Ethen John wrote: > I have a linux server which is running 2 similar instances of > Tomcats 7. > > Each of the tomcat instances has the exact same web application > (.war) deployed. > > The application is not actively used by the users, but the behavior > of the application on both the tomcat instances is looking very > sluggish. What are you using as your performance metric? > I checked the memory utilization of the server machine, which is > as follows: > > top - 12:47:20 up 182 days, 43 min, 1 user, load average: 0.09, > 0.16, 0.15 Tasks: 226 total, 1 running, 225 sleeping, 0 stopped, 0 > zombie Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, > 0.0%si, 0.0%st Mem: 15529M total, 11474M used, 4054M free, 485M > buffers Swap: 16383M total, 0M used, 16383M free, 9633M cached So you have a system with 16GiB of RAM. 16GiB of swap seems like a lot. I think Java tries very hard not to get swapped-out (might prohibit the OS from doing it), so you may find that you never use all that. The machine looks completely idle to me. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7866 root 0 > -20 43036 12m 2640 S 12 0.1 2479:02 scopeux 1 root 20 0 10548 824 > 692 S 0 0.0 1:41.80 init 2 root 20 0 0 0 0 S 0 0.0 0:00.16 > kthreadd 3 root 20 0 0 0 0 S 0 0.0 3:09.18 ksoftirqd/0 5 root 20 0 > 0 0 0 S 0 0.0 15:56.61 kworker/u:0 So... that's the useless part of the output. What about your JVM processes? > = > > yyy@xxx:~> free -m total used free shared buffers cached Mem: 15529 > 11469 4059 0 485 9633 -/+ buffers/cache: 1350 14178 Swap: 16383 0 > 16383 Okay, nothing new here -- top included this already. > == > > yyy@xxx:~> df -kh Filesystem Size Used Avail Use% Mounted on > /dev/mapper/vg00-rootvol 5.0G 1.4G 3.4G 29% / udev 7.6G 220K 7.6G > 1% /dev tmpfs 24G 0 24G 0% /dev/shm /dev/sda1 510M 48M 437M 10% > /boot /dev/mapper/vg00-homevol 992M 12M 929M 2% /home > /dev/mapper/vg00-tmpvol 2.0G 201M 1.7G 11% /tmp > /dev/mapper/vg00-varvol 7.9G 319M 7.2G 5% /var > /dev/mapper/vg00-crashvol 2.0G 68M 1.9G 4% /crashdump > /dev/mapper/vg00-auditvol 248M 11M 226M 5% /var/log/audit > /dev/mapper/vg00-optvol 12G 1.5G 9.8G 13% /opt > /dev/mapper/vg00-webdienste 99G 3.7G 95G 4% /webdienste The disk stats don't matter much. > Does the information above provide any pointers that can be > addressed further. Can you give us any information other than "I've got a server with a bunch of RAM and my application feels slow"? - From what you've shown us, there is no problem: the server is idle. > The strange thing is the used memory consumption of 11474M: Why is that strange? Memory exists to be used. Unused memory is a waste of silicon and power. What are the memory-related parameters for your JVMs? > Mem: 15529M total, 11474M used, 4054M free, 485M buffers > > I am also planning to have Lymbda probe installed on each of the > tomcat instances to get more details. Or you can just use "ps", but feel free to get heap-level information. > Afterwords, I am also planning to restart the server machine. That usually doesn't have a measurable effect on performance. > The application works fine on the other earlier version (5) of > Tomcat instance. It's just that we have now migrated the > application to Tomcat 7 with Java 6 and experiencing this > behavior. Glad to see you're all up-to-date for deployment in 2011. > The new server machine only running 2 tomcat 7 instances and > nothing else. It appears to be sitting idle. What's the problem? - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVulA4AAoJEBzwKT+lPKRYFH4QAJZsWHe+vtgMWSXFub/QmkFx bggJNpcVxlDJrs3PBTjWTYuhuttsb++K1IUWeCE3Jc+W3YKwMAFYeSuHiOY0vVcT 95/b0sW9ePrbWWP4TFFx2xX52Bhnoc3Kij2jZuMvQBN/rtpD595J9NusssZcIu6I CR/mGxzpsU42cDtWyB7Q9WYm0dQnhq3qA6LN+GwYPUvR4IaTfCn+e0WLShbH+i/k qW0SIUIvFh7Rz/wnFa8BGRCmlpWp5QPbsnNsS7SF4cqU/riWXtj+f1vt521wNlpj xEUXn5f3kDsIVu1LFvNd0IvqpmWvRpdLbPH3D7EYJafo2sA3me3LyJRq+YfGZHVg tTRNG4TrlQ/oiXs1QDerSgehCYvpMkp4dK1NMD5kfei9vCYAJXQJfNc1Pqec5ko3 ETt22jg4oflwxP0kw2i2F+KgLjm1xqFgVxr8g/NODjRvEs+mglxYEchrm2loPSdT sU0xQE0ipvf+2S6M1MaimKFb2E7LqnbDSqU/UtNh0hRmUTrYBnbaXXfV1LUKs48Q BWinFrie+auNMohLHmpD3moVO8VWQaalJa9JthMfw4GoNKIi6FgZHMEc3pKhEZSI TMZLR1vxu1sur3DmmUaIZ9Jn/oHYMVpG6fsLXVbS9TseAjCbqptiOxOeHNvSk0kK OMvZKeLtD65uyw/sTSGV =JtMt -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat8 startup service exit (return value 143)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Robert, On 7/30/15 2:28 AM, Robert Jacobs wrote: > We have double checked the environment settings we set and the only > thing we currently set is JAVA_HOME, which is set to > /usr/java/latest . This points to 1.8.0_20-b26. > > We also created a new virtual machine via virtualbox, also running > RHEL 7.1 and tomcat 8.0.18. We experience the exact same issues as > with our vmware virtual machine. Could it be that we are > experiencing this because we are using virtual machines? It should > not matter, right? Is it related to RHEL 7.1? > > We noticed that Tomcat startup times vary in terms of 900ms up to > 9000ms on this new virtual machine. If you are using SSL, the problem is likely lack of entropy. Virtual machines are often hamstrung by virtual devices not providing enough noise to gather entropy. Make sure your VM has access to the underlying host's entropy source. This can also happen when creating a SecureRandom instance for generating session ids. Previously-posted log messages seem to show that this is what is happening to you. If you do a lot of restarts on a VM in a short period of tie, you can drain your entropy source. > We also noticed that Tomcat returns 143 when deploying first > webapp (which is the Tomcat docs folder). When does the "143" get returned? Does the JVM exit and produce that as the return code? Or is jsvc producing that return code? Or is systemctl returning that code? It would be great to know what tool is returning this "143". That's not a standard OS exit code, at least not on any system I have access to. When you say "when deploying the first webapp" are you saying that the JVM exits when the first web application is trying to deploy, or something else? Is there a second web application? Does that ever start loading? - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVuk3iAAoJEBzwKT+lPKRYuEQP/3saDU+V8/8yEPouHiCtBv94 R54w71iTN5LqCQXyUjXJ/DUTGWpM0VFfuEpGNOaJNR3M88GWOsl36g70COVeMprZ FqkHk5/Gen/iMg9s7fOh+iePlfbGHajkohTZbpYFyJRkBcgidr1GaakFyAxFdo/m AlwcLuneVrwsF2PKwqdUQNNT11WcDpCZ9gJtJknosyEfbf6kQa7mpSFhoy2X10B+ k2Nc9lB/x2k6OHTDKBXdqf5yM4qoBUTNRmGUMaDQSlL6WC1/VLzC748ZJ3hpXtQr r3VUwdJQoHBbWwWzavOHPuKOaJwKuTcLctCcgA4jTinN7LkCI1MUTdLeSvWYtHR7 5VIxl/HKo5shRAKjM7rTr9abCfJ3K3RknIIgONEwKTGXBtRApqwKIfcp8viMjhtV Y1nrysGBaFYN64rws78pAKfwkk8PqAs/qou77+/oU+x56P7u/TBMRlG5ADtBydbC ycoH6lw6Z5fiRVDvWB8xsnswloTtIOHJfmz2B4JixqebjI5NdkbTP8YTFaIUpBZl 1JxABUWdsrp2wZ2edW2qihQqQ9PdtnRGzOzooHQY/8TGVXMaoIytxv3A8VNsYjyR vejUzcwWYifG4n5AtXa0jHzKpDbOhgpQWECj4ZLZtfrh0+QmzOvWSkiftIIljVrs y2fQA/lkU4KZNbOyOO7g =zQHy -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: "Too Many Connections" exceptions after moving to Tomcat 8
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jerry, On 7/29/15 3:25 PM, Jerry Malcolm wrote: > Well, it appears that we are slowly getting to the bottom of this. > But with every answer, I get a few more questions > > First, I installed the latest TC8 on my laptop, copied my > server.conf and conf/catalina folder to it and started it up just > to see what errors I got. After changing out an obsolete listener, > it came up. I found all of the parm exceptions in > stderr. So that question is cleared up. Thanks for clarifying > where to find that. If you have an obsolete Listener, you probably copied your server.xml from Tomcat 7 to Tomcat 8 which, while being less of a disaster than with previous version-pairs, is not good practice. Instead, start with the stock server.xml that comes with your Tomcat version and modify it to suit. These days, you should pretty much only have to configure the and elements, unless you have a particularly exotic configuration. > The site is a wedding vendor advertiser site that spans two major > cities. There is no user login. Simply a very huge online > catalog. I'm certain it's deployed only once. Whether I need that > many connections is a valid question. As far as I know, I haven't > hit the limit in normal operation until now. Could possibly reduce > the count if I collect statistics. Our user load is roughly 250 concurrently logged-in users per Tomcat node, and we have maxTotal="20". I never get alarms about hitting that maximum. Your requirements may be different. > I've been monitoring the production server logs all day watching to > be sure connection pool doesn't dry up again. About an hour ago, > there was a single huge dump in stdout of approx 2000 > 'logAbandoned' exceptions. They showed connections from 1am right > after my last bounce of the server thru 1:35pm. It looks like your startup process (likely loading and caching stuff from the db on launch) is leaky. That can run-up your connection could quite quickly. > The good news is with the stack trace on one of them I was able to > see the bug causing the leak. Good. > But why did it decide to wait over 12 hours accumulating abandoned > connections before dumping them back in the pool? I was about to say the following, but markt says it might be a bug in DBCP. I'll say it anyway: DBCP 2 looks like it only checks for abandoned connections "on borrow" so it might only log their abandonment when you see a flurry of connection-checkouts occurring, not when the connections are returned to the pool. DBCP 1 would complain pretty much immediately when the timeout was reached and the connection hadn't been returned. > I realize from now knowing the code bug that the leak is a slow > drip that is continually leaking on a regular basis. But since that > last 12-hour accumulated dump, the abandoning has returned to > silence. Since leaks are occurring regularly and would be timing > out regularly, shouldn't I see a similar 'slow drip' of > logAbandoned entries in stdout instead of a big dump every 12 > hours? > > It's going to take a day or two to fix the leak, test, and deploy. For testing, set maxTotal="1". You'll find your leaks *very quickly* that way, because everything will come grinding to a halt when you try to fetch that second connection from the pool. > If indeed abandoned connections are now correctly being returned to > the pool, then I presume we are back to working the way it did on > TC7. Still not sure why it started working now. But I guess once I > get the leak fixed and if TC8 is now configured to handle abandoned > connections, I'm good. Still would like to know about the > mega-dump vs. trickle of abandoned connections being logged. You should be able to run in testing with an upgraded DBCP 2. You might have to build it from trunk, though. I'm not sure if you are okay with that, but it might help you with your testing. Glad you are rooting-out some problems with your code. I highly recommend the use of findbugs: http://findbugs.sourceforge.net/ It automatically detects stuff like this in your code, as well as a whole bunch of other things. It sometimes finds false positives, but you can knock-up a configuration file to exclude those things in very specific cases if need be. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVukrfAAoJEBzwKT+lPKRYY20QAJw+fAfLXhTiTFadMRTeZFcX aU5oLi08BVPUq1/gkCAykZkRFTDGx+iGAo7QKPfjfwstX89wFmY50BWppo/MT6sA S0/nGQxwzSHI+lT1rxMAPhY/vq6PJXg92izkKejaF+8sy3uu5vbvzOviOcqtUAdD +3TDX2ybZpQI1rS4lCZRrCqVONSH0if8pooBLUeQBmnXuUDDuRfU5mQB/mT4XRcr MTWqjEU7/WgCmpTgdHyIEIImGNkey/79PAzxa9t8QjecdHpkCO4cE+5lmcw+0eL1 i/f2/+ujt5nDSv/n/+4PE9SEj2w70O0q9HNwmEOKBi9bv3lPEwzI38ixnm44/bqa rV+/BPoKd6mZUQuXsH4cJccGhaZ1Sn5MzC2BPgFfBSeK3yGnNm7k6U9iyQ7EoW2O HzAJquikdaPHKLvtmd76heMzxdSckDrlu2Xi8Jfp1exf/aTh426niunTZHrSRRPI lJFmUCRglpG9peOXxf+sH7pvO1DMcpt8ExMrtLvKXPbEQ6MRybHXS5Z0lWrlQRGg XPngxL66BHUVCTmrd4Q0RpysYkbtuO7
Re: programatically manipulating a JSP
Hello again, just for the archive: It turned out that it worked quite well to use Jericho for that. [1] - http://jericho.htmlparser.net/docs/index.html Best, Sandro Am 19.07.15 um 14:32 schrieb Sandro Boehme: Hello, I have a use case where I have to programatically add HTML and/or Scriptlet code to a JSP. It has to be placed beside an HTML tag with a known id or at the end or the beginning within an HTML tag with a known id. I would like to use Jasper 6.0.14 (or a newer version if that helps) but I haven't found tests in http://svn.apache.org/repos/asf/tomcat/tc7.0.x/trunk/test that come close to what I want (didn't check version 6.0.14 though) and I also haven't found documentation that I can use to get into the details. Does it makes sense to use https://theholyjava.wordpress.com/2011/06/10/hacking-jasper-to-get-object-model-of-a-jsp-page/ as a starting point to get it done or is there a better way to approach it in the mean time? I thought about regExp but it seemed to be too unreliably to me. Thanks for any advice, Sandro - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat8 startup service exit (return value 143)
Sorry to insist on that point, but have you tried to start Tomcat with an
ordinary user account? To do that change the port it listens to for
anything over 1024. Then start it with deamon.sh. If it works with one
account but not the other, there is a difference hiding somewhere. Also, I
suggest to work with a stock Tomcat.
I am on RHEL 5.11 running inside WMWare. We use Oracle Java 1.7.0u75.
2015-07-30 2:28 GMT-04:00 Robert Jacobs :
> Hello,
>
> We have double checked the environment settings we set and the only thing
> we currently set is JAVA_HOME, which is set to /usr/java/latest . This
> points to 1.8.0_20-b26.
>
> We also created a new virtual machine via virtualbox, also running RHEL 7.1
> and tomcat 8.0.18. We experience the exact same issues as with our vmware
> virtual machine. Could it be that we are experiencing this because we are
> using virtual machines? It should not matter, right? Is it related to RHEL
> 7.1?
>
> We noticed that Tomcat startup times vary in terms of 900ms up to 9000ms on
> this new virtual machine. We also noticed that Tomcat returns 143 when
> deploying first webapp (which is the Tomcat docs folder).
>
> Is there additional troubleshooting info which we can gather to provide you
> with some more input?
>
> We are lost here.
>
> Thanks again!
>
> Regards,
> Robert
>
> Op vr 24 jul. 2015 om 21:49 schreef Philippe Anctil <
> philippe.anc...@gmail.com>:
>
> > I have been struggling with an issue w.hich resulted in the same
> > message: "Service
> > exit with a return value of 143".
> >
> > In the end the problem was with a custom init script that was calling
> jsvc
> > directly. I did not bother to find the precise root cause though. When I
> > saw daemon.sh worked, I simply modified my init script to use it.
> >
> > daemon.sh is sensitive to many environment variables. Do you have some of
> > these variables exported? I suggest to run deamon.sh from a "clean"
> shell.
> > Define variables one by one as required and see if Tomcat finally starts
> > correctly.
> >
> >
> >
> >
> > 2015-07-24 6:53 GMT-04:00 Robert Jacobs :
> >
> > > On Fri, Jul 24, 2015 at 10:15 AM, Mark Eggers
> > > > > > wrote:
> > >
> > > > -BEGIN PGP SIGNED MESSAGE-
> > > > Hash: SHA1
> > > >
> > > > On 7/24/2015 12:59 AM, Robert Jacobs wrote:
> > > > > Hello,
> > > > >
> > > > > We are having problems getting tomcat to work with JSVC on RHEL 7.1
> > > > > (maipo). We found out that Tomcat restarts do not always result in
> > > > > expected behaviour. The pattern of this unexpected behavior
> > > > > (Service exit with a return value of 143) is random. This random
> > > > > pattern might depend on the deploy-time of Tomcat during startup.
> > > > >
> > > > > The problem might lay within the daemon functionality using JSVC.
> > > > >
> > > > > We have tried this one two machines; - Fedora 20 desktop ->
> > > > > everything works fine - RHEL 7.1 virtual machine -> random fails
> > > > >
> > > > > Tried versions of tomcat 8 include 18 and 24.
> > > > >
> > > > > We did see other people with this exact problem and error code, but
> > > > > these problems existed for tomcat 5 to 7. We could not find
> > > > > information regarding tomcat 8 yet.
> > > > >
> > > > > Our findings are as follows: systemctl start tomcat fail
> > > > > (143) systemctl stop tomcat fail (143) systemctl restart
> > > > > tomcatfail (143) etc/init.d/tomcat start pass
> > > > > etc/init.d/tomcat stop pass etc/init.d/tomcat restart
> > > > > fail (143)(note: combined stop/start) CATALINA_HOME/bin/daemon.sh
> > > > > startpass CATALINA_HOME/bin/daemon.sh stop pass
> > > > >
> > > > > General note for all stop commands: Every time this "Server
> > > > > startup" is not finished, each stop command results in the 143
> > > > > exit.
> > > > >
> > > > > The two methods of starting tomcat: init.d and systemctl are:
> > > > > ###init.d #!/bin/bash
> > > > >
> > > > > CATALINA_HOME="/opt/apache-tomcat/apache-tomcat-8.0.24"
> > > > >
> > > > > start() { sh $CATALINA_HOME/bin/daemon.sh start } stop() { sh
> > > > > $CATALINA_HOME/bin/daemon.sh stop }
> > > > >
> > > > > case $1 in start|stop) $1;; restart) stop; start;; *) echo "Run as
> > > > > $0 "; exit 1;; esac ###end of
> > > > > init.d
> > > > >
> > > > > ###systemctl [Unit] Description=Apache Tomcat
> > > > > Server
> > > > >
> > > > > [Install] WantedBy=multi-user.target
> > > > >
> > > > > [Service] Type=oneshot
> > > > > ExecStart=/opt/apache-tomcat/apache-tomcat-8.0.24/bin/daemon.sh
> > > > > start
> > > > > ExecStop=/opt/apache-tomcat/apache-tomcat-8.0.24/bin/daemon.sh
> > > > > stop User=adsswebserver Group=adsswebserver ###end of
> > > > > systemctl
> > > > >
> > > > > Server version:Apache Tomcat/8.0.24 Server built:
> > > > > Jul 1 2015 20:19:55 UTC Server number: 8.0.24.0 OS Name:
> > > > > Linux OS Version:
Re: memory trace of the server with 2 instances of tomcat 7 running
Ethen John wrote: hi all, I have a linux server which is running 2 similar instances of Tomcats 7. Each of the tomcat instances has the exact same web application (.war) deployed. The application is not actively used by the users, but the behavior of the application on both the tomcat instances is looking very sluggish. I checked the memory utilization of the server machine, which is as follows: top - 12:47:20 up 182 days, 43 min, 1 user, load average: 0.09, 0.16, 0.15 Tasks: 226 total, 1 running, 225 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15529M total, 11474M used, 4054M free, 485M buffers Swap: 16383M total, 0M used, 16383M free, 9633M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7866 root 0 -20 43036 12m 2640 S 12 0.1 2479:02 scopeux 1 root 20 0 10548 824 692 S 0 0.0 1:41.80 init 2 root 20 0 0 0 0 S 0 0.0 0:00.16 kthreadd 3 root 20 0 0 0 0 S 0 0.0 3:09.18 ksoftirqd/0 5 root 20 0 0 0 0 S 0 0.0 15:56.61 kworker/u:0 … Just a little thing here : under "top", if you enter "F" and then "n", it will show the processes in the order of memory usage. Did you do that ? (I am asking because I see no "java" processes above, and they tend to come on top of the list when there are any). = yyy@xxx:~> free -m total used free shared buffers cached Mem: 15529 11469 4059 0 485 9633 -/+ buffers/cache: 1350 14178 Swap: 16383 0 16383 == yyy@xxx:~> df -kh Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg00-rootvol 5.0G 1.4G 3.4G 29% / udev 7.6G 220K 7.6G 1% /dev tmpfs 24G 0 24G 0% /dev/shm /dev/sda1 510M 48M 437M 10% /boot /dev/mapper/vg00-homevol 992M 12M 929M 2% /home /dev/mapper/vg00-tmpvol 2.0G 201M 1.7G 11% /tmp /dev/mapper/vg00-varvol 7.9G 319M 7.2G 5% /var /dev/mapper/vg00-crashvol 2.0G 68M 1.9G 4% /crashdump /dev/mapper/vg00-auditvol 248M 11M 226M 5% /var/log/audit /dev/mapper/vg00-optvol 12G 1.5G 9.8G 13% /opt /dev/mapper/vg00-webdienste 99G 3.7G 95G 4% /webdienste - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
memory trace of the server with 2 instances of tomcat 7 running
hi all, I have a linux server which is running 2 similar instances of Tomcats 7. Each of the tomcat instances has the exact same web application (.war) deployed. The application is not actively used by the users, but the behavior of the application on both the tomcat instances is looking very sluggish. I checked the memory utilization of the server machine, which is as follows: top - 12:47:20 up 182 days, 43 min, 1 user, load average: 0.09, 0.16, 0.15 Tasks: 226 total, 1 running, 225 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 0.2%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15529M total, 11474M used, 4054M free, 485M buffers Swap: 16383M total, 0M used, 16383M free, 9633M cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7866 root 0 -20 43036 12m 2640 S 12 0.1 2479:02 scopeux 1 root 20 0 10548 824 692 S 0 0.0 1:41.80 init 2 root 20 0 0 0 0 S 0 0.0 0:00.16 kthreadd 3 root 20 0 0 0 0 S 0 0.0 3:09.18 ksoftirqd/0 5 root 20 0 0 0 0 S 0 0.0 15:56.61 kworker/u:0 … = yyy@xxx:~> free -m total used free shared buffers cached Mem: 15529 11469 4059 0 485 9633 -/+ buffers/cache: 1350 14178 Swap: 16383 0 16383 == yyy@xxx:~> df -kh Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg00-rootvol 5.0G 1.4G 3.4G 29% / udev 7.6G 220K 7.6G 1% /dev tmpfs 24G 0 24G 0% /dev/shm /dev/sda1 510M 48M 437M 10% /boot /dev/mapper/vg00-homevol 992M 12M 929M 2% /home /dev/mapper/vg00-tmpvol 2.0G 201M 1.7G 11% /tmp /dev/mapper/vg00-varvol 7.9G 319M 7.2G 5% /var /dev/mapper/vg00-crashvol 2.0G 68M 1.9G 4% /crashdump /dev/mapper/vg00-auditvol 248M 11M 226M 5% /var/log/audit /dev/mapper/vg00-optvol 12G 1.5G 9.8G 13% /opt /dev/mapper/vg00-webdienste 99G 3.7G 95G 4% /webdienste -- Does the information above provide any pointers that can be addressed further. The strange thing is the used memory consumption of 11474M: Mem: 15529M total, 11474M used, 4054M free, 485M buffers I am also planning to have Lymbda probe installed on each of the tomcat instances to get more details. Afterwords, I am also planning to restart the server machine. The application works fine on the other earlier version (5) of Tomcat instance. It's just that we have now migrated the application to Tomcat 7 with Java 6 and experiencing this behavior. The new server machine only running 2 tomcat 7 instances and nothing else. Any help shall he appreciated. thanks.
Re: Tomcat manager application not using custom ErrorReportingValve !
Thanks a lot Mark. On Thu, Jul 30, 2015 at 11:50 AM, Mark Thomas wrote: > On 30/07/2015 07:18, Utkarsh Dave wrote: > > Hi All, > > > > My application has a custom reporting valve in server.xml > > > >> errorReportValveClass="com..valves.CustomErrorReportValve" > > name="localhost" unpackWARs="true"> > > > > But when I try to access https:///manager > > I get normal error window page of (the tomcat error page is at > > /tomcat/webapps/manager/WEB-INF/jsp/403.jsp > > " > > 403 Unauthorized > > > > You are not authorized to view this page. If you have not changed any > > configuration files, please examine the file conf/tomcat-users.xml in > your > > installation. That file must contain the credentials to let you use this > > webapp. > > .." > > > > How to have the manager application use the custom error valve ? > > Application configured error pages take precedence over any error > reporting valve. > > > Do i need to configure manager application separately? > > You could remove the error page settings from the Manager app's web.xml. > > Mark > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
