RHEL 3.0, Tomcat 4.x and IPTables
Greetings all, I had posted earlier about an issue with Tomcat hanging, and also holding Apache processes hostage. I have narrowed the problem down to some kind of conflict with Tomcat/JVM and IPTables on RedHat Enterprise Linux 3.0 ES. When Tomcat hangs, what happens is, half the page is returned to the browser and the status bar sits at waiting for domain.com. As Soon as I issue service iptables stop the page finishes loading, and I cannot make any other pages half-load. Any apache process that were tied up waiting on tomcat stay hung though, and I have to either manually kill those PIDS or issue a SIGHUP to Apache. When only a few are hung, I kill them, as to not interrupt downloads for other users. However, when 92 are hung, I just SIGHUP apache. Obviously, this is an issue with not being able to run a firewall, as we don't have a hardware firewall in place. Does anyone know what kind of interference/issue IPTables would be causing Tomcat? I will note, this one website is the only Host on this instance. We do also have another tomcat instance which hosts 3 other domains, and those never hang. I am starting to think it might be some of the customers code, though I am not sure what it would be as I am certainly not a Java guru. I have asked him if he contacts outside services in his code, but he has not replied yet. I doubt it is this as I have started the firewall with ports 0-32767 open and it still hangs. Could this be some kind of memory collision between IPTables and Tomcat? As well, this setup worked flawlessly under RH 9.0 and RH 7.3. Just looking for ideas here, as this honestly makes no sense to me. thanks! -Sean Finkel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
Shapira, Yoav wrote: I already have. This exact same setup worked flawlessly under RH 7.3 and RH 9.0 Actually, I meant a non-Linux platform. But even your results so far suggest an RHEL-specific issue. Maybe try LD_ASSUME_KERNEL=2.2.5 instead of 2.4? I remember people having success with that in the past. You know what's funny? I stopped APF (the firewall) and it runs fine. I don't understand this. The needed ports were open (the shutdown, standalone and connector ports). I used the same firewall with the same ports open on the other servers/OSs and it worked fine. But I turn the firewall off and it stops hanging immediately. -Sean - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
Maybe I am missing something, but wouldn't that not be able to bind to
port 80 since Apache is already bound to it? There is a lot about the
Java world I am not familiar with, so maybe it can...
You would need separate ip addresses for apache and tomcat.
Aye I figured as much
And indeed that is what we are doing for him right now. However, my
previous attempts at fiddling with mod_rewrite to translate root request
/whatever.jsp into /appdir/whatever.jsp had failed. I now have this
working in a preliminary stage. Since I was unable to get mod_rewrite
processing the requests right, we just did a jkMount /* workername,
which passed every request off to tomcat, and since we had the root
context setup to serve from ~/public_html/appdir it all worked out fine.
But of course, this results in trying to pass PHP requests (and things
outside the appdir) to tomcat.
how about passing only requests that are not php to tomcat? You can do this
in httpd.conf:
LocationMatch ^/(?!phpdir)
jkUriSet ...
/LocationMatch
We are using mod_jk not mod_jk2. The above would only work with mod_jk2
correct? In any case, I have finally gotten this to work (only passing
off java requests to tomcat). For those interested:
in the site's root .htaccess file:
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/appdir/
RewriteRule ^(.*\.jsp)$ /appdir/$1 [L]
RewriteCond %{REQUEST_URI} !^/appdir/ [NC]
RewriteCond %{HTTP_REFERER} ^(.*)\.jsp.*$ [NC]
RewriteRule ^(.*)$ /appdir/$1 [R,NC]
then in httpd.conf, we just jkMount /appdir/*
Still having problems with Tomcat hanging Apache child processes. I
lowered the MaxRequestsPerChild in Apache to 10. Results in more cpu
work, but keeps the hung processes to a minimum. This morning there were
only 3 instead of 50-200 of them. They all seem to be hanging on image
files now. Weird
-Sean
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
I recall RHEL 3.0 comes with an NPTL-based 2.4, which will require setting LD_KERNEL_ASSUME=2.4 to disable the NPTL functionality for the Java process. That may be it. I am assuming you mean set this as an environment variable? I will give this a try and see what happens! Thank you for the suggestion! Well, that did not work. I was using the AJP13 connector. I have changed the private (Tomcat4) instance to use the Coyote connector to see if that makes a difference. It didn't before, and I doubt it will now. :( This is really becoming frustrating. At least it is not hanging Apache, as in, Apache still responds to *new* requests, it just has 50+ child process tied up waiting for previous Tomcat responses. -Sean - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
Shapira, Yoav wrote: Hi, Perhaps Tomcat standalone would be sufficient for your application requirements? No, because the user also wants access to PHP and other related Apache features (htacces, mod_rewrite, etc). -Sean - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
Sean Finkel wrote: Hello, First a brief background on the setup: We are running Apache 1.3.31 utilizing mod_jk (not jk2). We are running two instances of Tomcat. Previously, both were version 4. Currently, we have one shared instance running the latest 5.x release (just compiled yesterday). We have one customer running a private 4.x instance. With that said, we just moved to a new server (Dual Xeon, RHEL 3.0) and now the problem we are having is the customer who has this private instance has jsp pages that hang. But, it does not hang all the time. What's weird, is most of the time, I can stop the shared instance and the customer's pages will stop hanging. I would like to fix the hanging completely, obviously. Ok this is happening again right now. Here is some output from various programs: From Apache Status (blanked out the VHost) - these Apache child processes have been running for 15+ minutes waiting on Tomcat. They are not servicing new requests and then dieing like they should be (Apache hands *everything* off for this domain to Tomcat, including images): ( snip )--- *Srv* *PID* *Acc* *M* *CPU* *SS* *Req* *Conn* *Child* *Slot* *Host* *VHost* *Request* *0-0* 24535 0/21/1803 *W* 0.23 0 0.0 0.14 21.56 4.153.20.11 GET /images/pics/todayspic27.jpg HTTP/1.1 *1-0* 23257 0/39/2065 *W* 0.46 3452 0 0.0 0.27 12.33 4.153.20.11 GET /images/pics/todayspic16.jpg HTTP/1.1 *2-0* 23252 0/17/1868 *W* 0.20 3593 0 0.0 0.26 15.75 207.69.137.135 GET /images/pics/todayspic13.jpg HTTP/1.1 *3-0* 23377 0/22/1825 *W* 0.26 3445 0 0.0 0.34 20.29 4.153.20.11 GET /images/pics/todayspic21.jpg HTTP/1.1 *4-0* 23378 0/22/1839 *W* 0.38 3439 0 0.0 0.07 16.09 4.153.20.11 GET /images/pics/todayspic15.jpg HTTP/1.1 *5-0* 22810 0/43/1750 *W* 0.26 3584 0 0.0 0.54 19.31 207.69.137.135 GET /images/pics/todayspic12.jpg HTTP/1.1 *6-0* 23267 0/37/1788 *W* 0.68 3425 0 0.0 0.52 14.50 4.153.20.11 GET /images/pics/todayspic19.jpg HTTP/1.1 *7-0* 26919 0/33/1586 *W* 0.15 3143 0 0.0 0.27 12.33 4.153.20.11 GET /images/pics/todayspic36.jpg HTTP/1.1 *8-0* 23441 0/22/1532 *W* 0.28 3385 0 0.0 0.09 11.07 4.153.20.11 GET /images/pics/todayspic25.jpg HTTP/1.1 *9-0* 26920 0/30/1396 *W* 0.36 3147 0 0.0 0.17 8.23 4.153.20.11 GET /images/pics/todayspic34.jpg HTTP/1.1 -( /snip )- From A thread dump of the JVM - obviously there were more than three, but I only included three for the sake of brevity: ---( snip )--- Thread-20 daemon prio=1 tid=0x08192b68 nid=0x3560 in Object.wait() [a84db000..a84db87c] at java.lang.Object.wait(Native Method) - waiting on 0xab9bcba8 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:429) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:512) - locked 0xab9bcba8 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:534) Thread-19 daemon prio=1 tid=0x08195450 nid=0x3560 in Object.wait() [a855c000..a855c87c] at java.lang.Object.wait(Native Method) - waiting on 0xab9bcc10 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:429) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:512) - locked 0xab9bcc10 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:534) Thread-18 daemon prio=1 tid=0x0824b148 nid=0x3560 in Object.wait() [a85dd000..a85dd87c] at java.lang.Object.wait(Native Method) - waiting on 0xab7b9ae0 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:429) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:512) - locked 0xab7b9ae0 (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:534) - ( /snip )-- So, as another user suggested earler, it seems some blocking is happening. Also as I mentioned, I am on RHEL 3.0 and JDK version: 1.4.2_04. Does anyone have any ideas as to how to prevent the threads from locking up? I have already tried the LD_KERNEL_ASSUME=2.4 and that does not help. -Sean
Re: Tomcat causes Apache to hang. Any ideas?
(Apache hands *everything* off for this domain to Tomcat, including images): And yet you said Tomcat standalone wasn't an option for this installation? Too bad. You can do much of mod_rewrite with the balancer app, you can do much of .htaccess with Servlet security constraints, and if Tomcat already handles all the requests than you're losing performance by adding Apache and the connector layer. Sorry, this is a shared server, and he wants his site available on port 80. I suppose we could give him a dedicated IP. Currently, the only reason we hand everything off, is because he has his application hosted inside a directory (ie: ~/public_html/appdir) but wants it to be available to users as http://www.domainname.com/whatever.jsp instead of http://www.domainname.com/appdir/whatever.jsp. He also wants to use PHP on his main domain for forums or some such, once we get him to move his application out of appdir and into his website root. Once he does that, we can modify the jkMount apache directives to only pass off .jsp and /servlet/ requests instead of *everything* Unless you can suggest a way around this currently? From A thread dump of the JVM - obviously there were more than three, but I only included three for the sake of brevity: How did you pick these three threads? They were the only ones that had anything to do with apache (well, there were about 20 of them actually). But now it hits me, duh. Tomcat = Apache project. *sigh* I guess I thought the locked had something to do with it, though I now see it seems all threads say this. So, as another user suggested earler, it seems some blocking is happening. Also as I mentioned, I am on RHEL 3.0 and JDK version: 1.4.2_04. Does anyone have any ideas as to how to prevent the threads from locking up? I have already tried the LD_KERNEL_ASSUME=2.4 and that does not help. The next step would be to try your app on a different platform, IMHO, to try and tell if this is indeed an RHEL-related problem or something else. I already have. This exact same setup worked flawlessly under RH 7.3 and RH 9.0 however, moving (back) to either of those platforms is not an option. With that said however, I compiled tomcat under RH 7.3 and copied everything over. The JDK stayed the same throughout though. Could this be the problem? Should I recompile Tomcat? I *have* recompiled mod_jk on the new system though. thanks for your help! -Sean - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
Sorry, this is a shared server, and he wants his site available on port
80.
Tomcat standalone can run on port 80 without running as root by using
jsvc (from commons-daemon). There are examples and more information on
this configuration at
http://jakarta.apache.org/tomcat/faq/security.html#jsvcExample.
Maybe I am missing something, but wouldn't that not be able to bind to
port 80 since Apache is already bound to it? There is a lot about the
Java world I am not familiar with, so maybe it can...
reason we hand everything off, is because he has his application hosted
inside a directory (ie: ~/public_html/appdir) but wants it to be
available to users as http://www.domainname.com/whatever.jsp instead of
http://www.domainname.com/appdir/whatever.jsp.
You can configure Tomcat to have a Context whose docbase is
~/public_html/appdir but whose path is (the empty string, not null or
/).
And indeed that is what we are doing for him right now. However, my
previous attempts at fiddling with mod_rewrite to translate root request
/whatever.jsp into /appdir/whatever.jsp had failed. I now have this
working in a preliminary stage. Since I was unable to get mod_rewrite
processing the requests right, we just did a jkMount /* workername,
which passed every request off to tomcat, and since we had the root
context setup to serve from ~/public_html/appdir it all worked out fine.
But of course, this results in trying to pass PHP requests (and things
outside the appdir) to tomcat.
He also wants to use PHP
on his main domain for forums or some such, once we get him to move his
application out of appdir and into his website root.
This is the deal breaker I think, for a commercial organization anyways.
While you can use PHP on Tomcat, the performance is apparently not good
enough for a large scale installation. See
http://wiki.apache.org/jakarta-tomcat/UsingPhp for how to do this
anyways.
Well, I don't think I want to run PHP through Tomcat as we already have
Apache running with PHP and it runs solid (and quite snappy too).
What I am currently working on is this for his .htaccess (well, testing
on a test instance):
( snip )
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/appdir/
RewriteRule ^(.*\.jsp)$ /appdir/$1
---( /snip )-
This results in all jsp pages being sent to tomcat. I also added a
jkMount /appdir/* workername to the httpd.conf file. This means, that
in the user's JSP pages, if he uses absolute paths for images, css, etc
then they will be handed off to tomcat as well so that a complete page
is sent back.
The problem currently is, the user used relative paths for all his
images, css files, etc. So while the JSP is being served correctly from
Tomcat with the above .htaccess lines, it is not passing the css and
image files off, which is the expected behavior. So I am trying to find
some way to remedy this via .htaccess so the user doesn't have to change
his 50+ jsp pages =|
Thanks again for taking the time to answer my questions and provide your
suggestions. It really is much appreciated!
-Sean
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat causes Apache to hang. Any ideas?
Hello, First a brief background on the setup: We are running Apache 1.3.31 utilizing mod_jk (not jk2). We are running two instances of Tomcat. Previously, both were version 4. Currently, we have one shared instance running the latest 5.x release (just compiled yesterday). We have one customer running a private 4.x instance. With that said, we just moved to a new server (Dual Xeon, RHEL 3.0) and now the problem we are having is the customer who has this private instance has jsp pages that hang. But, it does not hang all the time. What's weird, is most of the time, I can stop the shared instance and the customer's pages will stop hanging. I would like to fix the hanging completely, obviously. Today though, the only thing that kept his pages from hanging was a full reboot. I am wondering if this is related to swap space usage. I don't know what else it would be, as I stopped/started his tomcat and apache numerous times. I tried using a different connector (AJP instead of Coyote). Could this last problem be due to running tomcat5 *and* tomcat 4? I would say no, as that makes no sense this would be the case, especially when it (still)occurred after stopping the Tomcat 5 instance. Essentially what happens is, Apache receives the request for the page, hands it off to Tomcat, tomcat returns half of the page (header/left menu) and just sits there. Hitting stop on the browser stops the transfer, however Tomcat never releases the Apache process. So we end up with dozens, sometimes hundreds of Apache processes that are hung by tomcat. This results in memory usages exceeding 2gb! Has anyone experienced similar problems or have any suggestions? -Sean Finkel - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat causes Apache to hang. Any ideas?
What kernel was running on the old box? Better put, what is different between the two machines? 7.3 on the old system. 9.0 on an intermediary system, with only the private instance installed (which never hung). I recall RHEL 3.0 comes with an NPTL-based 2.4, which will require setting LD_KERNEL_ASSUME=2.4 to disable the NPTL functionality for the Java process. That may be it. I am assuming you mean set this as an environment variable? I will give this a try and see what happens! Thank you for the suggestion! -Sean - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
