Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Well, it seems to be running pretty solidly like that. However, my only PHP
app, WordPress, can't even find its database connection now.
Moreover, I can't, no matter how much I crank up the logging, get Quercus to
log anything about anything it does. If I modify a page, I get an info line
indicating that the size changed (implying it's recompiling the page).
I had some problems with Quercus and 4.0.5 before, but they were limited to
certain constructs not rendering correctly. Why it's at all different running
with an un-make-ed version of resin is beyond me.
--
Rick
On Mar 23, 2010, at 13:57:33, Aaron Freeman wrote:
Yes, we just untar it and run it with something like the following script:
/**/
export JAVA_HOME=/opt/java/
export RESIN_HOME=/opt/[whatever]/ext/resin-pro-4.0.5
export SERVER_ROOT=/opt/[whatever]/server
$JAVA_HOME/bin/java \
-server \
-Djava.util.logging.manager=com.caucho.log.LogManagerImpl \
-Djava.security.egd=/dev/urandom \
-Dresin.home=${RESIN_HOME} \
-jar ${RESIN_HOME}/lib/resin.jar \
-conf ${SERVER_ROOT}/conf/resin.xml \
$*
/**/
Aaron
On 3/23/2010 3:50 PM, Rick Mann wrote:
I have not. I think that's probably a good idea, so late tonight I'll try
installing it fresh.
What's the right way to invoke it? Just not run make/make install?
On Mar 23, 2010, at 13:46:43, Aaron Freeman wrote:
Rick,
Out of curiosity did you trying running a vanilla version of 4.0.5 without
compiling (using Java sockets instead of the native sockets)? I am curious
if you still have the issue with it uncompiled.
Aaron
On 3/23/2010 2:15 PM, Rick Mann wrote:
On Mar 23, 2010, at 12:10:35, Scott Ferguson wrote:
Rick Mann wrote:
Scott, I've opened a bug for my 4.0.5 woes, including the config file
and logs from my much more modern Open Solaris machine:
http://bugs.caucho.com/view.php?id=3960
Thanks for the detailed bug report. We're not seeing the behavior here
yet, but the details should help track down the differences.
Did the way I built and installed it have anything to do with the
flakiness?
No virus found in this incoming message.
Checked by AVG -
www.avg.com
Version: 9.0.791 / Virus Database: 271.1.1/2765 - Release Date: 03/23/10
02:33:00
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
No virus found in this incoming message.
Checked by AVG -
www.avg.com
Version: 9.0.791 / Virus Database: 271.1.1/2765 - Release Date: 03/23/10
02:33:00
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
I spoke a bit too soon. WordPress behaves exactly like it did before; title of
blog posts don't appear anywhere. I had forgotten to tell resin where
resin-root was, and it didn't load the DB driver.
So, I feel like I'm pretty close to having a working 4.0.5, if it stays up and
stable for a while, and if I can address the WordPress titles issue.
On Mar 26, 2010, at 23:25:31, Rick Mann wrote:
Well, it seems to be running pretty solidly like that. However, my only PHP
app, WordPress, can't even find its database connection now.
Moreover, I can't, no matter how much I crank up the logging, get Quercus to
log anything about anything it does. If I modify a page, I get an info line
indicating that the size changed (implying it's recompiling the page).
I had some problems with Quercus and 4.0.5 before, but they were limited to
certain constructs not rendering correctly. Why it's at all different running
with an un-make-ed version of resin is beyond me.
--
Rick
On Mar 23, 2010, at 13:57:33, Aaron Freeman wrote:
Yes, we just untar it and run it with something like the following script:
/**/
export JAVA_HOME=/opt/java/
export RESIN_HOME=/opt/[whatever]/ext/resin-pro-4.0.5
export SERVER_ROOT=/opt/[whatever]/server
$JAVA_HOME/bin/java \
-server \
-Djava.util.logging.manager=com.caucho.log.LogManagerImpl \
-Djava.security.egd=/dev/urandom \
-Dresin.home=${RESIN_HOME} \
-jar ${RESIN_HOME}/lib/resin.jar \
-conf ${SERVER_ROOT}/conf/resin.xml \
$*
/**/
Aaron
On 3/23/2010 3:50 PM, Rick Mann wrote:
I have not. I think that's probably a good idea, so late tonight I'll try
installing it fresh.
What's the right way to invoke it? Just not run make/make install?
On Mar 23, 2010, at 13:46:43, Aaron Freeman wrote:
Rick,
Out of curiosity did you trying running a vanilla version of 4.0.5 without
compiling (using Java sockets instead of the native sockets)? I am
curious if you still have the issue with it uncompiled.
Aaron
On 3/23/2010 2:15 PM, Rick Mann wrote:
On Mar 23, 2010, at 12:10:35, Scott Ferguson wrote:
Rick Mann wrote:
Scott, I've opened a bug for my 4.0.5 woes, including the config file
and logs from my much more modern Open Solaris machine:
http://bugs.caucho.com/view.php?id=3960
Thanks for the detailed bug report. We're not seeing the behavior here
yet, but the details should help track down the differences.
Did the way I built and installed it have anything to do with the
flakiness?
No virus found in this incoming message.
Checked by AVG -
www.avg.com
Version: 9.0.791 / Virus Database: 271.1.1/2765 - Release Date: 03/23/10
02:33:00
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
No virus found in this incoming message.
Checked by AVG -
www.avg.com
Version: 9.0.791 / Virus Database: 271.1.1/2765 - Release Date: 03/23/10
02:33:00
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Rick Mann wrote: Scott, I've opened a bug for my 4.0.5 woes, including the config file and logs from my much more modern Open Solaris machine: http://bugs.caucho.com/view.php?id=3960 Thanks for the detailed bug report. We're not seeing the behavior here yet, but the details should help track down the differences. -- Scott On Mar 19, 2010, at 13:13:34, Rick Mann wrote: On Mar 19, 2010, at 12:40:23, Scott Ferguson wrote: Rick Mann wrote: Some additional information: I tried installing 4.0.5 on my better server (Resin-4.0.0, Nexenta/Open Solaris, recent, lots of RAM, faster CPUs), and got similar behavior. I got it up and running, noticed some oddness at first (java link errors running my app, slow page loads the first few times), then it seemed to get normal. Then about 5 minutes later, it stopped responding. I didn't investigate, but rather just switched back to using resin-4.0.0. No idea if it was the same issue. In the logs, I saw it restarting, but it never came back. Thanks for the update. Just to review the symptoms: 1. Both JVMs are around and responds to kill -QUIT (there have been cases where the JVM has been so frozen it doesn't respond to anything.) 2. Resin is not responding to connect() for either of two ports. 3. The thread dump shows active threads in the nativeAccept() of both ports. 4. Netstat looks normal (i.e. it's not showing lots of connections in strange states.) My stress testing isn't showing anything like this kind of behavior yet, but any further details would help narrow the tests. Scott, I finally just moved my webapps to my Open Solaris machine, which is much more modern, and has a lot more RAM available. (It is currently running resin-4.0.0, warts and all.) This will allow me to update the older machine at my leisure. The problem I'm facing now is twofold, and the first of these might be related. 1) Attempts to upgrade to resin-4.0.5 have failed. Resin seems to run fine, but then stops after an hour or two. Because this machine is more critical (it hosts the data for my iPhone app), I just switch back to resin-4.0.0, which has problems but is stable. More details about how I installed it below. 2) On both 4.0.0 and 4.0.5, I have problems with Quercus and WordPress. In 4.0.0, I get a StackOverflowError. In 4.0.5, I get a PHP report of a database connection error (and I believe that comes with a 500 response). Installation Details The configuration for both is exactly the same, as much as I can remember and glean from my history. I put the individual resin released in /usr/local/resin/versions(/resin-4.0.0, resin-4.0.5, etc.), and symlink one of those to /usr/local/resin/current. When I installed 4.0.0, I believe I set that up first, then ran ./configure --prefix=/usr/local/resin/current, then make, sudo make install. For 4.0.5, I put it in place, but left the symlink pointing to 4.0.0, and did configure --prefix=/usr/local/resin/versions/resin-4.0.5, make, make install (not sudo). This complains that it couldn't install /etc/resin/ stuff, but I already had that from the previous build. Now, when I switch between versions, I change the symlink, and blow away all the WEB-INF/work directories (I'm posting a separate question about that), sometimes I stop and restart the server via the resin.sh script I have (which is just a wrapper around: java -jar /usr/local/resin/current/lib/resin.jar -verbose -root-directory /var/www -conf /etc/resin/resin.xml $* But usually it restarts itself (because of the symlink change, I presume). When I get some time, I'll try with resin-4.0.4, but I'm supposed to be doing real work for my employer today. -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
On Mar 23, 2010, at 12:10:35, Scott Ferguson wrote: Rick Mann wrote: Scott, I've opened a bug for my 4.0.5 woes, including the config file and logs from my much more modern Open Solaris machine: http://bugs.caucho.com/view.php?id=3960 Thanks for the detailed bug report. We're not seeing the behavior here yet, but the details should help track down the differences. Did the way I built and installed it have anything to do with the flakiness? -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Rick, Out of curiosity did you trying running a vanilla version of 4.0.5 without compiling (using Java sockets instead of the native sockets)? I am curious if you still have the issue with it uncompiled. Aaron On 3/23/2010 2:15 PM, Rick Mann wrote: On Mar 23, 2010, at 12:10:35, Scott Ferguson wrote: Rick Mann wrote: Scott, I've opened a bug for my 4.0.5 woes, including the config file and logs from my much more modern Open Solaris machine: http://bugs.caucho.com/view.php?id=3960 Thanks for the detailed bug report. We're not seeing the behavior here yet, but the details should help track down the differences. Did the way I built and installed it have anything to do with the flakiness? No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.791 / Virus Database: 271.1.1/2765 - Release Date: 03/23/10 02:33:00 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
I have not. I think that's probably a good idea, so late tonight I'll try installing it fresh. What's the right way to invoke it? Just not run make/make install? On Mar 23, 2010, at 13:46:43, Aaron Freeman wrote: Rick, Out of curiosity did you trying running a vanilla version of 4.0.5 without compiling (using Java sockets instead of the native sockets)? I am curious if you still have the issue with it uncompiled. Aaron On 3/23/2010 2:15 PM, Rick Mann wrote: On Mar 23, 2010, at 12:10:35, Scott Ferguson wrote: Rick Mann wrote: Scott, I've opened a bug for my 4.0.5 woes, including the config file and logs from my much more modern Open Solaris machine: http://bugs.caucho.com/view.php?id=3960 Thanks for the detailed bug report. We're not seeing the behavior here yet, but the details should help track down the differences. Did the way I built and installed it have anything to do with the flakiness? No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.791 / Virus Database: 271.1.1/2765 - Release Date: 03/23/10 02:33:00 ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Scott, I've opened a bug for my 4.0.5 woes, including the config file and logs from my much more modern Open Solaris machine: http://bugs.caucho.com/view.php?id=3960 On Mar 19, 2010, at 13:13:34, Rick Mann wrote: On Mar 19, 2010, at 12:40:23, Scott Ferguson wrote: Rick Mann wrote: Some additional information: I tried installing 4.0.5 on my better server (Resin-4.0.0, Nexenta/Open Solaris, recent, lots of RAM, faster CPUs), and got similar behavior. I got it up and running, noticed some oddness at first (java link errors running my app, slow page loads the first few times), then it seemed to get normal. Then about 5 minutes later, it stopped responding. I didn't investigate, but rather just switched back to using resin-4.0.0. No idea if it was the same issue. In the logs, I saw it restarting, but it never came back. Thanks for the update. Just to review the symptoms: 1. Both JVMs are around and responds to kill -QUIT (there have been cases where the JVM has been so frozen it doesn't respond to anything.) 2. Resin is not responding to connect() for either of two ports. 3. The thread dump shows active threads in the nativeAccept() of both ports. 4. Netstat looks normal (i.e. it's not showing lots of connections in strange states.) My stress testing isn't showing anything like this kind of behavior yet, but any further details would help narrow the tests. Scott, I finally just moved my webapps to my Open Solaris machine, which is much more modern, and has a lot more RAM available. (It is currently running resin-4.0.0, warts and all.) This will allow me to update the older machine at my leisure. The problem I'm facing now is twofold, and the first of these might be related. 1) Attempts to upgrade to resin-4.0.5 have failed. Resin seems to run fine, but then stops after an hour or two. Because this machine is more critical (it hosts the data for my iPhone app), I just switch back to resin-4.0.0, which has problems but is stable. More details about how I installed it below. 2) On both 4.0.0 and 4.0.5, I have problems with Quercus and WordPress. In 4.0.0, I get a StackOverflowError. In 4.0.5, I get a PHP report of a database connection error (and I believe that comes with a 500 response). Installation Details The configuration for both is exactly the same, as much as I can remember and glean from my history. I put the individual resin released in /usr/local/resin/versions(/resin-4.0.0, resin-4.0.5, etc.), and symlink one of those to /usr/local/resin/current. When I installed 4.0.0, I believe I set that up first, then ran ./configure --prefix=/usr/local/resin/current, then make, sudo make install. For 4.0.5, I put it in place, but left the symlink pointing to 4.0.0, and did configure --prefix=/usr/local/resin/versions/resin-4.0.5, make, make install (not sudo). This complains that it couldn't install /etc/resin/ stuff, but I already had that from the previous build. Now, when I switch between versions, I change the symlink, and blow away all the WEB-INF/work directories (I'm posting a separate question about that), sometimes I stop and restart the server via the resin.sh script I have (which is just a wrapper around: java -jar /usr/local/resin/current/lib/resin.jar -verbose -root-directory /var/www -conf /etc/resin/resin.xml $* But usually it restarts itself (because of the symlink change, I presume). When I get some time, I'll try with resin-4.0.4, but I'm supposed to be doing real work for my employer today. -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Rick Mann wrote: Some additional information: I tried installing 4.0.5 on my better server (Resin-4.0.0, Nexenta/Open Solaris, recent, lots of RAM, faster CPUs), and got similar behavior. I got it up and running, noticed some oddness at first (java link errors running my app, slow page loads the first few times), then it seemed to get normal. Then about 5 minutes later, it stopped responding. I didn't investigate, but rather just switched back to using resin-4.0.0. No idea if it was the same issue. In the logs, I saw it restarting, but it never came back. Thanks for the update. Just to review the symptoms: 1. Both JVMs are around and responds to kill -QUIT (there have been cases where the JVM has been so frozen it doesn't respond to anything.) 2. Resin is not responding to connect() for either of two ports. 3. The thread dump shows active threads in the nativeAccept() of both ports. 4. Netstat looks normal (i.e. it's not showing lots of connections in strange states.) My stress testing isn't showing anything like this kind of behavior yet, but any further details would help narrow the tests. -- Scott ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
On Mar 19, 2010, at 12:40:23, Scott Ferguson wrote: Rick Mann wrote: Some additional information: I tried installing 4.0.5 on my better server (Resin-4.0.0, Nexenta/Open Solaris, recent, lots of RAM, faster CPUs), and got similar behavior. I got it up and running, noticed some oddness at first (java link errors running my app, slow page loads the first few times), then it seemed to get normal. Then about 5 minutes later, it stopped responding. I didn't investigate, but rather just switched back to using resin-4.0.0. No idea if it was the same issue. In the logs, I saw it restarting, but it never came back. Thanks for the update. Just to review the symptoms: 1. Both JVMs are around and responds to kill -QUIT (there have been cases where the JVM has been so frozen it doesn't respond to anything.) 2. Resin is not responding to connect() for either of two ports. 3. The thread dump shows active threads in the nativeAccept() of both ports. 4. Netstat looks normal (i.e. it's not showing lots of connections in strange states.) My stress testing isn't showing anything like this kind of behavior yet, but any further details would help narrow the tests. Scott, I finally just moved my webapps to my Open Solaris machine, which is much more modern, and has a lot more RAM available. (It is currently running resin-4.0.0, warts and all.) This will allow me to update the older machine at my leisure. The problem I'm facing now is twofold, and the first of these might be related. 1) Attempts to upgrade to resin-4.0.5 have failed. Resin seems to run fine, but then stops after an hour or two. Because this machine is more critical (it hosts the data for my iPhone app), I just switch back to resin-4.0.0, which has problems but is stable. More details about how I installed it below. 2) On both 4.0.0 and 4.0.5, I have problems with Quercus and WordPress. In 4.0.0, I get a StackOverflowError. In 4.0.5, I get a PHP report of a database connection error (and I believe that comes with a 500 response). Installation Details The configuration for both is exactly the same, as much as I can remember and glean from my history. I put the individual resin released in /usr/local/resin/versions(/resin-4.0.0, resin-4.0.5, etc.), and symlink one of those to /usr/local/resin/current. When I installed 4.0.0, I believe I set that up first, then ran ./configure --prefix=/usr/local/resin/current, then make, sudo make install. For 4.0.5, I put it in place, but left the symlink pointing to 4.0.0, and did configure --prefix=/usr/local/resin/versions/resin-4.0.5, make, make install (not sudo). This complains that it couldn't install /etc/resin/ stuff, but I already had that from the previous build. Now, when I switch between versions, I change the symlink, and blow away all the WEB-INF/work directories (I'm posting a separate question about that), sometimes I stop and restart the server via the resin.sh script I have (which is just a wrapper around: java -jar /usr/local/resin/current/lib/resin.jar -verbose -root-directory /var/www -conf /etc/resin/resin.xml $* But usually it restarts itself (because of the symlink change, I presume). When I get some time, I'll try with resin-4.0.4, but I'm supposed to be doing real work for my employer today. -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
My 4.0.5 install is really very slow. It just won't stay running (although ps shows many resin threads), and it's much, much slower than 3.0.23 was. I'm actually running fewer webapps currently, and my server is always idle. Configuration is slightly different than it used to be, a necessity of updating the config file(s), but I don't think it's radically different from what I had. ps -axwwu | grep resin on redhat 7.2 (I know, it's old) gives me many lines like this: root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00 /usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf /var/resin/resin.xml -log-directory /var/logs/resin -root-directory /var/resin/root -J-verbosegc console root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00 /usr/local/java-versions/jdk1.6.0_07/bin/java -server -Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m -verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory /var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory /var/logs/resin -root-directory /var/resin/root console I can't find where the -Xss1m and -Xmx256m are being set. According to the admin manual, it should default to -Xss2m. The verbose GC output seems reasonable. Every few seconds, a line like this. I'm pretty sure my app(s) aren't leaking, since they've run for months before under 3.0.23: [GC 41277K-35464K(63664K), 0.0226660 secs] Sigh. On Mar 18, 2010, at 00:47:39, Rick Mann wrote: Hi. My new insatll of 4.0.5 is running very badly. My server is basically unloaded, except for the requests I make, but connections are constantly dropped. I was running 3.0.23 without any real problems before this. Also, I don't see anything in any of the logs when this happens. Any suggestions? Thanks! -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Rick Mann wrote: My 4.0.5 install is really very slow. It just won't stay running (although ps shows many resin threads), and it's much, much slower than 3.0.23 was. I'm actually running fewer webapps currently, and my server is always idle. How are you measuring the slower? (ab? latency? startup? some particular page size?) If the server is idle, how is it slow? Also, I'm not sure I understand you mean by not-running if there are Resin threads. Do the two JVMs exist? What do the logs show? Configuration is slightly different than it used to be, a necessity of updating the config file(s), but I don't think it's radically different from what I had. ps -axwwu | grep resin on redhat 7.2 (I know, it's old) gives me many lines like this: Hmm. If it's showing separate ps lines for each thread, that's very old. Some of those very old linux versions only partially implemented some of the system calls we use, like epoll(). (So the configure script would discover epoll(), but it didn't actually work as advertised.) root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00 /usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf /var/resin/resin.xml -log-directory /var/logs/resin -root-directory /var/resin/root -J-verbosegc console root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00 /usr/local/java-versions/jdk1.6.0_07/bin/java -server -Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m -verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory /var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory /var/logs/resin -root-directory /var/resin/root console I can't find where the -Xss1m and -Xmx256m are being set. According to the admin manual, it should default to -Xss2m. You can change that in the resin.xml with a jvm-arg-Xss1m/jvm-arg. -- Scott The verbose GC output seems reasonable. Every few seconds, a line like this. I'm pretty sure my app(s) aren't leaking, since they've run for months before under 3.0.23: [GC 41277K-35464K(63664K), 0.0226660 secs] Sigh. On Mar 18, 2010, at 00:47:39, Rick Mann wrote: Hi. My new insatll of 4.0.5 is running very badly. My server is basically unloaded, except for the requests I make, but connections are constantly dropped. I was running 3.0.23 without any real problems before this. Also, I don't see anything in any of the logs when this happens. Any suggestions? Thanks! -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Thanks for getting back to me, Scott! Responses below
On Mar 18, 2010, at 12:37:21, Scott Ferguson wrote:
Rick Mann wrote:
My 4.0.5 install is really very slow. It just won't stay running (although
ps shows many resin threads), and it's much, much slower than 3.0.23 was.
I'm actually running fewer webapps currently, and my server is always idle.
How are you measuring the slower? (ab? latency? startup? some
particular page size?) If the server is idle, how is it slow?
Slower is subjective. Unfortunately, I can't currently run the old server to
actually measure times, but it's just slow. I can see individual requests
(images, CSS, etc) taking a long time to be fulfilled, whereas before, the
pages would just come up fully rendered. Time to first render is longer, too.
Also, I'm not sure I understand you mean by not-running if there are
Resin threads. Do the two JVMs exist? What do the logs show?
I left the server running late last night, tried to make requests of it this
morning, but I just got server not responding errors in the browser. Mind
you, these were not timeouts, but actual connection refusals.
logs show nothing with this config:
log-handler name=
level=all
path=/var/logs/resin/out.log
rollover-period=1W
archive-format=out-%Y-%W.log.gz
timestamp=[%m-%d %H:%M:%S.%s] {%{thread}}
format= ${log.level} (${log.name}) ${log.message}/
logger name=com.caucho.servlets level=warning/
logger name= level=warning/
I'm not sure how to verify that the JVMs exist, other than the ps command
outputs many lines for java (as described before). It's not that the server
died, it just stopped listening.
Configuration is slightly different than it used to be, a necessity of
updating the config file(s), but I don't think it's radically different from
what I had.
ps -axwwu | grep resin on redhat 7.2 (I know, it's old) gives me many lines
like this:
Hmm. If it's showing separate ps lines for each thread, that's very old.
Some of those very old linux versions only partially implemented some of
the system calls we use, like epoll(). (So the configure script would
discover epoll(), but it didn't actually work as advertised.)
So, that's an interesting and scary thought. However, I'm not using Resin
Professional, so I thought I didn't get native sockes.
root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00
/usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf
/var/resin/resin.xml -log-directory /var/logs/resin -root-directory
/var/resin/root -J-verbosegc console
root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00
/usr/local/java-versions/jdk1.6.0_07/bin/java -server
-Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1
-Djava.util.logging.manager=com.caucho.log.LogManagerImpl
-Djava.system.class.loader=com.caucho.loader.SystemClassLoader
-Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl
-Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m
-verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory
/var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory
/var/logs/resin -root-directory /var/resin/root console
I can't find where the -Xss1m and -Xmx256m are being set. According to the
admin manual, it should default to -Xss2m.
You can change that in the resin.xml with a jvm-arg-Xss1m/jvm-arg.
I used to have exactly those values specified in resin.xml, and thought perhaps
that was part of the problem, so I commented them out (to let the JVM and resin
pick defaults). I was still seeing those in the command, though, and didn't
know where they were coming from. That was something I had not been specifying
in my 3.0.23 installation, and so I thought they were causing more problems
than they were solving.
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Rick Mann wrote: Slower is subjective. Unfortunately, I can't currently run the old server to actually measure times, but it's just slow. I can see individual requests (images, CSS, etc) taking a long time to be fulfilled, whereas before, the pages would just come up fully rendered. Time to first render is longer, too. finer logging might show the issue, but there's a good chance it's related to the connection issue. I left the server running late last night, tried to make requests of it this morning, but I just got server not responding errors in the browser. Mind you, these were not timeouts, but actual connection refusals. Thanks. Refusing connections is a more helpful description. If possible, can you get a thread dump in that situation? There are a couple of possible scenarios: 1. After a while, Resin no longer has a thread listening for an accept(). (for example, if it doesn't properly spawn a new thread.) 2. The JVM is truly frozen. It is possible, by the way, that the older Linux is having trouble with the threading. They made huge improvements in the kernel's threading after the version you're using. logs show nothing with this config: logger name=com.caucho.servlets level=warning/ logger name= level=warning/ Ok, although, the warning level would only show pretty serious issues. So, that's an interesting and scary thought. However, I'm not using Resin Professional, so I thought I didn't get native sockes. Ok. That's one less variable. The main issue is the connection failures. That could cause your slowness as well. For example, if only one thread is listening for connections for some reason, the server would appear to be very slow. -- Scott root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00 /usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf /var/resin/resin.xml -log-directory /var/logs/resin -root-directory /var/resin/root -J-verbosegc console root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00 /usr/local/java-versions/jdk1.6.0_07/bin/java -server -Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m -verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory /var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory /var/logs/resin -root-directory /var/resin/root console I can't find where the -Xss1m and -Xmx256m are being set. According to the admin manual, it should default to -Xss2m. You can change that in the resin.xml with a jvm-arg-Xss1m/jvm-arg. I used to have exactly those values specified in resin.xml, and thought perhaps that was part of the problem, so I commented them out (to let the JVM and resin pick defaults). I was still seeing those in the command, though, and didn't know where they were coming from. That was something I had not been specifying in my 3.0.23 installation, and so I thought they were causing more problems than they were solving. ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
I replied with the thread dump, but don't see it appearing in the list. Seems like the list server is flaky. On Mar 18, 2010, at 14:37:25, Scott Ferguson wrote: Rick Mann wrote: Slower is subjective. Unfortunately, I can't currently run the old server to actually measure times, but it's just slow. I can see individual requests (images, CSS, etc) taking a long time to be fulfilled, whereas before, the pages would just come up fully rendered. Time to first render is longer, too. finer logging might show the issue, but there's a good chance it's related to the connection issue. I left the server running late last night, tried to make requests of it this morning, but I just got server not responding errors in the browser. Mind you, these were not timeouts, but actual connection refusals. Thanks. Refusing connections is a more helpful description. If possible, can you get a thread dump in that situation? There are a couple of possible scenarios: 1. After a while, Resin no longer has a thread listening for an accept(). (for example, if it doesn't properly spawn a new thread.) 2. The JVM is truly frozen. It is possible, by the way, that the older Linux is having trouble with the threading. They made huge improvements in the kernel's threading after the version you're using. logs show nothing with this config: logger name=com.caucho.servlets level=warning/ logger name= level=warning/ Ok, although, the warning level would only show pretty serious issues. So, that's an interesting and scary thought. However, I'm not using Resin Professional, so I thought I didn't get native sockes. Ok. That's one less variable. The main issue is the connection failures. That could cause your slowness as well. For example, if only one thread is listening for connections for some reason, the server would appear to be very slow. -- Scott root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00 /usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf /var/resin/resin.xml -log-directory /var/logs/resin -root-directory /var/resin/root -J-verbosegc console root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00 /usr/local/java-versions/jdk1.6.0_07/bin/java -server -Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m -verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory /var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory /var/logs/resin -root-directory /var/resin/root console I can't find where the -Xss1m and -Xmx256m are being set. According to the admin manual, it should default to -Xss2m. You can change that in the resin.xml with a jvm-arg-Xss1m/jvm-arg. I used to have exactly those values specified in resin.xml, and thought perhaps that was part of the problem, so I commented them out (to let the JVM and resin pick defaults). I was still seeing those in the command, though, and didn't know where they were coming from. That was something I had not been specifying in my 3.0.23 installation, and so I thought they were causing more problems than they were solving. ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Hmm, I can't seem to post longer email to the list. Here's a link to my last post: http://pastie.org/private/2c4lrjuj4oybdtzmj4gbzw On Mar 18, 2010, at 15:33:08, Rick Mann wrote: I replied with the thread dump, but don't see it appearing in the list. Seems like the list server is flaky. On Mar 18, 2010, at 14:37:25, Scott Ferguson wrote: Rick Mann wrote: Slower is subjective. Unfortunately, I can't currently run the old server to actually measure times, but it's just slow. I can see individual requests (images, CSS, etc) taking a long time to be fulfilled, whereas before, the pages would just come up fully rendered. Time to first render is longer, too. finer logging might show the issue, but there's a good chance it's related to the connection issue. I left the server running late last night, tried to make requests of it this morning, but I just got server not responding errors in the browser. Mind you, these were not timeouts, but actual connection refusals. Thanks. Refusing connections is a more helpful description. If possible, can you get a thread dump in that situation? There are a couple of possible scenarios: 1. After a while, Resin no longer has a thread listening for an accept(). (for example, if it doesn't properly spawn a new thread.) 2. The JVM is truly frozen. It is possible, by the way, that the older Linux is having trouble with the threading. They made huge improvements in the kernel's threading after the version you're using. logs show nothing with this config: logger name=com.caucho.servlets level=warning/ logger name= level=warning/ Ok, although, the warning level would only show pretty serious issues. So, that's an interesting and scary thought. However, I'm not using Resin Professional, so I thought I didn't get native sockes. Ok. That's one less variable. The main issue is the connection failures. That could cause your slowness as well. For example, if only one thread is listening for connections for some reason, the server would appear to be very slow. -- Scott root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00 /usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf /var/resin/resin.xml -log-directory /var/logs/resin -root-directory /var/resin/root -J-verbosegc console root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00 /usr/local/java-versions/jdk1.6.0_07/bin/java -server -Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m -verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory /var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory /var/logs/resin -root-directory /var/resin/root console I can't find where the -Xss1m and -Xmx256m are being set. According to the admin manual, it should default to -Xss2m. You can change that in the resin.xml with a jvm-arg-Xss1m/jvm-arg. I used to have exactly those values specified in resin.xml, and thought perhaps that was part of the problem, so I commented them out (to let the JVM and resin pick defaults). I was still seeing those in the command, though, and didn't know where they were coming from. That was something I had not been specifying in my 3.0.23 installation, and so I thought they were causing more problems than they were solving. ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Rick Mann wrote: Hmm, I can't seem to post longer email to the list. Here's a link to my last post: http://pastie.org/private/2c4lrjuj4oybdtzmj4gbzw Thanks. There is a size limit for the list. That trace looks normal, assuming the *.216 and *.217 are your two HTTP ports. I assume that's the connection-refused state? Each one has several threads in the nativeAccept() state. The situation that would be a problem is if there are no threads for the port waiting in accept(). When you're in a connection-refused, can you try both addresses, just to make sure? (It's possible, too, that netstat might show something in that state.) The JNI is used in Resin-OS for ports/sockets because we needed the JNI capability to handle the setuid (and we decided setuid really should be in Resin-OS.) And it may be better to set the thread-idle-max for your older OS to something smaller, like 10 or 15, because that OS may not be as good with lots of threads. -- Sctt On Mar 18, 2010, at 15:33:08, Rick Mann wrote: I replied with the thread dump, but don't see it appearing in the list. Seems like the list server is flaky. On Mar 18, 2010, at 14:37:25, Scott Ferguson wrote: Rick Mann wrote: Slower is subjective. Unfortunately, I can't currently run the old server to actually measure times, but it's just slow. I can see individual requests (images, CSS, etc) taking a long time to be fulfilled, whereas before, the pages would just come up fully rendered. Time to first render is longer, too. finer logging might show the issue, but there's a good chance it's related to the connection issue. I left the server running late last night, tried to make requests of it this morning, but I just got server not responding errors in the browser. Mind you, these were not timeouts, but actual connection refusals. Thanks. Refusing connections is a more helpful description. If possible, can you get a thread dump in that situation? There are a couple of possible scenarios: 1. After a while, Resin no longer has a thread listening for an accept(). (for example, if it doesn't properly spawn a new thread.) 2. The JVM is truly frozen. It is possible, by the way, that the older Linux is having trouble with the threading. They made huge improvements in the kernel's threading after the version you're using. logs show nothing with this config: logger name=com.caucho.servlets level=warning/ logger name= level=warning/ Ok, although, the warning level would only show pretty serious issues. So, that's an interesting and scary thought. However, I'm not using Resin Professional, so I thought I didn't get native sockes. Ok. That's one less variable. The main issue is the connection failures. That could cause your slowness as well. For example, if only one thread is listening for connections for some reason, the server would appear to be very slow. -- Scott root 20511 0.0 5.1 226964 26396 pts/2 S10:54 0:00 /usr/java/bin/java -jar /usr/local/resin/lib/resin.jar -conf /var/resin/resin.xml -log-directory /var/logs/resin -root-directory /var/resin/root -J-verbosegc console root 20517 0.0 6.8 450916 35108 pts/2 S10:54 0:00 /usr/local/java-versions/jdk1.6.0_07/bin/java -server -Djava.awt.headless=true -Dfile.encoding=utf-8 -Dresin.server=1 -Djava.util.logging.manager=com.caucho.log.LogManagerImpl -Djava.system.class.loader=com.caucho.loader.SystemClassLoader -Djavax.management.builder.initial=com.caucho.jmx.MBeanServerBuilderImpl -Djava.awt.headless=true -Dresin.home=/usr/local/resin/ -Xss1m -Xmx256m -verbosegc -verbosegc com.caucho.server.resin.Resin --root-directory /var/resin/root -conf /var/resin/resin.xml -socketwait 34280 -log-directory /var/logs/resin -root-directory /var/resin/root console I can't find where the -Xss1m and -Xmx256m are being set. According to the admin manual, it should default to -Xss2m. You can change that in the resin.xml with a jvm-arg-Xss1m/jvm-arg. I used to have exactly those values specified in resin.xml, and thought perhaps that was part of the problem, so I commented them out (to let the JVM and resin pick defaults). I was still seeing those in the command, though, and didn't know where they were coming from. That was something I had not been specifying in my 3.0.23 installation, and so I thought they were causing more problems than they were solving. ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
On Mar 18, 2010, at 16:30:02, Scott Ferguson wrote: That trace looks normal, assuming the *.216 and *.217 are your two HTTP ports. I assume that's the connection-refused state? Each one has several threads in the nativeAccept() state. The situation that would be a problem is if there are no threads for the port waiting in accept(). When you're in a connection-refused, can you try both addresses, just to make sure? (It's possible, too, that netstat might show something in that state.) I got to that connection-refused state again, and tried connecting to each IP address separately. They both behaved the same way, although apparently the server isn't completely dead. Every now and again, it will respond. But attempting to load a web page, it's obvious that many of the connections are being refused (no CSS, images, etc). netstat -an output in that state (sorry, I might've forgotten an option on that command): http://pastie.org/private/bttcmdyixr023xasvvsdzq -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
On Mar 18, 2010, at 16:30:02, Scott Ferguson wrote: And it may be better to set the thread-idle-max for your older OS to something smaller, like 10 or 15, because that OS may not be as good with lots of threads. thread-idle-max10/ seems to have no effect on the overall stability. -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] New 4.0.5 install constantly dropping connections on unloaded server
Some additional information: I tried installing 4.0.5 on my better server (Resin-4.0.0, Nexenta/Open Solaris, recent, lots of RAM, faster CPUs), and got similar behavior. I got it up and running, noticed some oddness at first (java link errors running my app, slow page loads the first few times), then it seemed to get normal. Then about 5 minutes later, it stopped responding. I didn't investigate, but rather just switched back to using resin-4.0.0. No idea if it was the same issue. In the logs, I saw it restarting, but it never came back. -- Rick ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
