[Resin-interest] Problem with Resin 3.1.6 Pro and keepalive
Hi
I have a test-server that runs a mix of Resin 2.1.17 and 3.0.23 Pro (havn't had
a reason to upgrade yet) using 2 different Apache 2.2 instances as frontends
for several vhosts on RHEL3u9 (2.4.21-50) with JDK1.6.0_03.
This setup is just fine, and performs as expected for all my needs.Howver I'm
in the process of rolling out Resin 3.1.6 Pro on a new server (since the 3.1
branch it's now stable I might as well upgrade).
I've set everything up - added a new Apache instance for the 3.1.6 version of
mod_caucho.so etc. - and started up a site. In resin.conf (see below) I've used
the stdout-log-tag to output to logs/resin-stdout.log, and the
stderr-log-tag to logs/resin-stderr.log.
Upon startup of a site this is written to resin-stdout.log:
[2008-06-17 16:28:01.467][16:28:01.467] Linux 2.4.21-50.EL i386
[2008-06-17 16:28:01.467][16:28:01.467] Java(TM) SE Runtime Environment
1.6.0_03-b05, ISO-8859-1, en
[2008-06-17 16:28:01.468][16:28:01.467] Java HotSpot(TM) Server VM
1.6.0_03-b05, 32, mixed mode, Sun Microsystems Inc.
[2008-06-17 16:28:01.468][16:28:01.468] user.name: root
[2008-06-17 16:28:01.468][16:28:01.468] resin.home =
/usr/local/resin/resin-pro-3.1.6/
[2008-06-17 16:28:01.469][16:28:01.469] resin.root = /usr/local/www/results_test
[2008-06-17 16:28:01.469][16:28:01.469] resin.conf =
/usr/local/www/results_test/conf/resin.conf
[2008-06-17 16:28:01.469][16:28:01.469]
[2008-06-17 16:28:16.800][16:28:16.800] Loaded Socket JNI library.
[2008-06-17 16:28:16.804][16:28:16.804] hmux listening to
localhost.localdomain:6875
[2008-06-17 16:28:16.816][16:28:16.816] Server[id=results_test,cluster=web]
active
[2008-06-17 16:28:16.818][16:28:16.818] Resin started in 17788ms
So far so good, however this also appears in resin-stderr.log at the same time:
[2008-06-17 16:28:01.507]Exception in thread resin-select-manager
java.io.IOException: failed to add EPOLL for pipe=47 (errno=-1)
[2008-06-17 16:28:01.508] at
com.caucho.server.port.JniSelectManager.initNative(Native Method)
[2008-06-17 16:28:01.508] at
com.caucho.server.port.JniSelectManager.run(JniSelectManager.java:274)
[2008-06-17 16:28:01.508] at java.lang.Thread.run(Thread.java:619)
Then upon accessing the site this appears in logs/resin-stdout.log:
[2008-06-17 16:28:18.377][16:28:18.377] Tcp[results_test,0] failed keepalive
(select)
[2008-06-17 16:28:32.685][16:28:32.684] Tcp[results_test,1] failed keepalive
(select)
[2008-06-17 16:28:32.705][16:28:32.705] Tcp[results_test,2] failed keepalive
(select)
[2008-06-17 16:28:32.725][16:28:32.725] Tcp[results_test,3] failed keepalive
(select)
[2008-06-17 16:28:32.754][16:28:32.754] Tcp[results_test,4] failed keepalive
(select)
[2008-06-17 16:28:32.785][16:28:32.784] Tcp[results_test,5] failed keepalive
(select)
[2008-06-17 16:28:32.825][16:28:32.825] Tcp[results_test,6] failed keepalive
(select)
.. and the site changes between a 503-error page (as if the site isn't started)
and a working site (using refresh in the browser a few times), so it's
obviously not working as it should.
This is my resin.conf:
resin xmlns=http://caucho.com/ns/resin;
xmlns:resin=http://caucho.com/ns/resin/core;
management
user name=xxx password=/
/management
stdout-log path=/usr/local/www/results_test/logs/resin-stdout.log
timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
stderr-log path=/usr/local/www/results_test/logs/resin-stderr.log
timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
log name= level=config path=stdout: timestamp=[%H:%M:%S.%s] /
logger name=com.caucho.java level=config/
logger name=com.caucho.loader level=config/
thread-pool
thread-max150/thread-max
spare-thread-min5/spare-thread-min
/thread-pool
cluster id=web
server-default
jvm-arg-Xmn32m/jvm-arg
jvm-arg-Xms64m/jvm-arg
jvm-arg-Xmx512m/jvm-arg
jvm-arg-server/jvm-arg
jvm-arg-XX:-OmitStackTraceInFastThrow/jvm-arg
/server-default
server id=results_test address=127.0.0.1 port=6875
watchdog-port7875/watchdog-port
keepalive-max100/keepalive-max
keepalive-timeout30s/keepalive-timeout
bind-ports-after-starttrue/bind-ports-after-start
/server
ignore-client-disconnecttrue/ignore-client-disconnect
host id= root-directory=/usr/local/www/results_test
class-loader
compiling-loader path=site/WEB-INF/classes/
library-loader path=site/WEB-INF/lib/
/class-loader
access-log path=/usr/local/www/results_test/logs/resin-access.log
format='%h %l %u %t %r %s %b %{Referer}i %{User-Agent}i'
rollover-period=1D/
stdout-log path=/usr/local/www/results_test/logs/resin-web-stdout.log
timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
stderr-log path=/usr/local/www/results_test/logs/resin-web-stderr.log
timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
web-app id=/ document-directory=site
resin:import path=WEB-INF/web.xml/
Re: [Resin-interest] Problem with Resin 3.1.6 Pro and keepalive
On Jun 17, 2008, at 9:47 AM, Jens Dueholm Christensen wrote:
So far so good, however this also appears in resin-stderr.log at the
same time:
[2008-06-17 16:28:01.507]Exception in thread resin-select-manager
java.io.IOException: failed to add EPOLL for pipe=47 (errno=-1)
[2008-06-17 16:28:01.508] at
com.caucho.server.port.JniSelectManager.initNative(Native Method)
[2008-06-17 16:28:01.508] at
com.caucho.server.port.JniSelectManager.run(JniSelectManager.java:274)
[2008-06-17 16:28:01.508] at java.lang.Thread.run(Thread.java:
619)
Can you try setting keepalive-select-enable=false in the server:
resin ...
...
cluster id=
...
server id=
http port=80/
keepalive-select-enablefalse/keepalive-select-enable
...
I believe your older Linux version doesn't correctly support EPOLL,
which the select manager uses to handle keepalives.
-- Scott
Then upon accessing the site this appears in logs/resin-stdout.log:
[2008-06-17 16:28:18.377][16:28:18.377] Tcp[results_test,0] failed
keepalive (select)
[2008-06-17 16:28:32.685][16:28:32.684] Tcp[results_test,1] failed
keepalive (select)
[2008-06-17 16:28:32.705][16:28:32.705] Tcp[results_test,2] failed
keepalive (select)
[2008-06-17 16:28:32.725][16:28:32.725] Tcp[results_test,3] failed
keepalive (select)
[2008-06-17 16:28:32.754][16:28:32.754] Tcp[results_test,4] failed
keepalive (select)
[2008-06-17 16:28:32.785][16:28:32.784] Tcp[results_test,5] failed
keepalive (select)
[2008-06-17 16:28:32.825][16:28:32.825] Tcp[results_test,6] failed
keepalive (select)
.. and the site changes between a 503-error page (as if the site
isn't started) and a working site (using refresh in the browser a
few times), so it's obviously not working as it should.
This is my resin.conf:
resin xmlns=http://caucho.com/ns/resin;
xmlns:resin=http://caucho.com/ns/resin/core
management
user name=xxx password=/
/management
stdout-log path=/usr/local/www/results_test/logs/resin-
stdout.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
stderr-log path=/usr/local/www/results_test/logs/resin-
stderr.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
log name= level=config path=stdout: timestamp=[%H:%M:%S.%s]
/
logger name=com.caucho.java level=config/
logger name=com.caucho.loader level=config/
thread-pool
thread-max150/thread-max
spare-thread-min5/spare-thread-min
/thread-pool
cluster id=web
server-default
jvm-arg-Xmn32m/jvm-arg
jvm-arg-Xms64m/jvm-arg
jvm-arg-Xmx512m/jvm-arg
jvm-arg-server/jvm-arg
jvm-arg-XX:-OmitStackTraceInFastThrow/jvm-arg
/server-default
server id=results_test address=127.0.0.1 port=6875
watchdog-port7875/watchdog-port
keepalive-max100/keepalive-max
keepalive-timeout30s/keepalive-timeout
bind-ports-after-starttrue/bind-ports-after-start
/server
ignore-client-disconnecttrue/ignore-client-disconnect
host id= root-directory=/usr/local/www/results_test
class-loader
compiling-loader path=site/WEB-INF/classes/
library-loader path=site/WEB-INF/lib/
/class-loader
access-log path=/usr/local/www/results_test/logs/resin-
access.log format='%h %l %u %t %r %s %b %{Referer}i %{User-
Agent}i' rollover-period=1D/
stdout-log path=/usr/local/www/results_test/logs/resin-web-
stdout.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
stderr-log path=/usr/local/www/results_test/logs/resin-web-
stderr.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
web-app id=/ document-directory=site
resin:import path=WEB-INF/web.xml/
session-config
session-max16384/session-max
enable-cookiestrue/enable-cookies
enable-url-rewritingtrue/enable-url-rewriting
/session-config
servlet servlet-name=resin-file servlet-
class=com.caucho.servlets.FileServlet/
servlet servlet-name=resin-jsp servlet-
class=com.caucho.jsp.JspServlet
init
load-tld-on-initfalse/load-tld-on-init
page-cache-max1024/page-cache-max
/init
load-on-startup/
/servlet
!-- invoker servlet for all standard WebPages and
WebCommands--
servlet-mapping url-pattern=/servlet/* servlet-
name=invoker/
servlet-mapping url-pattern=*.jsp servlet-name=resin-jsp/
... context params removed...
/web-app
/host
/cluster
/resin
I've looked at http://bugs.caucho.com/view.php?id=2555
http://bugs.caucho.com/view.php?id=2555
(first and almost onle relevant hit when googling for resin
failed keepalive and variations) and tried to add the keepalive-
select-max-tag, but to no avail (and I can't seem to find any
documentation on that tag?).
I think it's somehow related to the Exception in resin-select-
manager at startup, but I'm not sure, and
Re: [Resin-interest] Problem with Resin 3.1.6 Pro and keepalive
Hi Scott
That seems to have done the trick!
A bit more digging told me, that epoll(4) was introduced around 2.5.44, so that
was spot on - thanks for the hint!
Apart from the increased usage of sockets (or so I understand it from
http://www.caucho.com/resin/doc/server-tags.xtp#keepalive-select-enable) there
shouldn't be any other sideeffects?
Oh well - seems like I should start working on upgrading my testserver.. :)
Regards,
Jens Dueholm Christensen
Rambøll survey IT
From: [EMAIL PROTECTED] on behalf of Scott Ferguson
Sent: Tue 6/17/2008 19:07
To: General Discussion for the Resin application server
Subject: Re: [Resin-interest] Problem with Resin 3.1.6 Pro and keepalive
On Jun 17, 2008, at 9:47 AM, Jens Dueholm Christensen wrote:
So far so good, however this also appears in resin-stderr.log at the
same time:
[2008-06-17 16:28:01.507]Exception in thread resin-select-manager
java.io.IOException: failed to add EPOLL for pipe=47 (errno=-1)
[2008-06-17 16:28:01.508] at
com.caucho.server.port.JniSelectManager.initNative(Native Method)
[2008-06-17 16:28:01.508] at
com.caucho.server.port.JniSelectManager.run(JniSelectManager.java:274)
[2008-06-17 16:28:01.508] at java.lang.Thread.run(Thread.java:
619)
Can you try setting keepalive-select-enable=false in the server:
resin ...
...
cluster id=
...
server id=
http port=80/
keepalive-select-enablefalse/keepalive-select-enable
...
I believe your older Linux version doesn't correctly support EPOLL,
which the select manager uses to handle keepalives.
-- Scott
Then upon accessing the site this appears in logs/resin-stdout.log:
[2008-06-17 16:28:18.377][16:28:18.377] Tcp[results_test,0] failed
keepalive (select)
[2008-06-17 16:28:32.685][16:28:32.684] Tcp[results_test,1] failed
keepalive (select)
[2008-06-17 16:28:32.705][16:28:32.705] Tcp[results_test,2] failed
keepalive (select)
[2008-06-17 16:28:32.725][16:28:32.725] Tcp[results_test,3] failed
keepalive (select)
[2008-06-17 16:28:32.754][16:28:32.754] Tcp[results_test,4] failed
keepalive (select)
[2008-06-17 16:28:32.785][16:28:32.784] Tcp[results_test,5] failed
keepalive (select)
[2008-06-17 16:28:32.825][16:28:32.825] Tcp[results_test,6] failed
keepalive (select)
.. and the site changes between a 503-error page (as if the site
isn't started) and a working site (using refresh in the browser a
few times), so it's obviously not working as it should.
This is my resin.conf:
resin xmlns=http://caucho.com/ns/resin;
xmlns:resin=http://caucho.com/ns/resin/core
management
user name=xxx password=/
/management
stdout-log path=/usr/local/www/results_test/logs/resin-
stdout.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
stderr-log path=/usr/local/www/results_test/logs/resin-
stderr.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
log name= level=config path=stdout: timestamp=[%H:%M:%S.%s]
/
logger name=com.caucho.java level=config/
logger name=com.caucho.loader level=config/
thread-pool
thread-max150/thread-max
spare-thread-min5/spare-thread-min
/thread-pool
cluster id=web
server-default
jvm-arg-Xmn32m/jvm-arg
jvm-arg-Xms64m/jvm-arg
jvm-arg-Xmx512m/jvm-arg
jvm-arg-server/jvm-arg
jvm-arg-XX:-OmitStackTraceInFastThrow/jvm-arg
/server-default
server id=results_test address=127.0.0.1 port=6875
watchdog-port7875/watchdog-port
keepalive-max100/keepalive-max
keepalive-timeout30s/keepalive-timeout
bind-ports-after-starttrue/bind-ports-after-start
/server
ignore-client-disconnecttrue/ignore-client-disconnect
host id= root-directory=/usr/local/www/results_test
class-loader
compiling-loader path=site/WEB-INF/classes/
library-loader path=site/WEB-INF/lib/
/class-loader
access-log path=/usr/local/www/results_test/logs/resin-
access.log format='%h %l %u %t %r %s %b %{Referer}i %{User-
Agent}i' rollover-period=1D/
stdout-log path=/usr/local/www/results_test/logs/resin-web-
stdout.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
stderr-log path=/usr/local/www/results_test/logs/resin-web-
stderr.log timestamp=[%Y-%m-%d %H:%M:%S.%s] rollover-period=1D/
web-app id=/ document-directory=site
resin:import path=WEB-INF/web.xml/
session-config
session-max16384/session-max
enable-cookiestrue/enable-cookies
enable-url-rewritingtrue/enable-url-rewriting
/session-config
servlet servlet-name=resin-file servlet-
class=com.caucho.servlets.FileServlet/
servlet servlet-name=resin-jsp servlet-
class=com.caucho.jsp.JspServlet
init
load-tld-on-initfalse/load-tld-on-init
page-cache-max1024/page-cache-max
/init
load-on-startup/
/servlet
!-- invoker
