On Jan 12, 2009, at 1:19 PM, eros wrote:
Hi everyone,
I'm having the same problem using Ubuntu 8.04 server and Geronimo
Tomcat6
2.1.3, with Java version 1.6.0_07.
I can reproduce the problem quite easily (in fact, I can't get rid of
it...): I installed geronimo on 3 different computers:
- production server running Ubuntu 8.04 64bit (kernel 2.6.24-23)
- development server running Ubuntu 8.04 32bit (kernel 2.6.24-23)
- virtualbox (2.1.0) machine running Ubuntu 8.04 JeOS 32 bit (kernel
2.6.24-23)
The problem manifests itself on all these computers: whenever I try
to start
geronimo it gets stuck loading module 25/68, or at least that's what
/usr/share/geronimo-tomcat6/var/log/geronimo.out says:
Module 25/68 org.apache.geronimo.configs/tomcat6/2.1.3/car
the last line of the other log file
(/usr/share/geronimo-tomcat6/var/log/geronimo.log) while waiting for
Module
25/68 to load is:
[SystemProperties] Setting
Property=org.apache.catalina.STRICT_SERVLET_COMPLIANCE to Value=true
Any suggestiong as to what I should try to fix this would be very much
appreciated, and if yoy need more info just let me know.
I think you have a problem with system entropy that's causing a
problem generating pseudo-random numbers.
I dusted off an Ubuntu 8.0.4 server which I haven't been using lately
and was able to recreate your problem (or at least something that very
closely approximates your problem -- a 90 second delay while starting
the tomcat6 module).
While the server startup was stalled, I sent a kill -3 signal to the
server process (think I mentioned doing this earlier in this thread)
and got the following thread stack:
"main" prio=1 tid=0x0000000040117140 nid=0x14b0 runnable
[0x00007fff12f12000..0x00007fff12f150e0]
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:194)
at sun.security.provider.NativePRNG
$RandomIO.readFully(NativePRNG.java:185)
at sun.security.provider.NativePRNG
$RandomIO.implGenerateSeed(NativePRNG.java:202)
- locked <0x00007f51c4f48810> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access
$300(NativePRNG.java:108)
at
sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:102)
at java.security.SecureRandom.generateSeed(SecureRandom.java:
475)
at
com
.sun.net.ssl.internal.ssl.PKCS12KeyStore.getSalt(PKCS12KeyStore.java:
475)
at
com
.sun
.net.ssl.internal.ssl.PKCS12KeyStore.calculateMac(PKCS12KeyStore.java:
832)
at
com
.sun
.net.ssl.internal.ssl.PKCS12KeyStore.engineStore(PKCS12KeyStore.java:
786)
- locked <0x00007f51f6385438> (a
com.sun.net.ssl.internal.ssl.PKCS12KeyStore)
at java.security.KeyStore.store(KeyStore.java:1085)
at
org.apache.geronimo.crypto.KeystoreUtil.<clinit>(KeystoreUtil.java:101)
at
org
.apache
.geronimo
.tomcat
.TomcatManagerImpl.addSslConnectorAttributes(TomcatManagerImpl.java:450)
at
org
.apache
.geronimo.tomcat.TomcatManagerImpl.<clinit>(TomcatManagerImpl.java:101)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun
.reflect
.NativeConstructorAccessorImpl
.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun
.reflect
.DelegatingConstructorAccessorImpl
.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:
494)
at
org
.apache
.geronimo
.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:948)
at
org
.apache
.geronimo
.gbean
.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:
268)
at
org
.apache
.geronimo
.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
at
org
.apache
.geronimo
.gbean
.runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:124)
at
org
.apache
.geronimo
.gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:555)
at
org
.apache
.geronimo
.kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:379)
at
org
.apache
.geronimo
.kernel
.config
.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:456)
at
org
.apache
.geronimo
.kernel
.config
.KernelConfigurationManager.start(KernelConfigurationManager.java:188)
at
org
.apache
.geronimo
.kernel
.config
.SimpleConfigurationManager
.startConfiguration(SimpleConfigurationManager.java:562)
- locked <0x00007f51c5183ee8> (a
org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun
.reflect
.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org
.apache
.geronimo
.gbean
.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
at
org
.apache
.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
at
org
.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:
832)
at
org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
at
org
.apache
.geronimo
.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
at
org
.apache
.geronimo
.kernel
.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
at
org.apache.geronimo.kernel.config.EditableConfigurationManager$
$EnhancerByCGLIB$$a351628e.startConfiguration(<generated>)
at
org
.apache
.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:162)
at
org
.apache
.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:79)
at
org
.apache
.geronimo
.kernel
.util
.MainConfigurationBootstrapper.main(MainConfigurationBootstrapper.java:
45)
at
org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
at
org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
Once I entered the kill -3 (and the stacktraces printed), my server
startup resumed (because I'd just generated some new entropy). The
above is attempting to generate a pseudo-random number by reading
from /dev/random.
I suggest that you verify (using kill -3) what your server is doing
during the delay. If you see the above (or something pretty similar),
then you're hitting the same problem. Please let us know what you find.
One fix is to generate more entropy on your system (or live with this
slow startup behavior). I've run into this issue previously on linux
test systems... IIRC, I've seen people work around this with 'ln -s /
dev/urandom /dev/random'. I'd guess that this isn't a very secure
option. It's possible that we (Geronimo) might be able to do something
about this behavior... I'd have to spend some quality time with
org.apache.geronimo.crypto.KeyStoreUtil (see the static
initializer...). If you or somebody else is able to do a bit of
digging, that'd be great... Probably worth creating a Jira to track
this...
--kevan
