Re: Context fails to load during start of Tomcat, and stalls tomcat startup
Hi! Yes, it does use other webservices on the same webserver - I'll try to investigate if some of these are attempted during startup - this should be easily tested via tcpdump I guess. /Steffen On 10/18/12 5:00 PM, Ben Souther b...@souther.us wrote: Is it possible that the context in question depends on another context in your setup for something during startup? Do you have something in a context listener (or a servlet that gets deployed on startup) that makes a web service call to another context in your system during initialization? If so, what happens if that context is not available? Does it wait and try again or does it just hang? On Thu, 2012-10-18 at 16:08 +0200, Steffen Schumacher wrote: Hi! Running apache-tomcat 7.0.26 on FreeBSD 8.1@vmware, I've spotted a really curious issue. We are hosting ~8 different contexts on 4 servers all matching the above setup, and in normal operation we have no beef at all, except when the server guys needs to reboot the servers for whatever reason. When this happens, the a specific context prevents tomcat from starting up it simply stops doing anything after these lines: server17# bin/catalina.sh run Using CATALINA_BASE: /usr/local/apache-tomcat-7.0 Using CATALINA_HOME: /usr/local/apache-tomcat-7.0 Using CATALINA_TMPDIR: /usr/local/apache-tomcat-7.0/temp Using JRE_HOME:/usr/local Using CLASSPATH: /usr/local/apache-tomcat-7.0/bin/bootstrap.jar:/usr/local/apache-tomcat-7.0/ bin/tomcat-juli.jar Oct 18, 2012 1:06:03 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/local/diablo-jdk1.6.0/jre/lib/i386/server:/usr/local/diablo-jdk1.6.0/jr e/lib/i386:/usr/local/diablo-jdk1.6.0/jre/../lib/i386:/usr/java/packages/lib /i386:/lib:/usr/lib:/usr/local/lib Oct 18, 2012 1:06:03 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [http-bio-8180] Oct 18, 2012 1:06:03 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [http-bio-8443] Oct 18, 2012 1:06:04 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [ajp-bio-8009] Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1488 ms Oct 18, 2012 1:06:04 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Oct 18, 2012 1:06:04 PM org.apache.catalina.core.StandardEngine startInternalINFO: Starting Servlet Engine: Apache Tomcat/7.0.26 Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/XXX##10.4b2.war Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/.war Oct 18, 2012 1:06:12 PM com.sun.xml.ws.server.MonitorBase createRoot INFO: Metro monitoring rootname successfully set to: com.sun.metro:pp=/,type=WSEndpoint,name=/-RequestHandler-RequestHand lerPort Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate init INFO: WSSERVLET14: JAX-WS servlet initializing Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing Oct 18, 2012 1:06:12 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/##10.4b1.war Oct 18, 2012 1:06:15 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing And nothing happens now, even if I leave it for 1+ hours Now the ZZ(culprit) context is a perhaps not ultra-simple, as it is both a SOAP service (using metro) and also a SOAP client, but beyond that I wouldn't call it that complex. I can then remove the .war file and folder from webapps, and it will come up just nicely, and I can deploy the context via the html interface with now complaints. At this point, when the ZZZ context is deployed and running and I'll be good until the next reboot/restart of tomcat. The issue occurs regardless if I only have the context folder in webapps, or I also have the .war file present. I've used most of the day searching the web for solutions, but it is kinda hard when there is no indication of what is stalling things.. Could this be a tomcat issue? It is kinda hard to setup other (specific) versions of tomcat to test this, and I've tried 7.0.28, but its a completely separate environment with other versions of java etc
Re: Context fails to load during start of Tomcat, and stalls tomcat startup
It attempts to initialize a port to a local soap service, which has not yet been deployed - this is detected by the fact that a HTTP GET is made locally for the wsdl of said service, which is never responded to (expectedly so). So I guess this is the smoking gun I've been looking for, and the only option for me is to implement better handling of this, so that timeout occurs reasonably fast, and an attempt is made later on if the wsdl wasn't available at the time of tomcat-startup. So thanks for now, I think this is most likely what I needed! /Steffen On 10/19/12 12:30 PM, Steffen Schumacher s...@tdc.net wrote: Hi! Yes, it does use other webservices on the same webserver - I'll try to investigate if some of these are attempted during startup - this should be easily tested via tcpdump I guess. /Steffen On 10/18/12 5:00 PM, Ben Souther b...@souther.us wrote: Is it possible that the context in question depends on another context in your setup for something during startup? Do you have something in a context listener (or a servlet that gets deployed on startup) that makes a web service call to another context in your system during initialization? If so, what happens if that context is not available? Does it wait and try again or does it just hang? On Thu, 2012-10-18 at 16:08 +0200, Steffen Schumacher wrote: Hi! Running apache-tomcat 7.0.26 on FreeBSD 8.1@vmware, I've spotted a really curious issue. We are hosting ~8 different contexts on 4 servers all matching the above setup, and in normal operation we have no beef at all, except when the server guys needs to reboot the servers for whatever reason. When this happens, the a specific context prevents tomcat from starting up it simply stops doing anything after these lines: server17# bin/catalina.sh run Using CATALINA_BASE: /usr/local/apache-tomcat-7.0 Using CATALINA_HOME: /usr/local/apache-tomcat-7.0 Using CATALINA_TMPDIR: /usr/local/apache-tomcat-7.0/temp Using JRE_HOME:/usr/local Using CLASSPATH: /usr/local/apache-tomcat-7.0/bin/bootstrap.jar:/usr/local/apache-tomcat-7.0/ bin/tomcat-juli.jar Oct 18, 2012 1:06:03 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/local/diablo-jdk1.6.0/jre/lib/i386/server:/usr/local/diablo-jdk1.6.0/jr e/lib/i386:/usr/local/diablo-jdk1.6.0/jre/../lib/i386:/usr/java/packages/lib /i386:/lib:/usr/lib:/usr/local/lib Oct 18, 2012 1:06:03 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [http-bio-8180] Oct 18, 2012 1:06:03 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [http-bio-8443] Oct 18, 2012 1:06:04 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [ajp-bio-8009] Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1488 ms Oct 18, 2012 1:06:04 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Oct 18, 2012 1:06:04 PM org.apache.catalina.core.StandardEngine startInternalINFO: Starting Servlet Engine: Apache Tomcat/7.0.26 Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/XXX##10.4b2.war Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/.war Oct 18, 2012 1:06:12 PM com.sun.xml.ws.server.MonitorBase createRoot INFO: Metro monitoring rootname successfully set to: com.sun.metro:pp=/,type=WSEndpoint,name=/-RequestHandler-RequestHand lerPort Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate init INFO: WSSERVLET14: JAX-WS servlet initializing Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing Oct 18, 2012 1:06:12 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/##10.4b1.war Oct 18, 2012 1:06:15 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing And nothing happens now, even if I leave it for 1+ hours Now the ZZ(culprit) context is a perhaps not ultra-simple, as it is both a SOAP service (using metro) and also a SOAP client, but beyond that I wouldn't call it that complex. I can then remove the .war file and folder from webapps, and it will come up just nicely, and I can deploy the context
Context fails to load during start of Tomcat, and stalls tomcat startup
Hi! Running apache-tomcat 7.0.26 on FreeBSD 8.1@vmware, I've spotted a really curious issue. We are hosting ~8 different contexts on 4 servers all matching the above setup, and in normal operation we have no beef at all, except when the server guys needs to reboot the servers for whatever reason. When this happens, the a specific context prevents tomcat from starting up it simply stops doing anything after these lines: server17# bin/catalina.sh run Using CATALINA_BASE: /usr/local/apache-tomcat-7.0 Using CATALINA_HOME: /usr/local/apache-tomcat-7.0 Using CATALINA_TMPDIR: /usr/local/apache-tomcat-7.0/temp Using JRE_HOME:/usr/local Using CLASSPATH: /usr/local/apache-tomcat-7.0/bin/bootstrap.jar:/usr/local/apache-tomcat-7.0/ bin/tomcat-juli.jar Oct 18, 2012 1:06:03 PM org.apache.catalina.core.AprLifecycleListener init INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/local/diablo-jdk1.6.0/jre/lib/i386/server:/usr/local/diablo-jdk1.6.0/jr e/lib/i386:/usr/local/diablo-jdk1.6.0/jre/../lib/i386:/usr/java/packages/lib /i386:/lib:/usr/lib:/usr/local/lib Oct 18, 2012 1:06:03 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [http-bio-8180] Oct 18, 2012 1:06:03 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [http-bio-8443] Oct 18, 2012 1:06:04 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler [ajp-bio-8009] Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 1488 ms Oct 18, 2012 1:06:04 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Oct 18, 2012 1:06:04 PM org.apache.catalina.core.StandardEngine startInternalINFO: Starting Servlet Engine: Apache Tomcat/7.0.26 Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/XXX##10.4b2.war Oct 18, 2012 1:06:04 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/.war Oct 18, 2012 1:06:12 PM com.sun.xml.ws.server.MonitorBase createRoot INFO: Metro monitoring rootname successfully set to: com.sun.metro:pp=/,type=WSEndpoint,name=/-RequestHandler-RequestHand lerPort Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletDelegate init INFO: WSSERVLET14: JAX-WS servlet initializing Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing Oct 18, 2012 1:06:12 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing Oct 18, 2012 1:06:12 PM org.apache.catalina.startup.HostConfig deployWAR INFO: Deploying web application archive /usr/local/apache-tomcat-7.0/webapps/##10.4b1.war Oct 18, 2012 1:06:15 PM com.sun.xml.ws.transport.http.servlet.WSServletContextListener contextInitialized INFO: WSSERVLET12: JAX-WS context listener initializing And nothing happens now, even if I leave it for 1+ hours Now the ZZ(culprit) context is a perhaps not ultra-simple, as it is both a SOAP service (using metro) and also a SOAP client, but beyond that I wouldn't call it that complex. I can then remove the .war file and folder from webapps, and it will come up just nicely, and I can deploy the context via the html interface with now complaints. At this point, when the ZZZ context is deployed and running and I'll be good until the next reboot/restart of tomcat. The issue occurs regardless if I only have the context folder in webapps, or I also have the .war file present. I've used most of the day searching the web for solutions, but it is kinda hard when there is no indication of what is stalling things.. Could this be a tomcat issue? It is kinda hard to setup other (specific) versions of tomcat to test this, and I've tried 7.0.28, but its a completely separate environment with other versions of java etc, but there I run into other issues related to JAX/Metro, so thats not much help as it is. Any help is greatly appreciated! /Steffen I guess it could be the same underlying cause but why
