Hi Chris, I did some more troubleshooting by enabling Tomcat debug logs. It turns out that the class loading is failing due to FileNotFoundException for antlr-runtime.jar file. I am not sure why is that the case since I can see that jar file at the location mentioned.
Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass FINE: loadClass(org.apache.solr.util.IOUtils, false) Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass FINE: Searching local repositories Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader findClass FINE: findClass(org.apache.solr.util.IOUtils) Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader openJARs FINE: Failed to open JAR java.io.FileNotFoundException: /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar (No such file or directory) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:215) at java.util.zip.ZipFile.<init>(ZipFile.java:145) at java.util.jar.JarFile.<init>(JarFile.java:154) at java.util.jar.JarFile.<init>(JarFile.java:118) at org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:2754) at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2994) at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2789) at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1159) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1647) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526) at org.apache.solr.cloud.Overseer.doClose(Overseer.java:1253) at org.apache.solr.cloud.Overseer.close(Overseer.java:1246) at org.apache.solr.cloud.ZkController.close(ZkController.java:500) at org.apache.solr.core.ZkContainer.close(ZkContainer.java:280) at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:374) at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:205) at org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFilterConfig.java:360) at org.apache.catalina.core.StandardContext.filterStop(StandardContext.java:4111) at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4880) at org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1279) at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:307) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1392) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1656) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1645) at java.lang.Thread.run(Thread.java:745) I also verified that this jar file has appropriate permissions set. sudo -u solr ls -all /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar -rw-r--r-- 1 solr solr 167735 Feb 10 10:00 /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar Please let me have your feedback on this. Thanks Hrishikesh On Mon, Feb 8, 2016 at 6:52 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hrishikesh, > > On 2/8/16 9:50 AM, Christopher Schultz wrote: > > Hrishikesh, > > > > On 2/6/16 1:17 PM, Hrishikesh Gadre wrote: > >> Thanks for the reply. Let me try this out. But do you think its a > >> bug in Tomcat ? > > > > No. There's nothing Tomcat can do about this, aside from allowing > > your application to load /more/ classes on shutdown. What's > > happening is that your application (or one of its libraries) is > > being very sloppy. > > > >> Because as an application developer I should be able to invoke > >> arbitrary application logic during the > >> ServletFilter::destroy(...) method without bothering about the > >> class loading related issues (Note that the jar files for these > >> classes are available inside WEB-INF/lib directory and the > >> shutdown process completes successfully most of the times > >> without raising NoClassDefFoundErrors). > > > > I just read the SO reference you posted and, in that case (if the > > answer by "miljanm" is accurate), the library isn't properly using > > its own ClassLoader if it's using a private one. > > > > It's not clear to me which ClassLoader is failing to find > > c.g.c.c.RemovalCause. Usually, the WebappClassLoader will issue a > > WARN or INFO message to the log saying that a class is being > > loaded during shutdown. Do you see any such messages in any of the > > log files? If not, this may be *entirely* a library issue. > > Also, see this SO issue for an example of what Tomcat usually says: > http://stackoverflow.com/questions/9920974/what-is-the-proper-way-to-shu > tdown-threads-when-tomcat-closes?rq=1 > > - -chris > -----BEGIN PGP SIGNATURE----- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAla4q8oACgkQ9CaO5/Lv0PCz1wCglWVQGY4lpdIMWGGmX0byCA6a > vuwAni8c+i6U0AYoUUftoTX79a2CDyzr > =goJV > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
