Hi Chris, I added a check to see if this jar file exists (and is readable) both inside the ServletFilter init(...) and destroy(...) methods. Most of the times the check passes. But when I get NoClassDefFoundError, I see that check is passing during invocation of init(...) method but not during destroy(...) method. This is really weird behavior.
https://gist.github.com/hgadre/efd959ad1d0f1793b8b2 Thanks Hrishikesh On Wed, Feb 10, 2016 at 1:26 PM, Hrishikesh Gadre <gadre.s...@gmail.com> wrote: > Hi Chris, > > Thanks for the response. > > >>This looks like a different issue than you originally reported. Has > that other issue gone away? Or is this the root cause of the > originally-reported issue? > > I think it is the root cause for the original issue. As per my > understanding, the FileNotFoundException results in findClass method > (WebappClassLoader.java > line 1647) returning a ClassNotFoundException. After this, I see following > log entries (sorry didn't include them in my previous email). > > Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader > loadClass > > FINE: loadClass(org.apache.solr.util.IOUtils, false) > > Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader > loadClass > > FINE: Searching local repositories > > Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader > findClass > > FINE: findClass(org.apache.solr.util.IOUtils) > > Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader > openJARs > > FINE: Failed to open JAR > > java.io.FileNotFoundException: > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar > (No such file or directory) > > ... > > Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader > loadClass > > FINE: Delegating to parent classloader at end: > org.apache.catalina.loader.StandardClassLoader@65f40964 > > > It looks like the WebappClassLoader delegates the loading to the parent > class loader (WebappClassLoader.java line 1662). Since this jar file is > inside the WEB-INF/lib only, I think the parent class loader also can not > load it resulting in ClassNotFoundException in the end. Since the > FileNotFoundException is logged at the FINE level, it wasn't apparent until > I turned on FINE level logs for Tomcat. > > I also verified that the effective Tomcat user has "execute" permissions > for the entire > path (/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/). Here is > the output of namei command, > > namei -l > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar > > f: > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar > > drwxr-xr-x root root / > > drwxr-xr-x root root var > > drwxr-xr-x root root lib > > drwxr-xr-x solr solr solr > > drwxr-xr-x solr solr tomcat-deployment > > drwxr-xr-x solr solr webapps > > drwxr-xr-x solr solr solr > > drwxr-xr-x solr solr WEB-INF > > drwxr-xr-x solr solr lib > > -rw-r--r-- solr solr antlr-runtime-3.5.jar > > > Thanks > > Hrishikesh > > > > > > > On Wed, Feb 10, 2016 at 12:30 PM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Hdishikesh, >> >> On 2/10/16 2:12 PM, Hrishikesh Gadre wrote: >> > Hi Chris, >> > >> > I did some more troubleshooting by enabling Tomcat debug logs. It >> > turns out that the class loading is failing due to >> > FileNotFoundException for antlr-runtime.jar file. I am not sure why >> > is that the case since I can see that jar file at the location >> > mentioned. >> > >> > Feb 10, 2016 10:00:24 AM >> > org.apache.catalina.loader.WebappClassLoader loadClass FINE: >> > loadClass(org.apache.solr.util.IOUtils, false) Feb 10, 2016 >> > 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass >> > FINE: Searching local repositories Feb 10, 2016 10:00:24 AM >> > org.apache.catalina.loader.WebappClassLoader findClass FINE: >> > findClass(org.apache.solr.util.IOUtils) Feb 10, 2016 10:00:24 AM >> > org.apache.catalina.loader.WebappClassLoader openJARs FINE: Failed >> > to open JAR java.io.FileNotFoundException: >> > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime >> - -3.5.jar >> > >> > >> (No such file or directory) >> > at java.util.zip.ZipFile.open(Native Method) at >> > java.util.zip.ZipFile.<init>(ZipFile.java:215) at >> > java.util.zip.ZipFile.<init>(ZipFile.java:145) at >> > java.util.jar.JarFile.<init>(JarFile.java:154) at >> > java.util.jar.JarFile.<init>(JarFile.java:118) at >> > org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoade >> r.java:2754) >> > >> > >> at >> > org.apache.catalina.loader.WebappClassLoader.findResourceInternal(Weba >> ppClassLoader.java:2994) >> > >> > >> at >> > org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappC >> lassLoader.java:2789) >> > >> > >> at >> > org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoad >> er.java:1159) >> > >> > >> at >> > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoad >> er.java:1647) >> > >> > >> at >> > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoad >> er.java:1526) >> > >> > >> at org.apache.solr.cloud.Overseer.doClose(Overseer.java:1253) >> > at org.apache.solr.cloud.Overseer.close(Overseer.java:1246) at >> > org.apache.solr.cloud.ZkController.close(ZkController.java:500) at >> > org.apache.solr.core.ZkContainer.close(ZkContainer.java:280) at >> > org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:374) >> > >> > >> at >> > org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter. >> java:205) >> > >> > >> at >> > org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFi >> lterConfig.java:360) >> > >> > >> at >> > org.apache.catalina.core.StandardContext.filterStop(StandardContext.ja >> va:4111) >> > >> > >> at org.apache.catalina.core.StandardContext.stop(StandardContext.java:48 >> 80) >> > at >> > org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java: >> 1279) >> > >> > >> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385) >> > at >> > org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java: >> 307) >> > >> > >> at >> > org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(Lifecycle >> Support.java:142) >> > >> > >> at >> > org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase >> .java:1392) >> > >> > >> at >> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr >> ocessChildren(ContainerBase.java:1656) >> > >> > >> at >> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr >> ocessChildren(ContainerBase.java:1665) >> > >> > >> at >> > org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.ru >> n(ContainerBase.java:1645) >> > >> > >> at java.lang.Thread.run(Thread.java:745) >> > >> > I also verified that this jar file has appropriate permissions >> > set. >> > >> > sudo -u solr ls -all >> > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime >> - -3.5.jar >> > >> > -rw-r--r-- 1 solr solr 167735 Feb 10 10:00 >> > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime >> - -3.5.jar >> > >> > >> Please let me have your feedback on this. >> >> You also need to make sure that every directory from the root has >> "execute" privileges for the effective Tomcat user. >> >> This looks like a different issue than you originally reported. Has >> that other issue gone away? Or is this the root cause of the >> originally-reported issue? >> >> - -chris >> -----BEGIN PGP SIGNATURE----- >> Comment: GPGTools - http://gpgtools.org >> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >> >> iEYEARECAAYFAla7ncsACgkQ9CaO5/Lv0PCZdACePaJi6JBfsd/EvAsSip+FUm3Q >> LMgAn0nuXI0hrbh2mqoh7U8U4x+4+/4X >> =cCGU >> -----END PGP SIGNATURE----- >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> >
