subject:"Fwd\: NoClassDefFoundError during graceful shutdown"

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-10 Thread Hrishikesh Gadre

Hi Chris,

I did some more troubleshooting by enabling Tomcat debug logs. It turns out
that the class loading is failing due to FileNotFoundException for
antlr-runtime.jar file. I am not sure why is that the case since I can see
that jar file at the location mentioned.

Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
loadClass
FINE: loadClass(org.apache.solr.util.IOUtils, false)
Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
loadClass
FINE:   Searching local repositories
Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
findClass
FINE: findClass(org.apache.solr.util.IOUtils)
Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
openJARs
FINE: Failed to open JAR
java.io.FileNotFoundException:
/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
(No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:215)
at java.util.zip.ZipFile.(ZipFile.java:145)
at java.util.jar.JarFile.(JarFile.java:154)
at java.util.jar.JarFile.(JarFile.java:118)
at
org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:2754)
at
org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2994)
at
org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2789)
at
org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1159)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1647)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at org.apache.solr.cloud.Overseer.doClose(Overseer.java:1253)
at org.apache.solr.cloud.Overseer.close(Overseer.java:1246)
at org.apache.solr.cloud.ZkController.close(ZkController.java:500)
at org.apache.solr.core.ZkContainer.close(ZkContainer.java:280)
at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:374)
at
org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:205)
at
org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFilterConfig.java:360)
at
org.apache.catalina.core.StandardContext.filterStop(StandardContext.java:4111)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4880)
at
org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1279)
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385)
at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:307)
at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1392)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1656)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665)
at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1645)
at java.lang.Thread.run(Thread.java:745)

I also verified that this jar file has appropriate permissions set.

sudo -u solr ls -all
/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar

-rw-r--r-- 1 solr solr 167735 Feb 10 10:00
/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
Please let me have your feedback on this.

Thanks
Hrishikesh



On Mon, Feb 8, 2016 at 6:52 AM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hrishikesh,
>
> On 2/8/16 9:50 AM, Christopher Schultz wrote:
> > Hrishikesh,
> >
> > On 2/6/16 1:17 PM, Hrishikesh Gadre wrote:
> >> Thanks for the reply. Let me try this out. But do you think its a
> >> bug in Tomcat ?
> >
> > No. There's nothing Tomcat can do about this, aside from allowing
> > your application to load /more/ classes on shutdown. What's
> > happening is that your application (or one of its libraries) is
> > being very sloppy.
> >
> >> Because as an application developer I should be able to invoke
> >> arbitrary application logic during the
> >> ServletFilter::destroy(...) method without bothering about the
> >> class loading related issues (Note that the jar files for these
> >> classes are available inside WEB-INF/lib directory and the
> >> shutdown process completes successfully most of the times
> >> without raising NoClassDefFoundErrors).
> >
> > I just read the SO reference you posted and, in that case (if the
> > answer by "miljanm" is accurate), the library isn't properly using
> > its own ClassLoader if it's using a private one.
> >
> > It's not clear to me which ClassLoader is failing to find
> > c.g.c.c.RemovalCause. Usually, the WebappClassLoader will issue a
> > WARN or INFO message to the log saying that a class is being
> > loaded during shutdown. Do you see any such messages in any of the
> > log files? If not, this may be *entirely* a library issue.
>
>

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-10 Thread Hrishikesh Gadre

Hi Chris,

Thanks for the response.

>>This looks like a different issue than you originally reported. Has
that other issue gone away? Or is this the root cause of the
originally-reported issue?

I think it is the root cause for the original issue. As per my
understanding, the FileNotFoundException results in findClass method
(WebappClassLoader.java
line 1647) returning a ClassNotFoundException. After this, I see following
log entries (sorry didn't include them in my previous email).

Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
loadClass

FINE: loadClass(org.apache.solr.util.IOUtils, false)

Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
loadClass

FINE:   Searching local repositories

Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
findClass

FINE: findClass(org.apache.solr.util.IOUtils)

Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
openJARs

FINE: Failed to open JAR

java.io.FileNotFoundException:
/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
(No such file or directory)

...

Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
loadClass

FINE:   Delegating to parent classloader at end:
org.apache.catalina.loader.StandardClassLoader@65f40964


It looks like the WebappClassLoader delegates the loading to the parent
class loader (WebappClassLoader.java line 1662). Since this jar file is
inside the WEB-INF/lib only, I think the parent class loader also can not
load it resulting in ClassNotFoundException in the end. Since the
FileNotFoundException is logged at the FINE level, it wasn't apparent until
I turned on FINE level logs for Tomcat.

I also verified that the effective Tomcat user has "execute" permissions
for the entire
path (/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/). Here is
the output of namei command,

namei -l
/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar

f:
/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar

drwxr-xr-x root root /

drwxr-xr-x root root var

drwxr-xr-x root root lib

drwxr-xr-x solr solr solr

drwxr-xr-x solr solr tomcat-deployment

drwxr-xr-x solr solr webapps

drwxr-xr-x solr solr solr

drwxr-xr-x solr solr WEB-INF

drwxr-xr-x solr solr lib

-rw-r--r-- solr solr antlr-runtime-3.5.jar


Thanks

Hrishikesh






On Wed, Feb 10, 2016 at 12:30 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hdishikesh,
>
> On 2/10/16 2:12 PM, Hrishikesh Gadre wrote:
> > Hi Chris,
> >
> > I did some more troubleshooting by enabling Tomcat debug logs. It
> > turns out that the class loading is failing due to
> > FileNotFoundException for antlr-runtime.jar file. I am not sure why
> > is that the case since I can see that jar file at the location
> > mentioned.
> >
> > Feb 10, 2016 10:00:24 AM
> > org.apache.catalina.loader.WebappClassLoader loadClass FINE:
> > loadClass(org.apache.solr.util.IOUtils, false) Feb 10, 2016
> > 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass
> > FINE:   Searching local repositories Feb 10, 2016 10:00:24 AM
> > org.apache.catalina.loader.WebappClassLoader findClass FINE:
> > findClass(org.apache.solr.util.IOUtils) Feb 10, 2016 10:00:24 AM
> > org.apache.catalina.loader.WebappClassLoader openJARs FINE: Failed
> > to open JAR java.io.FileNotFoundException:
> > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime
> - -3.5.jar
> >
> >
> (No such file or directory)
> > at java.util.zip.ZipFile.open(Native Method) at
> > java.util.zip.ZipFile.(ZipFile.java:215) at
> > java.util.zip.ZipFile.(ZipFile.java:145) at
> > java.util.jar.JarFile.(JarFile.java:154) at
> > java.util.jar.JarFile.(JarFile.java:118) at
> > org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoade
> r.java:2754)
> >
> >
> at
> > org.apache.catalina.loader.WebappClassLoader.findResourceInternal(Weba
> ppClassLoader.java:2994)
> >
> >
> at
> > org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappC
> lassLoader.java:2789)
> >
> >
> at
> > org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoad
> er.java:1159)
> >
> >
> at
> > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoad
> er.java:1647)
> >
> >
> at
> > org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoad
> er.java:1526)
> >
> >
> at org.apache.solr.cloud.Overseer.doClose(Overseer.java:1253)
> > at org.apache.solr.cloud.Overseer.close(Overseer.java:1246) at
> > org.apache.solr.cloud.ZkController.close(ZkController.java:500) at
> > org.apache.solr.core.ZkContainer.close(ZkContainer.java:280) at
> > org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:374)
> >
> >
> at
> > org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.
> java:205)
> >
> >
> at
> > org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFi

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-10 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hdishikesh,

On 2/10/16 2:12 PM, Hrishikesh Gadre wrote:
> Hi Chris,
> 
> I did some more troubleshooting by enabling Tomcat debug logs. It
> turns out that the class loading is failing due to
> FileNotFoundException for antlr-runtime.jar file. I am not sure why
> is that the case since I can see that jar file at the location
> mentioned.
> 
> Feb 10, 2016 10:00:24 AM
> org.apache.catalina.loader.WebappClassLoader loadClass FINE:
> loadClass(org.apache.solr.util.IOUtils, false) Feb 10, 2016
> 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass 
> FINE:   Searching local repositories Feb 10, 2016 10:00:24 AM
> org.apache.catalina.loader.WebappClassLoader findClass FINE:
> findClass(org.apache.solr.util.IOUtils) Feb 10, 2016 10:00:24 AM
> org.apache.catalina.loader.WebappClassLoader openJARs FINE: Failed
> to open JAR java.io.FileNotFoundException: 
> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime
- -3.5.jar
>
> 
(No such file or directory)
> at java.util.zip.ZipFile.open(Native Method) at
> java.util.zip.ZipFile.(ZipFile.java:215) at
> java.util.zip.ZipFile.(ZipFile.java:145) at
> java.util.jar.JarFile.(JarFile.java:154) at
> java.util.jar.JarFile.(JarFile.java:118) at 
> org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoade
r.java:2754)
>
> 
at
> org.apache.catalina.loader.WebappClassLoader.findResourceInternal(Weba
ppClassLoader.java:2994)
>
> 
at
> org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappC
lassLoader.java:2789)
>
> 
at
> org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoad
er.java:1159)
>
> 
at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoad
er.java:1647)
>
> 
at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoad
er.java:1526)
>
> 
at org.apache.solr.cloud.Overseer.doClose(Overseer.java:1253)
> at org.apache.solr.cloud.Overseer.close(Overseer.java:1246) at
> org.apache.solr.cloud.ZkController.close(ZkController.java:500) at
> org.apache.solr.core.ZkContainer.close(ZkContainer.java:280) at
> org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:374)
>
> 
at
> org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.
java:205)
>
> 
at
> org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFi
lterConfig.java:360)
>
> 
at
> org.apache.catalina.core.StandardContext.filterStop(StandardContext.ja
va:4111)
>
> 
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:48
80)
> at 
> org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:
1279)
>
> 
at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385)
> at 
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:
307)
>
> 
at
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(Lifecycle
Support.java:142)
>
> 
at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase
.java:1392)
>
> 
at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr
ocessChildren(ContainerBase.java:1656)
>
> 
at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.pr
ocessChildren(ContainerBase.java:1665)
>
> 
at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.ru
n(ContainerBase.java:1645)
>
> 
at java.lang.Thread.run(Thread.java:745)
> 
> I also verified that this jar file has appropriate permissions
> set.
> 
> sudo -u solr ls -all 
> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime
- -3.5.jar
>
>  -rw-r--r-- 1 solr solr 167735 Feb 10 10:00 
> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime
- -3.5.jar
>
> 
Please let me have your feedback on this.

You also need to make sure that every directory from the root has
"execute" privileges for the effective Tomcat user.

This looks like a different issue than you originally reported. Has
that other issue gone away? Or is this the root cause of the
originally-reported issue?

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAla7ncsACgkQ9CaO5/Lv0PCZdACePaJi6JBfsd/EvAsSip+FUm3Q
LMgAn0nuXI0hrbh2mqoh7U8U4x+4+/4X
=cCGU
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-10 Thread Hrishikesh Gadre

Hi Chris,

I added a check to see if this jar file exists (and is readable) both
inside the ServletFilter init(...) and destroy(...) methods. Most of the
times the check passes. But when I get NoClassDefFoundError, I see that
check is passing during invocation of init(...) method but not during
destroy(...) method. This is really weird behavior.

https://gist.github.com/hgadre/efd959ad1d0f1793b8b2

Thanks
Hrishikesh


On Wed, Feb 10, 2016 at 1:26 PM, Hrishikesh Gadre 
wrote:

> Hi Chris,
>
> Thanks for the response.
>
> >>This looks like a different issue than you originally reported. Has
> that other issue gone away? Or is this the root cause of the
> originally-reported issue?
>
> I think it is the root cause for the original issue. As per my
> understanding, the FileNotFoundException results in findClass method 
> (WebappClassLoader.java
> line 1647) returning a ClassNotFoundException. After this, I see following
> log entries (sorry didn't include them in my previous email).
>
> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
> loadClass
>
> FINE: loadClass(org.apache.solr.util.IOUtils, false)
>
> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
> loadClass
>
> FINE:   Searching local repositories
>
> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
> findClass
>
> FINE: findClass(org.apache.solr.util.IOUtils)
>
> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
> openJARs
>
> FINE: Failed to open JAR
>
> java.io.FileNotFoundException:
> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
> (No such file or directory)
>
> ...
>
> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
> loadClass
>
> FINE:   Delegating to parent classloader at end:
> org.apache.catalina.loader.StandardClassLoader@65f40964
>
>
> It looks like the WebappClassLoader delegates the loading to the parent
> class loader (WebappClassLoader.java line 1662). Since this jar file is
> inside the WEB-INF/lib only, I think the parent class loader also can not
> load it resulting in ClassNotFoundException in the end. Since the
> FileNotFoundException is logged at the FINE level, it wasn't apparent until
> I turned on FINE level logs for Tomcat.
>
> I also verified that the effective Tomcat user has "execute" permissions
> for the entire
> path (/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/). Here is
> the output of namei command,
>
> namei -l
> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
>
> f:
> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
>
> drwxr-xr-x root root /
>
> drwxr-xr-x root root var
>
> drwxr-xr-x root root lib
>
> drwxr-xr-x solr solr solr
>
> drwxr-xr-x solr solr tomcat-deployment
>
> drwxr-xr-x solr solr webapps
>
> drwxr-xr-x solr solr solr
>
> drwxr-xr-x solr solr WEB-INF
>
> drwxr-xr-x solr solr lib
>
> -rw-r--r-- solr solr antlr-runtime-3.5.jar
>
>
> Thanks
>
> Hrishikesh
>
>
>
>
>
>
> On Wed, Feb 10, 2016 at 12:30 PM, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Hdishikesh,
>>
>> On 2/10/16 2:12 PM, Hrishikesh Gadre wrote:
>> > Hi Chris,
>> >
>> > I did some more troubleshooting by enabling Tomcat debug logs. It
>> > turns out that the class loading is failing due to
>> > FileNotFoundException for antlr-runtime.jar file. I am not sure why
>> > is that the case since I can see that jar file at the location
>> > mentioned.
>> >
>> > Feb 10, 2016 10:00:24 AM
>> > org.apache.catalina.loader.WebappClassLoader loadClass FINE:
>> > loadClass(org.apache.solr.util.IOUtils, false) Feb 10, 2016
>> > 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass
>> > FINE:   Searching local repositories Feb 10, 2016 10:00:24 AM
>> > org.apache.catalina.loader.WebappClassLoader findClass FINE:
>> > findClass(org.apache.solr.util.IOUtils) Feb 10, 2016 10:00:24 AM
>> > org.apache.catalina.loader.WebappClassLoader openJARs FINE: Failed
>> > to open JAR java.io.FileNotFoundException:
>> > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime
>> - -3.5.jar
>> >
>> >
>> (No such file or directory)
>> > at java.util.zip.ZipFile.open(Native Method) at
>> > java.util.zip.ZipFile.(ZipFile.java:215) at
>> > java.util.zip.ZipFile.(ZipFile.java:145) at
>> > java.util.jar.JarFile.(JarFile.java:154) at
>> > java.util.jar.JarFile.(JarFile.java:118) at
>> > org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoade
>> r.java:2754)
>> >
>> >
>> at
>> > org.apache.catalina.loader.WebappClassLoader.findResourceInternal(Weba
>> ppClassLoader.java:2994)
>> >
>> >
>> at
>> > org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappC
>> lassLoader.java:2789)
>> >
>> >
>> at
>> > org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoad
>> er.java:1159)
>> >
>> >
>> at
>> >

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-10 Thread Hrishikesh Gadre

Oh BTW the jar file still exits on the filesystem (i.e. its not being
touched during the execution of Tomcat).

On Wed, Feb 10, 2016 at 6:31 PM, Hrishikesh Gadre 
wrote:

> Hi Chris,
>
> I added a check to see if this jar file exists (and is readable) both
> inside the ServletFilter init(...) and destroy(...) methods. Most of the
> times the check passes. But when I get NoClassDefFoundError, I see that
> check is passing during invocation of init(...) method but not during
> destroy(...) method. This is really weird behavior.
>
> https://gist.github.com/hgadre/efd959ad1d0f1793b8b2
>
> Thanks
> Hrishikesh
>
>
> On Wed, Feb 10, 2016 at 1:26 PM, Hrishikesh Gadre 
> wrote:
>
>> Hi Chris,
>>
>> Thanks for the response.
>>
>> >>This looks like a different issue than you originally reported. Has
>> that other issue gone away? Or is this the root cause of the
>> originally-reported issue?
>>
>> I think it is the root cause for the original issue. As per my
>> understanding, the FileNotFoundException results in findClass method 
>> (WebappClassLoader.java
>> line 1647) returning a ClassNotFoundException. After this, I see following
>> log entries (sorry didn't include them in my previous email).
>>
>> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
>> loadClass
>>
>> FINE: loadClass(org.apache.solr.util.IOUtils, false)
>>
>> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
>> loadClass
>>
>> FINE:   Searching local repositories
>>
>> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
>> findClass
>>
>> FINE: findClass(org.apache.solr.util.IOUtils)
>>
>> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
>> openJARs
>>
>> FINE: Failed to open JAR
>>
>> java.io.FileNotFoundException:
>> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
>> (No such file or directory)
>>
>> ...
>>
>> Feb 10, 2016 10:00:24 AM org.apache.catalina.loader.WebappClassLoader
>> loadClass
>>
>> FINE:   Delegating to parent classloader at end:
>> org.apache.catalina.loader.StandardClassLoader@65f40964
>>
>>
>> It looks like the WebappClassLoader delegates the loading to the parent
>> class loader (WebappClassLoader.java line 1662). Since this jar file is
>> inside the WEB-INF/lib only, I think the parent class loader also can not
>> load it resulting in ClassNotFoundException in the end. Since the
>> FileNotFoundException is logged at the FINE level, it wasn't apparent until
>> I turned on FINE level logs for Tomcat.
>>
>> I also verified that the effective Tomcat user has "execute" permissions
>> for the entire
>> path (/var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/). Here is
>> the output of namei command,
>>
>> namei -l
>> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
>>
>> f:
>> /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime-3.5.jar
>>
>> drwxr-xr-x root root /
>>
>> drwxr-xr-x root root var
>>
>> drwxr-xr-x root root lib
>>
>> drwxr-xr-x solr solr solr
>>
>> drwxr-xr-x solr solr tomcat-deployment
>>
>> drwxr-xr-x solr solr webapps
>>
>> drwxr-xr-x solr solr solr
>>
>> drwxr-xr-x solr solr WEB-INF
>>
>> drwxr-xr-x solr solr lib
>>
>> -rw-r--r-- solr solr antlr-runtime-3.5.jar
>>
>>
>> Thanks
>>
>> Hrishikesh
>>
>>
>>
>>
>>
>>
>> On Wed, Feb 10, 2016 at 12:30 PM, Christopher Schultz <
>> ch...@christopherschultz.net> wrote:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA1
>>>
>>> Hdishikesh,
>>>
>>> On 2/10/16 2:12 PM, Hrishikesh Gadre wrote:
>>> > Hi Chris,
>>> >
>>> > I did some more troubleshooting by enabling Tomcat debug logs. It
>>> > turns out that the class loading is failing due to
>>> > FileNotFoundException for antlr-runtime.jar file. I am not sure why
>>> > is that the case since I can see that jar file at the location
>>> > mentioned.
>>> >
>>> > Feb 10, 2016 10:00:24 AM
>>> > org.apache.catalina.loader.WebappClassLoader loadClass FINE:
>>> > loadClass(org.apache.solr.util.IOUtils, false) Feb 10, 2016
>>> > 10:00:24 AM org.apache.catalina.loader.WebappClassLoader loadClass
>>> > FINE:   Searching local repositories Feb 10, 2016 10:00:24 AM
>>> > org.apache.catalina.loader.WebappClassLoader findClass FINE:
>>> > findClass(org.apache.solr.util.IOUtils) Feb 10, 2016 10:00:24 AM
>>> > org.apache.catalina.loader.WebappClassLoader openJARs FINE: Failed
>>> > to open JAR java.io.FileNotFoundException:
>>> > /var/lib/solr/tomcat-deployment/webapps/solr/WEB-INF/lib/antlr-runtime
>>> - -3.5.jar
>>> >
>>> >
>>> (No such file or directory)
>>> > at java.util.zip.ZipFile.open(Native Method) at
>>> > java.util.zip.ZipFile.(ZipFile.java:215) at
>>> > java.util.zip.ZipFile.(ZipFile.java:145) at
>>> > java.util.jar.JarFile.(JarFile.java:154) at
>>> > java.util.jar.JarFile.(JarFile.java:118) at
>>> > org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoade
>>> r.java:2754)
>>> >
>>> >
>>> at
>>> >

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-08 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hrishikesh,

On 2/8/16 9:50 AM, Christopher Schultz wrote:
> Hrishikesh,
> 
> On 2/6/16 1:17 PM, Hrishikesh Gadre wrote:
>> Thanks for the reply. Let me try this out. But do you think its a
>> bug in Tomcat ?
> 
> No. There's nothing Tomcat can do about this, aside from allowing 
> your application to load /more/ classes on shutdown. What's 
> happening is that your application (or one of its libraries) is 
> being very sloppy.
> 
>> Because as an application developer I should be able to invoke 
>> arbitrary application logic during the 
>> ServletFilter::destroy(...) method without bothering about the 
>> class loading related issues (Note that the jar files for these 
>> classes are available inside WEB-INF/lib directory and the 
>> shutdown process completes successfully most of the times
>> without raising NoClassDefFoundErrors).
> 
> I just read the SO reference you posted and, in that case (if the 
> answer by "miljanm" is accurate), the library isn't properly using 
> its own ClassLoader if it's using a private one.
> 
> It's not clear to me which ClassLoader is failing to find 
> c.g.c.c.RemovalCause. Usually, the WebappClassLoader will issue a 
> WARN or INFO message to the log saying that a class is being
> loaded during shutdown. Do you see any such messages in any of the
> log files? If not, this may be *entirely* a library issue.

Also, see this SO issue for an example of what Tomcat usually says:
http://stackoverflow.com/questions/9920974/what-is-the-proper-way-to-shu
tdown-threads-when-tomcat-closes?rq=1

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAla4q8oACgkQ9CaO5/Lv0PCz1wCglWVQGY4lpdIMWGGmX0byCA6a
vuwAni8c+i6U0AYoUUftoTX79a2CDyzr
=goJV
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-08 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hrishikesh,

On 2/8/16 9:50 AM, Christopher Schultz wrote:
> Hrishikesh,
> 
> On 2/6/16 1:17 PM, Hrishikesh Gadre wrote:
>> Thanks for the reply. Let me try this out. But do you think its
>> a bug in Tomcat ?
> 
> No. There's nothing Tomcat can do about this, aside from allowing
> your application to load /more/ classes on shutdown. What's
> happening is that your application (or one of its libraries) is
> being very sloppy.
> 
>> Because as an application developer I should be able to invoke 
>> arbitrary application logic during the
>> ServletFilter::destroy(...) method without bothering about the
>> class loading related issues (Note that the jar files for these
>> classes are available inside WEB-INF/lib directory and the
>> shutdown process completes successfully most of the times without
>> raising NoClassDefFoundErrors).
> 
> I just read the SO reference you posted and, in that case (if the 
> answer by "miljanm" is accurate), the library isn't properly using
> its own ClassLoader if it's using a private one.
> 
> It's not clear to me which ClassLoader is failing to find 
> c.g.c.c.RemovalCause. Usually, the WebappClassLoader will issue a
> WARN or INFO message to the log saying that a class is being loaded
> during shutdown. Do you see any such messages in any of the log
> files? If not, this may be *entirely* a library issue.

Also, see this SO issue for an example of what Tomcat usually says:
http://stackoverflow.com/questions/9920974/what-is-the-proper-way-to-shu
tdown-threads-when-tomcat-closes?rq=1

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAla4q8MACgkQ9CaO5/Lv0PDf+gCgj4iCSYFwKph6tpDkB5GzNYOu
8L8AnjuTH/f6frohS7gtEALSO4lKxQ6g
=zOP2
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-08 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hrishikesh,

On 2/6/16 1:17 PM, Hrishikesh Gadre wrote:
> Thanks for the reply. Let me try this out. But do you think its a
> bug in Tomcat ?

No. There's nothing Tomcat can do about this, aside from allowing your
application to load /more/ classes on shutdown. What's happening is
that your application (or one of its libraries) is being very sloppy.

> Because as an application developer I should be able to invoke 
> arbitrary application logic during the ServletFilter::destroy(...)
> method without bothering about the class loading related issues
> (Note that the jar files for these classes are available inside
> WEB-INF/lib directory and the shutdown process completes
> successfully most of the times without raising 
> NoClassDefFoundErrors).

I just read the SO reference you posted and, in that case (if the
answer by "miljanm" is accurate), the library isn't properly using its
own ClassLoader if it's using a private one.

It's not clear to me which ClassLoader is failing to find
c.g.c.c.RemovalCause. Usually, the WebappClassLoader will issue a WARN
or INFO message to the log saying that a class is being loaded during
shutdown. Do you see any such messages in any of the log files? If
not, this may be *entirely* a library issue.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAla4qzYACgkQ9CaO5/Lv0PDHbACgwnjRtzoKp6MxS+mqkm9egQjm
hdoAniUTCkProbKW3K2BG3wtgaCJJeYi
=R5EV
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-06 Thread Hrishikesh Gadre

Hi Chris,

Thanks for the reply. Let me try this out. But do you think its a bug in
Tomcat ? Because as an application developer I should be able to invoke
arbitrary application logic during the ServletFilter::destroy(...) method
without bothering about the class loading related issues (Note that the jar
files for these classes are available inside WEB-INF/lib directory and the
shutdown process completes successfully most of the times without raising
NoClassDefFoundErrors).

Please let me know.

Regards
Hrishikesh


On Fri, Feb 5, 2016 at 8:12 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hrishikesh,
>
> On 2/5/16 6:57 PM, Hrishikesh Gadre wrote:
> > We are getting NoClassDefFoundErrors during Tomcat graceful
> > shutdown process (i.e. using "catalina.sh stop" command). We are
> > using Tomcat with version 6.0.44. Note that this error can not be
> > reproduced consistently. I found following article on the
> > stackoverflow.com which suggests class loader related problems
> > during execution of JVM shutdown hook. But in this case, the error
> > is raised during the execution of ServletFilter::destroy(...)
> > method. So I am not sure if article is relevant.
>
> This kind of thing usually happens because Tomcat marks the
> WebappClassLoader as "shutting down" and it won't load any new
> classes. So, if your shutdown code ends up requiring new classes, it
> could fail.
>
> Is this reproducible? If possible, can you write a
> ServletContextListener that will simply load the class in question
> (com.google.common.cache.RemovalCause) and do nothing else?
>
> If that solves the problem, then the root issue is what I described
> above (and may be replaced by a series of other classes for the same
> reason; just load them all as a test to see if this is what's happening.
> )
>
> It's not a great solution, but it might fix the problem and allow your
> application to shutdown (actually) gracefully.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAla1cp4ACgkQ9CaO5/Lv0PB7EACfQvNwIHoHT89yeSKgP4+Fff4n
> gYUAoLLgQkISwi0zueBXPT2cBXenYBRt
> =yGJb
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Fwd: NoClassDefFoundError during graceful shutdown

2016-02-05 Thread Hrishikesh Gadre

Hi,

We are getting NoClassDefFoundErrors during Tomcat graceful shutdown
process (i.e. using "catalina.sh stop" command). We are using Tomcat with
version 6.0.44. Note that this error can not be reproduced consistently. I
found following article on the stackoverflow.com which suggests class
loader related problems during execution of JVM shutdown hook. But in this
case, the error is raised during the execution of
ServletFilter::destroy(...) method. So I am not sure if article is relevant.

http://stackoverflow.com/questions/19388201/could-not-load-org-apache-hadoop-util-shutdownhookmanager-when-shutdown-tomcat-s

Could you please provide any insight?

java.lang.NoClassDefFoundError: com/google/common/cache/RemovalCause

at
com.google.common.cache.LocalCache$Segment.remove(LocalCache.java:3147)

at com.google.common.cache.LocalCache.remove(LocalCache.java:4236)

at
org.apache.curator.framework.imps.NamespaceWatcherMap.remove(NamespaceWatcherMap.java:71)

at
org.apache.curator.framework.imps.CuratorFrameworkImpl.clearWatcherReferences(CuratorFrameworkImpl.java:189)

at
org.apache.curator.framework.recipes.cache.PathChildrenCache.close(PathChildrenCache.java:376)

at
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.stopThreads(ZKDelegationTokenSecretManager.java:456)

at
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.destroy(DelegationTokenManager.java:152)

at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.destroy(DelegationTokenAuthenticationHandler.java:153)

at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.destroy(AuthenticationFilter.java:401)

at
org.apache.solr.servlet.SolrHadoopAuthenticationFilter.destroy(SolrHadoopAuthenticationFilter.java:177)

at
org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFilterConfig.java:360)

at
org.apache.catalina.core.StandardContext.filterStop(StandardContext.java:4111)

at
org.apache.catalina.core.StandardContext.stop(StandardContext.java:4880)

at
org.apache.catalina.startup.HostConfig.checkResources(HostConfig.java:1279)

at
org.apache.catalina.startup.HostConfig.check(HostConfig.java:1385)

at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:307)

at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)

at
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1392)

at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1656)

at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1665)

at
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1645)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException:
com.google.common.cache.RemovalCause

at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1680)

at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)

Re: Fwd: NoClassDefFoundError during graceful shutdown

2016-02-05 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hrishikesh,

On 2/5/16 6:57 PM, Hrishikesh Gadre wrote:
> We are getting NoClassDefFoundErrors during Tomcat graceful 
> shutdown process (i.e. using "catalina.sh stop" command). We are 
> using Tomcat with version 6.0.44. Note that this error can not be 
> reproduced consistently. I found following article on the 
> stackoverflow.com which suggests class loader related problems 
> during execution of JVM shutdown hook. But in this case, the error 
> is raised during the execution of ServletFilter::destroy(...) 
> method. So I am not sure if article is relevant.

This kind of thing usually happens because Tomcat marks the
WebappClassLoader as "shutting down" and it won't load any new
classes. So, if your shutdown code ends up requiring new classes, it
could fail.

Is this reproducible? If possible, can you write a
ServletContextListener that will simply load the class in question
(com.google.common.cache.RemovalCause) and do nothing else?

If that solves the problem, then the root issue is what I described
above (and may be replaced by a series of other classes for the same
reason; just load them all as a test to see if this is what's happening.
)

It's not a great solution, but it might fix the problem and allow your
application to shutdown (actually) gracefully.

- -chris
-BEGIN PGP SIGNATURE-
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAla1cp4ACgkQ9CaO5/Lv0PB7EACfQvNwIHoHT89yeSKgP4+Fff4n
gYUAoLLgQkISwi0zueBXPT2cBXenYBRt
=yGJb
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

Fwd: NoClassDefFoundError during graceful shutdown

Re: Fwd: NoClassDefFoundError during graceful shutdown

11 matches

Site Navigation

Mail list logo

Footer information