Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 17/12/2010 15:38, Srikanth Konjarla wrote:
>
>
> On 12/17/10 3:01 AM, Pid wrote:
>> On 15/12/2010 15:37, Srikanth Konjarla wrote:
>>>
>>>
>>> On 12/15/10 7:13 AM, Pid wrote:
On 15/12/2010 02:40, Srikanth Konjarla wrote:
> I could catch Axis threadlocals from the app to clean up. However, I
> have a question wrt following tomcat's log at the time of undeploy of
> app. The message suggests that it is removing the same threadlocal
> twice. Is it because it was not removed in the first attempt?
>
> -
> ec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader
> clearThreadLocalMap
> SEVERE: A web application created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a
> value of type [org.apache.catalina.loader.WebappClassLoader] (value
> [WebappClassLoader
> delegate: false
> repositories:
> /WEB-INF/classes/
> --> Parent Classloader:
> org.apache.catalina.loader.standardclassloa...@6ee3849c
> ]) but failed to remove it when the web application was stopped. To
> prevent a memory leak, the ThreadLocal has been forcibly removed.
> Dec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader
> clearThreadLocalMap
> SEVERE: A web application created a ThreadLocal with key of type
> [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a
> value of type [org.apache.catalina.loader.WebappClassLoader] (value
> [WebappClassLoader
> delegate: false
> repositories:
> /WEB-INF/classes/
> --> Parent Classloader:
> org.apache.catalina.loader.standardclassloa...@6ee3849c
> ]) but failed to remove it when the web application was stopped. To
> prevent a memory leak, the ThreadLocal has been forcibly removed.
>
I don't think this is from Axis, does your code check other sources?
>>> Right. This is not from Axis. This is from one of the components of the
>>> app. I am wondering why the threalocal is not cleared at first time?
>>
>> Are you generically clearing all ThreadLocals, or just Axis related ones?
> I am clearing ThreadLocals pertaining to Axis. Also, clearing all
> threadlocals from context listener during "destroy" process of the webapp.
This runs in a different thread. You need to clear each request thread
after use. A Servlet Filter can do that, by calling the doFilter method
before clearing the thread.
public void doFilter(ServletRequest req, ServletResponse res) {
HttpServletRequest hreq = (HttpServletRequest) req;
HttpServletResponse hres = (HttpServletResponse) res;
chain.doFilter(hreq, hres);
myClearThreadLocals();
private void myClearThreadLocals() {
// ... etc
}
}
>>> Or is it one each for "threadLocals" and "inheritedthreadLocals"?
>>
>> Yes, you need to clear them separately.
> Right. The code mimics tomcat's cleaning process that exists in
> "WebappClassLoader" class.
>
> Srikanth
>
>>
>>
>> p
>>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
0x62590808.asc
Description: application/pgp-keys
signature.asc
Description: OpenPGP digital signature
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 12/17/10 3:01 AM, Pid wrote: > On 15/12/2010 15:37, Srikanth Konjarla wrote: >> >> >> On 12/15/10 7:13 AM, Pid wrote: >>> On 15/12/2010 02:40, Srikanth Konjarla wrote: I could catch Axis threadlocals from the app to clean up. However, I have a question wrt following tomcat's log at the time of undeploy of app. The message suggests that it is removing the same threadlocal twice. Is it because it was not removed in the first attempt? - ec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a value of type [org.apache.catalina.loader.WebappClassLoader] (value [WebappClassLoader delegate: false repositories: /WEB-INF/classes/ --> Parent Classloader: org.apache.catalina.loader.standardclassloa...@6ee3849c ]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Dec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a value of type [org.apache.catalina.loader.WebappClassLoader] (value [WebappClassLoader delegate: false repositories: /WEB-INF/classes/ --> Parent Classloader: org.apache.catalina.loader.standardclassloa...@6ee3849c ]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. >>> >>> I don't think this is from Axis, does your code check other sources? >> Right. This is not from Axis. This is from one of the components of the >> app. I am wondering why the threalocal is not cleared at first time? > > Are you generically clearing all ThreadLocals, or just Axis related ones? I am clearing ThreadLocals pertaining to Axis. Also, clearing all threadlocals from context listener during "destroy" process of the webapp. > >> Or is it one each for "threadLocals" and "inheritedthreadLocals"? > > Yes, you need to clear them separately. Right. The code mimics tomcat's cleaning process that exists in "WebappClassLoader" class. Srikanth > > > p > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 15/12/2010 15:37, Srikanth Konjarla wrote: > > > On 12/15/10 7:13 AM, Pid wrote: >> On 15/12/2010 02:40, Srikanth Konjarla wrote: >>> I could catch Axis threadlocals from the app to clean up. However, I >>> have a question wrt following tomcat's log at the time of undeploy of >>> app. The message suggests that it is removing the same threadlocal >>> twice. Is it because it was not removed in the first attempt? >>> >>> - >>> ec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader >>> clearThreadLocalMap >>> SEVERE: A web application created a ThreadLocal with key of type >>> [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a >>> value of type [org.apache.catalina.loader.WebappClassLoader] (value >>> [WebappClassLoader >>> delegate: false >>> repositories: >>> /WEB-INF/classes/ >>> --> Parent Classloader: >>> org.apache.catalina.loader.standardclassloa...@6ee3849c >>> ]) but failed to remove it when the web application was stopped. To >>> prevent a memory leak, the ThreadLocal has been forcibly removed. >>> Dec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader >>> clearThreadLocalMap >>> SEVERE: A web application created a ThreadLocal with key of type >>> [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a >>> value of type [org.apache.catalina.loader.WebappClassLoader] (value >>> [WebappClassLoader >>> delegate: false >>> repositories: >>> /WEB-INF/classes/ >>> --> Parent Classloader: >>> org.apache.catalina.loader.standardclassloa...@6ee3849c >>> ]) but failed to remove it when the web application was stopped. To >>> prevent a memory leak, the ThreadLocal has been forcibly removed. >>> >> >> I don't think this is from Axis, does your code check other sources? > Right. This is not from Axis. This is from one of the components of the > app. I am wondering why the threalocal is not cleared at first time? Are you generically clearing all ThreadLocals, or just Axis related ones? > Or is it one each for "threadLocals" and "inheritedthreadLocals"? Yes, you need to clear them separately. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 12/15/10 7:13 AM, Pid wrote: > On 15/12/2010 02:40, Srikanth Konjarla wrote: >> I could catch Axis threadlocals from the app to clean up. However, I >> have a question wrt following tomcat's log at the time of undeploy of >> app. The message suggests that it is removing the same threadlocal >> twice. Is it because it was not removed in the first attempt? >> >> - >> ec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader >> clearThreadLocalMap >> SEVERE: A web application created a ThreadLocal with key of type >> [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a >> value of type [org.apache.catalina.loader.WebappClassLoader] (value >> [WebappClassLoader >> delegate: false >> repositories: >> /WEB-INF/classes/ >> --> Parent Classloader: >> org.apache.catalina.loader.standardclassloa...@6ee3849c >> ]) but failed to remove it when the web application was stopped. To >> prevent a memory leak, the ThreadLocal has been forcibly removed. >> Dec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader >> clearThreadLocalMap >> SEVERE: A web application created a ThreadLocal with key of type >> [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a >> value of type [org.apache.catalina.loader.WebappClassLoader] (value >> [WebappClassLoader >> delegate: false >> repositories: >> /WEB-INF/classes/ >> --> Parent Classloader: >> org.apache.catalina.loader.standardclassloa...@6ee3849c >> ]) but failed to remove it when the web application was stopped. To >> prevent a memory leak, the ThreadLocal has been forcibly removed. >> > > I don't think this is from Axis, does your code check other sources? Right. This is not from Axis. This is from one of the components of the app. I am wondering why the threalocal is not cleared at first time? Or is it one each for "threadLocals" and "inheritedthreadLocals"? Srikanth > > > p > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 15/12/2010 02:40, Srikanth Konjarla wrote: > I could catch Axis threadlocals from the app to clean up. However, I > have a question wrt following tomcat's log at the time of undeploy of > app. The message suggests that it is removing the same threadlocal > twice. Is it because it was not removed in the first attempt? > > - > ec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a > value of type [org.apache.catalina.loader.WebappClassLoader] (value > [WebappClassLoader > delegate: false > repositories: > /WEB-INF/classes/ > --> Parent Classloader: > org.apache.catalina.loader.standardclassloa...@6ee3849c > ]) but failed to remove it when the web application was stopped. To > prevent a memory leak, the ThreadLocal has been forcibly removed. > Dec 15, 2010 1:05:28 AM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.threadlo...@14213040]) and a > value of type [org.apache.catalina.loader.WebappClassLoader] (value > [WebappClassLoader > delegate: false > repositories: > /WEB-INF/classes/ > --> Parent Classloader: > org.apache.catalina.loader.standardclassloa...@6ee3849c > ]) but failed to remove it when the web application was stopped. To > prevent a memory leak, the ThreadLocal has been forcibly removed. > I don't think this is from Axis, does your code check other sources? p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
f Axis - I don't think v1 is >>> expected to do any more releases. >>> >>> >>> p >>> >>> >>>> Srikanth >>>>> >>>>> >>>>> p >>>>> >>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>>>>> clearThreadLocalMap >>>>>> SEVERE: A web application created a ThreadLocal with key of type >>>>>> [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a >>>>>> value of type [org.apache.catalina.loader.WebappClassLoader] (value >>>>>> [WebappClassLoader >>>>>> delegate: false >>>>>> repositories: >>>>>> /WEB-INF/classes/ >>>>>> --> Parent Classloader: >>>>>> org.apache.catalina.loader.standardclassloa...@1eb3319f >>>>>> ]) but failed to remove it when the web application was stopped. To >>>>>> prevent a memory leak, the ThreadLocal has been forcibly removed. >>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>>>>> clearThreadLocalMap >>>>>> SEVERE: A web application created a ThreadLocal with key of type >>>>>> [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value >>>>>> [org.apache.xml.security.utils.unsyncbytearrayoutputstrea...@775d1479]) >>>>>> and a value of type [byte[]] (value [...@5faeed4]) but failed to remove >>>>>> it when the web application was stopped. To prevent a memory leak, the >>>>>> ThreadLocal has been forcibly removed. >>>>>> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >>>>>> clearThreadLocalMap >>>>>> SEVERE: A web application created a ThreadLocal with key of type >>>>>> [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value >>>>>> [org.apache.axis.utils.xmlutils$threadlocaldocumentbuil...@3a0ee334]) >>>>>> and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value >>>>>> [org.apache.xerces.jaxp.documentbuilderi...@61583db6]) but failed to >>>>>> remove it when the web application was stopped. To prevent a memory >>>>>> leak, the ThreadLocal has been forcibly removed. >>>>>> >>>>>> >>>>>> Srikanth >>>>>> >>>>>> On 12/11/10 2:26 AM, Pid wrote: >>>>>>> On 12/11/10 9:25 AM, Srikanth Konjarla wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: >>>>>>>> >>>>>>>>> On 11 Dec 2010, at 01:07, Srikanth Konjarla >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> BTW, I see some similarities with the following. >>>>>>>>>> >>>>>>>>>> https://jira.springframework.org/browse/SWS-533 >>>>>>>>> >>>>>>>>> Similarities in which sense? >>>>>>>>> The issue was closed as invalid. >>>>>>>> The similarity is that the thread is retained. >>>>>>> >>>>>>> This is not a problem. The threads exist in a pool and are reused. >>>>>>> >>>>>>> Chuck explained why the WAIT status isn't a problem. >>>>>>> >>>>>>> However, other than the case is invalid it does not provide detail why >>>>>>> it is invalid and why the thread is in WAIT state. >>>>>>> >>>>>>> Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to >>>>>>> your problem. >>>>>>> >>>>>>> If the thread is reused by tomcat, what happened to the references from >>>>>>> previous cycle. >>>>>>> >>>>>>> Which references? The ThreadLocals? >>>>>>> >>>>>>> If a ThreadLocal is created by an application, but not cleaned up, the >>>>>>> thread still contains the reference in the internal map of ThreadLocals. >>>>>&
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 13/12/2010 12:23, Pid wrote: > On 12/12/2010 18:18, Srikanth Konjarla wrote: >> >> >> On 12/12/10 8:28 AM, Pid * wrote: >>> On 11 Dec 2010, at 21:39, Srikanth Konjarla >>> wrote: >>> On Dec 11, 2010, at 1:04 PM, "Pid *" wrote: > On 11 Dec 2010, at 20:02, Srikanth Konjarla > wrote: > >> Pid, >> >> Thanks for your patience. Here is the output from catalina.out file >> while the webapp is being undeployed. As you can see there are few >> threadLocals that are cleaned up. >> >> -- >> Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader >> clearThreadLocalMap >> SEVERE: A web application created a ThreadLocal with key of type >> [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a >> value of type [org.apache.catalina.loader.WebappClassLoader] (value >> [WebappClassLoader >> delegate: false >> repositories: >> /WEB-INF/classes/ >> --> Parent Classloader: >> org.apache.catalina.loader.standardclassloa...@1eb3319f >> ]) but failed to remove it when the web application was stopped. To >> prevent a memory leak, the ThreadLocal has been forcibly removed. > > The above means that something is storing the WebappClassLoader in a > ThreadLocal. Is your app doing this? > > The two below I know about and I believe are defects in Axis 1.4. The only component that uses threadlocals in my app is Axis and I believe it is not cleaning up after completely. >>> >>> I agree. >>> Originally, I have started on tracking down Axis threads that are responsible. In this case, Axis is performing webservice client operations. >>> >>> Then you are also seeing an additional problem - namely that the >>> classloader itself is being stored in a threadlocal. I haven't seen >>> Axis do that so far. >> You are correct. I think it is Spring framework (also used in the >> application) that is storing classloader in threadlocal. > > Which version of Spring Framework are you using and how do you arrive at > that conclusion? > > Are you using any other libraries and if so, which exact versions are > you using? > > >> BTW, do you >> have any suggestions on where/how to track Axis threads so that I can >> clean them up from my app? > > I'm still looking into it - but I'll post here when I've got an answer. In the meantime you might consider moving to Axis2... p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
> [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value >>>>> [org.apache.axis.utils.xmlutils$threadlocaldocumentbuil...@3a0ee334]) >>>>> and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value >>>>> [org.apache.xerces.jaxp.documentbuilderi...@61583db6]) but failed to >>>>> remove it when the web application was stopped. To prevent a memory >>>>> leak, the ThreadLocal has been forcibly removed. >>>>> >>>>> >>>>> Srikanth >>>>> >>>>> On 12/11/10 2:26 AM, Pid wrote: >>>>>> On 12/11/10 9:25 AM, Srikanth Konjarla wrote: >>>>>>> >>>>>>> >>>>>>> On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: >>>>>>> >>>>>>>> On 11 Dec 2010, at 01:07, Srikanth Konjarla >>>>>>>> wrote: >>>>>>>> >>>>>>>>> BTW, I see some similarities with the following. >>>>>>>>> >>>>>>>>> https://jira.springframework.org/browse/SWS-533 >>>>>>>> >>>>>>>> Similarities in which sense? >>>>>>>> The issue was closed as invalid. >>>>>>> The similarity is that the thread is retained. >>>>>> >>>>>> This is not a problem. The threads exist in a pool and are reused. >>>>>> >>>>>> Chuck explained why the WAIT status isn't a problem. >>>>>> >>>>>> However, other than the case is invalid it does not provide detail why >>>>>> it is invalid and why the thread is in WAIT state. >>>>>> >>>>>> Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to >>>>>> your problem. >>>>>> >>>>>> If the thread is reused by tomcat, what happened to the references from >>>>>> previous cycle. >>>>>> >>>>>> Which references? The ThreadLocals? >>>>>> >>>>>> If a ThreadLocal is created by an application, but not cleaned up, the >>>>>> thread still contains the reference in the internal map of ThreadLocals. >>>>>> If the reference is to a class from a webapp which is subsequently >>>>>> unloaded, then the classloader will not be garbage collected. >>>>>> >>>>>> >>>>>> I would be interested in learning. >>>>>> >>>>>> PLEASE post the leak warnings from your logs. >>>>>> >>>>>> >>>>>> p >>>>>> >>>>>> >>>>>> >>>>>>>> You're using Axis 1.4 which I've been looking at recently, because I >>>>>>>> believe it causes a leak. >>>>>>> I believe the same but wanted to make sure that leaking threads are >>>>>>> captured and cleaned during the undeploy process. Additionally, I was >>>>>>> wondering how I can detect and locate runaway Axis threads with thread >>>>>>> locals. >>>>>>>> >>>>>>>> Please post the warnings from your logs so I can see of it's the same >>>>>>>> problem. >>>>>>> Will do. >>>>>>> >>>>>>> Srikanth >>>>>>>> >>>>>>>> >>>>>>>> p >>>>>>>> >>>>>>>>> >>>>>>>>> Srikanth >>>>>>>>> >>>>>>>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>>>>>>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>>>>>>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object >>>>>>>>>>> monitor) >>>>>>>>>> >>>>>>>>>>> as part of the thread local cleanup process at the time of undeploy, >>>>>>>>>>> tomcat removes few threadLocals but misses the threadLocals for the >>>>>>>>>>> thread in question. I am wondering about the tomcat thread still >>>>>>>>>>> being >>>>>>>>>>> in "WAIT" mode and been cleaned up. >>>>>>>>>> >>>>>>>>>> The WAIT mode is normal - the thread is sitting in the pool, waiting >>>>>>>>>> for a request to show up. I believe the ThreadLocal objects are >>>>>>>>>> discarded by letting such infected threads exit and creating new >>>>>>>>>> ones. That's an area still undergoing refinement, so you might want >>>>>>>>>> to try moving up to 6.0.29 and see if it makes any difference. Or >>>>>>>>>> try sending in enough concurrent requests to get all the threads >>>>>>>>>> busy and see if the references disappear after that. >>>>>>>>>> >>>>>>>>>> - Chuck >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE >>>>>>>>>> PROPRIETARY MATERIAL and is thus for use only by the intended >>>>>>>>>> recipient. If you received this in error, please contact the sender >>>>>>>>>> and delete the e-mail and its attachments from all computers. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> - >>>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
gt;>> On 12/11/10 9:25 AM, Srikanth Konjarla wrote: >>>>>> >>>>>> >>>>>> On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: >>>>>> >>>>>>> On 11 Dec 2010, at 01:07, Srikanth Konjarla >>>>>>> wrote: >>>>>>> >>>>>>>> BTW, I see some similarities with the following. >>>>>>>> >>>>>>>> https://jira.springframework.org/browse/SWS-533 >>>>>>> >>>>>>> Similarities in which sense? >>>>>>> The issue was closed as invalid. >>>>>> The similarity is that the thread is retained. >>>>> >>>>> This is not a problem. The threads exist in a pool and are reused. >>>>> >>>>> Chuck explained why the WAIT status isn't a problem. >>>>> >>>>> However, other than the case is invalid it does not provide detail why >>>>> it is invalid and why the thread is in WAIT state. >>>>> >>>>> Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to >>>>> your problem. >>>>> >>>>> If the thread is reused by tomcat, what happened to the references from >>>>> previous cycle. >>>>> >>>>> Which references? The ThreadLocals? >>>>> >>>>> If a ThreadLocal is created by an application, but not cleaned up, the >>>>> thread still contains the reference in the internal map of ThreadLocals. >>>>> If the reference is to a class from a webapp which is subsequently >>>>> unloaded, then the classloader will not be garbage collected. >>>>> >>>>> >>>>> I would be interested in learning. >>>>> >>>>> PLEASE post the leak warnings from your logs. >>>>> >>>>> >>>>> p >>>>> >>>>> >>>>> >>>>>>> You're using Axis 1.4 which I've been looking at recently, because I >>>>>>> believe it causes a leak. >>>>>> I believe the same but wanted to make sure that leaking threads are >>>>>> captured and cleaned during the undeploy process. Additionally, I was >>>>>> wondering how I can detect and locate runaway Axis threads with thread >>>>>> locals. >>>>>>> >>>>>>> Please post the warnings from your logs so I can see of it's the same >>>>>>> problem. >>>>>> Will do. >>>>>> >>>>>> Srikanth >>>>>>> >>>>>>> >>>>>>> p >>>>>>> >>>>>>>> >>>>>>>> Srikanth >>>>>>>> >>>>>>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>>>>>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>>>>>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object >>>>>>>>>> monitor) >>>>>>>>> >>>>>>>>>> as part of the thread local cleanup process at the time of undeploy, >>>>>>>>>> tomcat removes few threadLocals but misses the threadLocals for the >>>>>>>>>> thread in question. I am wondering about the tomcat thread still >>>>>>>>>> being >>>>>>>>>> in "WAIT" mode and been cleaned up. >>>>>>>>> >>>>>>>>> The WAIT mode is normal - the thread is sitting in the pool, waiting >>>>>>>>> for a request to show up. I believe the ThreadLocal objects are >>>>>>>>> discarded by letting such infected threads exit and creating new >>>>>>>>> ones. That's an area still undergoing refinement, so you might want >>>>>>>>> to try moving up to 6.0.29 and see if it makes any difference. Or >>>>>>>>> try sending in enough concurrent requests to get all the threads busy >>>>>>>>> and see if the references disappear after that. >>>>>>>>> >>>>>>>>> - Chuck >>>>>>>>> >>>>>>>>> >>>>>>>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE >>>>>>>>> PROPRIETARY MATERIAL and is thus for use only by the intended >>>>>>>>> recipient. If you received this in error, please contact the sender >>>>>>>>> and delete the e-mail and its attachments from all computers. >>>>>>>>> >>>>>>>>> >>>>>>>>> - >>>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>>> >>>>>>> >>>>>>> - >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>> >>>>>> >>>>>> - >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> >>>>> >>>> >>>> - >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
is that the thread is retained. >>>> >>>> This is not a problem. The threads exist in a pool and are reused. >>>> >>>> Chuck explained why the WAIT status isn't a problem. >>>> >>>> However, other than the case is invalid it does not provide detail why >>>> it is invalid and why the thread is in WAIT state. >>>> >>>> Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to >>>> your problem. >>>> >>>> If the thread is reused by tomcat, what happened to the references from >>>> previous cycle. >>>> >>>> Which references? The ThreadLocals? >>>> >>>> If a ThreadLocal is created by an application, but not cleaned up, the >>>> thread still contains the reference in the internal map of ThreadLocals. >>>> If the reference is to a class from a webapp which is subsequently >>>> unloaded, then the classloader will not be garbage collected. >>>> >>>> >>>> I would be interested in learning. >>>> >>>> PLEASE post the leak warnings from your logs. >>>> >>>> >>>> p >>>> >>>> >>>> >>>>>> You're using Axis 1.4 which I've been looking at recently, because I >>>>>> believe it causes a leak. >>>>> I believe the same but wanted to make sure that leaking threads are >>>>> captured and cleaned during the undeploy process. Additionally, I was >>>>> wondering how I can detect and locate runaway Axis threads with thread >>>>> locals. >>>>>> >>>>>> Please post the warnings from your logs so I can see of it's the same >>>>>> problem. >>>>> Will do. >>>>> >>>>> Srikanth >>>>>> >>>>>> >>>>>> p >>>>>> >>>>>>> >>>>>>> Srikanth >>>>>>> >>>>>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>>>>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>>>>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object >>>>>>>>> monitor) >>>>>>>> >>>>>>>>> as part of the thread local cleanup process at the time of undeploy, >>>>>>>>> tomcat removes few threadLocals but misses the threadLocals for the >>>>>>>>> thread in question. I am wondering about the tomcat thread still being >>>>>>>>> in "WAIT" mode and been cleaned up. >>>>>>>> >>>>>>>> The WAIT mode is normal - the thread is sitting in the pool, waiting >>>>>>>> for a request to show up. I believe the ThreadLocal objects are >>>>>>>> discarded by letting such infected threads exit and creating new ones. >>>>>>>> That's an area still undergoing refinement, so you might want to try >>>>>>>> moving up to 6.0.29 and see if it makes any difference. Or try >>>>>>>> sending in enough concurrent requests to get all the threads busy and >>>>>>>> see if the references disappear after that. >>>>>>>> >>>>>>>> - Chuck >>>>>>>> >>>>>>>> >>>>>>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE >>>>>>>> PROPRIETARY MATERIAL and is thus for use only by the intended >>>>>>>> recipient. If you received this in error, please contact the sender >>>>>>>> and delete the e-mail and its attachments from all computers. >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>>> >>>>>>> >>>>>>> - >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>> >>>>>> >>>>>> - >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> >>>>> >>>>> - >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
Locals? >>> >>> If a ThreadLocal is created by an application, but not cleaned up, the >>> thread still contains the reference in the internal map of ThreadLocals. >>> If the reference is to a class from a webapp which is subsequently >>> unloaded, then the classloader will not be garbage collected. >>> >>> >>> I would be interested in learning. >>> >>> PLEASE post the leak warnings from your logs. >>> >>> >>> p >>> >>> >>> >>>>> You're using Axis 1.4 which I've been looking at recently, because I >>>>> believe it causes a leak. >>>> I believe the same but wanted to make sure that leaking threads are >>>> captured and cleaned during the undeploy process. Additionally, I was >>>> wondering how I can detect and locate runaway Axis threads with thread >>>> locals. >>>>> >>>>> Please post the warnings from your logs so I can see of it's the same >>>>> problem. >>>> Will do. >>>> >>>> Srikanth >>>>> >>>>> >>>>> p >>>>> >>>>>> >>>>>> Srikanth >>>>>> >>>>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>>>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>>>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) >>>>>>> >>>>>>>> as part of the thread local cleanup process at the time of undeploy, >>>>>>>> tomcat removes few threadLocals but misses the threadLocals for the >>>>>>>> thread in question. I am wondering about the tomcat thread still being >>>>>>>> in "WAIT" mode and been cleaned up. >>>>>>> >>>>>>> The WAIT mode is normal - the thread is sitting in the pool, waiting >>>>>>> for a request to show up. I believe the ThreadLocal objects are >>>>>>> discarded by letting such infected threads exit and creating new ones. >>>>>>> That's an area still undergoing refinement, so you might want to try >>>>>>> moving up to 6.0.29 and see if it makes any difference. Or try sending >>>>>>> in enough concurrent requests to get all the threads busy and see if >>>>>>> the references disappear after that. >>>>>>> >>>>>>> - Chuck >>>>>>> >>>>>>> >>>>>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE >>>>>>> PROPRIETARY MATERIAL and is thus for use only by the intended >>>>>>> recipient. If you received this in error, please contact the sender and >>>>>>> delete the e-mail and its attachments from all computers. >>>>>>> >>>>>>> >>>>>>> - >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>>> >>>>>> >>>>>> - >>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>>> >>>>> >>>>> - >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>> >>>> - >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
Some short answers : - (for the moment) threads are always reused, even after an application is stopped. - tomcat 6.0.26 tries to clean the threadlocals which may provoke a leak, but 1) it was unsafe and has been disabled by default from 6.0.27 (see https://issues.apache.org/bugzilla/show_bug.cgi?id=48895 ) and 2) some leak through threadlocals might not be detected and thus not cleared, as explained in http://wiki.apache.org/tomcat/MemoryLeakProtection#webappClassInstanceAsThreadLocalIndirectValue - To improve over this situation, threads are now being renewed after an application is stopped. This change is currently only in SVN trunk for tomcat 7. See https://issues.apache.org/bugzilla/show_bug.cgi?id=49159 - please upgrade to 6.0.29 and post the logs again. There has been some improvement in the diagnostics. Sylvain On 11 déc. 2010, at 21:01, Srikanth Konjarla wrote: > Pid, > > Thanks for your patience. Here is the output from catalina.out file > while the webapp is being undeployed. As you can see there are few > threadLocals that are cleaned up. > > -- > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a > value of type [org.apache.catalina.loader.WebappClassLoader] (value > [WebappClassLoader > delegate: false > repositories: >/WEB-INF/classes/ > --> Parent Classloader: > org.apache.catalina.loader.standardclassloa...@1eb3319f > ]) but failed to remove it when the web application was stopped. To > prevent a memory leak, the ThreadLocal has been forcibly removed. > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a > value of type [org.apache.catalina.loader.WebappClassLoader] (value > [WebappClassLoader > delegate: false > repositories: >/WEB-INF/classes/ > --> Parent Classloader: > org.apache.catalina.loader.standardclassloa...@1eb3319f > ]) but failed to remove it when the web application was stopped. To > prevent a memory leak, the ThreadLocal has been forcibly removed. > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value > [org.apache.xml.security.utils.unsyncbytearrayoutputstrea...@775d1479]) > and a value of type [byte[]] (value [...@5faeed4]) but failed to remove > it when the web application was stopped. To prevent a memory leak, the > ThreadLocal has been forcibly removed. > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value > [org.apache.axis.utils.xmlutils$threadlocaldocumentbuil...@3a0ee334]) > and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value > [org.apache.xerces.jaxp.documentbuilderi...@61583db6]) but failed to > remove it when the web application was stopped. To prevent a memory > leak, the ThreadLocal has been forcibly removed. > > > Srikanth - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 11 Dec 2010, at 20:02, Srikanth Konjarla wrote: > Pid, > > Thanks for your patience. Here is the output from catalina.out file > while the webapp is being undeployed. As you can see there are few > threadLocals that are cleaned up. > > -- > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a > value of type [org.apache.catalina.loader.WebappClassLoader] (value > [WebappClassLoader > delegate: false > repositories: >/WEB-INF/classes/ > --> Parent Classloader: > org.apache.catalina.loader.standardclassloa...@1eb3319f > ]) but failed to remove it when the web application was stopped. To > prevent a memory leak, the ThreadLocal has been forcibly removed. The above means that something is storing the WebappClassLoader in a ThreadLocal. Is your app doing this? The two below I know about and I believe are defects in Axis 1.4. p > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a > value of type [org.apache.catalina.loader.WebappClassLoader] (value > [WebappClassLoader > delegate: false > repositories: >/WEB-INF/classes/ > --> Parent Classloader: > org.apache.catalina.loader.standardclassloa...@1eb3319f > ]) but failed to remove it when the web application was stopped. To > prevent a memory leak, the ThreadLocal has been forcibly removed. > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value > [org.apache.xml.security.utils.unsyncbytearrayoutputstrea...@775d1479]) > and a value of type [byte[]] (value [...@5faeed4]) but failed to remove > it when the web application was stopped. To prevent a memory leak, the > ThreadLocal has been forcibly removed. > Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader > clearThreadLocalMap > SEVERE: A web application created a ThreadLocal with key of type > [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value > [org.apache.axis.utils.xmlutils$threadlocaldocumentbuil...@3a0ee334]) > and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value > [org.apache.xerces.jaxp.documentbuilderi...@61583db6]) but failed to > remove it when the web application was stopped. To prevent a memory > leak, the ThreadLocal has been forcibly removed. > > > Srikanth > > On 12/11/10 2:26 AM, Pid wrote: >> On 12/11/10 9:25 AM, Srikanth Konjarla wrote: >>> >>> >>> On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: >>> >>>> On 11 Dec 2010, at 01:07, Srikanth Konjarla >>>> wrote: >>>> >>>>> BTW, I see some similarities with the following. >>>>> >>>>> https://jira.springframework.org/browse/SWS-533 >>>> >>>> Similarities in which sense? >>>> The issue was closed as invalid. >>> The similarity is that the thread is retained. >> >> This is not a problem. The threads exist in a pool and are reused. >> >> Chuck explained why the WAIT status isn't a problem. >> >> However, other than the case is invalid it does not provide detail why >> it is invalid and why the thread is in WAIT state. >> >> Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to >> your problem. >> >> If the thread is reused by tomcat, what happened to the references from >> previous cycle. >> >> Which references? The ThreadLocals? >> >> If a ThreadLocal is created by an application, but not cleaned up, the >> thread still contains the reference in the internal map of ThreadLocals. >> If the reference is to a class from a webapp which is subsequently >> unloaded, then the classloader will not be garbage collected. >> >> >> I would be interested in learning. >> >> PLEASE post the leak warnings from your logs. >> >> >> p >> >> >> >>>> You're using Axis 1.4 which I've been looking at recently, because I >>>> believe it causes a leak. >>> I believe the same but wanted to make sure that leaking threads are >
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
Pid, Thanks for your patience. Here is the output from catalina.out file while the webapp is being undeployed. As you can see there are few threadLocals that are cleaned up. -- Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a value of type [org.apache.catalina.loader.WebappClassLoader] (value [WebappClassLoader delegate: false repositories: /WEB-INF/classes/ --> Parent Classloader: org.apache.catalina.loader.standardclassloa...@1eb3319f ]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.threadlo...@7ee75db3]) and a value of type [org.apache.catalina.loader.WebappClassLoader] (value [WebappClassLoader delegate: false repositories: /WEB-INF/classes/ --> Parent Classloader: org.apache.catalina.loader.standardclassloa...@1eb3319f ]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [org.apache.xml.security.utils.UnsyncByteArrayOutputStream$1] (value [org.apache.xml.security.utils.unsyncbytearrayoutputstrea...@775d1479]) and a value of type [byte[]] (value [...@5faeed4]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Dec 10, 2010 8:46:56 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: A web application created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.xmlutils$threadlocaldocumentbuil...@3a0ee334]) and a value of type [org.apache.xerces.jaxp.DocumentBuilderImpl] (value [org.apache.xerces.jaxp.documentbuilderi...@61583db6]) but failed to remove it when the web application was stopped. To prevent a memory leak, the ThreadLocal has been forcibly removed. Srikanth On 12/11/10 2:26 AM, Pid wrote: > On 12/11/10 9:25 AM, Srikanth Konjarla wrote: >> >> >> On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: >> >>> On 11 Dec 2010, at 01:07, Srikanth Konjarla >>> wrote: >>> >>>> BTW, I see some similarities with the following. >>>> >>>> https://jira.springframework.org/browse/SWS-533 >>> >>> Similarities in which sense? >>> The issue was closed as invalid. >> The similarity is that the thread is retained. > > This is not a problem. The threads exist in a pool and are reused. > > Chuck explained why the WAIT status isn't a problem. > > However, other than the case is invalid it does not provide detail why > it is invalid and why the thread is in WAIT state. > > Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to > your problem. > > If the thread is reused by tomcat, what happened to the references from > previous cycle. > > Which references? The ThreadLocals? > > If a ThreadLocal is created by an application, but not cleaned up, the > thread still contains the reference in the internal map of ThreadLocals. > If the reference is to a class from a webapp which is subsequently > unloaded, then the classloader will not be garbage collected. > > > I would be interested in learning. > > PLEASE post the leak warnings from your logs. > > > p > > > >>> You're using Axis 1.4 which I've been looking at recently, because I >>> believe it causes a leak. >> I believe the same but wanted to make sure that leaking threads are captured >> and cleaned during the undeploy process. Additionally, I was wondering how I >> can detect and locate runaway Axis threads with thread locals. >>> >>> Please post the warnings from your logs so I can see of it's the same >>> problem. >> Will do. >> >> Srikanth >>> >>> >>> p >>> >>>> >>>> Srikanth >>>> >>>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (o
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 12/11/10 9:25 AM, Srikanth Konjarla wrote: > > > On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: > >> On 11 Dec 2010, at 01:07, Srikanth Konjarla >> wrote: >> >>> BTW, I see some similarities with the following. >>> >>> https://jira.springframework.org/browse/SWS-533 >> >> Similarities in which sense? >> The issue was closed as invalid. > The similarity is that the thread is retained. This is not a problem. The threads exist in a pool and are reused. Chuck explained why the WAIT status isn't a problem. However, other than the case is invalid it does not provide detail why it is invalid and why the thread is in WAIT state. Yes, it does. It also refers to JAXB, not Axis. So it's unrelated to your problem. If the thread is reused by tomcat, what happened to the references from previous cycle. Which references? The ThreadLocals? If a ThreadLocal is created by an application, but not cleaned up, the thread still contains the reference in the internal map of ThreadLocals. If the reference is to a class from a webapp which is subsequently unloaded, then the classloader will not be garbage collected. I would be interested in learning. PLEASE post the leak warnings from your logs. p >> You're using Axis 1.4 which I've been looking at recently, because I >> believe it causes a leak. > I believe the same but wanted to make sure that leaking threads are captured > and cleaned during the undeploy process. Additionally, I was wondering how I > can detect and locate runaway Axis threads with thread locals. >> >> Please post the warnings from your logs so I can see of it's the same >> problem. > Will do. > > Srikanth >> >> >> p >> >>> >>> Srikanth >>> >>> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) >>>> >>>>> as part of the thread local cleanup process at the time of undeploy, >>>>> tomcat removes few threadLocals but misses the threadLocals for the >>>>> thread in question. I am wondering about the tomcat thread still being >>>>> in "WAIT" mode and been cleaned up. >>>> >>>> The WAIT mode is normal - the thread is sitting in the pool, waiting for a >>>> request to show up. I believe the ThreadLocal objects are discarded by >>>> letting such infected threads exit and creating new ones. That's an area >>>> still undergoing refinement, so you might want to try moving up to 6.0.29 >>>> and see if it makes any difference. Or try sending in enough concurrent >>>> requests to get all the threads busy and see if the references disappear >>>> after that. >>>> >>>> - Chuck >>>> >>>> >>>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >>>> MATERIAL and is thus for use only by the intended recipient. If you >>>> received this in error, please contact the sender and delete the e-mail >>>> and its attachments from all computers. >>>> >>>> >>>> - >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On Dec 11, 2010, at 1:18 AM, "Pid *" wrote: > On 11 Dec 2010, at 01:07, Srikanth Konjarla > wrote: > >> BTW, I see some similarities with the following. >> >> https://jira.springframework.org/browse/SWS-533 > > Similarities in which sense? > The issue was closed as invalid. The similarity is that the thread is retained. However, other than the case is invalid it does not provide detail why it is invalid and why the thread is in WAIT state. If the thread is reused by tomcat, what happened to the references from previous cycle. I would be interested in learning. > > You're using Axis 1.4 which I've been looking at recently, because I > believe it causes a leak. I believe the same but wanted to make sure that leaking threads are captured and cleaned during the undeploy process. Additionally, I was wondering how I can detect and locate runaway Axis threads with thread locals. > > Please post the warnings from your logs so I can see of it's the same problem. Will do. Srikanth > > > p > >> >> Srikanth >> >> On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) >>> >>>> as part of the thread local cleanup process at the time of undeploy, >>>> tomcat removes few threadLocals but misses the threadLocals for the >>>> thread in question. I am wondering about the tomcat thread still being >>>> in "WAIT" mode and been cleaned up. >>> >>> The WAIT mode is normal - the thread is sitting in the pool, waiting for a >>> request to show up. I believe the ThreadLocal objects are discarded by >>> letting such infected threads exit and creating new ones. That's an area >>> still undergoing refinement, so you might want to try moving up to 6.0.29 >>> and see if it makes any difference. Or try sending in enough concurrent >>> requests to get all the threads busy and see if the references disappear >>> after that. >>> >>> - Chuck >>> >>> >>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >>> MATERIAL and is thus for use only by the intended recipient. If you >>> received this in error, please contact the sender and delete the e-mail and >>> its attachments from all computers. >>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 11 Dec 2010, at 01:07, Srikanth Konjarla wrote: > BTW, I see some similarities with the following. > > https://jira.springframework.org/browse/SWS-533 Similarities in which sense? The issue was closed as invalid. You're using Axis 1.4 which I've been looking at recently, because I believe it causes a leak. Please post the warnings from your logs so I can see of it's the same problem. p > > Srikanth > > On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) >> >>> as part of the thread local cleanup process at the time of undeploy, >>> tomcat removes few threadLocals but misses the threadLocals for the >>> thread in question. I am wondering about the tomcat thread still being >>> in "WAIT" mode and been cleaned up. >> >> The WAIT mode is normal - the thread is sitting in the pool, waiting for a >> request to show up. I believe the ThreadLocal objects are discarded by >> letting such infected threads exit and creating new ones. That's an area >> still undergoing refinement, so you might want to try moving up to 6.0.29 >> and see if it makes any difference. Or try sending in enough concurrent >> requests to get all the threads busy and see if the references disappear >> after that. >> >> - Chuck >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is thus for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all computers. >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
BTW, I see some similarities with the following. https://jira.springframework.org/browse/SWS-533 Srikanth On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) > >> as part of the thread local cleanup process at the time of undeploy, >> tomcat removes few threadLocals but misses the threadLocals for the >> thread in question. I am wondering about the tomcat thread still being >> in "WAIT" mode and been cleaned up. > > The WAIT mode is normal - the thread is sitting in the pool, waiting for a > request to show up. I believe the ThreadLocal objects are discarded by > letting such infected threads exit and creating new ones. That's an area > still undergoing refinement, so you might want to try moving up to 6.0.29 and > see if it makes any difference. Or try sending in enough concurrent requests > to get all the threads busy and see if the references disappear after that. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 12/10/10 4:11 PM, Pid * wrote: > On 10 Dec 2010, at 22:46, Srikanth Konjarla > wrote: > >> Chuck, >> >> You are right. It is apache Axis client that is responsible for >> threadLocals in this case. However, as part of the thread local cleanup >> process at the time of undeploy, tomcat removes few threadLocals but >> misses the threadLocals for the thread in question. I am wondering about >> the tomcat thread still being in "WAIT" mode and been cleaned up. > > Please upgrade to the latest 6.0 release. Monitor your catalina logs > on app unload, look for leak warnings. Will try that. > > Please report them and the version of Axis you're using here. Axis version is 1.4. Actually, the threadLocals in question are coming from Axis' MethodCache. Srikanth > > > p > > > >> >> Srikanth >> >> On 12/10/10 2:36 PM, Caldarale, Charles R wrote: From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] Subject: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) >>> Essentially, the thread has threadLocals object that has few references. >>> >>> And who put the ThreadLocal there? (Hint: it wasn't Tomcat; your webapp - >>> or a library it's using - is behaving badly, and not cleaning up after >>> itself.) >>> >>> - Chuck >>> >>> >>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >>> MATERIAL and is thus for use only by the intended recipient. If you >>> received this in error, please contact the sender and delete the e-mail and >>> its attachments from all computers. >>> >>> >>> - >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 12/10/10 3:41 PM, Caldarale, Charles R wrote: >> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >> Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) > >> as part of the thread local cleanup process at the time of undeploy, >> tomcat removes few threadLocals but misses the threadLocals for the >> thread in question. I am wondering about the tomcat thread still being >> in "WAIT" mode and been cleaned up. > > The WAIT mode is normal - the thread is sitting in the pool, waiting for a > request to show up. Does it mean that the thread is actually released to serve next request? As in thread is recycled for reuse? What happens to data that was referenced during the previous cycle? I believe the ThreadLocal objects are discarded by letting such infected threads exit and creating new ones. I was thinking on the same lines. As I have mentioned earlier, I see few messages in catalina log to the effect that threadLocals are forcibly removed. I was initially embarked on finding out Axis related threads that are leaking by checking threads in destroy method of application but, could not catch the thread in question. That's an area still undergoing refinement, so you might want to try moving up to 6.0.29 and see if it makes any difference. Ok. Will try that. Thanks Srikanth Or try sending in enough concurrent requests to get all the threads busy and see if the references disappear after that. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
On 10 Dec 2010, at 22:46, Srikanth Konjarla wrote: > Chuck, > > You are right. It is apache Axis client that is responsible for > threadLocals in this case. However, as part of the thread local cleanup > process at the time of undeploy, tomcat removes few threadLocals but > misses the threadLocals for the thread in question. I am wondering about > the tomcat thread still being in "WAIT" mode and been cleaned up. Please upgrade to the latest 6.0 release. Monitor your catalina logs on app unload, look for leak warnings. Please report them and the version of Axis you're using here. p > > Srikanth > > On 12/10/10 2:36 PM, Caldarale, Charles R wrote: >>> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >>> Subject: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) >> >>> Essentially, the thread has threadLocals object that has >>> few references. >> >> And who put the ThreadLocal there? (Hint: it wasn't Tomcat; your webapp - >> or a library it's using - is behaving badly, and not cleaning up after >> itself.) >> >> - Chuck >> >> >> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY >> MATERIAL and is thus for use only by the intended recipient. If you received >> this in error, please contact the sender and delete the e-mail and its >> attachments from all computers. >> >> >> - >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] > Subject: Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) > as part of the thread local cleanup process at the time of undeploy, > tomcat removes few threadLocals but misses the threadLocals for the > thread in question. I am wondering about the tomcat thread still being > in "WAIT" mode and been cleaned up. The WAIT mode is normal - the thread is sitting in the pool, waiting for a request to show up. I believe the ThreadLocal objects are discarded by letting such infected threads exit and creating new ones. That's an area still undergoing refinement, so you might want to try moving up to 6.0.29 and see if it makes any difference. Or try sending in enough concurrent requests to get all the threads busy and see if the references disappear after that. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
Chuck, You are right. It is apache Axis client that is responsible for threadLocals in this case. However, as part of the thread local cleanup process at the time of undeploy, tomcat removes few threadLocals but misses the threadLocals for the thread in question. I am wondering about the tomcat thread still being in "WAIT" mode and been cleaned up. Srikanth On 12/10/10 2:36 PM, Caldarale, Charles R wrote: >> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] >> Subject: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) > >> Essentially, the thread has threadLocals object that has >> few references. > > And who put the ThreadLocal there? (Hint: it wasn't Tomcat; your webapp - or > a library it's using - is behaving badly, and not cleaning up after itself.) > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: 6.0.26 java.lang.Thread.State: WAITING (on object monitor)
> From: Srikanth Konjarla [mailto:srikanth.konja...@gmail.com] > Subject: 6.0.26 java.lang.Thread.State: WAITING (on object monitor) > Essentially, the thread has threadLocals object that has > few references. And who put the ThreadLocal there? (Hint: it wasn't Tomcat; your webapp - or a library it's using - is behaving badly, and not cleaning up after itself.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
