Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark,

On 5/17/13 5:36 AM, Mark Thomas wrote:
 On 17/05/2013 09:28, Michael-O wrote:
 Hi folks,
 
 there's now a follow-up on the issue [1]. Recap:
 JreMemoryLeakPreventionListener reported that my webapp has
 spawned OracleTimeoutPollingThread because I have set a
 QueryTimeout in the JDBC Pool. Mark Thomas noted that this is
 likely a bug in the driver. I have taken action and created a
 service request with Oracle.
 
 My personal analysis with VisualVM: The thread is spawned when
 the first query is run. The thread keeps running when the webapp
 is shut down or undeployed. Classes remain in memory. The JDBC
 Pool does a simple Class.forName to load the driver, it neither
 uses the DriverManager loading nor does it actively unload the
 driver.
 
 Oracle answer: They were able to reproduce the issue. The
 technical analysis says:
 
 Hi Michael,
 
 I confirmned internally that this message from Tomcat can be
 ignored as there is no risk of any real leak from
 OracleTimeoutPollingThread thread. This thread is related to
 the JDBC driver which may be used by many apps simultaneously.
 Unloading the app does not unload the driver and should not and
 does not stop the thread.
 
 It seems to be the design behaviour.
 
 So my questions would be: 1. Is that still not a false positive?
 
 No, that is not a false positive. The response from Oracle is
 wrong.
 
 There is nothing wrong with the driver creating the thread or the
 thread continuing after the web application has stopped. The
 problem is as follows:
 
 1. When the JDBC driver creates the Thread, the Thread's context
 class loader (as returned by Thread.getContextClassLoader()) is set
 to the web application's class loader.
 
 2. The correct behaviour at this point would be for the Driver to
 set the Thread's context class loader to the class loader that
 loaded the Driver class when the Thread is created.

I have recently observed with MySQL Connector/J that setting the
context ClassLoader to something else (in their case, null) is not
sufficient[1].

The problem is that the Thread's inherited, cached ProtectionDomain
chain is still linked-up with the WebappClassLoader and so switching
the contextClassLoader after thread-creation is not effective.

I think the only way to do it is by having the code that creates the
Thread make a contextClassLoader switch first, then create the Thread,
then switch back.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRr5CuAAoJEBzwKT+lPKRY1OMQAJ6oQ4XjVzXq2imrG52HDix/
E1XNQYJGbjNIs+zS/JL8qjcKWDYGpykYobIZ5gNOCe5I+W419DSTOW9ff8/9djQF
Ui1IeLHe+NAUITxbqgrQjnxED0kkrBwrPDJLbr/+l4nW5ljKDkF9nv0EV+qDgx1L
cJ9ahr4FwsLo7NiHZ/4GjQ8T6gSyp3T2EpqCAE7gqDjYPyYT8i+GPLgxVHSFNzQy
8IYiqe7T7gPQwv8xrYQMks0L9HfnEdMrM4E+S1G5fIOP25VODR9BldMU8YvMKhLo
GXD+/dy0msl9IWoSKGoJMQmNQanh9uj74whmNQ3Ulf6Z7Jviz6OuHiukuNY7Awke
wL5U0J2ExJdFbHw8Jc67TvhmoIkmQ8N/nNI99RvZ5uVIHQ9dsg5hbRuc0V3E8ZJp
JSAVjqEUhHTC7E7DWwBqknAX4F1herCCeIBDJhh0HMJ+MYRWgSZTk/4ccEhEjkU8
1Yj7YFBaWucWGF4SPtac6YjItlWzGYNjVG5jKqkX1cYcIlfIJtuvPGy9nWDDl0Lv
pn1yWoirokwGssv5YEhUxeWTFnJoHD08ii17PJSkwBzL3l84ZVmIZbrodA+xWTNo
oTqVxjxSL0uOB7pI7TmGVWzm8GKLChoiHTwE88jt7KD1wLkRdwMDXdrCVlaZKAjq
pHTL8jry5JtriwOyQHhW
=sQg+
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Chuck,

On 5/22/13 1:42 PM, Caldarale, Charles R wrote:
 From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
 Subject: Re: Follow-up: Possible false-postive with
 JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and
 OracleTimeoutPollingThread
 
 I suspect that the DriverManager will always be loaded by the
 boot ClassLoader, since the default-dispatch for ClassLoaders is
 to chekc the parent first, then check yourself. The
 DriverManager is at the top-level (well, there is primordial, but
 that doesn't really count) ClassLoader so you'll always get
 that.
 
 Terminology alert: you're confusing the boot class loader with the 
 system class loader, and have erroneously labeled the actual boot 
 class loader as primordial.  Generally speaking, the boot class 
 loader is responsible for extracting JRE classes from rt.jar (and 
 others that come with the JRE), while the system class loader
 deals with those specified by the java.class.path setting
 (CLASSPATH for those still stuck on environment variables).

Thanks for the pedantry: I was in fact ignoring the difference between
the system and boot ClassLoaders. However, the primordial ClassLoader
does in fact exist, and does in fact load classes, and is not called
the boot ClassLoader.

Oracle describes the primordial ClassLoader as that which loads pretty
much everything in the java.* package space. Oracle also describes the
system ClassLoader as the delegation root ancestor of all class
loaders.[1]

In practice, the primordial class loader is identified by a null
reference (and is this difficult to inspect) and the system class
loader is represented by an ExtClassLoader. On top of that is an
AppClassLoader. I'll have to play around with some classes loaded via
endorsed directories to see if I can nail-down how to get a Class with
the ExtClassLoader as its class loader instead of AppClassLoader.

DriverManager's ClassLoader is in fact null, the primordial class loader.

You can test it yourself, and discover the ClassLoader hierarchy in
play with a simple program:

public class ClassLoaderTest
{
public static void main(String[] args)
{
printClassLoaders(ClassLoaderTest.class);
printClassLoaders(Thread.class);
printClassLoaders(Object.class);
printClassLoaders(java.sql.DriverManager.class);
}
public static void printClassLoaders(Class cls)
{
ClassLoader cl = cls.getClassLoader();

System.out.println(*** ClassLoaders for 
   + cls.getName() +  ***);
while(null != cl)
{
System.out.println(ClassLoader[
   + cls.getName() + ]:  + cl);
cl = cl.getParent();
}
System.out.println(ClassLoader[
   + cls.getName() + ]:  + cl);
}
}

Run under two different JVMs on my laptop, I get:

$ java -showversion ClassLoaderTest
java version 1.7.0_17
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

*** ClassLoaders for ClassLoaderTest ***
ClassLoader[ClassLoaderTest]: sun.misc.Launcher$AppClassLoader@63644028
ClassLoader[ClassLoaderTest]: sun.misc.Launcher$ExtClassLoader@4ab03512
ClassLoader[ClassLoaderTest]: null
*** ClassLoaders for java.lang.Thread ***
ClassLoader[java.lang.Thread]: null
*** ClassLoaders for java.lang.Object ***
ClassLoader[java.lang.Object]: null
*** ClassLoaders for java.sql.DriverManager ***
ClassLoader[java.sql.DriverManager]: null

$ java ClassLoaderTest
java version 1.6.0_45
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01-451, mixed mode)

*** ClassLoaders for ClassLoaderTest ***
ClassLoader[ClassLoaderTest]: sun.misc.Launcher$AppClassLoader@a6eb38a
ClassLoader[ClassLoaderTest]: sun.misc.Launcher$ExtClassLoader@69cd2e5f
ClassLoader[ClassLoaderTest]: null
*** ClassLoaders for java.lang.Thread ***
ClassLoader[java.lang.Thread]: null
*** ClassLoaders for java.lang.Object ***
ClassLoader[java.lang.Object]: null
*** ClassLoaders for java.sql.DriverManager ***
ClassLoader[java.sql.DriverManager]: null

$ java -showversion ClassLoaderTest
java version 1.6.0_45
Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406)
Java HotSpot(TM) Client VM (build 20.45-b01-451, mixed mode)

*** ClassLoaders for ClassLoaderTest ***
ClassLoader[ClassLoaderTest]: sun.misc.Launcher$AppClassLoader@cf2c80
ClassLoader[ClassLoaderTest]: sun.misc.Launcher$ExtClassLoader@1729854
ClassLoader[ClassLoaderTest]: null
*** ClassLoaders for java.lang.Thread ***
ClassLoader[java.lang.Thread]: null
*** ClassLoaders for java.lang.Object ***
ClassLoader[java.lang.Object]: null
*** ClassLoaders for java.sql.DriverManager ***
ClassLoader[java.sql.DriverManager]: null

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

All,

On 5/23/13 10:40 AM, Christopher Schultz wrote:
 Chuck,
 
 On 5/22/13 1:42 PM, Caldarale, Charles R wrote:
 From: Christopher Schultz [mailto:ch...@christopherschultz.net]
  Subject: Re: Follow-up: Possible false-postive with 
 JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and 
 OracleTimeoutPollingThread
 
 I suspect that the DriverManager will always be loaded by the 
 boot ClassLoader, since the default-dispatch for ClassLoaders
 is to chekc the parent first, then check yourself. The 
 DriverManager is at the top-level (well, there is primordial,
 but that doesn't really count) ClassLoader so you'll always
 get that.
 
 Terminology alert: you're confusing the boot class loader with
 the system class loader, and have erroneously labeled the actual
 boot class loader as primordial.  Generally speaking, the boot
 class loader is responsible for extracting JRE classes from
 rt.jar (and others that come with the JRE), while the system
 class loader deals with those specified by the java.class.path
 setting (CLASSPATH for those still stuck on environment
 variables).
 
 Thanks for the pedantry: I was in fact ignoring the difference
 between the system and boot ClassLoaders. However, the primordial
 ClassLoader does in fact exist, and does in fact load classes, and
 is not called the boot ClassLoader.
 
 Oracle describes the primordial ClassLoader as that which loads
 pretty much everything in the java.* package space. Oracle also
 describes the system ClassLoader as the delegation root ancestor
 of all class loaders.[1]
 
 In practice, the primordial class loader is identified by a null 
 reference (and is this difficult to inspect) and the system class 
 loader is represented by an ExtClassLoader. On top of that is an 
 AppClassLoader. I'll have to play around with some classes loaded
 via endorsed directories to see if I can nail-down how to get a
 Class with the ExtClassLoader as its class loader instead of
 AppClassLoader.
 
 DriverManager's ClassLoader is in fact null, the primordial class
 loader.
 
 You can test it yourself, and discover the ClassLoader hierarchy
 in play with a simple program:
 
 public class ClassLoaderTest { public static void main(String[]
 args) { printClassLoaders(ClassLoaderTest.class); 
 printClassLoaders(Thread.class); printClassLoaders(Object.class); 
 printClassLoaders(java.sql.DriverManager.class); } public static
 void printClassLoaders(Class cls) { ClassLoader cl =
 cls.getClassLoader();
 
 System.out.println(*** ClassLoaders for  + cls.getName() + 
 ***); while(null != cl) { System.out.println(ClassLoader[ +
 cls.getName() + ]:  + cl); cl = cl.getParent(); } 
 System.out.println(ClassLoader[ + cls.getName() + ]:  + cl); } 
 }
 
 Run under two different JVMs on my laptop, I get:
 
 $ java -showversion ClassLoaderTest java version 1.7.0_17 
 Java(TM) SE Runtime Environment (build 1.7.0_17-b02) Java
 HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
 
 *** ClassLoaders for ClassLoaderTest *** 
 ClassLoader[ClassLoaderTest]:
 sun.misc.Launcher$AppClassLoader@63644028 
 ClassLoader[ClassLoaderTest]:
 sun.misc.Launcher$ExtClassLoader@4ab03512 
 ClassLoader[ClassLoaderTest]: null *** ClassLoaders for
 java.lang.Thread *** ClassLoader[java.lang.Thread]: null ***
 ClassLoaders for java.lang.Object *** 
 ClassLoader[java.lang.Object]: null *** ClassLoaders for
 java.sql.DriverManager *** ClassLoader[java.sql.DriverManager]:
 null
 
 $ java ClassLoaderTest java version 1.6.0_45 Java(TM) SE Runtime
 Environment (build 1.6.0_45-b06-451-11M4406) Java HotSpot(TM)
 64-Bit Server VM (build 20.45-b01-451, mixed mode)
 
 *** ClassLoaders for ClassLoaderTest *** 
 ClassLoader[ClassLoaderTest]:
 sun.misc.Launcher$AppClassLoader@a6eb38a 
 ClassLoader[ClassLoaderTest]:
 sun.misc.Launcher$ExtClassLoader@69cd2e5f 
 ClassLoader[ClassLoaderTest]: null *** ClassLoaders for
 java.lang.Thread *** ClassLoader[java.lang.Thread]: null ***
 ClassLoaders for java.lang.Object *** 
 ClassLoader[java.lang.Object]: null *** ClassLoaders for
 java.sql.DriverManager *** ClassLoader[java.sql.DriverManager]:
 null
 
 $ java -showversion ClassLoaderTest java version 1.6.0_45 
 Java(TM) SE Runtime Environment (build 1.6.0_45-b06-451-11M4406) 
 Java HotSpot(TM) Client VM (build 20.45-b01-451, mixed mode)
 
 *** ClassLoaders for ClassLoaderTest *** 
 ClassLoader[ClassLoaderTest]:
 sun.misc.Launcher$AppClassLoader@cf2c80 
 ClassLoader[ClassLoaderTest]:
 sun.misc.Launcher$ExtClassLoader@1729854 
 ClassLoader[ClassLoaderTest]: null *** ClassLoaders for
 java.lang.Thread *** ClassLoader[java.lang.Thread]: null ***
 ClassLoaders for java.lang.Object *** 
 ClassLoader[java.lang.Object]: null *** ClassLoaders for
 java.sql.DriverManager *** ClassLoader[java.sql.DriverManager]:
 null
 
 -chris

Forgot my reference:
http://docs.oracle.com/javase/6/docs/technotes/guides/security/spec/security-spec.doc5.html
-BEGIN PGP SIGNATURE-
Version: 

RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Caldarale, Charles R]

 From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
 Subject: Re: Follow-up: Possible false-postive with 
 JreMemoryLeakPreventionListener 
 and Tomcat's JDBC Pool and OracleTimeoutPollingThread

 Thanks for the pedantry: I was in fact ignoring the difference between
 the system and boot ClassLoaders. However, the primordial ClassLoader
 does in fact exist, and does in fact load classes, and is not called
 the boot ClassLoader.

What you're calling the primordial class loader _is_ the boot class loader, 
identified by a null reference.  (The use of primordial in the page you 
referred to is unusual; it's known as the boot or bootstrap class loader in 
almost all other documentation.)  It's responsible for more than just the 
java.* packages, since it also loads all the com.sun.*, sun.*, and other 
JVM-vendor supplied classes.  I ignored the extensions class loader, since it's 
rarely used and was not pertinent to the topic.

 the system class loader is represented by an ExtClassLoader.  On top 
 of that is an AppClassLoader.

What you're looking at is JVM-internal classes (hence the $ in the names), that 
just happen to be the ones chosen in current versions of the JVM.  The name is 
an implementation detail; the internal class mechanism is used to handle 
privilege issues.  The internal names should not be construed as descriptive of 
the class loader hierarchy.

 On top of that is an AppClassLoader.

Only for programs that supply their own, such as Tomcat.  My comments concerned 
the JVM itself, not Tomcat.

 DriverManager's ClassLoader is in fact null, the primordial class loader.

Unless one configures a replacement DriverManager (I think there's a system 
property for that, but I'm not sure).

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

Am 2013-05-21 20:38, schrieb Mark Thomas:

On 21/05/2013 19:01, Michael-O wrote:

Mark,

I did receive an answer to the issue, citing your findings.
See verbatim copy below:

Hi Michael,

I received the following update from our developer:

**
The theoretical problem is that the thread is holding the app class
loader so it remains reachable and the app is never unloaded. So if the
user loads and unloads the app, the app classes remain in memory.
Subsequent loading and unloading of the app will not get pinned in
memory as the thread is already running so the subsequent loading and
unloading will not cause additional class loaders to be pinned. It is a
fixed size memory leak. It does not grow.

The thread suggests setting the context class loader to be the parent of
the default context class loader. This may work in Tomcat but it's
pretty random. I am researching the problem to determine what if
anything the driver should do to work across all containers. A Tomcat
specific fix is not acceptable.


Almost but not quite. The correct fix is to set the context class loader
of the Thread to the class loader that loaded the Driver or,
alternatively, the class loader that loaded the thread class.


As Mark Thomas pointed out it is critical that the driver is loaded in
the boot or system or container class loader, not the app class loader.
If the driver is in the app class loader then when the app is unloaded
the driver also should be unloaded. Unfortunately this doesn't work. The
driver itself remains reachable so the app class loader is reachable so
the app is never unloaded. At present there is no general way to solve
this problem. The necessary hooks are added in JDK 8.


I'd agree to this point.


Most users put the
driver in the container, not the app, so this is rarely a problem.


I don't agree with this. I often see this with JDBC drivers which is why
Tomcat has a pile of code specifically to unpick the mess this creates.


**

I will certainly have to fill out a bug to have it investigated officially.


That is good news.


Seems like they understood the problem. But I do doubt that this is a
fixed size moemory leak.


I think the point they are trying to make is that it is only the first
instance of the web application to be unloaded that will be pinned in
memory. Subsequent reloads won't trigger a leak. That is correct.


Mark,

here's the next update from Oracle on the issue:

1. For those with Oracle Support access, here's the bug id: 16841748
2. Quote: I don't think any system property is needed. We have a 
solution for the timeout thread that should work for all app containers.


So Oracle is working on a fix, my SR remains open until I receive an 
updated ojdbc.jar.


Michael


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Chuck,

On 5/23/13 12:01 PM, Caldarale, Charles R wrote:
 From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
 Subject: Re: Follow-up: Possible false-postive with
 JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and
 OracleTimeoutPollingThread
 
 Thanks for the pedantry: I was in fact ignoring the difference
 between the system and boot ClassLoaders. However, the primordial
 ClassLoader does in fact exist, and does in fact load classes,
 and is not called the boot ClassLoader.
 
 What you're calling the primordial class loader _is_ the boot
 class loader, identified by a null reference.  (The use of
 primordial in the page you referred to is unusual; it's known as
 the boot or bootstrap class loader in almost all other
 documentation.)  It's responsible for more than just the java.*
 packages, since it also loads all the com.sun.*, sun.*, and other
 JVM-vendor supplied classes.  I ignored the extensions class
 loader, since it's rarely used and was not pertinent to the topic.

While I haven't exactly implemented my own JVM or anything like that,
I have often heard the boot class loader referred to as the primordial
class loader. Either way, we're talking about the same thing.

 the system class loader is represented by an ExtClassLoader.
 On top of that is an AppClassLoader.
 
 What you're looking at is JVM-internal classes (hence the $ in the 
 names), that just happen to be the ones chosen in current versions
 of the JVM.  The name is an implementation detail; the internal
 class mechanism is used to handle privilege issues.  The internal
 names should not be construed as descriptive of the class loader 
 hierarchy.

I wasn't inferring anything from the names of the ClassLoader objects:
just showing what the hierarchy was given the code I wrote.

 On top of that is an AppClassLoader.
 
 Only for programs that supply their own, such as Tomcat. My
 comments concerned the JVM itself, not Tomcat.

Er, I meant that the AppClassLoader (an Oracle-specific thing) was
in the hierarchy.

The hierarchy I was showing was what actually happens when I run that
example class from the command-line, so Tomcat isn't involved.
Oracle's JVM has 2 ClassLoader objects between my class and the boot
class loader. Presumably one of them is for loading extensions
(ExtClassLoader) and then the other (Oracle's AppClassLoader) loads
stuff from the CLASSPATH (or -classpath or java.class.path).

When placing a JAR file into one of the paths found in java.ext.dirs,
the parent ClassLoader for those classes ends up being Oracle's
ExtClassLoader -- just underneath the ClassLoader that manages the
items on the java.class.path.

When I modified java.endorsed.dirs, the classes loaded from those
directories were loaded by the primordial/boot class loader.
Interesting. I would have expected a layer between them, but I guess
it makes sense as you have to be able to override things like
org.xml.*, etc.

So, at least in Oracle's JVM:

java.endorsed.dirs - boot class loader (null)
java.ext.dirs  - another class loader  (ExtClassLoader)
java.class.path- a third class loader  (AppClassLoader)

(In that order: most user classes will start at the bottom and search
upwards.)

- From within Tomcat, of course, there will be a longer chain from
webapp-loaded classes up to the top.

 DriverManager's ClassLoader is in fact null, the primordial class
 loader.
 
 Unless one configures a replacement DriverManager (I think there's
 a system property for that, but I'm not sure).

+1

...but who uses DriverManager anymore, anyway?

- -chris

-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRnpSbAAoJEBzwKT+lPKRYoP0QALrFCAJuCVfCP4BHNLhyGu3F
7YvT1IFS7zSUZlKDf44vOqRfgvQoUdU+Lpj/LTjUxVTEzbvF+5W3LmmQ7Efo3bJd
PA1Td2pqGJkUbcO7R1aSF2Xlaqzz9VzKvfjakenBq4MRzBCkzv06eNOoB+tEnyG1
uq+4CB5t4qS1kdEyZAI9wEPLK7wvHJFWZUk8s/NqK/X+rHA1yhzxWjWCOO+EjjZl
0TsQt44aVWhuL8X+goumQRfDkVjI0lpgOnhRQvnZo3b81H9zhhqvOzD9/JucOf+r
PwuSRIOLlLloCq1sCSNXgG0YT4lkJsReYmXM8mMjSY8EEa7aTeqZQzLfM1AOLNI4
Wk+fCwHDBxpnBP70oKISVRvEFVgvn4nQIDK9JxVikT/p+QhY6CZU7vSzZMnTDPHP
rXaJ2orhp4SIgb1Pr0VZ5RdLn785iEyxzqHVdONik+3XK/1azQ6JKe9TEEXXgVXS
AFuuajDm2yJhsYFeYmsKbgAO5L30jxU3/YhgQ0toyYvPi8n11w4J4hjn9gB6WQ2s
Rw+3nQ+25Jcww5sZ08nJdu+vcK/tWvjz6/B/vGrkpAilRZs5xxKAmwb4V+G5NiBs
Fr3ph5Ys5kV03axHZRRxhjxqfF/GZdeAVHDc57//Vd0S9zuyZ1CWmXVqAJXfrnil
fafc/jc6sAnd865xQ23X
=w6CF
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Caldarale, Charles R]

 From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
 Subject: Re: Follow-up: Possible false-postive with 
 JreMemoryLeakPreventionListener
 and Tomcat's JDBC Pool and OracleTimeoutPollingThread

 While I haven't exactly implemented my own JVM or anything like that,

I have...

 I have often heard the boot class loader referred to as the primordial
 class loader.

Both the JVM and Java Language specs refer to it as the bootstrap class loader:
http://docs.oracle.com/javase/specs/jvms/se7/html/jvms-5.html#jvms-5.3.1

The word primordial does not appear anywhere in the JVM spec, and only as a 
descriptive term for the Object class in the language spec.

 So, at least in Oracle's JVM:

Also in any JVM serious about running Java programs.

 java.endorsed.dirs - boot class loader (null)

The boot[strap] class loader, as noted.

 java.ext.dirs  - another class loader  (ExtClassLoader)

The extensions class loader, rarely used.

 java.class.path- a third class loader  (AppClassLoader)

The system class loader, officially 
(java.lang.ClassLoader.getSystemClassLoader()).

Interestingly enough, the above hierarchy does not appear in any spec I can 
find, although you'd be hard pressed to find a JVM that doesn't implement it 
that way.

Using AppClassLoader as the internal name for the system class loader is 
unfortunate, since it creates confusion with common usage and an actual 
Application Class Loader, as described in this link:

http://docs.oracle.com/cd/E19501-01/819-3659/beadf/index.html

That one does seem like a bit of overkill...

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Mark,

On 5/21/13 2:38 PM, Mark Thomas wrote:
 On 21/05/2013 19:01, Michael-O wrote:
 Mark,
 
 I did receive an answer to the issue, citing your findings. See
 verbatim copy below:
 
 Hi Michael,
 
 I received the following update from our developer:
 
 ** The
 theoretical problem is that the thread is holding the app class 
 loader so it remains reachable and the app is never unloaded. So
 if the user loads and unloads the app, the app classes remain in
 memory. Subsequent loading and unloading of the app will not get
 pinned in memory as the thread is already running so the
 subsequent loading and unloading will not cause additional class
 loaders to be pinned. It is a fixed size memory leak. It does not
 grow.
 
 The thread suggests setting the context class loader to be the
 parent of the default context class loader. This may work in
 Tomcat but it's pretty random. I am researching the problem to
 determine what if anything the driver should do to work across
 all containers. A Tomcat specific fix is not acceptable.
 
 Almost but not quite. The correct fix is to set the context class
 loader of the Thread to the class loader that loaded the Driver

+1

 or, alternatively, the class loader that loaded the thread class.

Do you mean java.lang.Thread (primordial)? Or whatever subclass they
are using? Maybe it would be more accurate to say the ClassLoader of
the runnable being run which covers thread-subclasses or standard
Runnables being run by a standard Thread.

 As Mark Thomas pointed out it is critical that the driver is
 loaded in the boot or system or container class loader, not the
 app class loader. If the driver is in the app class loader then
 when the app is unloaded the driver also should be unloaded.
 Unfortunately this doesn't work. The driver itself remains
 reachable so the app class loader is reachable so the app is
 never unloaded. At present there is no general way to solve this
 problem. The necessary hooks are added in JDK 8.
 
 I'd agree to this point.
 
 Most users put the driver in the container, not the app, so this
 is rarely a problem.
 
 I don't agree with this. I often see this with JDBC drivers which
 is why Tomcat has a pile of code specifically to unpick the mess
 this creates.

They are missing the point: the leak cannot be avoided, even if the
driver is (correctly) placed in the container-level ClassLoader and
not the application. It's frustrating when driver authors take so long
so understand what is going on and how easy (and reasonable) it is to
fix the problem.

 **
 
 I will certainly have to fill out a bug to have it investigated
 officially.
 
 That is good news.
 
 Seems like they understood the problem. But I do doubt that this
 is a fixed size moemory leak.
 
 I think the point they are trying to make is that it is only the
 first instance of the web application to be unloaded that will be
 pinned in memory. Subsequent reloads won't trigger a leak. That is
 correct.

That is little solace to folks who have more Class objects than just
about anything else. With a huge number of support libraries being
used to operate a fairly lean transaction-processing system, a single
reload could bust the heap and/or PermGen.

Just because it's a fixed-size memory leak doesn't mean it isn't worth
fixing. :(

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRnNGtAAoJEBzwKT+lPKRY6GUP/02KynMptUNCIZQp1ZafsfeY
ugNyicVw9BJ8Nw7tmW2vyRvFFNUDeXpZfS6iD5vSNU3QnSg/Sr5qs3oHSQHltjw8
VsBwVwc2Co6wHwR2FrW2LQda3TNHke5XCWmVVvJi+clJvl4zQ8LWINO+hmzOXgPL
O29hStrvkRhT5F5GBMnu9AIdFR8eccgv/BuLmQ3ASnoG7YRRiLSKkFFJuYzbKnFK
VoJwaJBK5QAbjqi/wAWF3mCOgqvzrozuV7RrrJ9Ah/5J74VQoBdIuvPG7oYDXu8n
bZBEIEhLrHvc6wUevXKR3K6xFGQxI+0+Nv5GmqjuehktHx/Cf3Td7c0hIPNw0sgN
VSASIhG0//x35e7lhzroeLjSUgvQNfcYH6d7JKropvkNQOsxV42ISVMVQouKLMJ+
fPEwy1ZyoUrJ6XCrzozyePaL5K18BlL8OGGs5yo5UWxoBonLvh20cJjGAkAh/Er2
a1Dwijc2ulkO4WnJn3RpKaCuEYCC8lBwakqAwLdzC5nJVQD5ZaQPcK8P63Zd5DeM
Kmkli3NEILi8JAaVHIoYB6fmd4VBH4sbGJbccISX65SyVr4HpnUbPcLgjBxlL0PG
RV8MwZ5fGmxpbVrXBbK7sO/a951kib+GJiMU8wlAqs0+cDc5kse/YzOEl+lc8iG1
krw5V+wCqB/NR8e+nTGQ
=lN0r
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Nick Williams]

On May 22, 2013, at 9:09 AM, Christopher Schultz wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256
 
 Mark,
 
 On 5/21/13 2:38 PM, Mark Thomas wrote:
 On 21/05/2013 19:01, Michael-O wrote:
 Mark,
 
 I did receive an answer to the issue, citing your findings. See
 verbatim copy below:
 
 Hi Michael,
 
 I received the following update from our developer:
 
 ** The
 theoretical problem is that the thread is holding the app class 
 loader so it remains reachable and the app is never unloaded. So
 if the user loads and unloads the app, the app classes remain in
 memory. Subsequent loading and unloading of the app will not get
 pinned in memory as the thread is already running so the
 subsequent loading and unloading will not cause additional class
 loaders to be pinned. It is a fixed size memory leak. It does not
 grow.
 
 The thread suggests setting the context class loader to be the
 parent of the default context class loader. This may work in
 Tomcat but it's pretty random. I am researching the problem to
 determine what if anything the driver should do to work across
 all containers. A Tomcat specific fix is not acceptable.
 
 Almost but not quite. The correct fix is to set the context class
 loader of the Thread to the class loader that loaded the Driver
 
 +1
 
 or, alternatively, the class loader that loaded the thread class.
 
 Do you mean java.lang.Thread (primordial)? Or whatever subclass they
 are using? Maybe it would be more accurate to say the ClassLoader of
 the runnable being run which covers thread-subclasses or standard
 Runnables being run by a standard Thread.

Or why not just java.sql.DriverManager.class.getClassLoader()? Since drivers 
are always stored in the DriverManager regardless of what class loader they're 
loaded in (hence why you shouldn't put the driver in WEB-INF/lib), isn't the 
alway-correct solution to use the class loader that loaded the DriverManager 
for this thread?

 
 As Mark Thomas pointed out it is critical that the driver is
 loaded in the boot or system or container class loader, not the
 app class loader. If the driver is in the app class loader then
 when the app is unloaded the driver also should be unloaded.
 Unfortunately this doesn't work. The driver itself remains
 reachable so the app class loader is reachable so the app is
 never unloaded. At present there is no general way to solve this
 problem. The necessary hooks are added in JDK 8.
 
 I'd agree to this point.
 
 Most users put the driver in the container, not the app, so this
 is rarely a problem.
 
 I don't agree with this. I often see this with JDBC drivers which
 is why Tomcat has a pile of code specifically to unpick the mess
 this creates.
 
 They are missing the point: the leak cannot be avoided, even if the
 driver is (correctly) placed in the container-level ClassLoader and
 not the application. It's frustrating when driver authors take so long
 so understand what is going on and how easy (and reasonable) it is to
 fix the problem.
 
 **
 
 I will certainly have to fill out a bug to have it investigated
 officially.
 
 That is good news.
 
 Seems like they understood the problem. But I do doubt that this
 is a fixed size moemory leak.
 
 I think the point they are trying to make is that it is only the
 first instance of the web application to be unloaded that will be
 pinned in memory. Subsequent reloads won't trigger a leak. That is
 correct.
 
 That is little solace to folks who have more Class objects than just
 about anything else. With a huge number of support libraries being
 used to operate a fairly lean transaction-processing system, a single
 reload could bust the heap and/or PermGen.
 
 Just because it's a fixed-size memory leak doesn't mean it isn't worth
 fixing. :(


Agreed.

Nick
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Nick,

On 5/22/13 10:18 AM, Nick Williams wrote:
 
 On May 22, 2013, at 9:09 AM, Christopher Schultz wrote:
 
 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
 
 Mark,
 
 On 5/21/13 2:38 PM, Mark Thomas wrote:
 On 21/05/2013 19:01, Michael-O wrote:
 Mark,
 
 I did receive an answer to the issue, citing your findings.
 See verbatim copy below:
 
 Hi Michael,
 
 I received the following update from our developer:
 
 **
 The theoretical problem is that the thread is holding the app
 class loader so it remains reachable and the app is never
 unloaded. So if the user loads and unloads the app, the app
 classes remain in memory. Subsequent loading and unloading of
 the app will not get pinned in memory as the thread is
 already running so the subsequent loading and unloading will
 not cause additional class loaders to be pinned. It is a
 fixed size memory leak. It does not grow.
 
 The thread suggests setting the context class loader to be
 the parent of the default context class loader. This may work
 in Tomcat but it's pretty random. I am researching the
 problem to determine what if anything the driver should do to
 work across all containers. A Tomcat specific fix is not
 acceptable.
 
 Almost but not quite. The correct fix is to set the context
 class loader of the Thread to the class loader that loaded the
 Driver
 
 +1
 
 or, alternatively, the class loader that loaded the thread
 class.
 
 Do you mean java.lang.Thread (primordial)? Or whatever subclass
 they are using? Maybe it would be more accurate to say the
 ClassLoader of the runnable being run which covers
 thread-subclasses or standard Runnables being run by a standard
 Thread.
 
 Or why not just java.sql.DriverManager.class.getClassLoader()?
 Since drivers are always stored in the DriverManager regardless of
 what class loader they're loaded in (hence why you shouldn't put
 the driver in WEB-INF/lib), isn't the alway-correct solution to use
 the class loader that loaded the DriverManager for this thread?

I suspect that the DriverManager will always be loaded by the boot
ClassLoader, since the default-dispatch for ClassLoaders is to chekc
the parent first, then check yourself. The DriverManager is at the
top-level (well, there is primordial, but that doesn't really count)
ClassLoader so you'll always get that.

I don't think that will work because the boot ClassLoader won't be
able to load the JDBC driver's classes, since they are in the
container's ClassLoader. (boot ClassLoader should be set to
CATLAINA_BASE/bin/*.jar while the container's ClassLoader will be set
to CATALINA_BASE/lib/*.jar).

Also remember that not all JDBC drivers are registering drivers:
some were explicitly-written *not* to register themselves because of
foolishness like this. When you are using a connection-pool, there's
no reason to register the driver with the DriverManager because the
pool itself acts kind of as the driver manager.

For example MySQL's Connector/J com.mysql.jdbc.Driver class extends
NonRegisteringDriver and adds only a static initializer that
basically does this:

 static {
DriverManager.register(new Driver());
 }

So if I wanted to (and I probably should!), I could change my driver
configuration from com.mysql.jdbc.Driver to
com.mysql.jdbc.NonRegisteringDriver and avoid the DriverManager
entirely. (And your suggestion above would not really be valid
anymore, since the DriverManager is not involved).

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJRnPPXAAoJEBzwKT+lPKRYLV8QAMIlybE7cMnT2XQTbmi9bB5+
RZqS+VHTCf6q36senH0KP5DF1b6d1XMcJEfkpGAOUsA7jye50NFbUf5HICOKqX2D
PSOt2GlDISHQvTyG3U2sBBZyxmtfHW9y13t+397qhC2X806WhbJGPWyWID9vCqX0
EcVF02jwTq6aWpI/aKasxuwS3BDpm7EUMloBX7XdHUYO8lR562rA68KWxGZO5CqD
x9Jho7wtUgoj2RlIB4/7mQQKO1k14upS9OFgqVbkLYRA2RLhT5eOHZTBT4jeIOYJ
gcNnF2f3hEx6rXeNaEdapn46wv1h4dVUNy4mvPi1THb0qb82NEsPYNIadV7CRDAT
eIjzLcjruab5FKlMMq8wRppXh7/+Y21Bd4f3xaPB2b9rmiuv0a4ixnpWGg4QH/3C
NZpKM1F+Lv4jkRaF45Je+Pqna3d/WTgN5kMwwVmC0q0f1Dab1nR0FmGBayRGat+/
qHodLD8bWZ2EVHBvmFXr3BHa4HAsNFSx6Wi630EOHuzFyNHUbkLP2MZxJiufM3+h
9uSDyDuoqWY16EauwHsnkouRAA8O56NlgJ8EALyO/+KV7mOFdjrT/O2VTYWIyVrf
sY3Kn+1UtLXEGf42p9K9fjdG/sGMD4PgJPryokg7UkyoxzP2mkWzOL4Rd1YtWzDD
0RraqDL/r9LszWrPilmN
=QT5q
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Nick Williams]

On May 22, 2013, at 11:35 AM, Christopher Schultz wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256
 
 Nick,
 
 On 5/22/13 10:18 AM, Nick Williams wrote:
 
 On May 22, 2013, at 9:09 AM, Christopher Schultz wrote:
 
 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
 
 Mark,
 
 On 5/21/13 2:38 PM, Mark Thomas wrote:
 On 21/05/2013 19:01, Michael-O wrote:
 Mark,
 
 I did receive an answer to the issue, citing your findings.
 See verbatim copy below:
 
 Hi Michael,
 
 I received the following update from our developer:
 
 **
 The theoretical problem is that the thread is holding the app
 class loader so it remains reachable and the app is never
 unloaded. So if the user loads and unloads the app, the app
 classes remain in memory. Subsequent loading and unloading of
 the app will not get pinned in memory as the thread is
 already running so the subsequent loading and unloading will
 not cause additional class loaders to be pinned. It is a
 fixed size memory leak. It does not grow.
 
 The thread suggests setting the context class loader to be
 the parent of the default context class loader. This may work
 in Tomcat but it's pretty random. I am researching the
 problem to determine what if anything the driver should do to
 work across all containers. A Tomcat specific fix is not
 acceptable.
 
 Almost but not quite. The correct fix is to set the context
 class loader of the Thread to the class loader that loaded the
 Driver
 
 +1
 
 or, alternatively, the class loader that loaded the thread
 class.
 
 Do you mean java.lang.Thread (primordial)? Or whatever subclass
 they are using? Maybe it would be more accurate to say the
 ClassLoader of the runnable being run which covers
 thread-subclasses or standard Runnables being run by a standard
 Thread.
 
 Or why not just java.sql.DriverManager.class.getClassLoader()?
 Since drivers are always stored in the DriverManager regardless of
 what class loader they're loaded in (hence why you shouldn't put
 the driver in WEB-INF/lib), isn't the alway-correct solution to use
 the class loader that loaded the DriverManager for this thread?
 
 I suspect that the DriverManager will always be loaded by the boot
 ClassLoader, since the default-dispatch for ClassLoaders is to chekc
 the parent first, then check yourself. The DriverManager is at the
 top-level (well, there is primordial, but that doesn't really count)
 ClassLoader so you'll always get that.
 
 I don't think that will work because the boot ClassLoader won't be
 able to load the JDBC driver's classes, since they are in the
 container's ClassLoader. (boot ClassLoader should be set to
 CATLAINA_BASE/bin/*.jar while the container's ClassLoader will be set
 to CATALINA_BASE/lib/*.jar).
 
 Also remember that not all JDBC drivers are registering drivers:
 some were explicitly-written *not* to register themselves because of
 foolishness like this. When you are using a connection-pool, there's
 no reason to register the driver with the DriverManager because the
 pool itself acts kind of as the driver manager.
 
 For example MySQL's Connector/J com.mysql.jdbc.Driver class extends
 NonRegisteringDriver and adds only a static initializer that
 basically does this:
 
 static {
DriverManager.register(new Driver());
 }
 
 So if I wanted to (and I probably should!), I could change my driver
 configuration from com.mysql.jdbc.Driver to
 com.mysql.jdbc.NonRegisteringDriver and avoid the DriverManager
 entirely. (And your suggestion above would not really be valid
 anymore, since the DriverManager is not involved).
 
 - -chris

Ahh. That makes sense.

And I will be changing my context.xml resource definitions now... :-)

Nick

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Caldarale, Charles R]

 From: Christopher Schultz [mailto:ch...@christopherschultz.net] 
 Subject: Re: Follow-up: Possible false-postive with 
 JreMemoryLeakPreventionListener 
 and Tomcat's JDBC Pool and OracleTimeoutPollingThread

 I suspect that the DriverManager will always be loaded by the boot
 ClassLoader, since the default-dispatch for ClassLoaders is to chekc
 the parent first, then check yourself. The DriverManager is at the
 top-level (well, there is primordial, but that doesn't really count)
 ClassLoader so you'll always get that.

Terminology alert: you're confusing the boot class loader with the system class 
loader, and have erroneously labeled the actual boot class loader as 
primordial.  Generally speaking, the boot class loader is responsible for 
extracting JRE classes from rt.jar (and others that come with the JRE), while 
the system class loader deals with those specified by the java.class.path 
setting (CLASSPATH for those still stuck on environment variables).

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

Am 2013-05-17 14:26, schrieb Mark Thomas:

On 17/05/2013 12:31, Michael-O wrote:

Hi Mark,

thanks again for the detailed answer, details inline.


Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr
Von: Mark Thomas ma...@apache.org
An: Tomcat Users List users@tomcat.apache.org
Betreff: Re: Follow-up: Possible false-postive with 
JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and 
OracleTimeoutPollingThread

On 17/05/2013 09:28, Michael-O wrote:

Hi folks,

there's now a follow-up on the issue [1].
Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned 
OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC Pool.
Mark Thomas noted that this is likely a bug in the driver. I have taken action 
and created a service request with Oracle.

My personal analysis with VisualVM:
The thread is spawned when the first query is run. The thread keeps running 
when the webapp is shut down or undeployed. Classes remain in memory.
The JDBC Pool does a simple Class.forName to load the driver, it neither uses 
the DriverManager loading nor does it actively unload the driver.

Oracle answer:
They were able to reproduce the issue. The technical analysis says:


Hi Michael,

I confirmned internally that this message from Tomcat can be ignored as there is no risk 
of any real leak from OracleTimeoutPollingThread thread.
This thread is related to the JDBC driver which may be used by many apps 
simultaneously. Unloading the app does not unload the driver and should not and 
does not stop the thread.

It seems to be the design behaviour.


So my questions would be:
1. Is that still not a false positive?


No, that is not a false positive. The response from Oracle is wrong.

There is nothing wrong with the driver creating the thread or the thread
continuing after the web application has stopped. The problem is as follows:

1. When the JDBC driver creates the Thread, the Thread's context class
loader (as returned by Thread.getContextClassLoader()) is set to the web
application's class loader.

2. The correct behaviour at this point would be for the Driver to set
the Thread's context class loader to the class loader that loaded the
Driver class when the Thread is created.

3. The memory leak occurs as follows:
- the web application is stopped
- Tomcat clears all references to the web application and its classes
- The web application should be eligible for garbage collection
- The JDBC driver is still loaded (as it should be)
- The JDBC driver retains a reference to the Thread (as it should)
- The thread retains a reference to the web application class loader
(this is the memory leak).

The reference chain is:
a) JDBC driver
   b) Thread
 c) Web application class loader
   d) Every class loaded by the web app

Everything from c) onwards should be eligible for garbage collection but
isn't because of the leak in the Thread that is retaining a reference to
the web application class loader.


This is what I would assume as correct behavior. If the thread is still 
attached ot the WebAppClassLoader, does that mean that the WebAppClassLoader 
cannot be garbage collected?


The correct behaviour is that there is no reference to c) held by b) and
therefore c) onwards would be eligible for GC (assuming the web app has
been stopped). The problem here is that because b) has a reference to
c), c) can't be GC'd and that is a memory leak.



2. Why does the JDBC Pool not unload the driver? That my cause the thread to 
stop after the last app has been stopped/undeployed.


1. Because the driver is in CATALINA_HOME/lib it is available to other
applications.


So first app, loads driver and keeps it in memory. Even if all apps are 
undeployed, driver remains cached and GCed when Tomcat is shut down?


The JDBC driver will be loaded though the services API as soon as any
code references the DriverManager.


2. JDBC drivers are never automatically unloaded. You have to do so
explicitly (not doing so is an other source of memory leaks when the
driver is packaged in WEB-INF/lib).


OK, this won't be the case. Driver is always shared here.


You need to go back to Oracle.


Yes, I will. We're paying probably a lot of money for he company-wide license.


I'd expect so.


From my own experience with Oracle commercial support for things Java

related, you'll have to go through the This is a bug. No it isn't. Yes
it is... cycle several times before you manage to get the issue in
front of someone with the necessary knowledge and skills to correctly
identify this is a bug in Oracle's JDBC driver. I'd recommend escalating
it through your account manager sooner rather than later.

Feel free to point Oracle support to this thread and/or my presentation
on memory leaks. If Oracle support require further help understanding
this issue they are, of course, free to join this mailing list.


Mark,

I did receive an answer to the issue, citing your findings.
See verbatim copy below:

Hi Michael,

I received the following 

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Mark Thomas]

On 21/05/2013 19:01, Michael-O wrote:
 Mark,
 
 I did receive an answer to the issue, citing your findings.
 See verbatim copy below:
 
 Hi Michael,
 
 I received the following update from our developer:
 
 **
 The theoretical problem is that the thread is holding the app class
 loader so it remains reachable and the app is never unloaded. So if the
 user loads and unloads the app, the app classes remain in memory.
 Subsequent loading and unloading of the app will not get pinned in
 memory as the thread is already running so the subsequent loading and
 unloading will not cause additional class loaders to be pinned. It is a
 fixed size memory leak. It does not grow.
 
 The thread suggests setting the context class loader to be the parent of
 the default context class loader. This may work in Tomcat but it's
 pretty random. I am researching the problem to determine what if
 anything the driver should do to work across all containers. A Tomcat
 specific fix is not acceptable.

Almost but not quite. The correct fix is to set the context class loader
of the Thread to the class loader that loaded the Driver or,
alternatively, the class loader that loaded the thread class.

 As Mark Thomas pointed out it is critical that the driver is loaded in
 the boot or system or container class loader, not the app class loader.
 If the driver is in the app class loader then when the app is unloaded
 the driver also should be unloaded. Unfortunately this doesn't work. The
 driver itself remains reachable so the app class loader is reachable so
 the app is never unloaded. At present there is no general way to solve
 this problem. The necessary hooks are added in JDK 8.

I'd agree to this point.

 Most users put the
 driver in the container, not the app, so this is rarely a problem.

I don't agree with this. I often see this with JDBC drivers which is why
Tomcat has a pile of code specifically to unpick the mess this creates.

 **
 
 I will certainly have to fill out a bug to have it investigated officially.

That is good news.

 Seems like they understood the problem. But I do doubt that this is a
 fixed size moemory leak.

I think the point they are trying to make is that it is only the first
instance of the web application to be unloaded that will be pinned in
memory. Subsequent reloads won't trigger a leak. That is correct.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

Am 2013-05-21 20:38, schrieb Mark Thomas:

Seems like they understood the problem. But I do doubt that this is a
fixed size moemory leak.


I think the point they are trying to make is that it is only the first
instance of the web application to be unloaded that will be pinned in
memory. Subsequent reloads won't trigger a leak. That is correct.


Does this mean that only one WebAppClassLoader stays in memory because 
the thread is launched once and attached to that class loader?


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Mark Thomas]

On 21/05/2013 19:47, Michael-O wrote:
 Am 2013-05-21 20:38, schrieb Mark Thomas:
 Seems like they understood the problem. But I do doubt that this is a
 fixed size moemory leak.

 I think the point they are trying to make is that it is only the first
 instance of the web application to be unloaded that will be pinned in
 memory. Subsequent reloads won't trigger a leak. That is correct.
 
 Does this mean that only one WebAppClassLoader stays in memory because
 the thread is launched once and attached to that class loader?

Exactly.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

Hi folks,

there's now a follow-up on the issue [1].
Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned 
OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC Pool.
Mark Thomas noted that this is likely a bug in the driver. I have taken action 
and created a service request with Oracle.

My personal analysis with VisualVM:
The thread is spawned when the first query is run. The thread keeps running 
when the webapp is shut down or undeployed. Classes remain in memory.
The JDBC Pool does a simple Class.forName to load the driver, it neither uses 
the DriverManager loading nor does it actively unload the driver.

Oracle answer:
They were able to reproduce the issue. The technical analysis says:

 Hi Michael,

 I confirmned internally that this message from Tomcat can be ignored as there 
 is no risk of any real leak from OracleTimeoutPollingThread thread.
 This thread is related to the JDBC driver which may be used by many apps 
 simultaneously. Unloading the app does not unload the driver and should not 
 and does not stop the thread.

 It seems to be the design behaviour.

So my questions would be:
1. Is that still not a false positive?
2. Why does the JDBC Pool not unload the driver? That my cause the thread to 
stop after the last app has been stopped/undeployed.

Thanks,

Michael

[1] http://www.mail-archive.com/users@tomcat.apache.org/msg106186.html

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Mark Thomas]

On 17/05/2013 09:28, Michael-O wrote:
 Hi folks,
 
 there's now a follow-up on the issue [1].
 Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned 
 OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC Pool.
 Mark Thomas noted that this is likely a bug in the driver. I have taken 
 action and created a service request with Oracle.
 
 My personal analysis with VisualVM:
 The thread is spawned when the first query is run. The thread keeps running 
 when the webapp is shut down or undeployed. Classes remain in memory.
 The JDBC Pool does a simple Class.forName to load the driver, it neither uses 
 the DriverManager loading nor does it actively unload the driver.
 
 Oracle answer:
 They were able to reproduce the issue. The technical analysis says:
 
 Hi Michael,

 I confirmned internally that this message from Tomcat can be ignored as 
 there is no risk of any real leak from OracleTimeoutPollingThread thread.
 This thread is related to the JDBC driver which may be used by many apps 
 simultaneously. Unloading the app does not unload the driver and should not 
 and does not stop the thread.

 It seems to be the design behaviour.
 
 So my questions would be:
 1. Is that still not a false positive?

No, that is not a false positive. The response from Oracle is wrong.

There is nothing wrong with the driver creating the thread or the thread
continuing after the web application has stopped. The problem is as follows:

1. When the JDBC driver creates the Thread, the Thread's context class
loader (as returned by Thread.getContextClassLoader()) is set to the web
application's class loader.

2. The correct behaviour at this point would be for the Driver to set
the Thread's context class loader to the class loader that loaded the
Driver class when the Thread is created.

3. The memory leak occurs as follows:
- the web application is stopped
- Tomcat clears all references to the web application and its classes
- The web application should be eligible for garbage collection
- The JDBC driver is still loaded (as it should be)
- The JDBC driver retains a reference to the Thread (as it should)
- The thread retains a reference to the web application class loader
(this is the memory leak).

The reference chain is:
a) JDBC driver
  b) Thread
c) Web application class loader
  d) Every class loaded by the web app

Everything from c) onwards should be eligible for garbage collection but
isn't because of the leak in the Thread that is retaining a reference to
the web application class loader.

 2. Why does the JDBC Pool not unload the driver? That my cause the thread to 
 stop after the last app has been stopped/undeployed.

1. Because the driver is in CATALINA_HOME/lib it is available to other
applications.

2. JDBC drivers are never automatically unloaded. You have to do so
explicitly (not doing so is an other source of memory leaks when the
driver is packaged in WEB-INF/lib).

You need to go back to Oracle.

Mark


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

Hi Mark,

thanks again for the detailed answer, details inline.

 Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr
 Von: Mark Thomas ma...@apache.org
 An: Tomcat Users List users@tomcat.apache.org
 Betreff: Re: Follow-up: Possible false-postive with 
 JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and 
 OracleTimeoutPollingThread

 On 17/05/2013 09:28, Michael-O wrote:
  Hi folks,
  
  there's now a follow-up on the issue [1].
  Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned 
  OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC 
  Pool.
  Mark Thomas noted that this is likely a bug in the driver. I have taken 
  action and created a service request with Oracle.
  
  My personal analysis with VisualVM:
  The thread is spawned when the first query is run. The thread keeps running 
  when the webapp is shut down or undeployed. Classes remain in memory.
  The JDBC Pool does a simple Class.forName to load the driver, it neither 
  uses the DriverManager loading nor does it actively unload the driver.
  
  Oracle answer:
  They were able to reproduce the issue. The technical analysis says:
  
  Hi Michael,
 
  I confirmned internally that this message from Tomcat can be ignored as 
  there is no risk of any real leak from OracleTimeoutPollingThread thread.
  This thread is related to the JDBC driver which may be used by many apps 
  simultaneously. Unloading the app does not unload the driver and should 
  not and does not stop the thread.
 
  It seems to be the design behaviour.
  
  So my questions would be:
  1. Is that still not a false positive?
 
 No, that is not a false positive. The response from Oracle is wrong.
 
 There is nothing wrong with the driver creating the thread or the thread
 continuing after the web application has stopped. The problem is as follows:
 
 1. When the JDBC driver creates the Thread, the Thread's context class
 loader (as returned by Thread.getContextClassLoader()) is set to the web
 application's class loader.
 
 2. The correct behaviour at this point would be for the Driver to set
 the Thread's context class loader to the class loader that loaded the
 Driver class when the Thread is created.
 
 3. The memory leak occurs as follows:
 - the web application is stopped
 - Tomcat clears all references to the web application and its classes
 - The web application should be eligible for garbage collection
 - The JDBC driver is still loaded (as it should be)
 - The JDBC driver retains a reference to the Thread (as it should)
 - The thread retains a reference to the web application class loader
 (this is the memory leak).
 
 The reference chain is:
 a) JDBC driver
   b) Thread
 c) Web application class loader
   d) Every class loaded by the web app
 
 Everything from c) onwards should be eligible for garbage collection but
 isn't because of the leak in the Thread that is retaining a reference to
 the web application class loader.

This is what I would assume as correct behavior. If the thread is still 
attached ot the WebAppClassLoader, does that mean that the WebAppClassLoader 
cannot be garbage collected?
 
  2. Why does the JDBC Pool not unload the driver? That my cause the thread 
  to stop after the last app has been stopped/undeployed.
 
 1. Because the driver is in CATALINA_HOME/lib it is available to other
 applications.

So first app, loads driver and keeps it in memory. Even if all apps are 
undeployed, driver remains cached and GCed when Tomcat is shut down?
 
 2. JDBC drivers are never automatically unloaded. You have to do so
 explicitly (not doing so is an other source of memory leaks when the
 driver is packaged in WEB-INF/lib).

OK, this won't be the case. Driver is always shared here.
 
 You need to go back to Oracle.

Yes, I will. We're paying probably a lot of money for he company-wide license.

Thanks again,

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Mark Thomas]

On 17/05/2013 12:31, Michael-O wrote:
 Hi Mark,
 
 thanks again for the detailed answer, details inline.
 
 Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr
 Von: Mark Thomas ma...@apache.org
 An: Tomcat Users List users@tomcat.apache.org
 Betreff: Re: Follow-up: Possible false-postive with 
 JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and 
 OracleTimeoutPollingThread

 On 17/05/2013 09:28, Michael-O wrote:
 Hi folks,

 there's now a follow-up on the issue [1].
 Recap: JreMemoryLeakPreventionListener reported that my webapp has spawned 
 OracleTimeoutPollingThread because I have set a QueryTimeout in the JDBC 
 Pool.
 Mark Thomas noted that this is likely a bug in the driver. I have taken 
 action and created a service request with Oracle.

 My personal analysis with VisualVM:
 The thread is spawned when the first query is run. The thread keeps running 
 when the webapp is shut down or undeployed. Classes remain in memory.
 The JDBC Pool does a simple Class.forName to load the driver, it neither 
 uses the DriverManager loading nor does it actively unload the driver.

 Oracle answer:
 They were able to reproduce the issue. The technical analysis says:

 Hi Michael,

 I confirmned internally that this message from Tomcat can be ignored as 
 there is no risk of any real leak from OracleTimeoutPollingThread thread.
 This thread is related to the JDBC driver which may be used by many apps 
 simultaneously. Unloading the app does not unload the driver and should 
 not and does not stop the thread.

 It seems to be the design behaviour.

 So my questions would be:
 1. Is that still not a false positive?

 No, that is not a false positive. The response from Oracle is wrong.

 There is nothing wrong with the driver creating the thread or the thread
 continuing after the web application has stopped. The problem is as follows:

 1. When the JDBC driver creates the Thread, the Thread's context class
 loader (as returned by Thread.getContextClassLoader()) is set to the web
 application's class loader.

 2. The correct behaviour at this point would be for the Driver to set
 the Thread's context class loader to the class loader that loaded the
 Driver class when the Thread is created.

 3. The memory leak occurs as follows:
 - the web application is stopped
 - Tomcat clears all references to the web application and its classes
 - The web application should be eligible for garbage collection
 - The JDBC driver is still loaded (as it should be)
 - The JDBC driver retains a reference to the Thread (as it should)
 - The thread retains a reference to the web application class loader
 (this is the memory leak).

 The reference chain is:
 a) JDBC driver
   b) Thread
 c) Web application class loader
   d) Every class loaded by the web app

 Everything from c) onwards should be eligible for garbage collection but
 isn't because of the leak in the Thread that is retaining a reference to
 the web application class loader.
 
 This is what I would assume as correct behavior. If the thread is still 
 attached ot the WebAppClassLoader, does that mean that the WebAppClassLoader 
 cannot be garbage collected?

The correct behaviour is that there is no reference to c) held by b) and
therefore c) onwards would be eligible for GC (assuming the web app has
been stopped). The problem here is that because b) has a reference to
c), c) can't be GC'd and that is a memory leak.


 2. Why does the JDBC Pool not unload the driver? That my cause the thread 
 to stop after the last app has been stopped/undeployed.

 1. Because the driver is in CATALINA_HOME/lib it is available to other
 applications.
 
 So first app, loads driver and keeps it in memory. Even if all apps are 
 undeployed, driver remains cached and GCed when Tomcat is shut down?

The JDBC driver will be loaded though the services API as soon as any
code references the DriverManager.

 2. JDBC drivers are never automatically unloaded. You have to do so
 explicitly (not doing so is an other source of memory leaks when the
 driver is packaged in WEB-INF/lib).
 
 OK, this won't be the case. Driver is always shared here.
  
 You need to go back to Oracle.
 
 Yes, I will. We're paying probably a lot of money for he company-wide license.

I'd expect so.

From my own experience with Oracle commercial support for things Java
related, you'll have to go through the This is a bug. No it isn't. Yes
it is... cycle several times before you manage to get the issue in
front of someone with the necessary knowledge and skills to correctly
identify this is a bug in Oracle's JDBC driver. I'd recommend escalating
it through your account manager sooner rather than later.

Feel free to point Oracle support to this thread and/or my presentation
on memory leaks. If Oracle support require further help understanding
this issue they are, of course, free to join this mailing list.

Mark


-
To unsubscribe, e-mail: 

Aw: Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

 Gesendet: Freitag, 17. Mai 2013 um 14:26 Uhr
 Von: Mark Thomas ma...@apache.org

 On 17/05/2013 12:31, Michael-O wrote:
  Hi Mark,
  
  thanks again for the detailed answer, details inline.
  
  Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr
  Von: Mark Thomas ma...@apache.org
  An: Tomcat Users List users@tomcat.apache.org
  Betreff: Re: Follow-up: Possible false-postive with 
  JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and 
  OracleTimeoutPollingThread
 
  On 17/05/2013 09:28, Michael-O wrote:
  Hi folks,
 
  there's now a follow-up on the issue [1].
  Recap: JreMemoryLeakPreventionListener reported that my webapp has 
  spawned OracleTimeoutPollingThread because I have set a QueryTimeout in 
  the JDBC Pool.
  Mark Thomas noted that this is likely a bug in the driver. I have taken 
  action and created a service request with Oracle.
 
  My personal analysis with VisualVM:
  The thread is spawned when the first query is run. The thread keeps 
  running when the webapp is shut down or undeployed. Classes remain in 
  memory.
  The JDBC Pool does a simple Class.forName to load the driver, it neither 
  uses the DriverManager loading nor does it actively unload the driver.
 
  Oracle answer:
  They were able to reproduce the issue. The technical analysis says:
 
  Hi Michael,
 
  I confirmned internally that this message from Tomcat can be ignored as 
  there is no risk of any real leak from OracleTimeoutPollingThread 
  thread.
  This thread is related to the JDBC driver which may be used by many apps 
  simultaneously. Unloading the app does not unload the driver and should 
  not and does not stop the thread.
 
  It seems to be the design behaviour.
 
  So my questions would be:
  1. Is that still not a false positive?
 
  No, that is not a false positive. The response from Oracle is wrong.
 
  There is nothing wrong with the driver creating the thread or the thread
  continuing after the web application has stopped. The problem is as 
  follows:
 
  1. When the JDBC driver creates the Thread, the Thread's context class
  loader (as returned by Thread.getContextClassLoader()) is set to the web
  application's class loader.
 
  2. The correct behaviour at this point would be for the Driver to set
  the Thread's context class loader to the class loader that loaded the
  Driver class when the Thread is created.
 
  3. The memory leak occurs as follows:
  - the web application is stopped
  - Tomcat clears all references to the web application and its classes
  - The web application should be eligible for garbage collection
  - The JDBC driver is still loaded (as it should be)
  - The JDBC driver retains a reference to the Thread (as it should)
  - The thread retains a reference to the web application class loader
  (this is the memory leak).
 
  The reference chain is:
  a) JDBC driver
b) Thread
  c) Web application class loader
d) Every class loaded by the web app
 
  Everything from c) onwards should be eligible for garbage collection but
  isn't because of the leak in the Thread that is retaining a reference to
  the web application class loader.
  
  This is what I would assume as correct behavior. If the thread is still 
  attached ot the WebAppClassLoader, does that mean that the 
  WebAppClassLoader cannot be garbage collected?
 
 The correct behaviour is that there is no reference to c) held by b) and
 therefore c) onwards would be eligible for GC (assuming the web app has
 been stopped). The problem here is that because b) has a reference to
 c), c) can't be GC'd and that is a memory leak.

Exactly, WebAppClassLoader keeps in a stale status.

  2. Why does the JDBC Pool not unload the driver? That my cause the thread 
  to stop after the last app has been stopped/undeployed.
 
  1. Because the driver is in CATALINA_HOME/lib it is available to other
  applications.
  
  So first app, loads driver and keeps it in memory. Even if all apps are 
  undeployed, driver remains cached and GCed when Tomcat is shut down?
 
 The JDBC driver will be loaded though the services API as soon as any
 code references the DriverManager.
 
  2. JDBC drivers are never automatically unloaded. You have to do so
  explicitly (not doing so is an other source of memory leaks when the
  driver is packaged in WEB-INF/lib).
  
  OK, this won't be the case. Driver is always shared here.
   
  You need to go back to Oracle.
  
  Yes, I will. We're paying probably a lot of money for he company-wide 
  license.
 
 I'd expect so.
 
 From my own experience with Oracle commercial support for things Java
 related, you'll have to go through the This is a bug. No it isn't. Yes
 it is... cycle several times before you manage to get the issue in
 front of someone with the necessary knowledge and skills to correctly
 identify this is a bug in Oracle's JDBC driver. I'd recommend escalating
 it through your account manager sooner rather than later.
 
 Feel free to point Oracle support to this thread and/or my 

RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Jeffrey Janner]

 -Original Message-
 From: Mark Thomas [mailto:ma...@apache.org]
 Sent: Friday, May 17, 2013 7:26 AM
 To: Tomcat Users List
 Subject: Re: Follow-up: Possible false-postive with
 JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and
 OracleTimeoutPollingThread
 
 On 17/05/2013 12:31, Michael-O wrote:
  Hi Mark,
 
  thanks again for the detailed answer, details inline.
 
  Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr
  Von: Mark Thomas ma...@apache.org
  An: Tomcat Users List users@tomcat.apache.org
  Betreff: Re: Follow-up: Possible false-postive with
  JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and
  OracleTimeoutPollingThread
 
  On 17/05/2013 09:28, Michael-O wrote:
  Hi folks,
 
  there's now a follow-up on the issue [1].
  Recap: JreMemoryLeakPreventionListener reported that my webapp has
 spawned OracleTimeoutPollingThread because I have set a QueryTimeout in
 the JDBC Pool.
  Mark Thomas noted that this is likely a bug in the driver. I have
 taken action and created a service request with Oracle.
 
  My personal analysis with VisualVM:
  The thread is spawned when the first query is run. The thread keeps
 running when the webapp is shut down or undeployed. Classes remain in
 memory.
  The JDBC Pool does a simple Class.forName to load the driver, it
 neither uses the DriverManager loading nor does it actively unload the
 driver.
 
  Oracle answer:
  They were able to reproduce the issue. The technical analysis says:
 
  Hi Michael,
 
  I confirmned internally that this message from Tomcat can be
 ignored as there is no risk of any real leak from
 OracleTimeoutPollingThread thread.
  This thread is related to the JDBC driver which may be used by
 many apps simultaneously. Unloading the app does not unload the driver
 and should not and does not stop the thread.
 
  It seems to be the design behaviour.
 
  So my questions would be:
  1. Is that still not a false positive?
 
  No, that is not a false positive. The response from Oracle is wrong.
 
  There is nothing wrong with the driver creating the thread or the
  thread continuing after the web application has stopped. The problem
 is as follows:
 
  1. When the JDBC driver creates the Thread, the Thread's context
  class loader (as returned by Thread.getContextClassLoader()) is set
  to the web application's class loader.
 
  2. The correct behaviour at this point would be for the Driver to
 set
  the Thread's context class loader to the class loader that loaded
 the
  Driver class when the Thread is created.
 
  3. The memory leak occurs as follows:
  - the web application is stopped
  - Tomcat clears all references to the web application and its
 classes
  - The web application should be eligible for garbage collection
  - The JDBC driver is still loaded (as it should be)
  - The JDBC driver retains a reference to the Thread (as it should)
  - The thread retains a reference to the web application class loader
  (this is the memory leak).
 
  The reference chain is:
  a) JDBC driver
b) Thread
  c) Web application class loader
d) Every class loaded by the web app
 
  Everything from c) onwards should be eligible for garbage collection
  but isn't because of the leak in the Thread that is retaining a
  reference to the web application class loader.
 
  This is what I would assume as correct behavior. If the thread is
 still attached ot the WebAppClassLoader, does that mean that the
 WebAppClassLoader cannot be garbage collected?
 
 The correct behaviour is that there is no reference to c) held by b)
 and therefore c) onwards would be eligible for GC (assuming the web app
 has been stopped). The problem here is that because b) has a reference
 to c), c) can't be GC'd and that is a memory leak.
 
 
  2. Why does the JDBC Pool not unload the driver? That my cause the
 thread to stop after the last app has been stopped/undeployed.
 
  1. Because the driver is in CATALINA_HOME/lib it is available to
  other applications.
 
  So first app, loads driver and keeps it in memory. Even if all apps
 are undeployed, driver remains cached and GCed when Tomcat is shut
 down?
 
 The JDBC driver will be loaded though the services API as soon as any
 code references the DriverManager.
 
  2. JDBC drivers are never automatically unloaded. You have to do so
  explicitly (not doing so is an other source of memory leaks when the
  driver is packaged in WEB-INF/lib).
 
  OK, this won't be the case. Driver is always shared here.
 
  You need to go back to Oracle.
 
  Yes, I will. We're paying probably a lot of money for he company-wide
 license.
 
 I'd expect so.
 
 From my own experience with Oracle commercial support for things Java
 related, you'll have to go through the This is a bug. No it isn't. Yes
 it is... cycle several times before you manage to get the issue in
 front of someone with the necessary knowledge and skills to correctly
 identify this is a bug in Oracle's JDBC driver. I'd recommend
 escalating it through your 

Aw: RE: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Michael-O]

  -Original Message-
  From: Mark Thomas [mailto:ma...@apache.org]
  Sent: Friday, May 17, 2013 7:26 AM
  To: Tomcat Users List
  Subject: Re: Follow-up: Possible false-postive with
  JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and
  OracleTimeoutPollingThread
  
  On 17/05/2013 12:31, Michael-O wrote:
   Hi Mark,
  
   thanks again for the detailed answer, details inline.
  
   Gesendet: Freitag, 17. Mai 2013 um 11:36 Uhr
   Von: Mark Thomas ma...@apache.org
   An: Tomcat Users List users@tomcat.apache.org
   Betreff: Re: Follow-up: Possible false-postive with
   JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and
   OracleTimeoutPollingThread
  
   On 17/05/2013 09:28, Michael-O wrote:
   Hi folks,
  
   there's now a follow-up on the issue [1].
   Recap: JreMemoryLeakPreventionListener reported that my webapp has
  spawned OracleTimeoutPollingThread because I have set a QueryTimeout in
  the JDBC Pool.
   Mark Thomas noted that this is likely a bug in the driver. I have
  taken action and created a service request with Oracle.
  
   My personal analysis with VisualVM:
   The thread is spawned when the first query is run. The thread keeps
  running when the webapp is shut down or undeployed. Classes remain in
  memory.
   The JDBC Pool does a simple Class.forName to load the driver, it
  neither uses the DriverManager loading nor does it actively unload the
  driver.
  
   Oracle answer:
   They were able to reproduce the issue. The technical analysis says:
  
   Hi Michael,
  
   I confirmned internally that this message from Tomcat can be
  ignored as there is no risk of any real leak from
  OracleTimeoutPollingThread thread.
   This thread is related to the JDBC driver which may be used by
  many apps simultaneously. Unloading the app does not unload the driver
  and should not and does not stop the thread.
  
   It seems to be the design behaviour.
  
   So my questions would be:
   1. Is that still not a false positive?
  
   No, that is not a false positive. The response from Oracle is wrong.
  
   There is nothing wrong with the driver creating the thread or the
   thread continuing after the web application has stopped. The problem
  is as follows:
  
   1. When the JDBC driver creates the Thread, the Thread's context
   class loader (as returned by Thread.getContextClassLoader()) is set
   to the web application's class loader.
  
   2. The correct behaviour at this point would be for the Driver to
  set
   the Thread's context class loader to the class loader that loaded
  the
   Driver class when the Thread is created.
  
   3. The memory leak occurs as follows:
   - the web application is stopped
   - Tomcat clears all references to the web application and its
  classes
   - The web application should be eligible for garbage collection
   - The JDBC driver is still loaded (as it should be)
   - The JDBC driver retains a reference to the Thread (as it should)
   - The thread retains a reference to the web application class loader
   (this is the memory leak).
  
   The reference chain is:
   a) JDBC driver
 b) Thread
   c) Web application class loader
 d) Every class loaded by the web app
  
   Everything from c) onwards should be eligible for garbage collection
   but isn't because of the leak in the Thread that is retaining a
   reference to the web application class loader.
  
   This is what I would assume as correct behavior. If the thread is
  still attached ot the WebAppClassLoader, does that mean that the
  WebAppClassLoader cannot be garbage collected?
  
  The correct behaviour is that there is no reference to c) held by b)
  and therefore c) onwards would be eligible for GC (assuming the web app
  has been stopped). The problem here is that because b) has a reference
  to c), c) can't be GC'd and that is a memory leak.
  
  
   2. Why does the JDBC Pool not unload the driver? That my cause the
  thread to stop after the last app has been stopped/undeployed.
  
   1. Because the driver is in CATALINA_HOME/lib it is available to
   other applications.
  
   So first app, loads driver and keeps it in memory. Even if all apps
  are undeployed, driver remains cached and GCed when Tomcat is shut
  down?
  
  The JDBC driver will be loaded though the services API as soon as any
  code references the DriverManager.
  
   2. JDBC drivers are never automatically unloaded. You have to do so
   explicitly (not doing so is an other source of memory leaks when the
   driver is packaged in WEB-INF/lib).
  
   OK, this won't be the case. Driver is always shared here.
  
   You need to go back to Oracle.
  
   Yes, I will. We're paying probably a lot of money for he company-wide
  license.
  
  I'd expect so.
  
  From my own experience with Oracle commercial support for things Java
  related, you'll have to go through the This is a bug. No it isn't. Yes
  it is... cycle several times before you manage to get the issue in
  front of someone with the 

Re: Follow-up: Possible false-postive with JreMemoryLeakPreventionListener and Tomcat's JDBC Pool and OracleTimeoutPollingThread

[Mark Thomas]

On 17/05/2013 15:34, Jeffrey Janner wrote:

 Michael and Mark - I happened to be reviewing how Oracle handles
 QueryTimeout yesterday on an unrelated issue and came across a
 passage in the JDBC Developers Guide (11g) that covered this
 Monitoring Thread (page E-3). My reading of it was that Oracle
 creates a single monitoring thread per JVM, so if there are multiple
 apps utilizing the Oracle driver, you certainly don't want one app
 shutting down and taking the monitor thread with it. I'm sure this
 one of the reasons that Mark says that the monitor thread should use
 the parent class loader.  Oracle should probably consider this. My
 question though is how is all this affected when the jdbc library is
 loaded on a per-app basis, i.e., it's not shared but loaded from each
 app's WEB-INF/lib folder?  Should the app then unload the driver when
 it shuts down to avoid memory leaks?  Or is there a special process
 that needs to be followed? Jeff

If the driver is in WEB-INF/lib things get really messy, very quickly.
You can make it work if only that application is using the driver (you
have to make sure you manually deregister the driver in a
ServletContextListener). If multiple apps try and use it, or if multiple
apps ship the same Driver, bad things happen.

There are multiple problems:

1. Using a driver automatically adds it to the DriverManager.
DriverManager is a JVM singleton. It maintains a list of loaded drivers
but does not key this list by class loader so there can only ever be one
instance of a Driver loaded even if different web applications ship
different versions of the same Driver.

2. If web app A loads the Driver and web app B tries to use it the
chances of a CNFE or similar are fairly high. You might get away with
it. You probably won't.

3. If webapp A loads the driver, webapp B manages to use it and then
webapp A is undeployed, assuming webapp A does the right thing and
deregisaters it,  webapp A will pull the driver out from underneath
webapp B.

Generally, putting it in CATALINA_HOME/lib is better. That also means
Tomcat can provide database connection pooling.

DriverManager is just one of the many JRE classes that make no
allowances for running in the class loader environments you get in J2EE
containers. LogManager is another.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org