Hi all.
I found that Using the JMX monitor the application will cause the GC time to
grow longer
Problem Description
When you use jmx to remotely monitor your application (such as VisualVM), after
a while (a few days), you will find that the application's GC time will be
longer and longer.
This problem exists in jdk7, jdk8, other versions have not been tried (the
following code is based on jdk7).
Pronlem Causes
When using the jmx monitor, the
javax.management.remote.rmi.RMIConnectionImpl#unwrap method will be called, and
each time a new ClassLoader (CombinedClassLoader) is created. When the newly
created ClassLoader loads the class, first find out if the class has been
loaded from the SystemDictionary according to the ClassLoader object and the
class name. If not, it will find or load it, and then add it to the
SystemDictionary. Because we create a ClassLoader object every time, the
SystemDictionary will definitely not find it, so every time we will put the new
ClassLoader object and the loaded class into the SystemDictionary.
Because our monitoring system(such as VisualVM) returns data in real time, we
constantly create new ClassLoader loading classes. As the time gets longer,
there will be more and more elements in the SystemDictionary. At this point,
the bucket's linked list may be very long, and the efficiency of the search
will be low.
When a young gc occurs, the SystemDictionary is scanned when a strong reference
is processed, so when there are more and more elements in the SystemDictionary,
the young gc will become slower and slower.
Debug Process
When you use jmx to monitor your application, the code goes to the
RMIConnectionImpl implementation class. When it goes to the unwrap method, it
creates a new ClassLoader each time, and then sets the ClassLoader to the
ClassLoader of the current thread context. code show as below.
javax.management.remote.rmi.RMIConnectionImpl#unwrap(
final MarshalledObject<?> mo,
final ClassLoader cl1,
final ClassLoader cl2,
final Class<T> wrappedClass)
|
try {
ClassLoader orderCL = AccessController.doPrivileged(
new PrivilegedExceptionAction<ClassLoader>() {
public ClassLoader run() throws Exception {
return new
CombinedClassLoader(Thread.currentThread().getContextClassLoader(),
new OrderClassLoaders(cl1, cl2));
}
}
);
return unwrap(mo, orderCL, wrappedClass);
} catch (PrivilegedActionException pe) {
|
javax.management.remote.rmi.RMIConnectionImpl#unwrap(
final MarshalledObject<?> mo,
final ClassLoader cl,
final Class<T> wrappedClass)
|
final ClassLoader old = AccessController.doPrivileged(new SetCcl(cl));
try {
return wrappedClass.cast(mo.get());
} catch (ClassNotFoundException cnfe) {
throw new UnmarshalException(cnfe.toString(), cnfe);
} finally {
AccessController.doPrivileged(new SetCcl(old));
}
|
This newly created ClassLoader will eventually be used in the
sun.rmi.server.LoaderHandler#loadClass(java.net.URL[], java.lang.String)
method. code show as below.
|
ClassLoader var2 = getRMIContextClassLoader();
if (loaderLog.isLoggable(Log.VERBOSE)) {
loaderLog.log(Log.VERBOSE, "(thread context class loader: " + var2 +
")");
}
SecurityManager var3 = System.getSecurityManager();
if (var3 == null) {
try {
Class var11 = loadClassForName(var1, false, var2);
if (loaderLog.isLoggable(Log.VERBOSE)) {
loaderLog.log(Log.VERBOSE, "class \"" + var1 + "\" found via
" + "thread context class loader " + "(no security manager: codebase disabled),
" + "defined by " + var11.getClassLoader());
}
return var11;
} catch (ClassNotFoundException var8) {
if (loaderLog.isLoggable(Log.BRIEF)) {
loaderLog.log(Log.BRIEF, "class \"" + var1 + "\" not found
via " + "thread context class loader " + "(no security manager: codebase
disabled)", var8);
}
throw new ClassNotFoundException(var8.getMessage() + " (no
security manager: RMI class loader disabled)", var8.getException());
}
}
|
|
private static ClassLoader getRMIContextClassLoader() {
return Thread.currentThread().getContextClassLoader();
}
|
|
private static Class<?> loadClassForName(String var0, boolean var1, ClassLoader
var2) throws ClassNotFoundException {
if (var2 == null) {
ReflectUtil.checkPackageAccess(var0);
}
return Class.forName(var0, var1, var2);
}
|
Finally, Class.forName will be called. At this point, the loader is the newly
created CombinedClassLoader. After the load is successful, it will be placed in
the SystemDictionary.
SystemDictionary
1. The data structure is a hashtable
2. The default bucket size is 1009 and does not support expansion
3. The way to resolve conflicts is to use a linked list
Young gc scan SystemDictionary code as follows
http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/file/69f46e2dbd83/src/share/vm/memory/sharedHeap.cpp
|
if (!_process_strong_tasks->is_task_claimed(SH_PS_SystemDictionary_oops_do)) {
if (so & SO_AllClasses) {
SystemDictionary::oops_do(roots);
} else if (so & SO_SystemClasses) {
SystemDictionary::always_strong_oops_do(roots);
}
}
|
http://hg.openjdk.java.net/jdk7u/jdk7u/hotspot/file/69f46e2dbd83/src/share/vm/classfile/dictionary.cpp
|
void Dictionary::always_strong_classes_do(OopClosure* blk) {
// Follow all system classes and temporary placeholders in dictionary
for (int index = 0; index < table_size(); index++) {
for (DictionaryEntry *probe = bucket(index);
probe != NULL;
probe = probe->next()) {
klassOop e = probe->klass();
oop class_loader = probe->loader();
if (is_strongly_reachable(class_loader, e)) {
blk->do_oop((oop*)probe->klass_addr());
if (class_loader != NULL) {
blk->do_oop(probe->loader_addr());
}
probe->protection_domain_set_oops_do(blk);
}
}
}
}
|
Hanyu King
