Hi, This is a similar blog post explaining the memory leak "Fixing Java's ByteBuffer native memory "leak"" and what i suggest is following.
Patching the leak Tony Printezis <https://twitter.com/TonyPrintezis> submitted a version of the patch he wrote for Twitter <http://mail.openjdk.java.net/pipermail/nio-dev/2016-January/003478.html>, which has been merged into JDK 9. I suggest running all services with -Djdk.nio.maxCachedBufferSize=262144 to ensure the JDK doesn't cache buffers larger than 256 kB. I would really love to have this get set as the default, but unfortunately that seems unlikely. However, if you are running on an older version, the source code for the Java libraries are available as part of OpenJDK <http://openjdk.java.net/>. It is actually quite easy to compile your own version of a single class, and replace it using the -Xbootclasspath/p option: 1. Get a copy of OpenJDK. You should follow the latest directions <http://openjdk.java.net/guide/repositories.html>, but at the time of writing, you can run: hg clone http://hg.openjdk.java.net/jdk8/jdk8/jdk/ 2. Patch jdk/src/share/classes/sun/nio/ch/Util.java. Replace the body of getTemporaryDirectBuffer with return ByteBuffer.allocateDirect(size);. Replace the body of offerFirstTemporaryDirectBuffer and offerLastTemporaryDirectBuffer with "free(buf);". You can also try applying my patch <https://www.evanjones.ca/> with: cd jdk; curl https://raw.githubusercontent.com/evanj/java-bytebuffer-leak/master/remove-bb-cache.diff | patch -p1 3. Compile this file with: mkdir out; javac -d out jdk/src/share/classes/sun/nio/ch/Util.java 4. Put this output directory on Java's "bootclasspath", which replaces the built-in classes, by running: java -Xbootclasspath/p:out (other arguments) [1] https://www.evanjones.ca/java-bytebuffer-leak.html Thanks shammi On Tue, Jun 4, 2019 at 11:15 AM Shammi Jayasinghe <[email protected]> wrote: > Hi, > > Recently with WSO2 EI server and API Manager, We came up with the subject > with some production deployments. When this happens, Either We had to free > up the memory with a restart with following command. > > # sync; echo 3 > /proc/sys/vm/drop_caches > > When researching on this, I could come up with following articles. [1] [2] > where it says the total memory is calculated as follows. > > Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss] > > > So, the max memory depends on number of threads and the ss value. In the > current startup scripts we have not configured the -Xss values AFAIK. So, > Have we come across this kind of issues before? > > Apart from that, I could read following as well. > > Furthermore, the JDK caches one or more DirectByteBuffers per thread, and > by default, there is no limit on the number or size of these buffers. As a > result, if a Java app creates many threads that perform I/O using > HeapByteBuffers, and/or these buffers are big, the JVM process may end up > using a lot of additional native memory that looks like a leak. Native > memory regions used by the given thread are released only when the thread > terminates, *and* subsequently, the GC reaches that thread’s > DirectByteBuffer instance(s). > > > When this issue occurs in the code that you cannot change, like a > third-party library, the only ways to address this problem are: > > 1. > > Reduce the number of I/O threads, if possible > 2. > > In JDK version 1.8u102 or newer, use -Djdk.nio.maxCachedBufferSize JVM > property to limit the per-thread DirectByteBuffer size. > > So, Any idea on how can we configure these parameters? > > > [1] > https://plumbr.io/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx > > [2] > https://dzone.com/articles/troubleshooting-problems-with-native-off-heap-memo > -- > Best Regards, > > * Shammi Jayasinghe* > > > *Senior Technical Lead* > *WSO2, Inc.* > *+1-812-391-7730* > *+1-812-327-3505* > > *http://shammijayasinghe.blogspot.com > <http://shammijayasinghe.blogspot.com>* > > -- Best Regards, * Shammi Jayasinghe* *Senior Technical Lead* *WSO2, Inc.* *+1-812-391-7730* *+1-812-327-3505* *http://shammijayasinghe.blogspot.com <http://shammijayasinghe.blogspot.com>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
