[
https://issues.apache.org/jira/browse/GROOVY-7498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14632260#comment-14632260
]
Max Kuchin commented on GROOVY-7498:
------------------------------------
Another example of VM.native_memory detail.diff, different web app, same
environment. We made a baseline at day 1, please find a detail.diff on day 2:
Native Memory Tracking:
Total: reserved=7026506KB +397530KB, committed=7026334KB +397550KB
- Java Heap (reserved=5447680KB, committed=5447680KB)
(mmap: reserved=5447680KB, committed=5447680KB)
- Class (reserved=6261KB +406KB, committed=6261KB +406KB)
(classes #21810 +230)
(malloc=6261KB +406KB, #73801 +1011)
- Thread (reserved=584KB -153KB, committed=584KB -153KB)
(thread #115 -30)
(stack: reserved=118220KB -30839KB,
committed=118220KB -30839KB)
(malloc=363KB -96KB, #467 -120)
(arena=221KB -57KB, #230 -60)
- Code (reserved=58922KB +265KB, committed=58782KB +285KB)
(malloc=9002KB +265KB, #11430 +211)
(mmap: reserved=49920KB, committed=49780KB +20KB)
- GC (reserved=414375KB +21372KB, committed=414375KB
+21372KB)
(malloc=212195KB +21372KB, #93189 +10399)
(mmap: reserved=202180KB, committed=202180KB)
- Compiler (reserved=382KB -140KB, committed=382KB -140KB)
(malloc=284KB -140KB, #402 -135)
(arena=98KB, #2)
- Internal (reserved=882616KB +392341KB, committed=882584KB
+392341KB)
(malloc=882584KB +392341KB, #1358120 +542334)
(mmap: reserved=32KB, committed=0KB)
- Symbol (reserved=22877KB +17KB, committed=22877KB +17KB)
(malloc=18937KB +17KB, #200719 +345)
(arena=3940KB, #1)
- Memory Tracking (reserved=74080KB +15007KB, committed=74080KB
+15007KB)
(malloc=74080KB +15007KB, #456 -578)
- Pooled Free Chunks (reserved=510KB -741KB, committed=510KB -741KB)
(malloc=510KB -741KB)
Details:
[0x00007ff6cc2428d8] CHeapObj<(unsigned short)1280>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=8426KB +815KB, #82615 +9477)
[0x00007ff6cc24ae89] Arena::operator new(unsigned long, unsigned short)+0x39
(malloc=10KB -2KB, #233 -60)
[0x00007ff6cc24b9e3] ChunkPool::allocate(unsigned long,
AllocFailStrategy::AllocFailEnum)+0x93
(malloc=4408KB -799KB, #369 -89)
[0x00007ff6cc24ba58] CHeapObj<(unsigned short)1792>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=379537KB +158200KB, #1279421 +532818)
[0x00007ff6cc2bbdb9] ArrayAllocator<unsigned long, (unsigned
short)1792>::allocate(unsigned long)+0x49
(malloc=43610KB +4754KB, #74886 +9508)
[0x00007ff6cc3b6258] CHeapObj<(unsigned short)256>::operator new(unsigned long,
unsigned char*)+0x28
(malloc=1497KB +46KB, #70233 +716)
[0x00007ff6cc3c9469] CodeBlob::set_oop_maps(OopMapSet*)+0x49
(malloc=8834KB +252KB, #10047 +162)
[0x00007ff6cc3d08d8] CHeapObj<(unsigned short)1024>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=97KB +13KB, #1146 +49)
[0x00007ff6cc3e7198] CHeapObj<(unsigned short)1536>::operator new(unsigned
long, unsigned char*)+0x28
(malloc=16KB -1KB, #178 -27)
[0x00007ff6cc44f0aa] Deoptimization::fetch_unroll_info_helper(JavaThread*)+0x9da
(malloc=152KB -74KB, #55 -27)
[0x00007ff6cc537bfd] GenericGrowableArray::raw_allocate(int)+0x9d
(malloc=459093KB +229383KB, #2623)
[0x00007ff6cc745bb6] MemBaseline::baseline_malloc_details(MemPointerArray
const*)+0x1f6
(malloc=2KB +2KB, #1 +1)
[0x00007ff6cc7464f3] MemBaseline::baseline_vm_details(MemPointerArray
const*)+0x493
(malloc=15KB +15KB, #1 +1)
[0x00007ff6cc747b20] MemPointerArrayImpl<VMCallsitePointer>::is_full()+0x40
(malloc=34KB +34KB, #1 +1)
[0x00007ff6cc7485a8] MemRecorder::MemRecorder()+0x208
(malloc=5241KB -7018KB, #436 -584)
[0x00007ff6cc74fbe4] MemPointerArrayImpl<MemPointerRecordEx>::is_full()+0x44
(malloc=67960KB +21640KB, #1)
[0x00007ff6cc74ffd6] MemPointerArrayImpl<SeqMemPointerRecordEx>::shrink()+0x86
(malloc=0KB -95KB, #0 -1)
[0x00007ff6cc750044] MemPointerArrayImpl<SeqMemPointerRecordEx>::is_full()+0x44
(malloc=672KB +432KB, #2 +1)
[0x00007ff6cc7c2a55] OopMapCache::OopMapCache()+0x45
(malloc=3198KB +360KB, #2558 +288)
[0x00007ff6cc7d1d88] CHeapObj<(unsigned short)512>::operator new(unsigned long,
unsigned char*)+0x28
(malloc=25KB -6KB, #116 -30)
[0x00007ff6cc7ef76b] ParkEvent::operator new(unsigned long)+0x3b
(malloc=311KB +4KB, #652 +8)
[0x00007ff6cc83a7cf] PtrQueueSet::allocate_buffer()+0xff
(malloc=13679KB +6507KB, #2771 +984)
[0x00007ff6cc8841e5] RSHashTable::RSHashTable(unsigned long)+0x85
(malloc=45793KB +12772KB, #2561 -31)
[0x00007ff6cc88422e] RSHashTable::RSHashTable(unsigned long)+0xce
(malloc=4579KB +1277KB, #2561 -31)
[0x00007ff6cc8c62f3] Symbol::operator new(unsigned long, int, Thread*)+0x53
(malloc=10461KB +17KB, #199730 +345)
[0x00007ff6cc9059b6] Thread::allocate(unsigned long, bool, unsigned short)+0x56
(malloc=328KB -87KB, #114 -30)
[0x00007ff6cc946e71] vframeArray::allocate(JavaThread*, int,
GrowableArray<compiledVFrame*>*, RegisterMap*, frame, frame, frame)+0x101
(malloc=112KB -60KB, #55 -27)
[0x00007ff6cc907034] JavaThread::run()+0x24
(mmap: reserved=102800KB -30839KB,
committed=102800KB -30839KB)
[0x00007ff6cc94a536] ReservedSpace::initialize(unsigned long, unsigned long,
bool, c+0x266
(mmap: reserved=11816KB, committed=11676KB +20KB)
At day 3, we can't get detail.diff and got an error "Native memory tracking has
been shutdown due to out of native memory".
We using large pages (-XX:+UseLargePages) so all java heap is in large memory
and we can measure native memory consumption looking at residential memory of
the java process. It was around 400M after start and on 3rd day becomes 2.241G.
> Native memory leak
> ------------------
>
> Key: GROOVY-7498
> URL: https://issues.apache.org/jira/browse/GROOVY-7498
> Project: Groovy
> Issue Type: Bug
> Affects Versions: 2.3.11
> Environment: Ubuntu 14.04 , grails 2.4.5 java 1.7.0_80, tomcat 7.0.56
> Reporter: Max Kuchin
>
> We are experiencing JVM native memory leak after upgrading our web
> application from grails 2.4.4 (groovy 2.3.7) to grails 2.4.5 (groovy 2.3.11).
> It disappears if we replace groovy-all-2.3.11.jar in web application with
> groovy-all-2.3.7.jar.
> As a result of the leak JVM native memory usage can grow to 4G in two weeks
> after start of web application.
> We'd enabled native memory tracking (-XX:NativeMemoryTracking=detail) to
> gather some details. Here are results of native memory summary snapshot made
> with command jcmd pid VM.native_memory summary
> Native memory summary after start of application:
> Total: reserved=6039782KB, committed=6039342KB
> - Java Heap (reserved=5447680KB, committed=5447680KB)
> (mmap: reserved=5447680KB, committed=5447680KB)
> - Class (reserved=2944KB, committed=2944KB)
> (classes #19490)
> (malloc=2944KB, #56242)
> - Thread (reserved=131202KB, committed=131202KB)
> (thread #127)
> (stack: reserved=130556KB, committed=130556KB)
> (malloc=402KB, #515)
> (arena=244KB, #254)
> - Code (reserved=54669KB, committed=54261KB)
> (malloc=4749KB, #7615)
> (mmap: reserved=49920KB, committed=49512KB)
> - GC (reserved=350261KB, committed=350261KB)
> (malloc=148081KB, #28742)
> (mmap: reserved=202180KB, committed=202180KB)
> - Compiler (reserved=399KB, committed=399KB)
> (malloc=301KB, #480)
> (arena=98KB, #2)
> - Internal (reserved=14366KB, committed=14334KB)
> (malloc=14334KB, #21616)
> (mmap: reserved=32KB, committed=0KB)
> - Symbol (reserved=20911KB, committed=20911KB)
> (malloc=17355KB, #180610)
> (arena=3556KB, #1)
> - Memory Tracking (reserved=14986KB, committed=14986KB)
> (malloc=14986KB, #492)
> - Pooled Free Chunks (reserved=2365KB, committed=2365KB)
> (malloc=2365KB)
> One day after start:
>
> Total: reserved=6610742KB , committed=6610518KB
> - Java Heap (reserved=5447680KB, committed=5447680KB)
> (mmap: reserved=5447680KB, committed=5447680KB)
> - Class (reserved=5525KB, committed=5525KB)
> (classes #21209)
> (malloc=5525KB, #71960)
> - Thread (reserved=646KB, committed=646KB)
> (thread #127)
> (stack: reserved=130556KB, committed=130556KB)
> (malloc=402KB +13KB, #515)
> (arena=244KB +8KB, #254)
> - Code (reserved=58182KB, committed=57990KB)
> (malloc=8262KB, #11074)
> (mmap: reserved=49920KB, committed=49728KB)
> - GC (reserved=423906KB, committed=423906KB)
> (malloc=221726KB, #82681)
> (mmap: reserved=202180KB, committed=202180KB)
> - Compiler (reserved=440KB, committed=440KB)
> (malloc=342KB +20KB, #513)
> (arena=98KB, #2)
> - Internal (reserved=472055KB, committed=472023KB)
> (malloc=472023KB, #746397)
> (mmap: reserved=32KB, committed=0KB)
> - Symbol (reserved=21668KB +7KB, committed=21668KB +7KB)
> (malloc=17760KB +7KB, #185311)
> (arena=3908KB, #1)
> - Memory Tracking (reserved=49894KB, committed=49894KB)
> (malloc=49894KB +2464KB, #536)
> - Pooled Free Chunks (reserved=191KB, committed=191KB)
> (malloc=191KB)
> There is substantial increase in Internal memory usage and object count:
> 14366KB, #21616 -> 472023KB, #746397
> Native memory summary, one day after start, same application, different
> groovy version (groovy-all-2.3.7.jar):
> Total: reserved=6098509KB, committed=6099309KB
> - Java Heap (reserved=5447680KB, committed=5447680KB)
> (mmap: reserved=5447680KB, committed=5447680KB)
> - Class (reserved=6263KB, committed=6263KB)
> (classes #21179)
> (malloc=6263KB, #71507)
> - Thread (reserved=117771KB, committed=118799KB)
> (thread #114)
> (stack: reserved=117192KB, committed=118220KB)
> (malloc=360KB, #463)
> (arena=219KB, #228)
> - Code (reserved=58131KB, committed=57935KB)
> (malloc=8211KB, #10858)
> (mmap: reserved=49920KB, committed=49724KB)
> - GC (reserved=383041KB, committed=383041KB)
> (malloc=180861KB, #80128)
> (mmap: reserved=202180KB, committed=202180KB)
> - Compiler (reserved=268KB, committed=268KB)
> (malloc=170KB, #313)
> (arena=98KB, #2)
> - Internal (reserved=41336KB, committed=41304KB)
> (malloc=41304KB, #69639)
> (mmap: reserved=32KB, committed=0KB)
> - Symbol (reserved=21696KB, committed=21696KB)
> (malloc=17788KB, #185505)
> (arena=3908KB, #1)
> - Memory Tracking (reserved=22135KB, committed=22135KB)
> (malloc=22135KB, #432)
> - Pooled Free Chunks (reserved=189KB, committed=189KB)
> (malloc=189KB)
> Internal memory consumption, one day after start, comparing groovy 2.3.7 to
> groovy 2.3.11:
> 41336KB, #69639 - groovy 2.3.7
> 472023KB, #746397 - groovy 2.3.11, more than 10 times increase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
