One last thing… Can you tell me what the capacity() of the “leaked” buffer is ?
> On 20 Jul 2016, at 19:11, Norman Maurer <[email protected]> wrote: > > Thanks Chris, > > this sounds really “fishy”. Let me try to debug this a bit more (not sure yet > how tho). > >> On 20 Jul 2016, at 19:06, Chris Conroy <[email protected] >> <mailto:[email protected]>> wrote: >> >> This leak resulted in exhaustion of 4G of NIO memory. The same application >> under Netty 3 only ever uses <200MB of NIO memory. I have run several >> experiments where I take some traffic, disable the node from serving >> traffic, and then force a full GC. The allocated NIO memory does not return >> to normal levels. >> >> Spot checking the byte buf handles from my heap dump, I see lots (all?) >> `recycle` and `lastRecycleIds` of `int -2147483648 = 0x80000000` >> >> The experiment yesterday with the recycler disabled did not result in any >> such leaks over several hours of traffic. >> >> On Wed, Jul 20, 2016 at 12:41 AM, 'Norman Maurer' via Netty discussions >> <[email protected] <mailto:[email protected]>> wrote: >> Are you actually sure these are really leaked ? >> >> The point of the “ThreadLocalUnsafeDirectByteBuf” is that it can be reused >> so it is expected that it not gets released after its written but put back >> in the recycler. Or you saying there are too many of these ? >> >> >>> On 19 Jul 2016, at 22:39, Chris Conroy <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Yes all the leaked buffers look to be ThreadLocalUnsafeDirectByteBuf >>> >>> Disabling the recycler w/o backpressure on slow readers resulted in several >>> massive allocation events. I was able to free them by forcing GC so no >>> leaks there. Memory usage with backpressure on slow readers is back down to >>> Netty 3 w/o backpressure levels. I don't see any leaks so far. >>> >>> Assuming that disabling the recycler does indeed prevent leaks, where do we >>> go from there? Any ideas of things to look for in my application that might >>> cause the recycler to get into a bad state? >>> >>> On Tue, Jul 19, 2016 at 3:40 PM, 'Norman Maurer' via Netty discussions >>> <[email protected] <mailto:[email protected]>> wrote: >>> Btw all the “leaking” buffers are always: >>> >>> ThreadLocalUnsafeDirectByteBuf >>> >>> ? >>> >>> >>>> On 19 Jul 2016, at 21:29, Chris Conroy <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> I ran with paranoid while observing the leak and got no messages from the >>>> ResourceLeakDetector. >>>> >>>> Initial results from recycler maxCapacity of 0 are looking positive with >>>> respect to the leak. However, I did see a large spike of NIO memory >>>> allocated (1G) where the Netty 3 version of this app receiving similar >>>> traffic peaks at about 1/10th that. I'll let this run for a while and >>>> report back later with the full results. >>>> >>>> On Tue, Jul 19, 2016 at 2:59 PM, 'Norman Maurer' via Netty discussions >>>> <[email protected] <mailto:[email protected]>> wrote: >>>> Can you do me a favour and run with: >>>> >>>> -Dio.netty.recycler.maxCapacity=0 >>>> >>>> And let me know if you still see some leaks ? >>>> >>>> >>>> >>>>> On 19 Jul 2016, at 20:50, Chris Conroy <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> 4.1.0.Final >>>>> >>>>> On Tue, Jul 19, 2016 at 2:48 PM, 'Norman Maurer' via Netty discussions >>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>> Are you using 4.0 or 4.1 ? >>>>> >>>>>> On 19 Jul 2016, at 20:42, Chris Conroy <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> I have not been able to reproduce locally yet, but I do see it in a >>>>>> cluster that takes a lot of varied traffic. The leak detector has not >>>>>> fired for this under advanced. I will give paranoid a shot to be safe, >>>>>> but it's my understanding that the leak detection framework is more for >>>>>> dealing with pooled byte buf misuse, but in this case I am exclusively >>>>>> using unpooled heap byte bufs: these are just the socket direct byte >>>>>> bufs that appear to be leaking. >>>>>> >>>>>> I meant to add this earlier: The path to GC root goes: >>>>>> >>>>>> io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf >>>>>> io.netty.util.Recycler$DefaultHandle >>>>>> io.netty.util.Recycler$DefaultHandle[] >>>>>> io.netty.util.Recycler#Stack >>>>>> java.lang.Object[] >>>>>> io.netty.util.internal.InternalThreadLocalMap >>>>>> ... (more thread local map refs up to java.lang.Thread) >>>>>> >>>>>> >>>>>> >>>>>> On Tuesday, July 19, 2016 at 2:17:25 PM UTC-4, Norman Maurer wrote: >>>>>> Can you provide a reproducer? Also did you try to run with paranoid leak >>>>>> detection >>>>>> >>>>>> Am 19.07.2016 um 20:04 schrieb Chris Conroy <[email protected] <>>: >>>>>> >>>>>>> Ah okay: I didn't see the calls to failFlushed since they occur above >>>>>>> the stanza I found suspicious. >>>>>>> >>>>>>> So, the above explanation is probably not correct. Still, I am seeing a >>>>>>> leak where DirectByteBufs are rooted to the recycler, and the speed at >>>>>>> which these buffers leak appears to be correlated with slow/partial >>>>>>> readers. >>>>>>> >>>>>>> On Monday, July 18, 2016 at 4:36:31 PM UTC-4, Norman Maurer wrote: >>>>>>> failFlushed(...) should be called to fail and release all flushed >>>>>>> messages. >>>>>>> >>>>>>> Are you saying this not happens? >>>>>>> >>>>>>> Am 18.07.2016 um 22:02 schrieb Chris Conroy <[email protected] <>>: >>>>>>> >>>>>>>> I’ve been trying to track down a NIO memory leak that occurs in a >>>>>>>> Netty application I am porting from Netty 3 to Netty 4. This leak does >>>>>>>> not occur in the Netty 3 version of the application. >>>>>>>> >>>>>>>> For now, I’m using only unpooled heap buffers in Netty 4, but NIO >>>>>>>> buffers do come into play for socket communication. >>>>>>>> >>>>>>>> I’ve captured a few heap dumps from affected instances, and in each it >>>>>>>> appears that the leaked DirectByteBuf java objects are rooted in an >>>>>>>> io.netty.util.Recycler. >>>>>>>> >>>>>>>> These buffers remain indefinitely: I can disable the application to >>>>>>>> drain traffic and force GCs, but the # of NIO buffers and NIO >>>>>>>> allocated space stays flat. >>>>>>>> >>>>>>>> The issue is likely related to slow readers. However, the leak >>>>>>>> persists long after all channels have been closed. >>>>>>>> >>>>>>>> I implemented a writability listener and the leak does appear to go >>>>>>>> away if I stop writing to a channel after it goes unwritable. This is >>>>>>>> good, but I’m still worried that this just makes the problem less >>>>>>>> likely since it’s still possible to write/flush and have pending data: >>>>>>>> writability just limits how much data will be buffered. >>>>>>>> >>>>>>>> Digging into ChannelOutBoundBuffer I see the following stanza in close: >>>>>>>> >>>>>>>> >>>>>>>> // Release all unflushed messages. >>>>>>>> try { >>>>>>>> Entry e = unflushedEntry; >>>>>>>> while (e != null) { >>>>>>>> // Just decrease; do not trigger any events via >>>>>>>> decrementPendingOutboundBytes() >>>>>>>> int size = e.pendingSize; >>>>>>>> TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size); >>>>>>>> >>>>>>>> if (!e.cancelled) { >>>>>>>> ReferenceCountUtil.safeRelease(e.msg); >>>>>>>> safeFail(e.promise, cause); >>>>>>>> } >>>>>>>> e = e.recycleAndGetNext(); >>>>>>>> } >>>>>>>> } finally { >>>>>>>> inFail = false; >>>>>>>> } >>>>>>>> clearNioBuffers(); >>>>>>>> This seems a bit curious to me: why are flushed buffers not released >>>>>>>> here? Since the leak seems to be rooted in the Recycler, this could be >>>>>>>> the culprit…What do you think? >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Netty discussions" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>>> an email to [email protected] <>. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Netty discussions" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected] <>. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Netty discussions" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to [email protected] >>>>>> <mailto:[email protected]>. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>> For more options, visit https://groups.google.com/d/optout >>>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Netty discussions" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected] >>>>> <mailto:[email protected]>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Netty discussions" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] >>>>> <mailto:[email protected]>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Netty discussions" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] >>>> <mailto:[email protected]>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com >>>> >>>> <https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com?utm_medium=email&utm_source=footer>. >>>> >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Netty discussions" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] >>>> <mailto:[email protected]>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Netty discussions" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com >>> >>> <https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com?utm_medium=email&utm_source=footer>. >>> >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Netty discussions" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Netty discussions" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com >> >> <https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com?utm_medium=email&utm_source=footer>. >> >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Netty discussions" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > -- You received this message because you are subscribed to the Google Groups "Netty discussions" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com. For more options, visit https://groups.google.com/d/optout.
