Sure thing. I'll give that a shot. On Wed, Jul 20, 2016 at 1:45 PM, 'Norman Maurer' via Netty discussions < [email protected]> wrote:
> Thanks I will have another look. > > One thing you could try is to use > “-Dio.netty.threadLocalDirectBufferSize=16” and see if the leaks are gone > then. > > This is not a solution but would help me a bit :) > > > On 20 Jul 2016, at 19:39, Chris Conroy <[email protected]> wrote: > > It's no problem! I'm sorry for all the back and forth. I'd just send you > the heap dump if I could, but alas it will be difficult to impossible to > sanitize it from sensitive data. (As an aside, I really wish there were > tools that let you interact with java heap dumps more programmatically...) > > In the particular heap dump I'm looking at, I have 12,762 such buffers. > Interestingly, I see 123k Recycler$DefaultHandles in the heap... > <Screen Shot 2016-07-20 at 1.31.40 PM.png> > > > Here's the only path to GC roots from the leaked byte bufs: > > <Screen Shot 2016-07-20 at 1.34.50 PM.png> > > The threads appear to all be server worker threads. The Biggest Objects - > Dominators view for strong references in YourKit shows me that the server > worker threads are the dominant roots in the heap: > > <Screen Shot 2016-07-20 at 1.38.19 PM.png> > > > > > > On Wed, Jul 20, 2016 at 1:22 PM, 'Norman Maurer' via Netty discussions < > [email protected]> wrote: > >> And how many of these you have ? >> >> Sorry for all the questions :( >> >> On 20 Jul 2016, at 19:19, Chris Conroy <[email protected]> wrote: >> >> Here's an example leaked buf >> >> <Screen Shot 2016-07-20 at 1.18.24 PM.png> >> >> On Wed, Jul 20, 2016 at 1:17 PM, 'Norman Maurer' via Netty discussions < >> [email protected]> wrote: >> >>> One last thing… >>> >>> Can you tell me what the capacity() of the “leaked” buffer is ? >>> >>> On 20 Jul 2016, at 19:11, Norman Maurer <[email protected]> >>> wrote: >>> >>> Thanks Chris, >>> >>> this sounds really “fishy”. Let me try to debug this a bit more (not >>> sure yet how tho). >>> >>> On 20 Jul 2016, at 19:06, Chris Conroy <[email protected]> wrote: >>> >>> This leak resulted in exhaustion of 4G of NIO memory. The same >>> application under Netty 3 only ever uses <200MB of NIO memory. I have run >>> several experiments where I take some traffic, disable the node from >>> serving traffic, and then force a full GC. The allocated NIO memory does >>> not return to normal levels. >>> >>> Spot checking the byte buf handles from my heap dump, I see lots (all?) >>> `recycle` and `lastRecycleIds` of `int -2147483648 = 0x80000000` >>> >>> The experiment yesterday with the recycler disabled did not result in >>> any such leaks over several hours of traffic. >>> >>> On Wed, Jul 20, 2016 at 12:41 AM, 'Norman Maurer' via Netty discussions >>> <[email protected]> wrote: >>> >>>> Are you actually sure these are really leaked ? >>>> >>>> The point of the “ThreadLocalUnsafeDirectByteBuf” is that it can be >>>> reused so it is expected that it not gets released after its written but >>>> put back in the recycler. Or you saying there are too many of these ? >>>> >>>> >>>> On 19 Jul 2016, at 22:39, Chris Conroy <[email protected]> wrote: >>>> >>>> Yes all the leaked buffers look to be ThreadLocalUnsafeDirectByteBuf >>>> >>>> Disabling the recycler w/o backpressure on slow readers resulted in >>>> several massive allocation events. I was able to free them by forcing GC so >>>> no leaks there. Memory usage with backpressure on slow readers is back down >>>> to Netty 3 w/o backpressure levels. I don't see any leaks so far. >>>> >>>> Assuming that disabling the recycler does indeed prevent leaks, where >>>> do we go from there? Any ideas of things to look for in my application that >>>> might cause the recycler to get into a bad state? >>>> >>>> On Tue, Jul 19, 2016 at 3:40 PM, 'Norman Maurer' via Netty discussions >>>> <[email protected]> wrote: >>>> >>>>> Btw all the “leaking” buffers are always: >>>>> >>>>> ThreadLocalUnsafeDirectByteBuf >>>>> >>>>> ? >>>>> >>>>> >>>>> On 19 Jul 2016, at 21:29, Chris Conroy <[email protected]> wrote: >>>>> >>>>> I ran with paranoid while observing the leak and got no messages from >>>>> the ResourceLeakDetector. >>>>> >>>>> Initial results from recycler maxCapacity of 0 are looking positive >>>>> with respect to the leak. However, I did see a large spike of NIO memory >>>>> allocated (1G) where the Netty 3 version of this app receiving similar >>>>> traffic peaks at about 1/10th that. I'll let this run for a while and >>>>> report back later with the full results. >>>>> >>>>> On Tue, Jul 19, 2016 at 2:59 PM, 'Norman Maurer' via Netty discussions >>>>> <[email protected]> wrote: >>>>> >>>>>> Can you do me a favour and run with: >>>>>> >>>>>> -Dio.netty.recycler.maxCapacity=0 >>>>>> >>>>>> And let me know if you still see some leaks ? >>>>>> >>>>>> >>>>>> >>>>>> On 19 Jul 2016, at 20:50, Chris Conroy <[email protected]> wrote: >>>>>> >>>>>> 4.1.0.Final >>>>>> >>>>>> On Tue, Jul 19, 2016 at 2:48 PM, 'Norman Maurer' via Netty >>>>>> discussions <[email protected]> wrote: >>>>>> >>>>>>> Are you using 4.0 or 4.1 ? >>>>>>> >>>>>>> On 19 Jul 2016, at 20:42, Chris Conroy <[email protected]> wrote: >>>>>>> >>>>>>> I have not been able to reproduce locally yet, but I do see it in a >>>>>>> cluster that takes a lot of varied traffic. The leak detector has not >>>>>>> fired >>>>>>> for this under advanced. I will give paranoid a shot to be safe, but >>>>>>> it's >>>>>>> my understanding that the leak detection framework is more for dealing >>>>>>> with >>>>>>> pooled byte buf misuse, but in this case I am exclusively using unpooled >>>>>>> heap byte bufs: these are just the socket direct byte bufs that appear >>>>>>> to >>>>>>> be leaking. >>>>>>> >>>>>>> I meant to add this earlier: The path to GC root goes: >>>>>>> >>>>>>> io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf >>>>>>> io.netty.util.Recycler$DefaultHandle >>>>>>> io.netty.util.Recycler$DefaultHandle[] >>>>>>> io.netty.util.Recycler#Stack >>>>>>> java.lang.Object[] >>>>>>> io.netty.util.internal.InternalThreadLocalMap >>>>>>> ... (more thread local map refs up to java.lang.Thread) >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tuesday, July 19, 2016 at 2:17:25 PM UTC-4, Norman Maurer wrote: >>>>>>>> >>>>>>>> Can you provide a reproducer? Also did you try to run with paranoid >>>>>>>> leak detection >>>>>>>> >>>>>>>> Am 19.07.2016 um 20:04 schrieb Chris Conroy <[email protected]>: >>>>>>>> >>>>>>>> Ah okay: I didn't see the calls to failFlushed since they occur >>>>>>>> above the stanza I found suspicious. >>>>>>>> >>>>>>>> So, the above explanation is probably not correct. Still, I am >>>>>>>> seeing a leak where DirectByteBufs are rooted to the recycler, and the >>>>>>>> speed at which these buffers leak appears to be correlated with >>>>>>>> slow/partial readers. >>>>>>>> >>>>>>>> On Monday, July 18, 2016 at 4:36:31 PM UTC-4, Norman Maurer wrote: >>>>>>>>> >>>>>>>>> failFlushed(...) should be called to fail and release all flushed >>>>>>>>> messages. >>>>>>>>> >>>>>>>>> Are you saying this not happens? >>>>>>>>> >>>>>>>>> Am 18.07.2016 um 22:02 schrieb Chris Conroy <[email protected]>: >>>>>>>>> >>>>>>>>> I’ve been trying to track down a NIO memory leak that occurs in a >>>>>>>>> Netty application I am porting from Netty 3 to Netty 4. This leak >>>>>>>>> does not >>>>>>>>> occur in the Netty 3 version of the application. >>>>>>>>> >>>>>>>>> For now, I’m using only unpooled heap buffers in Netty 4, but NIO >>>>>>>>> buffers do come into play for socket communication. >>>>>>>>> >>>>>>>>> I’ve captured a few heap dumps from affected instances, and in >>>>>>>>> each it appears that the leaked DirectByteBuf java objects are rooted >>>>>>>>> in an >>>>>>>>> io.netty.util.Recycler. >>>>>>>>> >>>>>>>>> These buffers remain indefinitely: I can disable the application >>>>>>>>> to drain traffic and force GCs, but the # of NIO buffers and NIO >>>>>>>>> allocated >>>>>>>>> space stays flat. >>>>>>>>> >>>>>>>>> The issue is likely related to slow readers. However, the leak >>>>>>>>> persists long after all channels have been closed. >>>>>>>>> >>>>>>>>> I implemented a writability listener and the leak does appear to >>>>>>>>> go away if I stop writing to a channel after it goes unwritable. This >>>>>>>>> is >>>>>>>>> good, but I’m still worried that this just makes the problem less >>>>>>>>> likely >>>>>>>>> since it’s still possible to write/flush and have pending data: >>>>>>>>> writability >>>>>>>>> just limits how much data will be buffered. >>>>>>>>> >>>>>>>>> Digging into ChannelOutBoundBuffer I see the following stanza in >>>>>>>>> close: >>>>>>>>> >>>>>>>>> >>>>>>>>> // Release all unflushed messages. >>>>>>>>> try { >>>>>>>>> Entry e = unflushedEntry; >>>>>>>>> while (e != null) { >>>>>>>>> // Just decrease; do not trigger any events via >>>>>>>>> decrementPendingOutboundBytes() >>>>>>>>> int size = e.pendingSize; >>>>>>>>> TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size); >>>>>>>>> >>>>>>>>> if (!e.cancelled) { >>>>>>>>> ReferenceCountUtil.safeRelease(e.msg); >>>>>>>>> safeFail(e.promise, cause); >>>>>>>>> } >>>>>>>>> e = e.recycleAndGetNext(); >>>>>>>>> } >>>>>>>>> } finally { >>>>>>>>> inFail = false; >>>>>>>>> } >>>>>>>>> clearNioBuffers(); >>>>>>>>> >>>>>>>>> This seems a bit curious to me: why are flushed buffers not >>>>>>>>> released here? Since the leak seems to be rooted in the Recycler, this >>>>>>>>> could be the culprit…What do you think? >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Netty discussions" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Netty discussions" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Netty discussions" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "Netty discussions" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com >>>>>>> <https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Netty discussions" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "Netty discussions" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com >>>>>> <https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Netty discussions" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Netty discussions" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com >>>>> <https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Netty discussions" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Netty discussions" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com >>>> <https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Netty discussions" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com >>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Netty discussions" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com >>> <https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Netty discussions" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com >> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Netty discussions" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com >> <https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > You received this message because you are subscribed to the Google Groups > "Netty discussions" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com > <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Netty discussions" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com > <https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Netty discussions" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDO6ZGZYvpXAuG32O2orLAfP28TTqbX9qScgUjVXktiCA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
