Yes all the leaked buffers look to be ThreadLocalUnsafeDirectByteBuf Disabling the recycler w/o backpressure on slow readers resulted in several massive allocation events. I was able to free them by forcing GC so no leaks there. Memory usage with backpressure on slow readers is back down to Netty 3 w/o backpressure levels. I don't see any leaks so far.
Assuming that disabling the recycler does indeed prevent leaks, where do we go from there? Any ideas of things to look for in my application that might cause the recycler to get into a bad state? On Tue, Jul 19, 2016 at 3:40 PM, 'Norman Maurer' via Netty discussions < [email protected]> wrote: > Btw all the “leaking” buffers are always: > > ThreadLocalUnsafeDirectByteBuf > > ? > > > On 19 Jul 2016, at 21:29, Chris Conroy <[email protected]> wrote: > > I ran with paranoid while observing the leak and got no messages from the > ResourceLeakDetector. > > Initial results from recycler maxCapacity of 0 are looking positive with > respect to the leak. However, I did see a large spike of NIO memory > allocated (1G) where the Netty 3 version of this app receiving similar > traffic peaks at about 1/10th that. I'll let this run for a while and > report back later with the full results. > > On Tue, Jul 19, 2016 at 2:59 PM, 'Norman Maurer' via Netty discussions < > [email protected]> wrote: > >> Can you do me a favour and run with: >> >> -Dio.netty.recycler.maxCapacity=0 >> >> And let me know if you still see some leaks ? >> >> >> >> On 19 Jul 2016, at 20:50, Chris Conroy <[email protected]> wrote: >> >> 4.1.0.Final >> >> On Tue, Jul 19, 2016 at 2:48 PM, 'Norman Maurer' via Netty discussions < >> [email protected]> wrote: >> >>> Are you using 4.0 or 4.1 ? >>> >>> On 19 Jul 2016, at 20:42, Chris Conroy <[email protected]> wrote: >>> >>> I have not been able to reproduce locally yet, but I do see it in a >>> cluster that takes a lot of varied traffic. The leak detector has not fired >>> for this under advanced. I will give paranoid a shot to be safe, but it's >>> my understanding that the leak detection framework is more for dealing with >>> pooled byte buf misuse, but in this case I am exclusively using unpooled >>> heap byte bufs: these are just the socket direct byte bufs that appear to >>> be leaking. >>> >>> I meant to add this earlier: The path to GC root goes: >>> >>> io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf >>> io.netty.util.Recycler$DefaultHandle >>> io.netty.util.Recycler$DefaultHandle[] >>> io.netty.util.Recycler#Stack >>> java.lang.Object[] >>> io.netty.util.internal.InternalThreadLocalMap >>> ... (more thread local map refs up to java.lang.Thread) >>> >>> >>> >>> On Tuesday, July 19, 2016 at 2:17:25 PM UTC-4, Norman Maurer wrote: >>>> >>>> Can you provide a reproducer? Also did you try to run with paranoid >>>> leak detection >>>> >>>> Am 19.07.2016 um 20:04 schrieb Chris Conroy <[email protected]>: >>>> >>>> Ah okay: I didn't see the calls to failFlushed since they occur above >>>> the stanza I found suspicious. >>>> >>>> So, the above explanation is probably not correct. Still, I am seeing a >>>> leak where DirectByteBufs are rooted to the recycler, and the speed at >>>> which these buffers leak appears to be correlated with slow/partial >>>> readers. >>>> >>>> On Monday, July 18, 2016 at 4:36:31 PM UTC-4, Norman Maurer wrote: >>>>> >>>>> failFlushed(...) should be called to fail and release all flushed >>>>> messages. >>>>> >>>>> Are you saying this not happens? >>>>> >>>>> Am 18.07.2016 um 22:02 schrieb Chris Conroy <[email protected]>: >>>>> >>>>> I’ve been trying to track down a NIO memory leak that occurs in a >>>>> Netty application I am porting from Netty 3 to Netty 4. This leak does not >>>>> occur in the Netty 3 version of the application. >>>>> >>>>> For now, I’m using only unpooled heap buffers in Netty 4, but NIO >>>>> buffers do come into play for socket communication. >>>>> >>>>> I’ve captured a few heap dumps from affected instances, and in each it >>>>> appears that the leaked DirectByteBuf java objects are rooted in an >>>>> io.netty.util.Recycler. >>>>> >>>>> These buffers remain indefinitely: I can disable the application to >>>>> drain traffic and force GCs, but the # of NIO buffers and NIO allocated >>>>> space stays flat. >>>>> >>>>> The issue is likely related to slow readers. However, the leak >>>>> persists long after all channels have been closed. >>>>> >>>>> I implemented a writability listener and the leak does appear to go >>>>> away if I stop writing to a channel after it goes unwritable. This is >>>>> good, >>>>> but I’m still worried that this just makes the problem less likely since >>>>> it’s still possible to write/flush and have pending data: writability just >>>>> limits how much data will be buffered. >>>>> >>>>> Digging into ChannelOutBoundBuffer I see the following stanza in close >>>>> : >>>>> >>>>> >>>>> // Release all unflushed messages. >>>>> try { >>>>> Entry e = unflushedEntry; >>>>> while (e != null) { >>>>> // Just decrease; do not trigger any events via >>>>> decrementPendingOutboundBytes() >>>>> int size = e.pendingSize; >>>>> TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size); >>>>> >>>>> if (!e.cancelled) { >>>>> ReferenceCountUtil.safeRelease(e.msg); >>>>> safeFail(e.promise, cause); >>>>> } >>>>> e = e.recycleAndGetNext(); >>>>> } >>>>> } finally { >>>>> inFail = false; >>>>> } >>>>> clearNioBuffers(); >>>>> >>>>> This seems a bit curious to me: why are flushed buffers not released >>>>> here? Since the leak seems to be rooted in the Recycler, this could be the >>>>> culprit…What do you think? >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Netty discussions" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Netty discussions" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com >>>> <https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Netty discussions" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com >>> <https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Netty discussions" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com >>> <https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Netty discussions" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com >> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Netty discussions" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com >> <https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > You received this message because you are subscribed to the Google Groups > "Netty discussions" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com > <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Netty discussions" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com > <https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Netty discussions" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
