Btw all the “leaking” buffers are always: ThreadLocalUnsafeDirectByteBuf
? > On 19 Jul 2016, at 21:29, Chris Conroy <[email protected]> wrote: > > I ran with paranoid while observing the leak and got no messages from the > ResourceLeakDetector. > > Initial results from recycler maxCapacity of 0 are looking positive with > respect to the leak. However, I did see a large spike of NIO memory allocated > (1G) where the Netty 3 version of this app receiving similar traffic peaks at > about 1/10th that. I'll let this run for a while and report back later with > the full results. > > On Tue, Jul 19, 2016 at 2:59 PM, 'Norman Maurer' via Netty discussions > <[email protected] <mailto:[email protected]>> wrote: > Can you do me a favour and run with: > > -Dio.netty.recycler.maxCapacity=0 > > And let me know if you still see some leaks ? > > > >> On 19 Jul 2016, at 20:50, Chris Conroy <[email protected] >> <mailto:[email protected]>> wrote: >> >> 4.1.0.Final >> >> On Tue, Jul 19, 2016 at 2:48 PM, 'Norman Maurer' via Netty discussions >> <[email protected] <mailto:[email protected]>> wrote: >> Are you using 4.0 or 4.1 ? >> >>> On 19 Jul 2016, at 20:42, Chris Conroy <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> I have not been able to reproduce locally yet, but I do see it in a cluster >>> that takes a lot of varied traffic. The leak detector has not fired for >>> this under advanced. I will give paranoid a shot to be safe, but it's my >>> understanding that the leak detection framework is more for dealing with >>> pooled byte buf misuse, but in this case I am exclusively using unpooled >>> heap byte bufs: these are just the socket direct byte bufs that appear to >>> be leaking. >>> >>> I meant to add this earlier: The path to GC root goes: >>> >>> io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf >>> io.netty.util.Recycler$DefaultHandle >>> io.netty.util.Recycler$DefaultHandle[] >>> io.netty.util.Recycler#Stack >>> java.lang.Object[] >>> io.netty.util.internal.InternalThreadLocalMap >>> ... (more thread local map refs up to java.lang.Thread) >>> >>> >>> >>> On Tuesday, July 19, 2016 at 2:17:25 PM UTC-4, Norman Maurer wrote: >>> Can you provide a reproducer? Also did you try to run with paranoid leak >>> detection >>> >>> Am 19.07.2016 um 20:04 schrieb Chris Conroy <[email protected] <>>: >>> >>>> Ah okay: I didn't see the calls to failFlushed since they occur above the >>>> stanza I found suspicious. >>>> >>>> So, the above explanation is probably not correct. Still, I am seeing a >>>> leak where DirectByteBufs are rooted to the recycler, and the speed at >>>> which these buffers leak appears to be correlated with slow/partial >>>> readers. >>>> >>>> On Monday, July 18, 2016 at 4:36:31 PM UTC-4, Norman Maurer wrote: >>>> failFlushed(...) should be called to fail and release all flushed messages. >>>> >>>> Are you saying this not happens? >>>> >>>> Am 18.07.2016 um 22:02 schrieb Chris Conroy <[email protected] <>>: >>>> >>>>> I’ve been trying to track down a NIO memory leak that occurs in a Netty >>>>> application I am porting from Netty 3 to Netty 4. This leak does not >>>>> occur in the Netty 3 version of the application. >>>>> >>>>> For now, I’m using only unpooled heap buffers in Netty 4, but NIO buffers >>>>> do come into play for socket communication. >>>>> >>>>> I’ve captured a few heap dumps from affected instances, and in each it >>>>> appears that the leaked DirectByteBuf java objects are rooted in an >>>>> io.netty.util.Recycler. >>>>> >>>>> These buffers remain indefinitely: I can disable the application to drain >>>>> traffic and force GCs, but the # of NIO buffers and NIO allocated space >>>>> stays flat. >>>>> >>>>> The issue is likely related to slow readers. However, the leak persists >>>>> long after all channels have been closed. >>>>> >>>>> I implemented a writability listener and the leak does appear to go away >>>>> if I stop writing to a channel after it goes unwritable. This is good, >>>>> but I’m still worried that this just makes the problem less likely since >>>>> it’s still possible to write/flush and have pending data: writability >>>>> just limits how much data will be buffered. >>>>> >>>>> Digging into ChannelOutBoundBuffer I see the following stanza in close: >>>>> >>>>> >>>>> // Release all unflushed messages. >>>>> try { >>>>> Entry e = unflushedEntry; >>>>> while (e != null) { >>>>> // Just decrease; do not trigger any events via >>>>> decrementPendingOutboundBytes() >>>>> int size = e.pendingSize; >>>>> TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size); >>>>> >>>>> if (!e.cancelled) { >>>>> ReferenceCountUtil.safeRelease(e.msg); >>>>> safeFail(e.promise, cause); >>>>> } >>>>> e = e.recycleAndGetNext(); >>>>> } >>>>> } finally { >>>>> inFail = false; >>>>> } >>>>> clearNioBuffers(); >>>>> This seems a bit curious to me: why are flushed buffers not released >>>>> here? Since the leak seems to be rooted in the Recycler, this could be >>>>> the culprit…What do you think? >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Netty discussions" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] <>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Netty discussions" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] <>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Netty discussions" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Netty discussions" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com >> >> <https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com?utm_medium=email&utm_source=footer>. >> >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Netty discussions" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Netty discussions" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe > <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com > > <https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com?utm_medium=email&utm_source=footer>. > > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "Netty discussions" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com > > <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Netty discussions" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com. For more options, visit https://groups.google.com/d/optout.
