Comments inside.. > On 29 Jul 2016, at 01:10, 'Chris Conroy' via Netty discussions > <[email protected]> wrote: > > Ping: What do you think about a global recycler instead of many thread-local > recyclers? > > Im not sure this can be done without too much overhead. But if you want to cook up a PR and show it with benchmarks I would be interested for sure :)
> Also, can you provide some more context on the rationale behind the recycler? > Especially with the PooledByteBufAllocator, NIO allocations should be very > cheap, so why bother to reuse the buffers? > > Its because of object allocation. It basically reuses the “ByteBuf” container object (non the actual memory here). Im working on another fix for the problem you see. And you also may be interested in these: Allow to limit the maximum number of WeakOrderQueue instances per The… <https://github.com/netty/netty/pull/5592> Introduce allocation / pooling ratio in Recycler <https://github.com/netty/netty/pull/5594> Set Recycler DEFAULT_INITIAL_MAX_CAPACITY to a more sane value <https://github.com/netty/netty/pull/5589> Ensure shared capacity is updated correctly when WeakOrderQueue is co… <https://github.com/netty/netty/pull/5577> > > On Mon, Jul 25, 2016 at 2:00 PM, Chris Conroy <[email protected] > <mailto:[email protected]>> wrote: > If the recycler is used by each EventLoopGroup, then it probably should have > a per EventLoopGroup configuration since you’ll need lower thresholds for > more threads. In practice I’m only seeing much usage of the recycler on one > of my EventLoopGroup s but I would be worried about running out of memory > unnecessarily in some other situation where another group ends up buffering a > large amount of data due to some slowdown. > > This would be a bit easier to configure safe automatic defaults if it were a > global (instead of per thread) recycler. How crazy would that be? Without > that, the recyclers need to be small enough to multiply per thread in the app > or there needs to be some kind of coordination mechanism to disable recycler > growth in some threads if other threads are currently using a lot of > capacity. There also might be some value in expiring older buffers so that > after high pressure periods they are able to be reclaimed (I have had success > doing this in MRU object pools elsewhere) > > > For HTTP applications, the default chunk size is 4k, so I imagine most > buffers would be under that size? I’m currently seeing good memory usage > results without too much extra GC using -Dio.netty.recycler.maxCapacity=4096 > -Dio.netty.threadLocalDirectBufferSize=8192 but I haven’t explored too many > other options yet. I did not seem to get much utilization with a thread local > buffer size threshold of 4k for whatever reason though. > > > On Mon, Jul 25, 2016 at 1:37 PM, 'Norman Maurer' via Netty discussions > <[email protected] <mailto:[email protected]>> wrote: > Yeah I agree the default is not super good… What you think would be a “sane” > default ? > >> On 25 Jul 2016, at 19:30, Chris Conroy <[email protected] >> <mailto:[email protected]>> wrote: >> >> Nope: we use fixed size thread pools and are not killing threads. >> >> I'm not saying that we've gotten that much allocated (we'd OOME well before >> then), but the defaults seem to imply that the recycler will hold on to 17G >> (per thread?) of memory if you were to allocate that much. This seems wrong. >> >> I'm running a few experiments changing the recycler max capacity and the >> thread local direct buffer size which I think should unblock us. >> >> On Fri, Jul 22, 2016 at 8:43 AM, 'Norman Maurer' via Netty discussions >> <[email protected] <mailto:[email protected]>> wrote: >> Hey Chris, >> >> the thing is that I would not expect to have so much stuff in there as it >> should pick things from the Stack per thread. Or do you create and destroy a >> lot of threads ? >> >> >>> On 22 Jul 2016, at 00:09, Chris Conroy <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> I was a bit delayed as I had to roll back for an unrelated reason. >>> >>> Using “-Dio.netty.threadLocalDirectBufferSize=16” appears to fix the leak. >>> >>> Can you help me understand how much memory we should expect to sit resident >>> in the recycler under the default settings? The documentation is very >>> scarce... At first glance, the default of 266k objects in the recycler and >>> 64k thread local buffer size limit would consume 17GB of NIO memory? But, >>> that can't be right... >>> >>> https://github.com/netty/netty/issues/3166 >>> <https://github.com/netty/netty/issues/3166> and >>> https://github.com/netty/netty/issues/4146 >>> <https://github.com/netty/netty/issues/4146> seem relevant here. >>> >>> On Wed, Jul 20, 2016 at 1:54 PM, Chris Conroy <[email protected] >>> <mailto:[email protected]>> wrote: >>> Sure thing. I'll give that a shot. >>> >>> On Wed, Jul 20, 2016 at 1:45 PM, 'Norman Maurer' via Netty discussions >>> <[email protected] <mailto:[email protected]>> wrote: >>> Thanks I will have another look. >>> >>> One thing you could try is to use >>> “-Dio.netty.threadLocalDirectBufferSize=16” and see if the leaks are gone >>> then. >>> >>> This is not a solution but would help me a bit :) >>> >>> >>>> On 20 Jul 2016, at 19:39, Chris Conroy <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> It's no problem! I'm sorry for all the back and forth. I'd just send you >>>> the heap dump if I could, but alas it will be difficult to impossible to >>>> sanitize it from sensitive data. (As an aside, I really wish there were >>>> tools that let you interact with java heap dumps more programmatically...) >>>> >>>> In the particular heap dump I'm looking at, I have 12,762 such buffers. >>>> Interestingly, I see 123k Recycler$DefaultHandles in the heap... >>>> <Screen Shot 2016-07-20 at 1.31.40 PM.png> >>>> >>>> >>>> Here's the only path to GC roots from the leaked byte bufs: >>>> >>>> <Screen Shot 2016-07-20 at 1.34.50 PM.png> >>>> >>>> The threads appear to all be server worker threads. The Biggest Objects - >>>> Dominators view for strong references in YourKit shows me that the server >>>> worker threads are the dominant roots in the heap: >>>> >>>> <Screen Shot 2016-07-20 at 1.38.19 PM.png> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jul 20, 2016 at 1:22 PM, 'Norman Maurer' via Netty discussions >>>> <[email protected] <mailto:[email protected]>> wrote: >>>> And how many of these you have ? >>>> >>>> Sorry for all the questions :( >>>> >>>>> On 20 Jul 2016, at 19:19, Chris Conroy <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Here's an example leaked buf >>>>> >>>>> <Screen Shot 2016-07-20 at 1.18.24 PM.png> >>>>> >>>>> On Wed, Jul 20, 2016 at 1:17 PM, 'Norman Maurer' via Netty discussions >>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>> One last thing… >>>>> >>>>> Can you tell me what the capacity() of the “leaked” buffer is ? >>>>> >>>>>> On 20 Jul 2016, at 19:11, Norman Maurer <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Thanks Chris, >>>>>> >>>>>> this sounds really “fishy”. Let me try to debug this a bit more (not >>>>>> sure yet how tho). >>>>>> >>>>>>> On 20 Jul 2016, at 19:06, Chris Conroy <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> This leak resulted in exhaustion of 4G of NIO memory. The same >>>>>>> application under Netty 3 only ever uses <200MB of NIO memory. I have >>>>>>> run several experiments where I take some traffic, disable the node >>>>>>> from serving traffic, and then force a full GC. The allocated NIO >>>>>>> memory does not return to normal levels. >>>>>>> >>>>>>> Spot checking the byte buf handles from my heap dump, I see lots (all?) >>>>>>> `recycle` and `lastRecycleIds` of `int -2147483648 <tel:2147483648> = >>>>>>> 0x80000000` >>>>>>> >>>>>>> The experiment yesterday with the recycler disabled did not result in >>>>>>> any such leaks over several hours of traffic. >>>>>>> >>>>>>> On Wed, Jul 20, 2016 at 12:41 AM, 'Norman Maurer' via Netty discussions >>>>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>>>> Are you actually sure these are really leaked ? >>>>>>> >>>>>>> The point of the “ThreadLocalUnsafeDirectByteBuf” is that it can be >>>>>>> reused so it is expected that it not gets released after its written >>>>>>> but put back in the recycler. Or you saying there are too many of these >>>>>>> ? >>>>>>> >>>>>>> >>>>>>>> On 19 Jul 2016, at 22:39, Chris Conroy <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Yes all the leaked buffers look to be ThreadLocalUnsafeDirectByteBuf >>>>>>>> >>>>>>>> Disabling the recycler w/o backpressure on slow readers resulted in >>>>>>>> several massive allocation events. I was able to free them by forcing >>>>>>>> GC so no leaks there. Memory usage with backpressure on slow readers >>>>>>>> is back down to Netty 3 w/o backpressure levels. I don't see any leaks >>>>>>>> so far. >>>>>>>> >>>>>>>> Assuming that disabling the recycler does indeed prevent leaks, where >>>>>>>> do we go from there? Any ideas of things to look for in my application >>>>>>>> that might cause the recycler to get into a bad state? >>>>>>>> >>>>>>>> On Tue, Jul 19, 2016 at 3:40 PM, 'Norman Maurer' via Netty discussions >>>>>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>>>>> Btw all the “leaking” buffers are always: >>>>>>>> >>>>>>>> ThreadLocalUnsafeDirectByteBuf >>>>>>>> >>>>>>>> ? >>>>>>>> >>>>>>>> >>>>>>>>> On 19 Jul 2016, at 21:29, Chris Conroy <[email protected] >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> I ran with paranoid while observing the leak and got no messages from >>>>>>>>> the ResourceLeakDetector. >>>>>>>>> >>>>>>>>> Initial results from recycler maxCapacity of 0 are looking positive >>>>>>>>> with respect to the leak. However, I did see a large spike of NIO >>>>>>>>> memory allocated (1G) where the Netty 3 version of this app receiving >>>>>>>>> similar traffic peaks at about 1/10th that. I'll let this run for a >>>>>>>>> while and report back later with the full results. >>>>>>>>> >>>>>>>>> On Tue, Jul 19, 2016 at 2:59 PM, 'Norman Maurer' via Netty >>>>>>>>> discussions <[email protected] <mailto:[email protected]>> >>>>>>>>> wrote: >>>>>>>>> Can you do me a favour and run with: >>>>>>>>> >>>>>>>>> -Dio.netty.recycler.maxCapacity=0 >>>>>>>>> >>>>>>>>> And let me know if you still see some leaks ? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 19 Jul 2016, at 20:50, Chris Conroy <[email protected] >>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>> >>>>>>>>>> 4.1.0.Final >>>>>>>>>> >>>>>>>>>> On Tue, Jul 19, 2016 at 2:48 PM, 'Norman Maurer' via Netty >>>>>>>>>> discussions <[email protected] <mailto:[email protected]>> >>>>>>>>>> wrote: >>>>>>>>>> Are you using 4.0 or 4.1 ? >>>>>>>>>> >>>>>>>>>>> On 19 Jul 2016, at 20:42, Chris Conroy <[email protected] >>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>> >>>>>>>>>>> I have not been able to reproduce locally yet, but I do see it in a >>>>>>>>>>> cluster that takes a lot of varied traffic. The leak detector has >>>>>>>>>>> not fired for this under advanced. I will give paranoid a shot to >>>>>>>>>>> be safe, but it's my understanding that the leak detection >>>>>>>>>>> framework is more for dealing with pooled byte buf misuse, but in >>>>>>>>>>> this case I am exclusively using unpooled heap byte bufs: these are >>>>>>>>>>> just the socket direct byte bufs that appear to be leaking. >>>>>>>>>>> >>>>>>>>>>> I meant to add this earlier: The path to GC root goes: >>>>>>>>>>> >>>>>>>>>>> io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf >>>>>>>>>>> io.netty.util.Recycler$DefaultHandle >>>>>>>>>>> io.netty.util.Recycler$DefaultHandle[] >>>>>>>>>>> io.netty.util.Recycler#Stack >>>>>>>>>>> java.lang.Object[] >>>>>>>>>>> io.netty.util.internal.InternalThreadLocalMap >>>>>>>>>>> ... (more thread local map refs up to java.lang.Thread) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tuesday, July 19, 2016 at 2:17:25 PM UTC-4, Norman Maurer wrote: >>>>>>>>>>> Can you provide a reproducer? Also did you try to run with paranoid >>>>>>>>>>> leak detection >>>>>>>>>>> >>>>>>>>>>> Am 19.07.2016 um 20:04 schrieb Chris Conroy <[email protected] >>>>>>>>>>> <>>: >>>>>>>>>>> >>>>>>>>>>>> Ah okay: I didn't see the calls to failFlushed since they occur >>>>>>>>>>>> above the stanza I found suspicious. >>>>>>>>>>>> >>>>>>>>>>>> So, the above explanation is probably not correct. Still, I am >>>>>>>>>>>> seeing a leak where DirectByteBufs are rooted to the recycler, and >>>>>>>>>>>> the speed at which these buffers leak appears to be correlated >>>>>>>>>>>> with slow/partial readers. >>>>>>>>>>>> >>>>>>>>>>>> On Monday, July 18, 2016 at 4:36:31 PM UTC-4, Norman Maurer wrote: >>>>>>>>>>>> failFlushed(...) should be called to fail and release all flushed >>>>>>>>>>>> messages. >>>>>>>>>>>> >>>>>>>>>>>> Are you saying this not happens? >>>>>>>>>>>> >>>>>>>>>>>> Am 18.07.2016 um 22:02 schrieb Chris Conroy <[email protected] >>>>>>>>>>>> <>>: >>>>>>>>>>>> >>>>>>>>>>>>> I’ve been trying to track down a NIO memory leak that occurs in a >>>>>>>>>>>>> Netty application I am porting from Netty 3 to Netty 4. This leak >>>>>>>>>>>>> does not occur in the Netty 3 version of the application. >>>>>>>>>>>>> >>>>>>>>>>>>> For now, I’m using only unpooled heap buffers in Netty 4, but NIO >>>>>>>>>>>>> buffers do come into play for socket communication. >>>>>>>>>>>>> >>>>>>>>>>>>> I’ve captured a few heap dumps from affected instances, and in >>>>>>>>>>>>> each it appears that the leaked DirectByteBuf java objects are >>>>>>>>>>>>> rooted in an io.netty.util.Recycler. >>>>>>>>>>>>> >>>>>>>>>>>>> These buffers remain indefinitely: I can disable the application >>>>>>>>>>>>> to drain traffic and force GCs, but the # of NIO buffers and NIO >>>>>>>>>>>>> allocated space stays flat. >>>>>>>>>>>>> >>>>>>>>>>>>> The issue is likely related to slow readers. However, the leak >>>>>>>>>>>>> persists long after all channels have been closed. >>>>>>>>>>>>> >>>>>>>>>>>>> I implemented a writability listener and the leak does appear to >>>>>>>>>>>>> go away if I stop writing to a channel after it goes unwritable. >>>>>>>>>>>>> This is good, but I’m still worried that this just makes the >>>>>>>>>>>>> problem less likely since it’s still possible to write/flush and >>>>>>>>>>>>> have pending data: writability just limits how much data will be >>>>>>>>>>>>> buffered. >>>>>>>>>>>>> >>>>>>>>>>>>> Digging into ChannelOutBoundBuffer I see the following stanza in >>>>>>>>>>>>> close: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> // Release all unflushed messages. >>>>>>>>>>>>> try { >>>>>>>>>>>>> Entry e = unflushedEntry; >>>>>>>>>>>>> while (e != null) { >>>>>>>>>>>>> // Just decrease; do not trigger any events via >>>>>>>>>>>>> decrementPendingOutboundBytes() >>>>>>>>>>>>> int size = e.pendingSize; >>>>>>>>>>>>> TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size); >>>>>>>>>>>>> >>>>>>>>>>>>> if (!e.cancelled) { >>>>>>>>>>>>> ReferenceCountUtil.safeRelease(e.msg); >>>>>>>>>>>>> safeFail(e.promise, cause); >>>>>>>>>>>>> } >>>>>>>>>>>>> e = e.recycleAndGetNext(); >>>>>>>>>>>>> } >>>>>>>>>>>>> } finally { >>>>>>>>>>>>> inFail = false; >>>>>>>>>>>>> } >>>>>>>>>>>>> clearNioBuffers(); >>>>>>>>>>>>> This seems a bit curious to me: why are flushed buffers not >>>>>>>>>>>>> released here? Since the leak seems to be rooted in the Recycler, >>>>>>>>>>>>> this could be the culprit…What do you think? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "Netty discussions" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>>>> send an email to [email protected] <>. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>>> Groups "Netty discussions" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>>> send an email to [email protected] <>. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com >>>>>>>>>>>> >>>>>>>>>>>> <https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>> Groups "Netty discussions" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>> send an email to [email protected] >>>>>>>>>>> <mailto:[email protected]>. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>>> the Google Groups "Netty discussions" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>>>> [email protected] >>>>>>>>>> <mailto:[email protected]>. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>>>>> >>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>> Groups "Netty discussions" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected] >>>>>>>>>> <mailto:[email protected]>. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "Netty discussions" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>>> [email protected] >>>>>>>>> <mailto:[email protected]>. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Netty discussions" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected] >>>>>>>>> <mailto:[email protected]>. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in the >>>>>>>> Google Groups "Netty discussions" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected] >>>>>>>> <mailto:[email protected]>. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Netty discussions" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>>> an email to [email protected] >>>>>>>> <mailto:[email protected]>. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in the >>>>>>> Google Groups "Netty discussions" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Netty discussions" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Netty discussions" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected] >>>>> <mailto:[email protected]>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Netty discussions" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] >>>>> <mailto:[email protected]>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Netty discussions" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] >>>> <mailto:[email protected]>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com >>>> >>>> <https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com?utm_medium=email&utm_source=footer>. >>>> >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Netty discussions" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] >>>> <mailto:[email protected]>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Netty discussions" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com >>> >>> <https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com?utm_medium=email&utm_source=footer>. >>> >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Netty discussions" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAWh8EQDe-D0hiY%2B4BXbLvDdtjUMDNoyY1B4_bBpYYNJg%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAWh8EQDe-D0hiY%2B4BXbLvDdtjUMDNoyY1B4_bBpYYNJg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Netty discussions" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/0328BF3C-794D-4821-8665-2C6C88DAF8E0%40googlemail.com >> >> <https://groups.google.com/d/msgid/netty/0328BF3C-794D-4821-8665-2C6C88DAF8E0%40googlemail.com?utm_medium=email&utm_source=footer>. >> >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Netty discussions" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBzJ84qpabpppj_SRQD6N5OVLN6kQnCFEV8nMYZ4Pcdjg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBzJ84qpabpppj_SRQD6N5OVLN6kQnCFEV8nMYZ4Pcdjg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Netty discussions" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe > <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/320953AF-FBE7-4C5E-A84C-F3A7CBEDFB80%40googlemail.com > > <https://groups.google.com/d/msgid/netty/320953AF-FBE7-4C5E-A84C-F3A7CBEDFB80%40googlemail.com?utm_medium=email&utm_source=footer>. > > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > > -- > You received this message because you are subscribed to the Google Groups > "Netty discussions" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDk8%3DhNfW1C6nDzj7wPEMZm7m9bkq%3D95fYX7hvnnR%3D0JQ%40mail.gmail.com > > <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDk8%3DhNfW1C6nDzj7wPEMZm7m9bkq%3D95fYX7hvnnR%3D0JQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Netty discussions" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/netty/F879B76C-11F8-4FB8-87AE-1A1B00EECF08%40googlemail.com. For more options, visit https://groups.google.com/d/optout.
