> On 25 Jul 2016, at 20:00, Chris Conroy <[email protected]> wrote: > > If the recycler is used by each EventLoopGroup, then it probably should have > a per EventLoopGroup configuration since you’ll need lower thresholds for > more threads. In practice I’m only seeing much usage of the recycler on one > of my EventLoopGroup s but I would be worried about running out of memory > unnecessarily in some other situation where another group ends up buffering a > large amount of data due to some slowdown. > >
At the moment these Recyclers are not point to an EventLoopGroup as these are no even aware of anything like EventLoops. For example you may use buffers without EventLoops at all. > This would be a bit easier to configure safe automatic defaults if it were a > global (instead of per thread) recycler. How crazy would that be? Without > that, the recyclers need to be small enough to multiply per thread in the app > or there needs to be some kind of coordination mechanism to disable recycler > growth in some threads if other threads are currently using a lot of > capacity. There also might be some value in expiring older buffers so that > after high pressure periods they are able to be reclaimed (I have had success > doing this in MRU object pools elsewhere) > > So you are talking about automatically drop stuff if its not used for X timeframe ? > > For HTTP applications, the default chunk size is 4k, so I imagine most > buffers would be under that size? I’m currently seeing good memory usage > results without too much extra GC using -Dio.netty.recycler.maxCapacity=4096 > -Dio.netty.threadLocalDirectBufferSize=8192 but I haven’t explored too many > other options yet. I did not seem to get much utilization with a thread local > buffer size threshold of 4k for whatever reason though. > > > On Mon, Jul 25, 2016 at 1:37 PM, 'Norman Maurer' via Netty discussions > <[email protected] <mailto:[email protected]>> wrote: > Yeah I agree the default is not super good… What you think would be a “sane” > default ? > >> On 25 Jul 2016, at 19:30, Chris Conroy <[email protected] >> <mailto:[email protected]>> wrote: >> >> Nope: we use fixed size thread pools and are not killing threads. >> >> I'm not saying that we've gotten that much allocated (we'd OOME well before >> then), but the defaults seem to imply that the recycler will hold on to 17G >> (per thread?) of memory if you were to allocate that much. This seems wrong. >> >> I'm running a few experiments changing the recycler max capacity and the >> thread local direct buffer size which I think should unblock us. >> >> On Fri, Jul 22, 2016 at 8:43 AM, 'Norman Maurer' via Netty discussions >> <[email protected] <mailto:[email protected]>> wrote: >> Hey Chris, >> >> the thing is that I would not expect to have so much stuff in there as it >> should pick things from the Stack per thread. Or do you create and destroy a >> lot of threads ? >> >> >>> On 22 Jul 2016, at 00:09, Chris Conroy <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> I was a bit delayed as I had to roll back for an unrelated reason. >>> >>> Using “-Dio.netty.threadLocalDirectBufferSize=16” appears to fix the leak. >>> >>> Can you help me understand how much memory we should expect to sit resident >>> in the recycler under the default settings? The documentation is very >>> scarce... At first glance, the default of 266k objects in the recycler and >>> 64k thread local buffer size limit would consume 17GB of NIO memory? But, >>> that can't be right... >>> >>> https://github.com/netty/netty/issues/3166 >>> <https://github.com/netty/netty/issues/3166> and >>> https://github.com/netty/netty/issues/4146 >>> <https://github.com/netty/netty/issues/4146> seem relevant here. >>> >>> On Wed, Jul 20, 2016 at 1:54 PM, Chris Conroy <[email protected] >>> <mailto:[email protected]>> wrote: >>> Sure thing. I'll give that a shot. >>> >>> On Wed, Jul 20, 2016 at 1:45 PM, 'Norman Maurer' via Netty discussions >>> <[email protected] <mailto:[email protected]>> wrote: >>> Thanks I will have another look. >>> >>> One thing you could try is to use >>> “-Dio.netty.threadLocalDirectBufferSize=16” and see if the leaks are gone >>> then. >>> >>> This is not a solution but would help me a bit :) >>> >>> >>>> On 20 Jul 2016, at 19:39, Chris Conroy <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> It's no problem! I'm sorry for all the back and forth. I'd just send you >>>> the heap dump if I could, but alas it will be difficult to impossible to >>>> sanitize it from sensitive data. (As an aside, I really wish there were >>>> tools that let you interact with java heap dumps more programmatically...) >>>> >>>> In the particular heap dump I'm looking at, I have 12,762 such buffers. >>>> Interestingly, I see 123k Recycler$DefaultHandles in the heap... >>>> <Screen Shot 2016-07-20 at 1.31.40 PM.png> >>>> >>>> >>>> Here's the only path to GC roots from the leaked byte bufs: >>>> >>>> <Screen Shot 2016-07-20 at 1.34.50 PM.png> >>>> >>>> The threads appear to all be server worker threads. The Biggest Objects - >>>> Dominators view for strong references in YourKit shows me that the server >>>> worker threads are the dominant roots in the heap: >>>> >>>> <Screen Shot 2016-07-20 at 1.38.19 PM.png> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jul 20, 2016 at 1:22 PM, 'Norman Maurer' via Netty discussions >>>> <[email protected] <mailto:[email protected]>> wrote: >>>> And how many of these you have ? >>>> >>>> Sorry for all the questions :( >>>> >>>>> On 20 Jul 2016, at 19:19, Chris Conroy <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Here's an example leaked buf >>>>> >>>>> <Screen Shot 2016-07-20 at 1.18.24 PM.png> >>>>> >>>>> On Wed, Jul 20, 2016 at 1:17 PM, 'Norman Maurer' via Netty discussions >>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>> One last thing… >>>>> >>>>> Can you tell me what the capacity() of the “leaked” buffer is ? >>>>> >>>>>> On 20 Jul 2016, at 19:11, Norman Maurer <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Thanks Chris, >>>>>> >>>>>> this sounds really “fishy”. Let me try to debug this a bit more (not >>>>>> sure yet how tho). >>>>>> >>>>>>> On 20 Jul 2016, at 19:06, Chris Conroy <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> This leak resulted in exhaustion of 4G of NIO memory. The same >>>>>>> application under Netty 3 only ever uses <200MB of NIO memory. I have >>>>>>> run several experiments where I take some traffic, disable the node >>>>>>> from serving traffic, and then force a full GC. The allocated NIO >>>>>>> memory does not return to normal levels. >>>>>>> >>>>>>> Spot checking the byte buf handles from my heap dump, I see lots (all?) >>>>>>> `recycle` and `lastRecycleIds` of `int -2147483648 <tel:2147483648> = >>>>>>> 0x80000000` >>>>>>> >>>>>>> The experiment yesterday with the recycler disabled did not result in >>>>>>> any such leaks over several hours of traffic. >>>>>>> >>>>>>> On Wed, Jul 20, 2016 at 12:41 AM, 'Norman Maurer' via Netty discussions >>>>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>>>> Are you actually sure these are really leaked ? >>>>>>> >>>>>>> The point of the “ThreadLocalUnsafeDirectByteBuf” is that it can be >>>>>>> reused so it is expected that it not gets released after its written >>>>>>> but put back in the recycler. Or you saying there are too many of these >>>>>>> ? >>>>>>> >>>>>>> >>>>>>>> On 19 Jul 2016, at 22:39, Chris Conroy <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Yes all the leaked buffers look to be ThreadLocalUnsafeDirectByteBuf >>>>>>>> >>>>>>>> Disabling the recycler w/o backpressure on slow readers resulted in >>>>>>>> several massive allocation events. I was able to free them by forcing >>>>>>>> GC so no leaks there. Memory usage with backpressure on slow readers >>>>>>>> is back down to Netty 3 w/o backpressure levels. I don't see any leaks >>>>>>>> so far. >>>>>>>> >>>>>>>> Assuming that disabling the recycler does indeed prevent leaks, where >>>>>>>> do we go from there? Any ideas of things to look for in my application >>>>>>>> that might cause the recycler to get into a bad state? >>>>>>>> >>>>>>>> On Tue, Jul 19, 2016 at 3:40 PM, 'Norman Maurer' via Netty discussions >>>>>>>> <[email protected] <mailto:[email protected]>> wrote: >>>>>>>> Btw all the “leaking” buffers are always: >>>>>>>> >>>>>>>> ThreadLocalUnsafeDirectByteBuf >>>>>>>> >>>>>>>> ? >>>>>>>> >>>>>>>> >>>>>>>>> On 19 Jul 2016, at 21:29, Chris Conroy <[email protected] >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> I ran with paranoid while observing the leak and got no messages from >>>>>>>>> the ResourceLeakDetector. >>>>>>>>> >>>>>>>>> Initial results from recycler maxCapacity of 0 are looking positive >>>>>>>>> with respect to the leak. However, I did see a large spike of NIO >>>>>>>>> memory allocated (1G) where the Netty 3 version of this app receiving >>>>>>>>> similar traffic peaks at about 1/10th that. I'll let this run for a >>>>>>>>> while and report back later with the full results. >>>>>>>>> >>>>>>>>> On Tue, Jul 19, 2016 at 2:59 PM, 'Norman Maurer' via Netty >>>>>>>>> discussions <[email protected] <mailto:[email protected]>> >>>>>>>>> wrote: >>>>>>>>> Can you do me a favour and run with: >>>>>>>>> >>>>>>>>> -Dio.netty.recycler.maxCapacity=0 >>>>>>>>> >>>>>>>>> And let me know if you still see some leaks ? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 19 Jul 2016, at 20:50, Chris Conroy <[email protected] >>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>> >>>>>>>>>> 4.1.0.Final >>>>>>>>>> >>>>>>>>>> On Tue, Jul 19, 2016 at 2:48 PM, 'Norman Maurer' via Netty >>>>>>>>>> discussions <[email protected] <mailto:[email protected]>> >>>>>>>>>> wrote: >>>>>>>>>> Are you using 4.0 or 4.1 ? >>>>>>>>>> >>>>>>>>>>> On 19 Jul 2016, at 20:42, Chris Conroy <[email protected] >>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>> >>>>>>>>>>> I have not been able to reproduce locally yet, but I do see it in a >>>>>>>>>>> cluster that takes a lot of varied traffic. The leak detector has >>>>>>>>>>> not fired for this under advanced. I will give paranoid a shot to >>>>>>>>>>> be safe, but it's my understanding that the leak detection >>>>>>>>>>> framework is more for dealing with pooled byte buf misuse, but in >>>>>>>>>>> this case I am exclusively using unpooled heap byte bufs: these are >>>>>>>>>>> just the socket direct byte bufs that appear to be leaking. >>>>>>>>>>> >>>>>>>>>>> I meant to add this earlier: The path to GC root goes: >>>>>>>>>>> >>>>>>>>>>> io.netty.buffer.ByteBufUtil$ThreadLocalUnsafeDirectByteBuf >>>>>>>>>>> io.netty.util.Recycler$DefaultHandle >>>>>>>>>>> io.netty.util.Recycler$DefaultHandle[] >>>>>>>>>>> io.netty.util.Recycler#Stack >>>>>>>>>>> java.lang.Object[] >>>>>>>>>>> io.netty.util.internal.InternalThreadLocalMap >>>>>>>>>>> ... (more thread local map refs up to java.lang.Thread) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tuesday, July 19, 2016 at 2:17:25 PM UTC-4, Norman Maurer wrote: >>>>>>>>>>> Can you provide a reproducer? Also did you try to run with paranoid >>>>>>>>>>> leak detection >>>>>>>>>>> >>>>>>>>>>> Am 19.07.2016 um 20:04 schrieb Chris Conroy <[email protected] >>>>>>>>>>> <>>: >>>>>>>>>>> >>>>>>>>>>>> Ah okay: I didn't see the calls to failFlushed since they occur >>>>>>>>>>>> above the stanza I found suspicious. >>>>>>>>>>>> >>>>>>>>>>>> So, the above explanation is probably not correct. Still, I am >>>>>>>>>>>> seeing a leak where DirectByteBufs are rooted to the recycler, and >>>>>>>>>>>> the speed at which these buffers leak appears to be correlated >>>>>>>>>>>> with slow/partial readers. >>>>>>>>>>>> >>>>>>>>>>>> On Monday, July 18, 2016 at 4:36:31 PM UTC-4, Norman Maurer wrote: >>>>>>>>>>>> failFlushed(...) should be called to fail and release all flushed >>>>>>>>>>>> messages. >>>>>>>>>>>> >>>>>>>>>>>> Are you saying this not happens? >>>>>>>>>>>> >>>>>>>>>>>> Am 18.07.2016 um 22:02 schrieb Chris Conroy <[email protected] >>>>>>>>>>>> <>>: >>>>>>>>>>>> >>>>>>>>>>>>> I’ve been trying to track down a NIO memory leak that occurs in a >>>>>>>>>>>>> Netty application I am porting from Netty 3 to Netty 4. This leak >>>>>>>>>>>>> does not occur in the Netty 3 version of the application. >>>>>>>>>>>>> >>>>>>>>>>>>> For now, I’m using only unpooled heap buffers in Netty 4, but NIO >>>>>>>>>>>>> buffers do come into play for socket communication. >>>>>>>>>>>>> >>>>>>>>>>>>> I’ve captured a few heap dumps from affected instances, and in >>>>>>>>>>>>> each it appears that the leaked DirectByteBuf java objects are >>>>>>>>>>>>> rooted in an io.netty.util.Recycler. >>>>>>>>>>>>> >>>>>>>>>>>>> These buffers remain indefinitely: I can disable the application >>>>>>>>>>>>> to drain traffic and force GCs, but the # of NIO buffers and NIO >>>>>>>>>>>>> allocated space stays flat. >>>>>>>>>>>>> >>>>>>>>>>>>> The issue is likely related to slow readers. However, the leak >>>>>>>>>>>>> persists long after all channels have been closed. >>>>>>>>>>>>> >>>>>>>>>>>>> I implemented a writability listener and the leak does appear to >>>>>>>>>>>>> go away if I stop writing to a channel after it goes unwritable. >>>>>>>>>>>>> This is good, but I’m still worried that this just makes the >>>>>>>>>>>>> problem less likely since it’s still possible to write/flush and >>>>>>>>>>>>> have pending data: writability just limits how much data will be >>>>>>>>>>>>> buffered. >>>>>>>>>>>>> >>>>>>>>>>>>> Digging into ChannelOutBoundBuffer I see the following stanza in >>>>>>>>>>>>> close: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> // Release all unflushed messages. >>>>>>>>>>>>> try { >>>>>>>>>>>>> Entry e = unflushedEntry; >>>>>>>>>>>>> while (e != null) { >>>>>>>>>>>>> // Just decrease; do not trigger any events via >>>>>>>>>>>>> decrementPendingOutboundBytes() >>>>>>>>>>>>> int size = e.pendingSize; >>>>>>>>>>>>> TOTAL_PENDING_SIZE_UPDATER.addAndGet(this, -size); >>>>>>>>>>>>> >>>>>>>>>>>>> if (!e.cancelled) { >>>>>>>>>>>>> ReferenceCountUtil.safeRelease(e.msg); >>>>>>>>>>>>> safeFail(e.promise, cause); >>>>>>>>>>>>> } >>>>>>>>>>>>> e = e.recycleAndGetNext(); >>>>>>>>>>>>> } >>>>>>>>>>>>> } finally { >>>>>>>>>>>>> inFail = false; >>>>>>>>>>>>> } >>>>>>>>>>>>> clearNioBuffers(); >>>>>>>>>>>>> This seems a bit curious to me: why are flushed buffers not >>>>>>>>>>>>> released here? Since the leak seems to be rooted in the Recycler, >>>>>>>>>>>>> this could be the culprit…What do you think? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>> Google Groups "Netty discussions" group. >>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>>>> send an email to [email protected] <>. >>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com >>>>>>>>>>>>> >>>>>>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADssKFcs-WCc8%2Br2RWrvbgg3csaJPdcsXL_mCD5yG2bg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>>> Groups "Netty discussions" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>>> send an email to [email protected] <>. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com >>>>>>>>>>>> >>>>>>>>>>>> <https://groups.google.com/d/msgid/netty/b66894c3-1e65-4235-9201-b4f1dca11a81%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>>> Groups "Netty discussions" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>>> send an email to [email protected] >>>>>>>>>>> <mailto:[email protected]>. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/d/msgid/netty/e823494b-caf1-4b1f-b629-405bbdbf4c40%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>>> the Google Groups "Netty discussions" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>>>> [email protected] >>>>>>>>>> <mailto:[email protected]>. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/netty/F788648D-2C4E-4031-BD2A-EFFAEED64BDC%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>>>>> >>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>>> Groups "Netty discussions" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected] >>>>>>>>>> <mailto:[email protected]>. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAiQseAviHAkbE-eoybDWQsGY1ek6LtXkMCebcMd4WU4g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>>> the Google Groups "Netty discussions" group. >>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>>> [email protected] >>>>>>>>> <mailto:[email protected]>. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/netty/925935D9-92F1-4516-B9E9-524991BF053E%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>>>> >>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Netty discussions" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected] >>>>>>>>> <mailto:[email protected]>. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKDfjjLkj1yytjhrU564K3Z36SOaqfuPcER-CK3_vrK16g%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to a topic in the >>>>>>>> Google Groups "Netty discussions" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected] >>>>>>>> <mailto:[email protected]>. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/netty/A77C4A03-0A98-4104-9791-7F161A3AAD91%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>> <https://groups.google.com/d/optout>. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Netty discussions" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>>> an email to [email protected] >>>>>>>> <mailto:[email protected]>. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKCA8g-4E0AJCf%2BopC2NMufmQBGcWnMXHkwbDj-veW7x8w%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in the >>>>>>> Google Groups "Netty discussions" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/netty/D30BEE2E-9119-4FD3-A144-F6B765B189E8%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Netty discussions" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>>> an email to [email protected] >>>>>>> <mailto:[email protected]>. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKADHSVtqFoqMRak34S23OsDrZTEfADSTMpKGa3bguU-ig%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>>>> For more options, visit https://groups.google.com/d/optout >>>>>>> <https://groups.google.com/d/optout>. >>>>>> >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "Netty discussions" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected] >>>>> <mailto:[email protected]>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/297B68FF-2B8F-4EE6-92B3-81856A901A7A%40googlemail.com?utm_medium=email&utm_source=footer>. >>>>> >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Netty discussions" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected] >>>>> <mailto:[email protected]>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKABgsVZaToNgE-UqK04v_fsy508Gwd35BTyq3_rZtfZ5Q%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>>> For more options, visit https://groups.google.com/d/optout >>>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Netty discussions" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected] >>>> <mailto:[email protected]>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com >>>> >>>> <https://groups.google.com/d/msgid/netty/F4977516-4D85-4995-AA5D-AFDC2EC18911%40googlemail.com?utm_medium=email&utm_source=footer>. >>>> >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "Netty discussions" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected] >>>> <mailto:[email protected]>. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAMJyi3yLcSVunKLeRdtjJ7%2BQY0ExDq%2Bemb%2BoK9z8kg9A%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>>> For more options, visit https://groups.google.com/d/optout >>>> <https://groups.google.com/d/optout>. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Netty discussions" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >>> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com >>> >>> <https://groups.google.com/d/msgid/netty/F3DB7987-51F1-49A4-B77D-49AD2A270FE4%40googlemail.com?utm_medium=email&utm_source=footer>. >>> >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Netty discussions" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected] >>> <mailto:[email protected]>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAWh8EQDe-D0hiY%2B4BXbLvDdtjUMDNoyY1B4_bBpYYNJg%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKAWh8EQDe-D0hiY%2B4BXbLvDdtjUMDNoyY1B4_bBpYYNJg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Netty discussions" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe >> <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/0328BF3C-794D-4821-8665-2C6C88DAF8E0%40googlemail.com >> >> <https://groups.google.com/d/msgid/netty/0328BF3C-794D-4821-8665-2C6C88DAF8E0%40googlemail.com?utm_medium=email&utm_source=footer>. >> >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Netty discussions" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBzJ84qpabpppj_SRQD6N5OVLN6kQnCFEV8nMYZ4Pcdjg%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBzJ84qpabpppj_SRQD6N5OVLN6kQnCFEV8nMYZ4Pcdjg%40mail.gmail.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to a topic in the Google > Groups "Netty discussions" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe > <https://groups.google.com/d/topic/netty/Ve4lnRvFXjM/unsubscribe>. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/320953AF-FBE7-4C5E-A84C-F3A7CBEDFB80%40googlemail.com > > <https://groups.google.com/d/msgid/netty/320953AF-FBE7-4C5E-A84C-F3A7CBEDFB80%40googlemail.com?utm_medium=email&utm_source=footer>. > > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "Netty discussions" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBJSvf3o9XAKLHYGkbgFvVhsRmx1CEuAJg1YQWOwAaDHA%40mail.gmail.com > > <https://groups.google.com/d/msgid/netty/CA%2B%3DgZKBJSvf3o9XAKLHYGkbgFvVhsRmx1CEuAJg1YQWOwAaDHA%40mail.gmail.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Netty discussions" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/netty/50787371-A607-41F1-A140-27BC84398453%40googlemail.com. For more options, visit https://groups.google.com/d/optout.
