On May 24, 2017, at 11:37 AM, Dunkin, Nick <[email protected]<mailto:[email protected]>> wrote:
Hi Kapil, This is very interesting, thanks. Excuse this newbie question though: What is the lifetime of “in use” objects in the ioBufs? That is determined by your RAM cache size and algorithm (CLFUS or LRU). I don’t understand why we are not seeing reuse in these structures. In your artificial example, why would ATS “keep allocating 2M chunks until RAM cache limit is reached”. At what point are those allocated 2M chunks eligible for reuse? Once the they are de-allocated by RAM cache as part of LRU/CLFUS, they are put back into the free/reclaimable pool, and can be re-used. But the important point is that the low level memory mgmt code doesn’t actually free the memory - so memory allocated by chunks can only increase not shrink. My example was an extreme case to make a point. Thanks, Nick From: "Kapil Sharma (kapsharm)" <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, May 24, 2017 at 11:29 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Understanding ioBufAllocator behvaiour On plateauing - not necessarily; we do see the memory consumption increasing continuously in our deployments as well. It depends on the pattern of segment sizes over time. ATS uses power of 2 allocators for memory pool - there are 15 of those, ranging from 128bytes to 2M if my memory serves me right - and these are per thread! ATS will choose an optimal allocator for the segments. As Alan mentioned, once chunk are allocated, they are never freed. Here is a totally artificial example just to make the point (please correct if my understanding is flawed): * the traffic pattern was such that initially only 2M allocators were used then ATS will keep allocating 2M chunks until RAM cache limit (lets say it is 64GB) is reached. * Now traffic pattern changed (smaller fragment requests), and only 1M allocators are used, ATS will now keep allocating 1M chunks, again capping at 64GB. But in the end ATS would have allocated 128GB well over RAM cache size limit…. In the past a there was some prototype of reclaimable buffer support added in ATS, but I believe it was removed in 7.0? Also there is recent discussion of adding jmalloc? On May 24, 2017, at 11:01 AM, Alan Carroll <[email protected]<mailto:[email protected]>> wrote: One issue is that memory never moves between the iobuf sizes. Once a chunk of memory is used for a specific iobuf slot, it's there forever. But unless something is leaking, the total size should eventually plateau, certainly within less than a day if you have a basically constant load. There will be some growth due to blocks being kept in thread local allocation pools, but again that should level in less time than you've run. On Wednesday, May 24, 2017, 9:50:39 AM CDT, Dunkin, Nick <[email protected]<mailto:[email protected]>> wrote: Hi Alan, This is 7.0.0 I only see this behavior on ioBufAllocator[0], [4] and [5]. The other ioBufAllocators’ usage looks as I would expect (i.e. allocated goes up then flat), so I was thinking it was more likely something to do with my configuration or use-case. I’d also just like to understand, at a high level, how the ioBufAllocators are used. Thanks, Nick From: Alan Carroll <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, May 24, 2017 at 10:33 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: Understanding ioBufAllocator behvaiour Honestly it sounds like a leak. Can you specify which version of Traffic Server this is? On Wednesday, May 24, 2017, 8:22:46 AM CDT, Dunkin, Nick <[email protected]<mailto:[email protected]>> wrote: Hi I have a load test that I’ve been running for a number of days now. I’m using the memory dump logging in traffic.out and I’m trying to understand how Traffic Server allocates and reuses memory. I’m still quite new to Traffic Server. Nearly all of the memory traces look as I would expect, i.e. memory is allocated and reused over the lifetime of the test. However my readings from ioBufAllocator[0] show a continual increase in allocated AND used. I am attaching a graph. (FYI – This graph covers approximately 3 days of continual load test.) I would have expected to start seeing reuse in ioBufAllocator by now, like I do in the other ioBufAllocators. Can someone help me understand what I’m seeing? Many thanks, Nick Dunkin Nick Dunkin Principal Engineer o: 678.258.4071 e: [email protected]<mailto:[email protected]> 4375 River Green Pkwy # 100, Duluth, GA 30096, USA <image001.png> <image001.png>
