On Thu, Jul 1, 2021 at 10:07 AM Damjan Marion <dmar...@me.com> wrote:
> > > > On 01.07.2021., at 16:12, Matthew Smith <mgsm...@netgate.com> wrote: > > > > > > > > On Thu, Jul 1, 2021 at 6:36 AM Damjan Marion <dmar...@me.com> wrote: > > > > > > > On 01.07.2021., at 11:12, Benoit Ganne (bganne) via lists.fd.io > <bganne=cisco....@lists.fd.io> wrote: > > > > > >> Yes, allowing dynamic heap growth sounds like it could be better. > > >> Alternatively... if memory allocations could fail and something more > > >> graceful than VPP exiting could occur, that may also be better. E.g. > if > > >> I'm adding a route and try to allocate a counter for it and that > fails, it > > >> would be better to refuse to add the route than to exit and take the > > >> network down. > > >> > > >> I realize that neither of those options is easy to do btw. I'm just > trying > > >> to figure out how to make it easier and more forgiving for users to > set up > > >> their configuration without making them learn about various memory > > >> parameters. > > > > > > Understood, but setting a very high default will just make users of > smaller config puzzled too 😊 and I think changing all memory allocation > callsites to check for NULL would be a big paradigm change in VPP. > > > That's why I think a dynamically growing heap might be better but I do > not really know what would be the complexity. > > > That said, you can probably change the default in your own build and > that should work. > > > > > > > Fully agree wirth Benoit. We should not increase heap size default value. > > > > Things are actually a bit more complicated. For performance reasons > people should use > > hugepages whenever they are available, but they are also not default. > > When hugepages are used all pages are immediately backed with physical > memory. > > > > So different use cases require different heap configurations and end > user needs to tune that. > > Same applies for other things like stats segment page size which again > may impact forwarding > > performance significantly. > > > > If messing with startup.conf is too complicated for end user, some nice > configuration script may be helpful. > > Or just throwing few startup.confs into extras/startup_configs. > > > > Dynamic heap is possible, but not straight forward, as at some places we > use offsets > > to the start of the heap, so additional allocation cannot be anywhere. > > Also it will not help in some cases, i.e. when 1G hugepage is used for > heap, growing up to 2G > > will fail if 2nd 1G page is not pre-allocated. > > > > > > Sorry for not being clear. I was not advocating any change to defaults > in VPP code in gerrit. I was trying to figure out the impact of changing > the default value written in startup.conf by the management plane I work > on. And also have a conversation on whether there are ways that it could be > made easier to tune memory parameters correctly. > > ok, so let me try to answer your original questions: > > > It's my understanding that when you set the size of the main heap or the > stat segment in startup.conf, the size you specify is used to set up > virtual address space and the system does not actually allocate that full > amount of memory to VPP. I think when VPP tries to read/write addresses > within the address space, then memory is requested from the system to back > the chunk of address space containing the address being accessed. Is my > understanding correct(ish)? > > heap-size parameter defines size of memory mapping created for the heap. > With the normal 4K pages mapping is not backed by physical memory. Instead, > first time you try to access specific page CPU will generate page fault, > and kernel will handle it by allocating 4k chunk of physical memory to back > that specific virtual address and setup MMU mapping for that page. > > In VPP we don’t have reverse process, even if all memory allocations which > use specific 4k page are freed, that 4K page will not be returned to > kernel, as kernel simply doesn’t know that specific page is not in use > anymore. > Solution would be to somehow track number of memory allocations sharing > single 4K page and call madvise() system call when last one is freed... > > If you are using hugepages, all virtual memory is immediately backed by > physical memory so VPP with 32G of hugepage heap will use 32G of physical > memory as long as VPP is running. > > If you do `show memory main-heap` you will actually see how many physical > pages are allocated: > > vpp# show memory main-heap > Thread 0 vpp_main > base 0x7f6f95c9f000, size 1g, locked, unmap-on-destroy, name 'main heap' > page stats: page-size 4K, total 262144, mapped 50702, not-mapped 211442 > numa 1: 50702 pages, 198.05m bytes > total: 1023.99M, used: 115.51M, free: 908.49M, trimmable: 905.75M > > > Out of this you can see that heap is using 4K pages, 262144 total, and > 50702 are mapped to physical memory. > All 50702 pages are using memory on numa node 1. > > So effectively VPP is using around 198 MB of physical memory for heap > while real heap usage is only 115 MB. > Such a big difference is mainly caused by one place in our code which > temporary allocates ~200M of memory for > temporary vector. That vector is freed but unused pages are never given > back to kernel. > > > > > > When I add a large number of routes to the FIB (>1M), I have seen VPP > crash when the main heap or stats segment run out of space. I am wondering > if it makes sense to just set the heap sizes to some huge value that I am > confident will not be exceeded. If memory is not allocated unless it's > needed, it seems like that would be ok to do. > > For such a big FIB size, you likely want to use hugepages to avoid TLB > misses which can significantly hurt performance. If you decide to go that > way, then you will be able to set heap size only to value which is smaller > than amount of free physical memory in the system and that memory will not > be available for anything else... > > HTH, > > — > Damjan > > Thanks for this explanation Damjan. It helps a lot. -Matt
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19688): https://lists.fd.io/g/vpp-dev/message/19688 Mute This Topic: https://lists.fd.io/mt/83856384/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
