Great find! I think the linked patch that disables MADV_FREE would be good to include in native-toolchain.
We have done the most intensive stress/perf testing for Impala on older kernels, especially CentOS7/RHEL7, so I think always using MADV_DONTNEED is safer than having different behaviour on different OSes. We could also run into trouble with containers, if Impala is compiled in a container with a newer userspace but running on an older host. On Mon, May 18, 2020 at 6:05 AM LiFu He <hel...@gmail.com> wrote: > Thanks Tim ^_^ > I have found the answer: > https://github.com/gperftools/gperftools/issues/780 > By the way, is it necessary to disable MADV_FREE while the linux > kernel version is >= 4.5? If yes, I will add a patch to native-toolchain :) > > > Tim Armstrong <tarmstr...@cloudera.com> 于2020年5月16日周六 上午9:21写道： > > > That is strange, I would expect TCMalloc's unmapped bytes to be *not* > > counted as part of the process RSS. We depend on TCMalloc's > > tcmalloc.aggressive_memory_decommit setting to release all freed memory > to > > the OS. If the memory page size on your system is bigger than TCMalloc > > expects, or something along those lines, it could cause those symptoms. > > > > On Fri, May 15, 2020 at 4:04 PM LiFu He <hel...@gmail.com> wrote: > > > > > Hi, > > > Recently I deployed an Impala(3.2) cluster on aarch64 platform with > some > > > modifications, and then ran some big queries that are from TPC-DS. In > the > > > meanwhile, I found the memory of Impalad was not released to OS even > > though > > > the queries were all finished. I looked through the impalad's metrics > tab > > > and really can't figure out where these memory(50.33GB) gone. And > > according > > > to the decription of /memz, it seems the tcmalloc(2.5) had released its > > > memory to OS. Here are the /memz and /metrics info: > > > https://photos.app.goo.gl/1Ze9WGyEuyMSCs6b9 > > > Any tips will be appreciated. Thanks in advance :) > > > > > > -- > > > > >
