On 07/16/2018 10:34 AM, Claudio Freire wrote:
On Fri, Jul 13, 2018 at 5:43 PM Andrew Dunstan
<andrew.duns...@2ndquadrant.com> wrote:
On 07/13/2018 09:44 AM, Heikki Linnakangas wrote:
On 13/07/18 01:39, Andrew Dunstan wrote:
On 07/12/2018 06:34 PM, Alvaro Herrera wrote:
On 2018-Jul-12, Andrew Dunstan wrote:
I fully understand. I think this needs to go back to "Waiting on
Author".
Why? Heikki's patch applies fine and passes the regression tests.
Well, I understood Claudio was going to do some more work (see
upthread).
Claudio raised a good point, that doing small pallocs leads to
fragmentation, and in particular, it might mean that we can't give
back the memory to the OS. The default glibc malloc() implementation
has a threshold of 4 or 32 MB or something like that - allocations
larger than the threshold are mmap()'d, and can always be returned to
the OS. I think a simple solution to that is to allocate larger
chunks, something like 32-64 MB at a time, and carve out the
allocations for the nodes from those chunks. That's pretty
straightforward, because we don't need to worry about freeing the
nodes in retail. Keep track of the current half-filled chunk, and
allocate a new one when it fills up.
Google seems to suggest the default threshold is much lower, like 128K.
Still, making larger allocations seems sensible. Are you going to work
on that?
Below a few MB the threshold is dynamic, and if a block bigger than
128K but smaller than the higher threshold (32-64MB IIRC) is freed,
the dynamic threshold is set to the size of the freed block.
See M_MMAP_MAX and M_MMAP_THRESHOLD in the man page for mallopt[1]
So I'd suggest allocating blocks bigger than M_MMAP_MAX.
[1] http://man7.org/linux/man-pages/man3/mallopt.3.html
That page says:
M_MMAP_MAX
This parameter specifies the maximum number of allocation
requests that may be simultaneously serviced using mmap(2).
This parameter exists because some systems have a limited
number of internal tables for use by mmap(2), and using more
than a few of them may degrade performance.
The default value is 65,536, a value which has no special
significance and which serves only as a safeguard. Setting
this parameter to 0 disables the use of mmap(2) for servicing
large allocation requests.
I'm confused about the relevance.
cheers
andrew
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
