Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, this patch is marked as committed in CF application but the second part (generational allocator) was AFAICS never committed. Does anybody plan to push this forward? -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] PATCH: two slab-like memory allocators

On 28 February 2017 at 01:02, Andres Freund wrote: > Hi, > > On 2017-02-27 03:17:32 -0800, Andres Freund wrote: >> I'll work on getting slab committed first, and then review / edit / >> commit generation.c later. One first note there is that I'm wondering >> if generation.c

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/07/2017 12:19 AM, Andres Freund wrote: On 2017-03-02 22:51:09 +0100, Tomas Vondra wrote: Attaches is the last part of the patch series, rebased to current master and adopting the new chunk header approach. Something seems to have gone awry while sending that - the attachement is a

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-03-02 22:51:09 +0100, Tomas Vondra wrote: > Attaches is the last part of the patch series, rebased to current master and > adopting the new chunk header approach. Something seems to have gone awry while sending that - the attachement is a whopping 0 bytes... -- Sent via pgsql-hackers

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-03-06 23:32:30 +0100, Tomas Vondra wrote: > > > The question however is whether this won't make the optimization > > > pointless. > > > I also, wonder how much we save by this optimization and how widely it's > > > used? Can someone point me to some numbers? > > > > I don't recall

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/06/2017 08:08 PM, Andres Freund wrote: Hi, On 2017-03-06 19:49:56 +0100, Tomas Vondra wrote: On 03/06/2017 07:05 PM, Robert Haas wrote: On Mon, Mar 6, 2017 at 12:44 PM, Andres Freund wrote: On 2017-03-06 12:40:18 -0500, Robert Haas wrote: On Wed, Mar 1, 2017 at

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-03-06 19:49:56 +0100, Tomas Vondra wrote: > On 03/06/2017 07:05 PM, Robert Haas wrote: > > On Mon, Mar 6, 2017 at 12:44 PM, Andres Freund wrote: > > > On 2017-03-06 12:40:18 -0500, Robert Haas wrote: > > > > On Wed, Mar 1, 2017 at 5:55 PM, Andres Freund

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/06/2017 07:05 PM, Robert Haas wrote: On Mon, Mar 6, 2017 at 12:44 PM, Andres Freund wrote: On 2017-03-06 12:40:18 -0500, Robert Haas wrote: On Wed, Mar 1, 2017 at 5:55 PM, Andres Freund wrote: The issue was that on 32bit platforms the Datum

Re: [HACKERS] PATCH: two slab-like memory allocators

On Mon, Mar 6, 2017 at 12:44 PM, Andres Freund wrote: > On 2017-03-06 12:40:18 -0500, Robert Haas wrote: >> On Wed, Mar 1, 2017 at 5:55 PM, Andres Freund wrote: >> > The issue was that on 32bit platforms the Datum returned by some >> > functions

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-03-06 12:40:18 -0500, Robert Haas wrote: > On Wed, Mar 1, 2017 at 5:55 PM, Andres Freund wrote: > > The issue was that on 32bit platforms the Datum returned by some > > functions (int2int4_sum in this case) isn't actually a separately > > allocated Datum, but rather

Re: [HACKERS] PATCH: two slab-like memory allocators

On Wed, Mar 1, 2017 at 5:55 PM, Andres Freund wrote: > The issue was that on 32bit platforms the Datum returned by some > functions (int2int4_sum in this case) isn't actually a separately > allocated Datum, but rather just something embedded in a larger > struct. That,

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/04/2017 02:58 AM, Andres Freund wrote: On 2017-03-01 22:19:30 -0800, Andres Freund wrote: On 2017-03-02 04:36:23 +0100, Tomas Vondra wrote: I've noticed two minor typos: 1) That is solved this by creating ... - extra "this" 2) Given this, routines like pfree their corresponding

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-03-01 22:19:30 -0800, Andres Freund wrote: > On 2017-03-02 04:36:23 +0100, Tomas Vondra wrote: > > I've noticed two minor typos: > > > > 1) That is solved this by creating ... > >- extra "this" > > > > 2) Given this, routines like pfree their corresponding context ... > >-

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/01/2017 05:29 AM, Andres Freund wrote: On 2017-02-28 20:18:35 -0800, Andres Freund wrote: - Andres, hoping the buildfarm turns greener Oh well, that didn't work. Investigating. Attaches is the last part of the patch series, rebased to current master and adopting the new chunk header

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-03-02 04:47:13 +0100, Tomas Vondra wrote: > On 03/01/2017 11:55 PM, Andres Freund wrote: > > On 2017-02-28 20:29:36 -0800, Andres Freund wrote: > > > On 2017-02-28 20:18:35 -0800, Andres Freund wrote: > > > > - Andres, hoping the buildfarm turns greener > > > > > > Oh well, that didn't

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-03-02 04:36:23 +0100, Tomas Vondra wrote: > I've noticed two minor typos: > > 1) That is solved this by creating ... >- extra "this" > > 2) Given this, routines like pfree their corresponding context ... >- missing "find" or "determine" Will fix. > I also see you've explicitly

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/01/2017 11:55 PM, Andres Freund wrote: On 2017-02-28 20:29:36 -0800, Andres Freund wrote: On 2017-02-28 20:18:35 -0800, Andres Freund wrote: - Andres, hoping the buildfarm turns greener Oh well, that didn't work. Investigating. The fix for that was fairly trivial, and the buildfarm

Re: [HACKERS] PATCH: two slab-like memory allocators

On 03/01/2017 05:18 AM, Andres Freund wrote: On 2017-02-28 10:41:22 -0800, Andres Freund wrote: Hi, On 2017-02-27 23:44:20 -0800, Andres Freund wrote: *preliminary* patch attached. This needs a good bit of polishing (primarily comment work, verifying that valgrind works), but I'm too tired

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-28 20:29:36 -0800, Andres Freund wrote: > On 2017-02-28 20:18:35 -0800, Andres Freund wrote: > > - Andres, hoping the buildfarm turns greener > > Oh well, that didn't work. Investigating. The fix for that was fairly trivial, and the buildfarm has cooled down. The issue was that on

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-28 20:18:35 -0800, Andres Freund wrote: > - Andres, hoping the buildfarm turns greener Oh well, that didn't work. Investigating. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-28 10:41:22 -0800, Andres Freund wrote: > Hi, > > On 2017-02-27 23:44:20 -0800, Andres Freund wrote: > > *preliminary* patch attached. This needs a good bit of polishing > > (primarily comment work, verifying that valgrind works), but I'm too > > tired now. > > > > I'm not quite sure

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-27 23:44:20 -0800, Andres Freund wrote: > *preliminary* patch attached. This needs a good bit of polishing > (primarily comment work, verifying that valgrind works), but I'm too > tired now. > > I'm not quite sure how to deal with mmgr/README - it's written as kind > of a

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, *preliminary* patch attached. This needs a good bit of polishing (primarily comment work, verifying that valgrind works), but I'm too tired now. I'm not quite sure how to deal with mmgr/README - it's written as kind of a historical document, and the "Mechanisms to Allow Multiple Types of

Re: [HACKERS] PATCH: two slab-like memory allocators

Andres Freund writes: > Hm, that should be doable with something like > #if MAXIMUM_ALIGNOF > 4 && SIZEOF_VOID_P == 4 > which'd probably be better documentation than a macro that hides this > (arguing internally whether SIZEOF_VOID_P or SIZEOF_SIZE_T) is better. Not sure

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-27 22:57:24 -0500, Tom Lane wrote: > If the slab allocator would be happier with just a MemoryContext pointer > as chunk header, I think we should push in this direction rather than > invent some short-term hack. It would - it really doesn't need the size, because it's the same

Re: [HACKERS] PATCH: two slab-like memory allocators

Andres Freund writes: > Independently of this, we really should redefine StandardChunkHeader to > be only the MemoryContext. There's no need to have size/requested_size > part of StandardChunkHeader, given there's > MemoryContextMethods->get_chunk_space(). Yeah, perhaps.

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-28 01:44:42 +0100, Tomas Vondra wrote: > On 02/27/2017 06:42 PM, Andres Freund wrote: > > Yea, I hadn't yet realized when writing that that termite actually, > > despite running on ppc64, compiles a 32bit postgres. Will thus > > duplicate StandardChunkHeader's contents in to slab.c :( -

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/27/2017 06:42 PM, Andres Freund wrote: On 2017-02-27 12:27:48 -0500, Tom Lane wrote: Andres Freund writes: The best theory I have so far that I have is that slab.c's idea of StandardChunkHeader's size doesn't match what mcxt.c think it is (because slab.c simply

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/27/2017 04:07 PM, Andres Freund wrote: On February 27, 2017 6:14:20 AM PST, Tomas Vondra wrote: On 02/27/2017 01:02 PM, Andres Freund wrote: Hi, On 2017-02-27 03:17:32 -0800, Andres Freund wrote: I'll work on getting slab committed first, and then

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/27/2017 06:40 PM, Andres Freund wrote: On 2017-02-27 18:04:41 +0100, Petr Jelinek wrote: On 27/02/17 18:00, Andres Freund wrote: FWIW I think the ppc64 machines are failing because of unrelated issue (changes to integer timestamps). We should probably look at 32bit machines first.

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-27 12:27:48 -0500, Tom Lane wrote: > Andres Freund writes: > > The best theory I have so far that I have is that slab.c's idea of > > StandardChunkHeader's size doesn't match what mcxt.c think it is > > (because slab.c simply embeds StandardChunkHeader, but mcxt

Re: [HACKERS] PATCH: two slab-like memory allocators

Andres Freund writes: > The best theory I have so far that I have is that slab.c's idea of > StandardChunkHeader's size doesn't match what mcxt.c think it is > (because slab.c simply embeds StandardChunkHeader, but mcxt uses > MAXALIGN(sizeof(StandardChunkHeader))). That's

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-27 18:04:41 +0100, Petr Jelinek wrote: > On 27/02/17 18:00, Andres Freund wrote: > > > >> FWIW I think the ppc64 machines are failing because of unrelated issue > >> (changes to integer timestamps). We should probably look at 32bit machines > >> first. > > > > Don't think so - termite

Re: [HACKERS] PATCH: two slab-like memory allocators

Andres Freund writes: >> FWIW I think the ppc64 machines are failing because of unrelated issue >> (changes to integer timestamps). We should probably look at 32bit machines >> first. > Don't think so - termite is ppc64 afaics, and the failure doesn't look > integer timestamp

Re: [HACKERS] PATCH: two slab-like memory allocators

On 27/02/17 18:00, Andres Freund wrote: > >> FWIW I think the ppc64 machines are failing because of unrelated issue >> (changes to integer timestamps). We should probably look at 32bit machines >> first. > > Don't think so - termite is ppc64 afaics, and the failure doesn't look > integer

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-27 17:56:08 +0100, Tomas Vondra wrote: > No, I don't, but I'll ping Craig. I might ping him, but it's ~4AM in > Australia, though, so it'll take time. Did the same... ;) > FWIW I think the ppc64 machines are failing because of unrelated issue > (changes to integer timestamps).

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/27/2017 05:48 PM, Andres Freund wrote: On 2017-02-27 07:55:32 -0800, Andres Freund wrote: On 2017-02-27 10:32:25 -0500, Tom Lane wrote: Andres Freund writes: And pushed slab and its usage. Will have a look at generation.c tomorrow. Perhaps first you need to find

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-27 15:11:40 +0100, Tomas Vondra wrote: > > I've not yet reviewed the generational allocator yet, but during these > > measurements I get: > > postgres[3970][1]=# select count(*) FROM pg_logical_slot_get_changes('ttt', > > NULL, NULL); > > WARNING: 01000: problem in Generation

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-27 07:55:32 -0800, Andres Freund wrote: > On 2017-02-27 10:32:25 -0500, Tom Lane wrote: > > Andres Freund writes: > > > And pushed slab and its usage. Will have a look at generation.c > > > tomorrow. > > > > Perhaps first you need to find out why so much of the

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-27 10:32:25 -0500, Tom Lane wrote: > Andres Freund writes: > > And pushed slab and its usage. Will have a look at generation.c > > tomorrow. > > Perhaps first you need to find out why so much of the buildfarm > is unhappy. Will do, after a morning coffee. -

Re: [HACKERS] PATCH: two slab-like memory allocators

Andres Freund writes: > And pushed slab and its usage. Will have a look at generation.c > tomorrow. Perhaps first you need to find out why so much of the buildfarm is unhappy. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] PATCH: two slab-like memory allocators

On February 27, 2017 6:14:20 AM PST, Tomas Vondra wrote: >On 02/27/2017 01:02 PM, Andres Freund wrote: >> Hi, >> >> On 2017-02-27 03:17:32 -0800, Andres Freund wrote: >>> I'll work on getting slab committed first, and then review / edit / >>> commit generation.c

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/27/2017 01:02 PM, Andres Freund wrote: Hi, On 2017-02-27 03:17:32 -0800, Andres Freund wrote: I'll work on getting slab committed first, and then review / edit / commit generation.c later. One first note there is that I'm wondering if generation.c is a too generic filename. And pushed

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/27/2017 12:17 PM, Andres Freund wrote: Hi, On 2017-02-24 14:10:38 -0800, Andres Freund wrote: I've not yet looked a lot at the next type of context - I want to get this much committed first... I plan to give this another pass sometime this weekend and then push soon. Before committing

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-27 03:17:32 -0800, Andres Freund wrote: > I'll work on getting slab committed first, and then review / edit / > commit generation.c later. One first note there is that I'm wondering > if generation.c is a too generic filename. And pushed slab and its usage. Will have a look at

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-24 14:10:38 -0800, Andres Freund wrote: > I've not yet looked a lot at the next type of context - I want to get > this much committed first... > > I plan to give this another pass sometime this weekend and then push > soon. Before committing I wanted to make sure that

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-21 01:43:46 +0100, Tomas Vondra wrote: > Attached is v9 of this patch series. This addresses most of the points > raised in the review, namely: Cool, thanks. > 3) get rid of the block-level bitmap tracking free chunks > > Instead of the bitmap, I've used a simple singly-linked

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, Attached is v9 of this patch series. This addresses most of the points raised in the review, namely: 1) change most 'debug' stuff to 'static inline' in memdebug.h 2) fixed and reworded a bunch of comments 3) get rid of the block-level bitmap tracking free chunks Instead of the bitmap,

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/14/2017 03:22 AM, Andres Freund wrote: Hi, On 2017-02-11 14:40:18 +0100, Tomas Vondra wrote: On 02/11/2017 02:33 AM, Andres Freund wrote: I have a hard time believing this the cache efficiency of linked lists (which may or may not be real in this case) out-weights this, but if you want

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2017-02-11 14:40:18 +0100, Tomas Vondra wrote: > On 02/11/2017 02:33 AM, Andres Freund wrote: > > > I have a hard time believing this the cache efficiency of linked lists > > > (which may or may not be real in this case) out-weights this, but if you > > > want to try, be my guest. > > > >

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/11/2017 02:33 AM, Andres Freund wrote: On 2017-02-11 02:13:59 +0100, Tomas Vondra wrote: On 02/09/2017 10:37 PM, Andres Freund wrote: Hi, On 2016-12-13 01:45:13 +0100, Tomas Vondra wrote: src/backend/utils/mmgr/Makefile | 2 +- src/backend/utils/mmgr/aset.c | 115

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-11 02:13:59 +0100, Tomas Vondra wrote: > > > + /* move the whole block to the right place in the freelist */ > > > + dlist_delete(>node); > > > + dlist_push_head(>freelist[block->nfree], >node); > > > > Hm. What if we, instead of the array of doubly linked lists, just kept > > a

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2017-02-11 02:13:59 +0100, Tomas Vondra wrote: > On 02/09/2017 10:37 PM, Andres Freund wrote: > > Hi, > > > > On 2016-12-13 01:45:13 +0100, Tomas Vondra wrote: > > > src/backend/utils/mmgr/Makefile | 2 +- > > > src/backend/utils/mmgr/aset.c | 115 + > >

Re: [HACKERS] PATCH: two slab-like memory allocators

On 02/09/2017 10:37 PM, Andres Freund wrote: Hi, On 2016-12-13 01:45:13 +0100, Tomas Vondra wrote: src/backend/utils/mmgr/Makefile | 2 +- src/backend/utils/mmgr/aset.c | 115 + src/backend/utils/mmgr/memdebug.c | 131

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, On 2016-12-13 01:45:13 +0100, Tomas Vondra wrote: > src/backend/utils/mmgr/Makefile | 2 +- > src/backend/utils/mmgr/aset.c | 115 + > src/backend/utils/mmgr/memdebug.c | 131 > ++ > src/include/utils/memdebug.h

Re: [HACKERS] PATCH: two slab-like memory allocators

On Tue, Dec 13, 2016 at 10:32 AM, Petr Jelinek wrote: > Okay, this version looks good to me, marked as RfC. The patches still apply, moved to CF 2017-03 with same status: RfC. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] PATCH: two slab-like memory allocators

On 13/12/16 01:45, Tomas Vondra wrote: > On 12/12/2016 11:39 PM, Tomas Vondra wrote: >> On 12/12/2016 05:05 AM, Petr Jelinek wrote: >>> >>> I'd be happy with this patch now (as in committer ready) except that it >>> does have some merge conflicts after the recent commits, so rebase is >>> needed.

Re: [HACKERS] PATCH: two slab-like memory allocators

On 12/12/2016 11:39 PM, Tomas Vondra wrote: On 12/12/2016 05:05 AM, Petr Jelinek wrote: I'd be happy with this patch now (as in committer ready) except that it does have some merge conflicts after the recent commits, so rebase is needed. Attached is a rebased version of the patch, resolving

Re: [HACKERS] PATCH: two slab-like memory allocators

On 12/12/2016 05:05 AM, Petr Jelinek wrote: I'd be happy with this patch now (as in committer ready) except that it does have some merge conflicts after the recent commits, so rebase is needed. Attached is a rebased version of the patch, resolving the Makefile merge conflicts. regards --

Re: [HACKERS] PATCH: two slab-like memory allocators

On 01/12/16 03:26, Tomas Vondra wrote: > > Dne 11/27/2016 v 11:02 PM Andres Freund napsal(a): >> On 2016-11-27 22:21:49 +0100, Petr Jelinek wrote: >>> On 27/11/16 21:47, Andres Freund wrote: Hi, >> +typedef struct SlabBlockData *SlabBlock;/* forward >> reference */

Re: [HACKERS] PATCH: two slab-like memory allocators

On Thu, Dec 1, 2016 at 1:26 PM, Tomas Vondra wrote: > > > Dne 11/27/2016 v 11:02 PM Andres Freund napsal(a): > >> On 2016-11-27 22:21:49 +0100, Petr Jelinek wrote: >> >>> On 27/11/16 21:47, Andres Freund wrote: >>> Hi, +typedef struct SlabBlockData

Re: [HACKERS] PATCH: two slab-like memory allocators

Dne 11/27/2016 v 11:02 PM Andres Freund napsal(a): On 2016-11-27 22:21:49 +0100, Petr Jelinek wrote: On 27/11/16 21:47, Andres Freund wrote: Hi, +typedef struct SlabBlockData *SlabBlock; /* forward reference */ +typedef struct SlabChunkData *SlabChunk; Can we please not

Re: [HACKERS] PATCH: two slab-like memory allocators

On 2016-11-27 22:21:49 +0100, Petr Jelinek wrote: > On 27/11/16 21:47, Andres Freund wrote: > > Hi, > > > >>> +typedef struct SlabBlockData *SlabBlock; /* forward reference */ > >>> +typedef struct SlabChunkData *SlabChunk; > >>> > >>> Can we please not continue hiding pointers behind

Re: [HACKERS] PATCH: two slab-like memory allocators

On 27/11/16 21:47, Andres Freund wrote: > Hi, > >>> +typedef struct SlabBlockData *SlabBlock; /* forward reference */ >>> +typedef struct SlabChunkData *SlabChunk; >>> >>> Can we please not continue hiding pointers behind typedefs? It's a bad >>> pattern, and that it's fairly widely

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, > > +typedef struct SlabBlockData *SlabBlock; /* forward reference */ > > +typedef struct SlabChunkData *SlabChunk; > > > > Can we please not continue hiding pointers behind typedefs? It's a bad > > pattern, and that it's fairly widely used isn't a good excuse to > > introduce

Re: [HACKERS] PATCH: two slab-like memory allocators

On 11/27/2016 07:25 PM, Petr Jelinek wrote: On 15/11/16 01:44, Tomas Vondra wrote: Attached is v6 of the patch series, fixing most of the points: * common bits (valgrind/randomization/wipe) moved to memdebug.h/c Instead of introducing a new header file, I've added the prototypes to memdebug.h

Re: [HACKERS] PATCH: two slab-like memory allocators

On 15/11/16 01:44, Tomas Vondra wrote: > Attached is v6 of the patch series, fixing most of the points: > > * common bits (valgrind/randomization/wipe) moved to memdebug.h/c > > Instead of introducing a new header file, I've added the prototypes to > memdebug.h (which was already used for the

Re: [HACKERS] PATCH: two slab-like memory allocators

Attached is v6 of the patch series, fixing most of the points: * common bits (valgrind/randomization/wipe) moved to memdebug.h/c Instead of introducing a new header file, I've added the prototypes to memdebug.h (which was already used for the valgrind stuff anyway), and the implementations to

Re: [HACKERS] PATCH: two slab-like memory allocators

On 11/12/2016 08:12 PM, Andres Freund wrote: Hi, Subject: [PATCH 1/2] slab allocator diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 6ad7e7d..520f295 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, Subject: [PATCH 1/2] slab allocator diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 6ad7e7d..520f295 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c I'd

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/25/16 4:48 PM, Tomas Vondra wrote: The main issue that bugs me is the name of the Gen allocator, but I don't have a good naming ideas :( The basic characteristics of Gen is that it does not reuse space released by pfree(), relying on the fact that the whole block will become free. That

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/1/16 7:34 PM, Tomas Vondra wrote: +/* otherwise add it to the proper freelist bin */ Looks like something went missing... :) Ummm? The patch contains this: +/* otherwise add it to the proper freelist bin */ +if (set->freelist[block->nfree]) +

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/23/2016 05:26 PM, Petr Jelinek wrote: On 23/10/16 16:26, Tomas Vondra wrote: On 10/22/2016 08:30 PM, Tomas Vondra wrote: ... Moreover, the slab/gen allocators proposed here seem like a better fit for reorderbuffer, e.g. because they release memory. I haven't looked at sb_alloc too

Re: [HACKERS] PATCH: two slab-like memory allocators

On 23/10/16 16:26, Tomas Vondra wrote: > On 10/22/2016 08:30 PM, Tomas Vondra wrote: >> On 10/20/2016 04:43 PM, Robert Haas wrote: >>> >>> ... >>> >>> The sb_alloc allocator I proposed a couple of years ago would work >>> well for this case, I think. >>> >> >> Maybe, but it does not follow the

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/22/2016 08:30 PM, Tomas Vondra wrote: On 10/20/2016 04:43 PM, Robert Haas wrote: >> ... The sb_alloc allocator I proposed a couple of years ago would work well for this case, I think. Maybe, but it does not follow the Memory Context design at all, if I understand it correctly. I was

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/20/2016 04:43 PM, Robert Haas wrote: On Tue, Oct 18, 2016 at 6:27 PM, Petr Jelinek wrote: I agree though that the usability beyond the ReoderBuffer is limited because everything that will want to use it for part of allocations will get much more complicated by the

Re: [HACKERS] PATCH: two slab-like memory allocators

On Tue, Oct 18, 2016 at 6:27 PM, Petr Jelinek wrote: > I agree though that the usability beyond the ReoderBuffer is limited > because everything that will want to use it for part of allocations will > get much more complicated by the fact that it will have to use two >

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/19/2016 02:51 PM, Tomas Vondra wrote: ... > Yeah. There are three contexts in reorder buffers: - changes (fixed size) - txns (fixed size) - tuples (variable size) The first two work perfectly fine with Slab. The last one (tuples) is used to allocate variable-sized bits, so I've tried

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/19/2016 12:27 AM, Petr Jelinek wrote: > On 18/10/16 22:25, Robert Haas wrote: >> On Wed, Oct 5, 2016 at 12:22 AM, Tomas Vondra >> wrote: >>> attached is v3 of the patches, with a few minor fixes in Slab, and much >>> larger fixes in GenSlab. >>> >>> Slab

Re: [HACKERS] PATCH: two slab-like memory allocators

On 18/10/16 22:25, Robert Haas wrote: > On Wed, Oct 5, 2016 at 12:22 AM, Tomas Vondra > wrote: >> attached is v3 of the patches, with a few minor fixes in Slab, and much >> larger fixes in GenSlab. >> >> Slab (minor fixes) >> -- >> - Removed the

Re: [HACKERS] PATCH: two slab-like memory allocators

On Wed, Oct 5, 2016 at 12:22 AM, Tomas Vondra wrote: > attached is v3 of the patches, with a few minor fixes in Slab, and much > larger fixes in GenSlab. > > Slab (minor fixes) > -- > - Removed the unnecessary memset() of new blocks in SlabAlloc(),

Re: [HACKERS] PATCH: two slab-like memory allocators

On Tue, Oct 4, 2016 at 10:11 PM, Tomas Vondra For GenSlab the situation is less clear, as there probably are ways to make > it work, but I'd vote to keep it simple for now, and simply do elog(ERROR) > in the realloc() methods - both for Slab and GenSlab. The current use case > (reorderbuffer)

Re: [HACKERS] PATCH: two slab-like memory allocators

On 05/10/16 03:11, Tomas Vondra wrote: > On 10/04/2016 09:44 PM, John Gorman wrote: >> >> Remind me again why we cannot realloc in place for sizes >> smaller than chunkSize? GenSlab is happy to allocate smaller >> sizes and put them into the fixed size chunks. >> >> Maybe SlabAlloc can be happy

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, attached is v3 of the patches, with a few minor fixes in Slab, and much larger fixes in GenSlab. Slab (minor fixes) -- - Removed the unnecessary memset() of new blocks in SlabAlloc(), although we still need to zero the free bitmap at the end of the block. - Renamed

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/04/2016 09:44 PM, John Gorman wrote: SlabContext has a parent context. It can delegate requests that it cannot handle to the parent context which is GenSlab. Genslab can send them to the sister aset context. But Slab may also be used separately, not just as part of GenSlab (actually,

Re: [HACKERS] PATCH: two slab-like memory allocators

SlabContext has a parent context. It can delegate requests that it cannot handle to the parent context which is GenSlab. Genslab can send them to the sister aset context. This could handle all reallocs so there will be no surprises. Remind me again why we cannot realloc in place for sizes

Re: [HACKERS] PATCH: two slab-like memory allocators

On Sun, Oct 2, 2016 at 10:15 AM, Tomas Vondra wrote: > One more comment about GenSlab, particularly about unpredictability of the > repalloc() behavior. It works for "large" chunks allocated in the AllocSet > part, and mostly does not work for "small" chunks

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/02/2016 12:23 AM, John Gorman wrote: I reproduced the quadradic pfree performance problem and verified that these patches solved it. The slab.c data structures and functions contain no quadradic components. I noticed the sizing loop in SlabContextCreate() and came up with a similar

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/02/2016 12:23 AM, John Gorman wrote: I reproduced the quadradic pfree performance problem and verified that these patches solved it. The slab.c data structures and functions contain no quadradic components. I noticed the sizing loop in SlabContextCreate() and came up with a similar

Re: [HACKERS] PATCH: two slab-like memory allocators

On 10/02/2016 01:53 AM, Jim Nasby wrote: On 9/26/16 9:10 PM, Tomas Vondra wrote: Attached is v2 of the patch, updated based on the review. That means: +/* make sure the block can store at least one chunk (with 1B for a bitmap)? */ (and the comment below it) I find the question to be

Re: [HACKERS] PATCH: two slab-like memory allocators

On 9/26/16 9:10 PM, Tomas Vondra wrote: Attached is v2 of the patch, updated based on the review. That means: + /* make sure the block can store at least one chunk (with 1B for a bitmap)? */ (and the comment below it) I find the question to be confusing... I think these would be better as

Re: [HACKERS] PATCH: two slab-like memory allocators

I reproduced the quadradic pfree performance problem and verified that these patches solved it. The slab.c data structures and functions contain no quadradic components. I noticed the sizing loop in SlabContextCreate() and came up with a similar formula to determine chunksPerBlock that you

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi, Attached is v2 of the patch, updated based on the review. That means: - Better comment explaining how free chunks are tracked in Slab context. - Removed the unused SlabPointerIsValid macro. - Modified the comment before SlabChunkData, explaining how it relates to StandardChunkHeader. -

Re: [HACKERS] PATCH: two slab-like memory allocators

On 25/09/16 22:17, Tomas Vondra wrote: > On 09/25/2016 08:48 PM, Petr Jelinek wrote: > >> Slab: >> In general it seems understandable, the initial description helps to >> understand what's happening well enough. >> >> One thing I don't understand however is why the freelist is both >> array and

Re: [HACKERS] PATCH: two slab-like memory allocators

On 09/25/2016 08:48 PM, Petr Jelinek wrote: Hi Tomas, On 02/08/16 17:44, Tomas Vondra wrote: This patch actually includes two new memory allocators (not one). Very brief summary (for more detailed explanation of the ideas, see comments at the beginning of slab.c and genslab.c): Slab *

Re: [HACKERS] PATCH: two slab-like memory allocators

Hi Tomas, On 02/08/16 17:44, Tomas Vondra wrote: > > This patch actually includes two new memory allocators (not one). Very > brief summary (for more detailed explanation of the ideas, see comments > at the beginning of slab.c and genslab.c): > > Slab > > * suitable for fixed-length

[HACKERS] PATCH: two slab-like memory allocators

Hi, Back in the bug #14231 thread [1], dealing with performance issues in reorderbuffer due to excessive number of expensive free() calls, I've proposed to resolve that by a custom slab-like memory allocator, suitable for fixed-size allocations. I'd like to put this into the next CF, and