Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-09-15 15:39:49 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-09-14 23:29:05 -0400, Tom Lane wrote: > >> FWIW, I'm not on board with that. I think the version of typedefs.list > >> in the tree should reflect the last official pgindent run. > > > Why? I

Re: [HACKERS] POC: Sharing record typmods between backends

Andres Freund writes: > On 2017-09-14 23:29:05 -0400, Tom Lane wrote: >> FWIW, I'm not on board with that. I think the version of typedefs.list >> in the tree should reflect the last official pgindent run. > Why? I see pretty much no upside to that. You can't reindent

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-09-14 23:29:05 -0400, Tom Lane wrote: > Thomas Munro writes: > > On Fri, Sep 15, 2017 at 3:03 PM, Andres Freund wrote: > >> - added typedefs to typedefs.list > > > Should I do this manually with future patches? I think there's sort of

Re: [HACKERS] POC: Sharing record typmods between backends

Thomas Munro writes: > On Fri, Sep 15, 2017 at 3:03 PM, Andres Freund wrote: >> - added typedefs to typedefs.list > Should I do this manually with future patches? FWIW, I'm not on board with that. I think the version of typedefs.list in the

Re: [HACKERS] POC: Sharing record typmods between backends

On Fri, Sep 15, 2017 at 3:03 PM, Andres Freund wrote: > On 2017-09-04 18:14:39 +1200, Thomas Munro wrote: >> Thanks for the review and commits so far. Here's a rebased, debugged >> and pgindented version of the remaining patches. > > I've pushed this with minor modifications:

Re: [HACKERS] POC: Sharing record typmods between backends

Hi, On 2017-09-04 18:14:39 +1200, Thomas Munro wrote: > Thanks for the review and commits so far. Here's a rebased, debugged > and pgindented version of the remaining patches. I've pushed this with minor modifications: - added typedefs to typedefs.list - re-pgindented, there were some missing

Re: [HACKERS] POC: Sharing record typmods between backends

Thanks for the review and commits so far. Here's a rebased, debugged and pgindented version of the remaining patches. I ran pgindent with --list-of-typedefs="SharedRecordTableKey,SharedRecordTableEntry,SharedTypmodTableEntry,SharedRecordTypmodRegistry,Session" to fix some weirdness around these

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-21 11:02:52 +1200, Thomas Munro wrote: > 2. Andres didn't like what I did to DecrTupleDescRefCount, namely > allowing to run when there is no ResourceOwner. I now see that this > is probably an indication of a different problem; even if there were a > worker ResourceOwner as he

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, Aug 23, 2017 at 11:58 PM, Thomas Munro wrote: > On Wed, Aug 23, 2017 at 5:46 PM, Andres Freund wrote: >> Notes for possible followup commits of the dshash API: >> - nontrivial portions of dsahash are essentially critical sections lest >>

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, Aug 23, 2017 at 12:42 PM, Andres Freund wrote: > I don't think that's sufficient. make, and especially check-world, > should have a decent coverage of the code locally. Without having to > know about options like force_parallel_mode=regress. As e.g. evidenced > by the

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-23 09:45:38 -0400, Robert Haas wrote: > On Wed, Aug 23, 2017 at 1:46 AM, Andres Freund wrote: > > For later commits in the series: > > - Afaict the whole shared tupledesc stuff, as tqueue.c before, is > > entirely untested. This baffles me. See also [1]. I can

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, Aug 23, 2017 at 1:46 AM, Andres Freund wrote: > For later commits in the series: > - Afaict the whole shared tupledesc stuff, as tqueue.c before, is > entirely untested. This baffles me. See also [1]. I can force the code > to be reached with

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, Aug 23, 2017 at 11:58 PM, Thomas Munro wrote: > On Wed, Aug 23, 2017 at 5:46 PM, Andres Freund wrote: >> - Afaict GetSessionDsmHandle() uses the current rather than >> TopMemoryContext. Try running the regression tests under >>

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, Aug 23, 2017 at 5:46 PM, Andres Freund wrote: > Committing 0003. This'll probably need further adjustment, but I think > it's good to make progress here. Thanks! > Changes: > - pgindent'ed after adding the necessary typedefs to typedefs.list > - replaced INT64CONST w

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-22 16:41:23 -0700, Andres Freund wrote: > On 2017-08-21 11:02:52 +1200, Thomas Munro wrote: > > On Mon, Aug 21, 2017 at 6:17 AM, Andres Freund wrote: > > > Pushing 0001, 0002 now. > > > > > > - rebased after conflicts > > > - fixed a significant number of too long

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-21 11:02:52 +1200, Thomas Munro wrote: > On Mon, Aug 21, 2017 at 6:17 AM, Andres Freund wrote: > > Pushing 0001, 0002 now. > > > > - rebased after conflicts > > - fixed a significant number of too long lines > > - removed a number of now superflous linebreaks > >

Re: [HACKERS] POC: Sharing record typmods between backends

On Mon, Aug 21, 2017 at 10:18 AM, Thomas Munro wrote: > On Mon, Aug 21, 2017 at 6:17 AM, Andres Freund wrote: >> I think it'd be a good idea to backpatch the addition of >> TupleDescAttr(tupledesc, n) to make future backpatching easier. What do

Re: [HACKERS] POC: Sharing record typmods between backends

On Mon, Aug 21, 2017 at 6:17 AM, Andres Freund wrote: > I think it'd be a good idea to backpatch the addition of > TupleDescAttr(tupledesc, n) to make future backpatching easier. What do > others think? +1 That would also provide a way for extension developers to be able to

Re: [HACKERS] POC: Sharing record typmods between backends

On Mon, Aug 21, 2017 at 6:17 AM, Andres Freund wrote: > Pushing 0001, 0002 now. > > - rebased after conflicts > - fixed a significant number of too long lines > - removed a number of now superflous linebreaks Thanks! Please find attached a rebased version of the rest of the

Re: [HACKERS] POC: Sharing record typmods between backends

Hi, Pushing 0001, 0002 now. - rebased after conflicts - fixed a significant number of too long lines - removed a number of now superflous linebreaks I think it'd be a good idea to backpatch the addition of TupleDescAttr(tupledesc, n) to make future backpatching easier. What do others think?

Re: [HACKERS] POC: Sharing record typmods between backends

On Tue, Aug 15, 2017 at 8:34 PM, Andres Freund wrote: > On 2017-08-15 20:30:16 -0400, Robert Haas wrote: >> On Tue, Aug 15, 2017 at 6:06 PM, Andres Freund wrote: >> > Interesting. I was apparently thinking slightly differently. I'd have >> > thought we'd

Re: [HACKERS] POC: Sharing record typmods between backends

Will respond to the actionable code review points separately with a new patch set, but first: On Wed, Aug 16, 2017 at 10:06 AM, Andres Freund wrote: > On 2017-08-15 17:44:55 +1200, Thomas Munro wrote: >> > @@ -99,12 +72,9 @@ CreateTemplateTupleDesc(int natts, bool hasoid) >>

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-15 20:30:16 -0400, Robert Haas wrote: > On Tue, Aug 15, 2017 at 6:06 PM, Andres Freund wrote: > > Interesting. I was apparently thinking slightly differently. I'd have > > thought we'd have Session struct in statically allocated shared > > memory. Which'd then have

Re: [HACKERS] POC: Sharing record typmods between backends

On Tue, Aug 15, 2017 at 6:06 PM, Andres Freund wrote: > Interesting. I was apparently thinking slightly differently. I'd have > thought we'd have Session struct in statically allocated shared > memory. Which'd then have dsa_handle, dshash_table_handle, ... members. Sounds an

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-15 17:44:55 +1200, Thomas Munro wrote: > > @@ -99,12 +72,9 @@ CreateTemplateTupleDesc(int natts, bool hasoid) > > > > /* > > * CreateTupleDesc > > - * This function allocates a new TupleDesc pointing to a given > > + * This function allocates a new TupleDesc

Re: [HACKERS] POC: Sharing record typmods between backends

On Tue, Aug 15, 2017 at 5:44 PM, Thomas Munro wrote: > On Mon, Aug 14, 2017 at 12:32 PM, Andres Freund wrote: >> But architecturally I'm still not sure I quite like the a bit ad-hoc >> manner session state is defined here. I think we much more

Re: [HACKERS] POC: Sharing record typmods between backends

On Mon, Aug 14, 2017 at 12:32 PM, Andres Freund wrote: > Review for 0001: > > I think you made a few long lines even longer, like: > > @@ -1106,11 +1106,11 @@ pltcl_trigger_handler(PG_FUNCTION_ARGS, > pltcl_call_state *call_state, >

Re: [HACKERS] POC: Sharing record typmods between backends

Hi, On 2017-08-11 20:39:13 +1200, Thomas Munro wrote: > Please find attached a new patch series. Review for 0001: I think you made a few long lines even longer, like: @@ -1106,11 +1106,11 @@ pltcl_trigger_handler(PG_FUNCTION_ARGS, pltcl_call_state *call_state,

Re: [HACKERS] POC: Sharing record typmods between backends

Robert Haas writes: > On Sat, Aug 12, 2017 at 11:30 PM, Andres Freund wrote: >> That seems to involve a lot more than this though, given that currently >> the stats collector data doesn't entirely have to be in memory. I've >> seen sites with a lot of

Re: [HACKERS] POC: Sharing record typmods between backends

On Sat, Aug 12, 2017 at 11:30 PM, Andres Freund wrote: > That seems to involve a lot more than this though, given that currently > the stats collector data doesn't entirely have to be in memory. I've > seen sites with a lot of databases with quite some per-database stats >

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-12 22:52:57 -0400, Robert Haas wrote: > On Fri, Aug 11, 2017 at 9:55 PM, Andres Freund wrote: > > Well, most of the potential usecases for dsmhash I've heard about so > > far, don't actually benefit much from incremental growth. In nearly all > > the implementations

Re: [HACKERS] POC: Sharing record typmods between backends

On Fri, Aug 11, 2017 at 9:55 PM, Andres Freund wrote: > Well, most of the potential usecases for dsmhash I've heard about so > far, don't actually benefit much from incremental growth. In nearly all > the implementations I've seen incremental move ends up requiring more >

Re: [HACKERS] POC: Sharing record typmods between backends

Thanks for your feedback. Here are two parts that jumped out at me. I'll address the other parts in a separate email. On Sat, Aug 12, 2017 at 1:55 PM, Andres Freund wrote: >> This is complicated, and in the category that I would normally want a >> stack of heavy unit tests

Re: [HACKERS] POC: Sharing record typmods between backends

Hi, On 2017-08-11 20:39:13 +1200, Thomas Munro wrote: > Please find attached a new patch series. I apologise in advance for > 0001 and note that the patchset now weighs in at ~75kB compressed. > Here are my in-line replies to your two reviews: Replying to a few points here, then I'll do a pass

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-08-11 11:14:44 -0400, Robert Haas wrote: > On Fri, Aug 11, 2017 at 4:39 AM, Thomas Munro > wrote: > > OK. Now it's ds_hash_table.{c,h}, where "ds" stands for "dynamic > > shared". Better? If we were to do other data structures in DSA > > memory they could

Re: [HACKERS] POC: Sharing record typmods between backends

On Fri, Aug 11, 2017 at 4:39 AM, Thomas Munro wrote: > OK. Now it's ds_hash_table.{c,h}, where "ds" stands for "dynamic > shared". Better? If we were to do other data structures in DSA > memory they could follow that style: ds_red_black_tree.c, ds_vector.c, >

Re: [HACKERS] POC: Sharing record typmods between backends

Hi, Please find attached a new patch series. I apologise in advance for 0001 and note that the patchset now weighs in at ~75kB compressed. Here are my in-line replies to your two reviews: On Tue, Jul 25, 2017 at 10:09 PM, Andres Freund wrote: > It does concern me that we're

Re: [HACKERS] POC: Sharing record typmods between backends

Hi, diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 9fd7b4e019b..97c0125a4ba 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -337,17 +337,75 @@ DecrTupleDescRefCount(TupleDesc tupdesc) {

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-07-10 21:39:09 +1200, Thomas Munro wrote: > Here's a new version that introduces a per-session DSM segment to hold > the shared record typmod registry (and maybe more things later). You like to switch it up. *.patchset.tgz??? ;) It does concern me that we're growing yet another somewhat

Re: [HACKERS] POC: Sharing record typmods between backends

On Thu, Jun 1, 2017 at 6:29 AM, Andres Freund wrote: > On May 31, 2017 11:28:18 AM PDT, Robert Haas wrote: >>> On 2017-05-31 13:27:28 -0400, Dilip Kumar wrote: [ ... various discussion in support of using DHT ... ] Ok, good. Here's a new version that

Re: [HACKERS] POC: Sharing record typmods between backends

On May 31, 2017 11:28:18 AM PDT, Robert Haas wrote: >On Wed, May 31, 2017 at 1:46 PM, Andres Freund >wrote: >> On 2017-05-31 13:27:28 -0400, Dilip Kumar wrote: >>> On Wed, May 31, 2017 at 12:53 PM, Robert Haas > wrote: >>> >

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, May 31, 2017 at 1:46 PM, Andres Freund wrote: > On 2017-05-31 13:27:28 -0400, Dilip Kumar wrote: >> On Wed, May 31, 2017 at 12:53 PM, Robert Haas wrote: >> > Well, SH_TYPE's members SH_ELEMENT_TYPE *data and void *private_data >> > are not going

Re: [HACKERS] POC: Sharing record typmods between backends

On 2017-05-31 13:27:28 -0400, Dilip Kumar wrote: > On Wed, May 31, 2017 at 12:53 PM, Robert Haas wrote: > > Well, SH_TYPE's members SH_ELEMENT_TYPE *data and void *private_data > > are not going to work in DSM, because they are pointers. You can > > doubtless come up with

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, May 31, 2017 at 12:53 PM, Robert Haas wrote: > Well, SH_TYPE's members SH_ELEMENT_TYPE *data and void *private_data > are not going to work in DSM, because they are pointers. You can > doubtless come up with a way around that problem, but I guess the > question is

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, May 31, 2017 at 11:16 AM, Dilip Kumar wrote: > I agree with you. But, if I understand the use case correctly we need > to store the TupleDesc for the RECORD in shared hash so that it can be > shared across multiple processes. I think this can be achieved with > the

Re: [HACKERS] POC: Sharing record typmods between backends

On Wed, May 31, 2017 at 10:57 AM, Robert Haas wrote: >> Simplehash provides an option to provide your own allocator function >> to it. So in the allocator function, you can allocate memory from DSA. >> After it reaches some threshold it expands the size (double) and it >>

Re: [HACKERS] POC: Sharing record typmods between backends

On Tue, May 30, 2017 at 2:45 AM, Dilip Kumar wrote: > On Tue, May 30, 2017 at 1:09 AM, Thomas Munro > wrote: >>> * Perhaps simplehash + an LWLock would be better than dht, but I >>> haven't looked into that. Can it be convinced to work in

Re: [HACKERS] POC: Sharing record typmods between backends

On Tue, May 30, 2017 at 1:09 AM, Thomas Munro wrote: >> * Perhaps simplehash + an LWLock would be better than dht, but I >> haven't looked into that. Can it be convinced to work in DSA memory >> and to grow on demand? Simplehash provides an option to provide your

Re: [HACKERS] POC: Sharing record typmods between backends

On Fri, Apr 7, 2017 at 5:21 PM, Thomas Munro wrote: > * It would be nice for the SharedRecordTypeRegistry to be able to > survive longer than a single parallel query, perhaps in a per-session > DSM segment. Perhaps eventually we will want to consider a >

[HACKERS] POC: Sharing record typmods between backends

Hi hackers, Tuples can have type RECORDOID and a typmod number that identifies a "blessed" TupleDesc in a backend-private cache. To support the sharing of such tuples through shared memory and temporary files, I think we need a typmod registry in shared memory. Here's a proof-of-concept patch