Re: [HACKERS] Changeset Extraction v7.6.1
On 2014-06-01 00:50:58 -0500, Jim Nasby wrote: > On 5/31/14, 9:11 AM, Andres Freund wrote: > >On 2014-02-21 15:14:15 -0600, Jim Nasby wrote: > >>On 2/17/14, 7:31 PM, Robert Haas wrote: > >>>But do you really want to keep that snapshot around long enough to > >>>copy the entire database? I bet you don't: if the database is big, > >>>holding back xmin for long enough to copy the whole thing isn't likely > >>>to be fun. > >> > >>I can confirm that this would be epic fail, at least for londiste. It takes > >>about 3 weeks for a new copy of a ~2TB database. There's no way that'd work > >>with one snapshot. (Granted, copy performance in londiste is rather > >>lackluster, but still...) > > > >I'd marked this email as todo: > >If you have such a huge database you can, with logical decoding at > >least, use a basebackup using pg_basebackup or pg_start/stop_backup() > >and roll forwards from that... That'll hopefull make such huge copies > >much faster. > Just keep in mind that one of the use cases for logical replication is > upgrades. Should still be fine. Make a physical copy; pg_upgrade; catchup via logical rep. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Changeset Extraction v7.6.1
On 5/31/14, 9:11 AM, Andres Freund wrote: On 2014-02-21 15:14:15 -0600, Jim Nasby wrote: On 2/17/14, 7:31 PM, Robert Haas wrote: But do you really want to keep that snapshot around long enough to copy the entire database? I bet you don't: if the database is big, holding back xmin for long enough to copy the whole thing isn't likely to be fun. I can confirm that this would be epic fail, at least for londiste. It takes about 3 weeks for a new copy of a ~2TB database. There's no way that'd work with one snapshot. (Granted, copy performance in londiste is rather lackluster, but still...) I'd marked this email as todo: If you have such a huge database you can, with logical decoding at least, use a basebackup using pg_basebackup or pg_start/stop_backup() and roll forwards from that... That'll hopefull make such huge copies much faster. Just keep in mind that one of the use cases for logical replication is upgrades. -- Jim C. Nasby, Data Architect j...@nasby.net 512.569.9461 (cell) http://jim.nasby.net -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch
On Sat, May 31, 2014 at 9:44 PM, johnlumby wrote: > I'll try to do some measuring of performance with: > a) git head > b) git head + patch as-is > c) git head + patch without aio_suspend in foreign processes (just re-read) > d) git head + patch with a lwlock (or whatever works) instead of aio_suspend > > a-c will be the fastest, d might take some while. > > I'll let you know of the results as I get them. > > > Claudio, I am not quite sure if what I am submitting now is > quite the same as any of yours. As I promised before, but have > not yet done, I will package one or two of my benchmarks and > send them in. It's a tad different. c will not do polling on the foreign process, I will just let PG do the read again. d will be like polling, but without the associated CPU overhead. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch
On 05/31/14 20:44, johnlumby wrote: On 05/30/14 09:36, Claudio Freire wrote: Good point. I have included the guts of your little test program (modified to do polling) into the existing autoconf test program that decides on the #define USE_AIO_ATOMIC_BUILTIN_COMP_SWAP. See config/c-library.m4. I hope this goes some way to answer your concern about robustness, as at least now if the implementation changes in some way that renders the polling ineffective, it will be caught in configure. I meant to add that by including this test, which involves a fork(), in the autoconf tester, on Windows USE_AIO_ATOMIC_BUILTIN_COMP_SWAP would always by un-defined. (But could then be defined manually if someone wanted to give it a try)
Re: [HACKERS] Re: Logical slots not mentioned in CREATE_REPLICATION_SLOT for replication protocol docs
On Sat, May 31, 2014 at 11:02 PM, Andres Freund wrote: > On 2014-05-30 08:12:35 +0900, Michael Paquier wrote: >> On Fri, May 30, 2014 at 5:31 AM, Robert Haas wrote: >> > Thanks, this looks good. But shouldn't the bit about output plugin >> > options mention say something like: >> > >> > ( option_name option_argument [, ...] ) >> > >> > ...instead of just: >> > >> > ( option [, ...] ) >> > ? >> Yes, true. Here is an updated patch. > > I've pushed the patch with some additional changes. Hope they make sense > to you. Thanks, those changes look good. Limiting the uses of literal and noticing that plugin options can have empty values are good catches that I didn't get in my patch. > I wonder whether START_REPLICATION ... LOGICAL should be folded into > ... PHYSICAL since they have most things in common. But it'd be a > awfully complicated description. Yeah let's keep them separated, grouping them may be painful. Btw, I think that we should explicitly list the types of messages that a logical receiver can receive and use, aka: - 'w' for stream data - 'k' for keepalive messages - 'r' for feedback The logical messages have the same format as in the physical replication case, the only difference being that the message 'h' cannot be used. Thoughts? Regards, -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Changeset Extraction v7.6.1
On 2014-02-21 15:14:15 -0600, Jim Nasby wrote: > On 2/17/14, 7:31 PM, Robert Haas wrote: > >But do you really want to keep that snapshot around long enough to > >copy the entire database? I bet you don't: if the database is big, > >holding back xmin for long enough to copy the whole thing isn't likely > >to be fun. > > I can confirm that this would be epic fail, at least for londiste. It takes > about 3 weeks for a new copy of a ~2TB database. There's no way that'd work > with one snapshot. (Granted, copy performance in londiste is rather > lackluster, but still...) I'd marked this email as todo: If you have such a huge database you can, with logical decoding at least, use a basebackup using pg_basebackup or pg_start/stop_backup() and roll forwards from that... That'll hopefull make such huge copies much faster. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] FOR [SHARE|UPDATE] NOWAIT may still block in EvalPlanQualFetch
Hi, On 2014-01-31 16:28:08 -0500, Bruce Momjian wrote: > On Fri, Aug 2, 2013 at 04:00:03PM +0800, Craig Ringer wrote: > > FOR SHARE|UPDATE NOWAIT will still block if they have to follow a ctid > > chain because the call to EvalPlanQualFetch doesn't take a param for > > noWait, so it doesn't know not to block if the updated row can't be locked. > > > > The attached patch against master includes an isolationtester spec to > > demonstrate this issue and a proposed fix. Builds with the fix applied > > pass "make check" and isolationtester "make installcheck". > > > > To reach this point you need to apply the patch in > > http://www.postgresql.org/message-id/51fb4305.3070...@2ndquadrant.com > > first, otherwise you'll get stuck there and won't touch the code changed > > in this patch. > > The above looks like a legitimate patch that was not applied: > > http://www.postgresql.org/message-id/51fb6703.9090...@2ndquadrant.com > > The patch mentioned in the text above was applied, I think. Craig: I think you should add this to the next CF. Seems like something we should fix, but which isn't super urgent. But when the skip locked stuff comes in it'll be more relevant. Might also need a rebase. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: Logical slots not mentioned in CREATE_REPLICATION_SLOT for replication protocol docs
On 2014-05-30 08:12:35 +0900, Michael Paquier wrote: > On Fri, May 30, 2014 at 5:31 AM, Robert Haas wrote: > > Thanks, this looks good. But shouldn't the bit about output plugin > > options mention say something like: > > > > ( option_name option_argument [, ...] ) > > > > ...instead of just: > > > > ( option [, ...] ) > > ? > Yes, true. Here is an updated patch. I've pushed the patch with some additional changes. Hope they make sense to you. I wonder whether START_REPLICATION ... LOGICAL should be folded into ... PHYSICAL since they have most things in common. But it'd be a awfully complicated description. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
