Re: [HACKERS] Re: Logical slots not mentioned in CREATE_REPLICATION_SLOT for replication protocol docs

2014-05-31 Thread Andres Freund
On 2014-05-30 08:12:35 +0900, Michael Paquier wrote:
 On Fri, May 30, 2014 at 5:31 AM, Robert Haas robertmh...@gmail.com wrote:
  Thanks, this looks good.  But shouldn't the bit about output plugin
  options mention say something like:
 
  ( option_name option_argument [, ...] )
 
  ...instead of just:
 
  ( option [, ...] )
  ?
 Yes, true. Here is an updated patch.

I've pushed the patch with some additional changes. Hope they make sense
to you.

I wonder whether START_REPLICATION ... LOGICAL should be folded into
... PHYSICAL since they have most things in common. But it'd be a
awfully complicated description.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] FOR [SHARE|UPDATE] NOWAIT may still block in EvalPlanQualFetch

2014-05-31 Thread Andres Freund
Hi,

On 2014-01-31 16:28:08 -0500, Bruce Momjian wrote:
 On Fri, Aug  2, 2013 at 04:00:03PM +0800, Craig Ringer wrote:
  FOR SHARE|UPDATE NOWAIT will still block if they have to follow a ctid
  chain because the call to EvalPlanQualFetch doesn't take a param for
  noWait, so it doesn't know not to block if the updated row can't be locked.
  
  The attached patch against master includes an isolationtester spec to
  demonstrate this issue and a proposed fix. Builds with the fix applied
  pass make check and isolationtester make installcheck.
  
  To reach this point you need to apply the patch in
  http://www.postgresql.org/message-id/51fb4305.3070...@2ndquadrant.com
  first, otherwise you'll get stuck there and won't touch the code changed
  in this patch.
 
 The above looks like a legitimate patch that was not applied:
 
   http://www.postgresql.org/message-id/51fb6703.9090...@2ndquadrant.com
 
 The patch mentioned in the text above was applied, I think.

Craig: I think you should add this to the next CF. Seems like something
we should fix, but which isn't super urgent. But when the skip locked
stuff comes in it'll be more relevant.
Might also need a rebase.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Changeset Extraction v7.6.1

2014-05-31 Thread Andres Freund
On 2014-02-21 15:14:15 -0600, Jim Nasby wrote:
 On 2/17/14, 7:31 PM, Robert Haas wrote:
 But do you really want to keep that snapshot around long enough to
 copy the entire database?  I bet you don't: if the database is big,
 holding back xmin for long enough to copy the whole thing isn't likely
 to be fun.
 
 I can confirm that this would be epic fail, at least for londiste. It takes 
 about 3 weeks for a new copy of a ~2TB database. There's no way that'd work 
 with one snapshot. (Granted, copy performance in londiste is rather 
 lackluster, but still...)

I'd marked this email as todo:
If you have such a huge database you can, with logical decoding at
least, use a basebackup using pg_basebackup or pg_start/stop_backup()
and roll forwards from that... That'll hopefull make such huge copies
much faster.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: Logical slots not mentioned in CREATE_REPLICATION_SLOT for replication protocol docs

2014-05-31 Thread Michael Paquier
On Sat, May 31, 2014 at 11:02 PM, Andres Freund and...@2ndquadrant.com wrote:
 On 2014-05-30 08:12:35 +0900, Michael Paquier wrote:
 On Fri, May 30, 2014 at 5:31 AM, Robert Haas robertmh...@gmail.com wrote:
  Thanks, this looks good.  But shouldn't the bit about output plugin
  options mention say something like:
 
  ( option_name option_argument [, ...] )
 
  ...instead of just:
 
  ( option [, ...] )
  ?
 Yes, true. Here is an updated patch.

 I've pushed the patch with some additional changes. Hope they make sense
 to you.
Thanks, those changes look good. Limiting the uses of literal and
noticing that plugin options can have empty values are good catches
that I didn't get in my patch.

 I wonder whether START_REPLICATION ... LOGICAL should be folded into
 ... PHYSICAL since they have most things in common. But it'd be a
 awfully complicated description.
Yeah let's keep them separated, grouping them may be painful. Btw, I
think that we should explicitly list the types of messages that a
logical receiver can receive and use, aka:
- 'w' for stream data
- 'k' for keepalive messages
- 'r' for feedback
The logical messages have the same format as in the physical
replication case, the only difference being that the message 'h'
cannot be used. Thoughts?
Regards,
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-31 Thread johnlumby

On 05/31/14 20:44, johnlumby wrote:

On 05/30/14 09:36, Claudio Freire wrote:

Good point. I have included the guts of your little test program
(modified to do polling) into the existing autoconf test program
that decides on the
#define  USE_AIO_ATOMIC_BUILTIN_COMP_SWAP.
See config/c-library.m4.
I hope this goes some way to answer your concern about robustness,
as at least now if the implementation changes in some way that
renders the polling ineffective,  it will be caught in configure.



I meant to add that by including this test,  which involves a fork(),
in the autoconf tester,   on Windows
USE_AIO_ATOMIC_BUILTIN_COMP_SWAP would always by un-defined.
(But could then be defined manually if someone wanted to give it a try)



Re: [HACKERS] Extended Prefetching using Asynchronous IO - proposal and patch

2014-05-31 Thread Claudio Freire
On Sat, May 31, 2014 at 9:44 PM, johnlumby johnlu...@hotmail.com wrote:
 I'll try to do some measuring of performance with:
 a) git head
 b) git head + patch as-is
 c) git head + patch without aio_suspend in foreign processes (just re-read)
 d) git head + patch with a lwlock (or whatever works) instead of aio_suspend

 a-c will be the fastest, d might take some while.

 I'll let you know of the results as I get them.


 Claudio,  I am not quite sure if what I am submitting now is
 quite the same as any of yours.   As I promised before,  but have
 not yet done,  I will package one or two of my benchmarks and
 send them in.


It's a tad different. c will not do polling on the foreign process, I
will just let PG do the read again. d will be like polling, but
without the associated CPU overhead.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Changeset Extraction v7.6.1

2014-05-31 Thread Jim Nasby

On 5/31/14, 9:11 AM, Andres Freund wrote:

On 2014-02-21 15:14:15 -0600, Jim Nasby wrote:

On 2/17/14, 7:31 PM, Robert Haas wrote:

But do you really want to keep that snapshot around long enough to
copy the entire database?  I bet you don't: if the database is big,
holding back xmin for long enough to copy the whole thing isn't likely
to be fun.


I can confirm that this would be epic fail, at least for londiste. It takes 
about 3 weeks for a new copy of a ~2TB database. There's no way that'd work 
with one snapshot. (Granted, copy performance in londiste is rather lackluster, 
but still...)


I'd marked this email as todo:
If you have such a huge database you can, with logical decoding at
least, use a basebackup using pg_basebackup or pg_start/stop_backup()
and roll forwards from that... That'll hopefull make such huge copies
much faster.


Just keep in mind that one of the use cases for logical replication is upgrades.
--
Jim C. Nasby, Data Architect   j...@nasby.net
512.569.9461 (cell) http://jim.nasby.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Changeset Extraction v7.6.1

2014-05-31 Thread Andres Freund
On 2014-06-01 00:50:58 -0500, Jim Nasby wrote:
 On 5/31/14, 9:11 AM, Andres Freund wrote:
 On 2014-02-21 15:14:15 -0600, Jim Nasby wrote:
 On 2/17/14, 7:31 PM, Robert Haas wrote:
 But do you really want to keep that snapshot around long enough to
 copy the entire database?  I bet you don't: if the database is big,
 holding back xmin for long enough to copy the whole thing isn't likely
 to be fun.
 
 I can confirm that this would be epic fail, at least for londiste. It takes 
 about 3 weeks for a new copy of a ~2TB database. There's no way that'd work 
 with one snapshot. (Granted, copy performance in londiste is rather 
 lackluster, but still...)
 
 I'd marked this email as todo:
 If you have such a huge database you can, with logical decoding at
 least, use a basebackup using pg_basebackup or pg_start/stop_backup()
 and roll forwards from that... That'll hopefull make such huge copies
 much faster.

 Just keep in mind that one of the use cases for logical replication is 
 upgrades.

Should still be fine. Make a physical copy; pg_upgrade; catchup via
logical rep.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers