Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-17 Thread Andres Freund
On 2015-02-17 12:18:41 +0900, Fujii Masao wrote:
 On Fri, Feb 13, 2015 at 9:18 AM, Andres Freund and...@2ndquadrant.com wrote:
  On 2015-02-12 11:44:05 -0800, Sergey Konoplev wrote:
  On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund and...@2ndquadrant.com 
  wrote:
   This obviously should not be the case. I'll have a look in a couple of 
   hours. Until then you can likely  just work around the problem by 
   creating the archive_status directory.
 
  Thank you. Just let me know if you need some extra info or debugging.
 
  No need for debugging. It's plain and simply a (cherry-pick) conflict I
  resolved wrongly during backpatching. 9.3, 9.4 and master do not have
  that problem. That whole fix was quite painful because every single
  release had significantly different code :(. pg_basebackup/ is pretty
  messy.
  I'm not sure why my testsuite didn't trigger that problem. Possibly
  because a retry makes things work :(
 
  Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier
  releases don't have pg_receivexlog)
 
 Are you planning to back-patch the fix to 9.2?

Now done. Thanks Sergey, Fujii. And sorry for the 9.2 screwup.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-17 Thread Andres Freund
On 2015-02-17 12:18:41 +0900, Fujii Masao wrote:
 On Fri, Feb 13, 2015 at 9:18 AM, Andres Freund and...@2ndquadrant.com wrote:
  Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier
  releases don't have pg_receivexlog)
 
 Are you planning to back-patch the fix to 9.2?

Yes, but I want to look through all versions, to make sure there's no
other merge resolution mistakes lurking.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-16 Thread Fujii Masao
On Fri, Feb 13, 2015 at 9:18 AM, Andres Freund and...@2ndquadrant.com wrote:
 On 2015-02-12 11:44:05 -0800, Sergey Konoplev wrote:
 On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund and...@2ndquadrant.com 
 wrote:
  This obviously should not be the case. I'll have a look in a couple of 
  hours. Until then you can likely  just work around the problem by creating 
  the archive_status directory.

 Thank you. Just let me know if you need some extra info or debugging.

 No need for debugging. It's plain and simply a (cherry-pick) conflict I
 resolved wrongly during backpatching. 9.3, 9.4 and master do not have
 that problem. That whole fix was quite painful because every single
 release had significantly different code :(. pg_basebackup/ is pretty
 messy.
 I'm not sure why my testsuite didn't trigger that problem. Possibly
 because a retry makes things work :(

 Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier
 releases don't have pg_receivexlog)

Are you planning to back-patch the fix to 9.2?

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Sergey Konoplev
Hi,

On Mon, Jan 5, 2015 at 4:34 AM, Andres Freund and...@2ndquadrant.com wrote:
  pg_receivexlog: could not create archive status file
  mmm/archive_status/00010003.done: No such file or
  directory
 
  Dang. Stupid typo. And my tests didn't catch it, because I had
  archive_directory in the target directory :(

I started getting these errors after upgrading from 9.2.8 to 9.2.10.
Is it something critical that requires version downgrade or I can just
ignore that errors?

-- 
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray...@gmail.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Sergey Konoplev
On Thu, Feb 12, 2015 at 11:13 AM, Andres Freund and...@2ndquadrant.com wrote:
I started getting these errors after upgrading from 9.2.8 to 9.2.10.
Is it something critical that requires version downgrade or I can just
ignore that errors?

 What errors are you getting in precisely which circumstances? You're using 
 pg-receivexlog?

Errors like this one

pg_receivexlog: could not create archive status file
/mnt/archive/wal/archive_status/000402AF00B7.done: No
such file or directory
pg_receivexlog: disconnected..

on

Linux xyz 3.2.0-76-generic #111-Ubuntu SMP
PostgreSQL 9.2.10

Yes, I use pg_receivexlog. I also use a wrapper/watchdog script around
pg_receivexlog which tracks failures and restarts the latter.

The WAL files time correlates with the pg_receivexlog failures.

postgres@xyz:~$ ls -ltr /mnt/archive/wal/ | tail
-rw--- 1 postgres postgres 16777216 Feb 12 10:58 000402B60011
-rw--- 1 postgres postgres 16777216 Feb 12 11:02 000402B60012
-rw--- 1 postgres postgres 16777216 Feb 12 11:06 000402B60013
-rw--- 1 postgres postgres 16777216 Feb 12 11:11 000402B60014
-rw--- 1 postgres postgres 16777216 Feb 12 11:15 000402B60015
-rw--- 1 postgres postgres 16777216 Feb 12 11:19 000402B60016
-rw--- 1 postgres postgres 16777216 Feb 12 11:23 000402B60017
-rw--- 1 postgres postgres 16777216 Feb 12 11:27 000402B60018
-rw--- 1 postgres postgres 16777216 Feb 12 11:30 000402B60019
-rw--- 1 postgres postgres 16777216 Feb 12 11:32
000402B6001A.partial

postgres@xyz:~$ cat /var/log/pgcookbook/manage_pitr-wal.log | tail
Thu Feb 12 11:15:18 PST 2015 ERROR manage_pitr.sh: Problem occured
during WAL archiving: pg_receivexlog: could not create archive status
file /mnt/archive/wal/archive_status/000402B60015.done:
No such file or directory
pg_receivexlog: disconnected..
Thu Feb 12 11:19:33 PST 2015 ERROR manage_pitr.sh: Problem occured
during WAL archiving: pg_receivexlog: could not create archive status
file /mnt/archive/wal/archive_status/000402B60016.done:
No such file or directory
pg_receivexlog: disconnected..
Thu Feb 12 11:23:38 PST 2015 ERROR manage_pitr.sh: Problem occured
during WAL archiving: pg_receivexlog: could not create archive status
file /mnt/archive/wal/archive_status/000402B60017.done:
No such file or directory
pg_receivexlog: disconnected..
Thu Feb 12 11:27:32 PST 2015 ERROR manage_pitr.sh: Problem occured
during WAL archiving: pg_receivexlog: could not create archive status
file /mnt/archive/wal/archive_status/000402B60018.done:
No such file or directory
pg_receivexlog: disconnected..
Thu Feb 12 11:30:34 PST 2015 ERROR manage_pitr.sh: Problem occured
during WAL archiving: pg_receivexlog: could not create archive status
file /mnt/archive/wal/archive_status/000402B60019.done:
No such file or directory
pg_receivexlog: disconnected..

-- 
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray...@gmail.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Sergey Konoplev
On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund and...@2ndquadrant.com wrote:
 This obviously should not be the case. I'll have a look in a couple of hours. 
 Until then you can likely  just work around the problem by creating the 
 archive_status directory.

Thank you. Just let me know if you need some extra info or debugging.

-- 
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray...@gmail.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Andres Freund
On February 12, 2015 8:11:05 PM CET, Sergey Konoplev gray...@gmail.com wrote:
Hi,

On Mon, Jan 5, 2015 at 4:34 AM, Andres Freund and...@2ndquadrant.com
wrote:
  pg_receivexlog: could not create archive status file
  mmm/archive_status/00010003.done: No such file
or
  directory
 
  Dang. Stupid typo. And my tests didn't catch it, because I had
  archive_directory in the target directory :(

I started getting these errors after upgrading from 9.2.8 to 9.2.10.
Is it something critical that requires version downgrade or I can just
ignore that errors?

What errors are you getting in precisely which circumstances? You're using 
pg-receivexlog? 

-- 
Please excuse brevity and formatting - I am writing this on my mobile phone.

Andres Freund  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Andres Freund
Hi.

This obviously should not be the case. I'll have a look in a couple of hours. 
Until then you can likely  just work around the problem by creating the 
archive_status directory.
-- 
Please excuse brevity and formatting - I am writing this on my mobile phone.

Andres Freund  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Andres Freund
On 2015-02-12 11:44:05 -0800, Sergey Konoplev wrote:
 On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund and...@2ndquadrant.com 
 wrote:
  This obviously should not be the case. I'll have a look in a couple of 
  hours. Until then you can likely  just work around the problem by creating 
  the archive_status directory.
 
 Thank you. Just let me know if you need some extra info or debugging.

No need for debugging. It's plain and simply a (cherry-pick) conflict I
resolved wrongly during backpatching. 9.3, 9.4 and master do not have
that problem. That whole fix was quite painful because every single
release had significantly different code :(. pg_basebackup/ is pretty
messy.
I'm not sure why my testsuite didn't trigger that problem. Possibly
because a retry makes things work :(

Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier
releases don't have pg_receivexlog) and can quite easily be worked
around by creating the archive_status directory.

If you want to fix it locally, you just need to replace
ReceiveXlogStream(conn, startpos, timeline, NULL, basedir,
  stop_streaming, 
standby_message_timeout, false, true);
by
ReceiveXlogStream(conn, startpos, timeline, NULL, basedir,
  stop_streaming, 
standby_message_timeout, false, false);

Yes, that and pretty much all other functions in that directory have too
many parameters.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-02-12 Thread Sergey Konoplev
On Thu, Feb 12, 2015 at 4:18 PM, Andres Freund and...@2ndquadrant.com wrote:
 No need for debugging. It's plain and simply a (cherry-pick) conflict I
 resolved wrongly during backpatching. 9.3, 9.4 and master do not have
 that problem. That whole fix was quite painful because every single
 release had significantly different code :(. pg_basebackup/ is pretty
 messy.
 I'm not sure why my testsuite didn't trigger that problem. Possibly
 because a retry makes things work :(

 Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier
 releases don't have pg_receivexlog) and can quite easily be worked
 around by creating the archive_status directory.

The workaround works perfectly for me in this case, I'm going to
updrade it up to 9.4 anyway soon.

Thank you, Andres.

-- 
Kind regards,
Sergey Konoplev
PostgreSQL Consultant and DBA

http://www.linkedin.com/in/grayhemp
+1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979
gray...@gmail.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-01-05 Thread Fujii Masao
On Mon, Jan 5, 2015 at 6:22 PM, Andres Freund and...@2ndquadrant.com wrote:
 On 2015-01-05 16:22:56 +0900, Fujii Masao wrote:
 On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund and...@2ndquadrant.com wrote:
  On 2015-01-03 16:03:36 +0100, Andres Freund wrote:
  pg_basebackup really
  changed heavily since it's introduction. And desparately needs some
  restructuring.

 The patch seems to break pg_receivexlog. I got the following error message
 while running pg_receivexlog.

 pg_receivexlog: could not create archive status file
 mmm/archive_status/00010003.done: No such file or
 directory

 Dang. Stupid typo. And my tests didn't catch it, because I had
 archive_directory in the target directory :(

 At least it's only broken in master :/

 Thanks for the catch. Do you have some additional testsuite or did you
 catch it manually?

Manually... I just tested the tools and options which the patch may affect...

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-01-05 Thread Andres Freund
On 2015-01-05 16:22:56 +0900, Fujii Masao wrote:
 On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund and...@2ndquadrant.com wrote:
  On 2015-01-03 16:03:36 +0100, Andres Freund wrote:
  pg_basebackup really
  changed heavily since it's introduction. And desparately needs some
  restructuring.
 
 The patch seems to break pg_receivexlog. I got the following error message
 while running pg_receivexlog.

 pg_receivexlog: could not create archive status file
 mmm/archive_status/00010003.done: No such file or
 directory

Dang. Stupid typo. And my tests didn't catch it, because I had
archive_directory in the target directory :(

At least it's only broken in master :/

Thanks for the catch. Do you have some additional testsuite or did you
catch it manually?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-01-05 Thread Andres Freund
On 2015-01-05 18:49:06 +0900, Fujii Masao wrote:
 On Mon, Jan 5, 2015 at 6:22 PM, Andres Freund and...@2ndquadrant.com wrote:
  On 2015-01-05 16:22:56 +0900, Fujii Masao wrote:
  On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund and...@2ndquadrant.com 
  wrote:
   On 2015-01-03 16:03:36 +0100, Andres Freund wrote:
   pg_basebackup really
   changed heavily since it's introduction. And desparately needs some
   restructuring.
 
  The patch seems to break pg_receivexlog. I got the following error message
  while running pg_receivexlog.
 
  pg_receivexlog: could not create archive status file
  mmm/archive_status/00010003.done: No such file or
  directory
 
  Dang. Stupid typo. And my tests didn't catch it, because I had
  archive_directory in the target directory :(
 
  At least it's only broken in master :/

I've pushed the trivial fix, and verified using my adapted testscript
that it works on all branches.

  Thanks for the catch. Do you have some additional testsuite or did you
  catch it manually?
 
 Manually... I just tested the tools and options which the patch may affect...

Thanks!

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-01-04 Thread Fujii Masao
On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund and...@2ndquadrant.com wrote:
 On 2015-01-03 16:03:36 +0100, Andres Freund wrote:
 On 2014-12-31 16:32:19 +0100, Andres Freund wrote:
  On 2014-12-05 16:18:02 +0900, Fujii Masao wrote:
   On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund and...@2ndquadrant.com 
   wrote:
So I think we just need to make pg_basebackup create to .ready
files.
  
   s/.ready/.done? If yes, +1.
 
  That unfortunately requires changes to both backend and pg_basebackup to
  support fetch and stream modes respectively.
 
  I've attached a preliminary patch for this. I'd appreciate feedback. I
  plan to commit it in a couple of days, after some more
  testing/rereading.

 Attached are two updated patches that I am starting to backport
 now. I've fixed a couple minor oversights. And tested the patches.

 Pushed this after some major pain with backporting.

Thanks!

 pg_basebackup really
 changed heavily since it's introduction. And desparately needs some
 restructuring.

The patch seems to break pg_receivexlog. I got the following error message
while running pg_receivexlog.

pg_receivexlog: could not create archive status file
mmm/archive_status/00010003.done: No such file or
directory

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-01-03 Thread Andres Freund
On 2014-12-31 16:32:19 +0100, Andres Freund wrote:
 On 2014-12-05 16:18:02 +0900, Fujii Masao wrote:
  On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund and...@2ndquadrant.com 
  wrote:
   So I think we just need to make pg_basebackup create to .ready
   files.
  
  s/.ready/.done? If yes, +1.
 
 That unfortunately requires changes to both backend and pg_basebackup to
 support fetch and stream modes respectively.
 
 I've attached a preliminary patch for this. I'd appreciate feedback. I
 plan to commit it in a couple of days, after some more
 testing/rereading.

Attached are two updated patches that I am starting to backport
now. I've fixed a couple minor oversights. And tested the patches.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services
From 7140b41651a5a7a7bfdbe2ed7192bf0a2475e57f Mon Sep 17 00:00:00 2001
From: Andres Freund and...@anarazel.de
Date: Sat, 3 Jan 2015 15:57:49 +0100
Subject: [PATCH 1/2] Add pg_string_endswith as the start of a string helper
 library in src/common.

Backpatch to 9.3 where src/common was introduce, because a bugfix that
needs to be backpatched, requires the function. Earlier branches will
have to duplicate the code.
---
 src/backend/replication/slot.c | 21 ++---
 src/common/Makefile|  2 +-
 src/common/string.c| 43 ++
 src/include/common/string.h| 15 +++
 src/tools/msvc/Mkvcbuild.pm|  2 +-
 5 files changed, 62 insertions(+), 21 deletions(-)
 create mode 100644 src/common/string.c
 create mode 100644 src/include/common/string.h

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 937b669..8708616 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -40,6 +40,7 @@
 #include sys/stat.h
 
 #include access/transam.h
+#include common/string.h
 #include miscadmin.h
 #include replication/slot.h
 #include storage/fd.h
@@ -780,24 +781,6 @@ CheckSlotRequirements(void)
 }
 
 /*
- * Returns whether the string `str' has the postfix `end'.
- */
-static bool
-string_endswith(const char *str, const char *end)
-{
-	size_t		slen = strlen(str);
-	size_t		elen = strlen(end);
-
-	/* can't be a postfix if longer */
-	if (elen  slen)
-		return false;
-
-	/* compare the end of the strings */
-	str += slen - elen;
-	return strcmp(str, end) == 0;
-}
-
-/*
  * Flush all replication slots to disk.
  *
  * This needn't actually be part of a checkpoint, but it's a convenient
@@ -864,7 +847,7 @@ StartupReplicationSlots(void)
 			continue;
 
 		/* we crashed while a slot was being setup or deleted, clean up */
-		if (string_endswith(replication_de-d_name, .tmp))
+		if (pg_str_endswith(replication_de-d_name, .tmp))
 		{
 			if (!rmtree(path, true))
 			{
diff --git a/src/common/Makefile b/src/common/Makefile
index 7edbaaa..e5c345d 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -23,7 +23,7 @@ include $(top_builddir)/src/Makefile.global
 override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
 LIBS += $(PTHREAD_LIBS)
 
-OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o username.o wait_error.o
+OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o string.o username.o wait_error.o
 
 OBJS_FRONTEND = $(OBJS_COMMON) fe_memutils.o
 
diff --git a/src/common/string.c b/src/common/string.c
new file mode 100644
index 000..27e0743
--- /dev/null
+++ b/src/common/string.c
@@ -0,0 +1,43 @@
+/*-
+ *
+ * string.c
+ *		string handling helpers
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ *	  src/common/string.c
+ *
+ *-
+ */
+
+
+#ifndef FRONTEND
+#include postgres.h
+#else
+#include postgres_fe.h
+#endif
+
+#include common/string.h
+
+
+/*
+ * Returns whether the string `str' has the postfix `end'.
+ */
+bool
+pg_str_endswith(const char *str, const char *end)
+{
+	size_t		slen = strlen(str);
+	size_t		elen = strlen(end);
+
+	/* can't be a postfix if longer */
+	if (elen  slen)
+		return false;
+
+	/* compare the end of the strings */
+	str += slen - elen;
+	return strcmp(str, end) == 0;
+}
diff --git a/src/include/common/string.h b/src/include/common/string.h
new file mode 100644
index 000..0233858
--- /dev/null
+++ b/src/include/common/string.h
@@ -0,0 +1,15 @@
+/*
+ *	string.h
+ *		string handling helpers
+ *
+ *  Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ *  Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *	src/include/common/string.h
+ */
+#ifndef COMMON_STRING_H
+#define COMMON_STRING_H
+
+extern bool pg_str_endswith(const char *str, const char *end);
+
+#endif   /* COMMON_STRING_H */
diff --git 

Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2015-01-03 Thread Andres Freund
On 2015-01-03 16:03:36 +0100, Andres Freund wrote:
 On 2014-12-31 16:32:19 +0100, Andres Freund wrote:
  On 2014-12-05 16:18:02 +0900, Fujii Masao wrote:
   On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund and...@2ndquadrant.com 
   wrote:
So I think we just need to make pg_basebackup create to .ready
files.
   
   s/.ready/.done? If yes, +1.
  
  That unfortunately requires changes to both backend and pg_basebackup to
  support fetch and stream modes respectively.
  
  I've attached a preliminary patch for this. I'd appreciate feedback. I
  plan to commit it in a couple of days, after some more
  testing/rereading.
 
 Attached are two updated patches that I am starting to backport
 now. I've fixed a couple minor oversights. And tested the patches.

Pushed this after some major pain with backporting. pg_basebackup really
changed heavily since it's introduction. And desparately needs some
restructuring.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2014-12-31 Thread Andres Freund
Hi,

On 2014-12-05 16:18:02 +0900, Fujii Masao wrote:
 On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund and...@2ndquadrant.com wrote:
  So I think we just need to make pg_basebackup create to .ready
  files.
 
 s/.ready/.done? If yes, +1.

That unfortunately requires changes to both backend and pg_basebackup to
support fetch and stream modes respectively.

I've attached a preliminary patch for this. I'd appreciate feedback. I
plan to commit it in a couple of days, after some more
testing/rereading.

  Given that the walreceiver and restore_command already
  unconditionally do XLogArchiveForceDone() I think we'd follow the
  established precedent. Arguably it could make sense to archive files
  again on the standby after a promotion as they aren't guaranteed to have
  been on the then primary. But we don't have any infrastructure anyway
  for that and walsender doesn't do so, so it doesn't seem to make any
  sense to do that for pg_basebackup.
 
  Independent from this bug, there's also some debatable behaviour about
  what happens if a node with a high wal_keep_segments turns on
  archive_mode. Suddenly all those old files are archived... I think it
  might be a good idea to simply always create .done files when
  archive_mode is disabled while a wal segment is finished.
 
 +1

I tend to think that's a master only change. Agreed?

Greetings,

Andres Freund
From 3db116bc5b9465f555957bb11ac6cb8b20c18405 Mon Sep 17 00:00:00 2001
From: Andres Freund and...@anarazel.de
Date: Wed, 31 Dec 2014 14:50:57 +0100
Subject: [PATCH 1/2] Add pg_string_endswith as the start of a string helper
 library in src/common.

Backpatch to 9.3 where src/common was introduce, because a bugfix that
needs to be backpatched, requires the function. Earlier branches will
have to duplicate the code.
---
 src/backend/replication/slot.c | 21 ++---
 src/common/Makefile|  2 +-
 src/common/string.c| 43 ++
 src/include/common/string.h| 15 +++
 src/tools/msvc/Mkvcbuild.pm|  2 +-
 5 files changed, 62 insertions(+), 21 deletions(-)
 create mode 100644 src/common/string.c
 create mode 100644 src/include/common/string.h

diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 937b669..698ca6b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -40,6 +40,7 @@
 #include sys/stat.h
 
 #include access/transam.h
+#include common/string.h
 #include miscadmin.h
 #include replication/slot.h
 #include storage/fd.h
@@ -780,24 +781,6 @@ CheckSlotRequirements(void)
 }
 
 /*
- * Returns whether the string `str' has the postfix `end'.
- */
-static bool
-string_endswith(const char *str, const char *end)
-{
-	size_t		slen = strlen(str);
-	size_t		elen = strlen(end);
-
-	/* can't be a postfix if longer */
-	if (elen  slen)
-		return false;
-
-	/* compare the end of the strings */
-	str += slen - elen;
-	return strcmp(str, end) == 0;
-}
-
-/*
  * Flush all replication slots to disk.
  *
  * This needn't actually be part of a checkpoint, but it's a convenient
@@ -864,7 +847,7 @@ StartupReplicationSlots(void)
 			continue;
 
 		/* we crashed while a slot was being setup or deleted, clean up */
-		if (string_endswith(replication_de-d_name, .tmp))
+		if (pg_string_endswith(replication_de-d_name, .tmp))
 		{
 			if (!rmtree(path, true))
 			{
diff --git a/src/common/Makefile b/src/common/Makefile
index 7edbaaa..e5c345d 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -23,7 +23,7 @@ include $(top_builddir)/src/Makefile.global
 override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
 LIBS += $(PTHREAD_LIBS)
 
-OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o username.o wait_error.o
+OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o string.o username.o wait_error.o
 
 OBJS_FRONTEND = $(OBJS_COMMON) fe_memutils.o
 
diff --git a/src/common/string.c b/src/common/string.c
new file mode 100644
index 000..07a2aaf
--- /dev/null
+++ b/src/common/string.c
@@ -0,0 +1,43 @@
+/*-
+ *
+ * string.c
+ *		string handling helpers
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ *	  src/common/string.c
+ *
+ *-
+ */
+
+
+#ifndef FRONTEND
+#include postgres.h
+#else
+#include postgres_fe.h
+#endif
+
+#include common/string.h
+
+
+/*
+ * Returns whether the string `str' has the postfix `end'.
+ */
+bool
+pg_string_endswith(const char *str, const char *end)
+{
+	size_t		slen = strlen(str);
+	size_t		elen = strlen(end);
+
+	/* can't be a postfix if longer */
+	if (elen  slen)
+		return false;
+
+	/* compare the end of the strings */
+	str += slen - elen;
+	return strcmp(str, end) == 0;
+}
diff --git a/src/include/common/string.h 

Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode wal_keep_segments

2014-12-04 Thread Fujii Masao
On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund and...@2ndquadrant.com wrote:
 Hi,

 We've recently observed a case where, after a promotion, a postgres
 server suddenly started to archive a large amount of old WAL.

 After some digging the problem is this:

 pg_basebackup -X creates files in pg_xlog/ without creating the
 corresponding .done file. Note that walreceiver *does* create them. The
 standby in this case, just like the master, had a significant
 wal_keep_segments.  RemoveOldXlogFiles() then, during recovery restart
 points, calls XLogArchiveCheckDone() which in turn does:
 /* Retry creation of the .ready file */
 XLogArchiveNotify(xlog);
 return false;
 if there's neither a .done nor a .ready file present and archive_mode is
 enabled. These segments then aren't removed because there's a .ready
 present and they're never archived as long as the node is a standby
 because we don't do archiving on standbys.
 Once the node is promoted archiver will be started and suddenly archive
 all these files - which might be months old.

 And additional, at first strange, nice detail is that a lot of the
 .ready files had nearly the same timestamps. Turns out that's due to
 wal_keep_segments. Initially RemoveOldXlogFiles() doesn't process the
 files because they're newer than allowed due to wal_keep_segments. Then
 every checkpoint a couple segments would be old enough to reach
 XLogArchiveCheckDone() which then'd create the .ready marker... But not
 all at once :)


 So I think we just need to make pg_basebackup create to .ready
 files.

s/.ready/.done? If yes, +1.

 Given that the walreceiver and restore_command already
 unconditionally do XLogArchiveForceDone() I think we'd follow the
 established precedent. Arguably it could make sense to archive files
 again on the standby after a promotion as they aren't guaranteed to have
 been on the then primary. But we don't have any infrastructure anyway
 for that and walsender doesn't do so, so it doesn't seem to make any
 sense to do that for pg_basebackup.

 Independent from this bug, there's also some debatable behaviour about
 what happens if a node with a high wal_keep_segments turns on
 archive_mode. Suddenly all those old files are archived... I think it
 might be a good idea to simply always create .done files when
 archive_mode is disabled while a wal segment is finished.

+1

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers