Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2015-02-17 12:18:41 +0900, Fujii Masao wrote: > On Fri, Feb 13, 2015 at 9:18 AM, Andres Freund wrote: > > On 2015-02-12 11:44:05 -0800, Sergey Konoplev wrote: > >> On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund > >> wrote: > >> > This obviously should not be the case. I'll have a look in a couple of > >> > hours. Until then you can likely just work around the problem by > >> > creating the archive_status directory. > >> > >> Thank you. Just let me know if you need some extra info or debugging. > > > > No need for debugging. It's plain and simply a (cherry-pick) conflict I > > resolved wrongly during backpatching. 9.3, 9.4 and master do not have > > that problem. That whole fix was quite painful because every single > > release had significantly different code :(. pg_basebackup/ is pretty > > messy. > > I'm not sure why my testsuite didn't trigger that problem. Possibly > > because a retry makes things work :( > > > > Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier > > releases don't have pg_receivexlog) > > Are you planning to back-patch the fix to 9.2? Now done. Thanks Sergey, Fujii. And sorry for the 9.2 screwup. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2015-02-17 12:18:41 +0900, Fujii Masao wrote: > On Fri, Feb 13, 2015 at 9:18 AM, Andres Freund wrote: > > Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier > > releases don't have pg_receivexlog) > > Are you planning to back-patch the fix to 9.2? Yes, but I want to look through all versions, to make sure there's no other merge resolution mistakes lurking. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Fri, Feb 13, 2015 at 9:18 AM, Andres Freund wrote: > On 2015-02-12 11:44:05 -0800, Sergey Konoplev wrote: >> On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund >> wrote: >> > This obviously should not be the case. I'll have a look in a couple of >> > hours. Until then you can likely just work around the problem by creating >> > the archive_status directory. >> >> Thank you. Just let me know if you need some extra info or debugging. > > No need for debugging. It's plain and simply a (cherry-pick) conflict I > resolved wrongly during backpatching. 9.3, 9.4 and master do not have > that problem. That whole fix was quite painful because every single > release had significantly different code :(. pg_basebackup/ is pretty > messy. > I'm not sure why my testsuite didn't trigger that problem. Possibly > because a retry makes things work :( > > Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier > releases don't have pg_receivexlog) Are you planning to back-patch the fix to 9.2? Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Thu, Feb 12, 2015 at 4:18 PM, Andres Freund wrote: > No need for debugging. It's plain and simply a (cherry-pick) conflict I > resolved wrongly during backpatching. 9.3, 9.4 and master do not have > that problem. That whole fix was quite painful because every single > release had significantly different code :(. pg_basebackup/ is pretty > messy. > I'm not sure why my testsuite didn't trigger that problem. Possibly > because a retry makes things work :( > > Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier > releases don't have pg_receivexlog) and can quite easily be worked > around by creating the archive_status directory. The workaround works perfectly for me in this case, I'm going to updrade it up to 9.4 anyway soon. Thank you, Andres. -- Kind regards, Sergey Konoplev PostgreSQL Consultant and DBA http://www.linkedin.com/in/grayhemp +1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979 gray...@gmail.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2015-02-12 11:44:05 -0800, Sergey Konoplev wrote: > On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund > wrote: > > This obviously should not be the case. I'll have a look in a couple of > > hours. Until then you can likely just work around the problem by creating > > the archive_status directory. > > Thank you. Just let me know if you need some extra info or debugging. No need for debugging. It's plain and simply a (cherry-pick) conflict I resolved wrongly during backpatching. 9.3, 9.4 and master do not have that problem. That whole fix was quite painful because every single release had significantly different code :(. pg_basebackup/ is pretty messy. I'm not sure why my testsuite didn't trigger that problem. Possibly because a retry makes things work :( Somewhat uckily it's 9.2 only (9.3, 9.4 and master look correct, earlier releases don't have pg_receivexlog) and can quite easily be worked around by creating the archive_status directory. If you want to fix it locally, you just need to replace ReceiveXlogStream(conn, startpos, timeline, NULL, basedir, stop_streaming, standby_message_timeout, false, true); by ReceiveXlogStream(conn, startpos, timeline, NULL, basedir, stop_streaming, standby_message_timeout, false, false); Yes, that and pretty much all other functions in that directory have too many parameters. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Thu, Feb 12, 2015 at 11:40 AM, Andres Freund wrote: > This obviously should not be the case. I'll have a look in a couple of hours. > Until then you can likely just work around the problem by creating the > archive_status directory. Thank you. Just let me know if you need some extra info or debugging. -- Kind regards, Sergey Konoplev PostgreSQL Consultant and DBA http://www.linkedin.com/in/grayhemp +1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979 gray...@gmail.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
Hi. This obviously should not be the case. I'll have a look in a couple of hours. Until then you can likely just work around the problem by creating the archive_status directory. -- Please excuse brevity and formatting - I am writing this on my mobile phone. Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Thu, Feb 12, 2015 at 11:13 AM, Andres Freund wrote: >>I started getting these errors after upgrading from 9.2.8 to 9.2.10. >>Is it something critical that requires version downgrade or I can just >>ignore that errors? > > What errors are you getting in precisely which circumstances? You're using > pg-receivexlog? Errors like this one pg_receivexlog: could not create archive status file "/mnt/archive/wal/archive_status/000402AF00B7.done": No such file or directory pg_receivexlog: disconnected.. on Linux xyz 3.2.0-76-generic #111-Ubuntu SMP PostgreSQL 9.2.10 Yes, I use pg_receivexlog. I also use a wrapper/watchdog script around pg_receivexlog which tracks failures and restarts the latter. The WAL files time correlates with the pg_receivexlog failures. postgres@xyz:~$ ls -ltr /mnt/archive/wal/ | tail -rw--- 1 postgres postgres 16777216 Feb 12 10:58 000402B60011 -rw--- 1 postgres postgres 16777216 Feb 12 11:02 000402B60012 -rw--- 1 postgres postgres 16777216 Feb 12 11:06 000402B60013 -rw--- 1 postgres postgres 16777216 Feb 12 11:11 000402B60014 -rw--- 1 postgres postgres 16777216 Feb 12 11:15 000402B60015 -rw--- 1 postgres postgres 16777216 Feb 12 11:19 000402B60016 -rw--- 1 postgres postgres 16777216 Feb 12 11:23 000402B60017 -rw--- 1 postgres postgres 16777216 Feb 12 11:27 000402B60018 -rw--- 1 postgres postgres 16777216 Feb 12 11:30 000402B60019 -rw--- 1 postgres postgres 16777216 Feb 12 11:32 000402B6001A.partial postgres@xyz:~$ cat /var/log/pgcookbook/manage_pitr-wal.log | tail Thu Feb 12 11:15:18 PST 2015 ERROR manage_pitr.sh: Problem occured during WAL archiving: pg_receivexlog: could not create archive status file "/mnt/archive/wal/archive_status/000402B60015.done": No such file or directory pg_receivexlog: disconnected.. Thu Feb 12 11:19:33 PST 2015 ERROR manage_pitr.sh: Problem occured during WAL archiving: pg_receivexlog: could not create archive status file "/mnt/archive/wal/archive_status/000402B60016.done": No such file or directory pg_receivexlog: disconnected.. Thu Feb 12 11:23:38 PST 2015 ERROR manage_pitr.sh: Problem occured during WAL archiving: pg_receivexlog: could not create archive status file "/mnt/archive/wal/archive_status/000402B60017.done": No such file or directory pg_receivexlog: disconnected.. Thu Feb 12 11:27:32 PST 2015 ERROR manage_pitr.sh: Problem occured during WAL archiving: pg_receivexlog: could not create archive status file "/mnt/archive/wal/archive_status/000402B60018.done": No such file or directory pg_receivexlog: disconnected.. Thu Feb 12 11:30:34 PST 2015 ERROR manage_pitr.sh: Problem occured during WAL archiving: pg_receivexlog: could not create archive status file "/mnt/archive/wal/archive_status/000402B60019.done": No such file or directory pg_receivexlog: disconnected.. -- Kind regards, Sergey Konoplev PostgreSQL Consultant and DBA http://www.linkedin.com/in/grayhemp +1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979 gray...@gmail.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On February 12, 2015 8:11:05 PM CET, Sergey Konoplev wrote: >Hi, > >On Mon, Jan 5, 2015 at 4:34 AM, Andres Freund >wrote: >>> >> pg_receivexlog: could not create archive status file >>> >> "mmm/archive_status/00010003.done": No such file >or >>> >> directory >>> > >>> > Dang. Stupid typo. And my tests didn't catch it, because I had >>> > archive_directory in the target directory :( > >I started getting these errors after upgrading from 9.2.8 to 9.2.10. >Is it something critical that requires version downgrade or I can just >ignore that errors? What errors are you getting in precisely which circumstances? You're using pg-receivexlog? -- Please excuse brevity and formatting - I am writing this on my mobile phone. Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
Hi, On Mon, Jan 5, 2015 at 4:34 AM, Andres Freund wrote: >> >> pg_receivexlog: could not create archive status file >> >> "mmm/archive_status/00010003.done": No such file or >> >> directory >> > >> > Dang. Stupid typo. And my tests didn't catch it, because I had >> > archive_directory in the target directory :( I started getting these errors after upgrading from 9.2.8 to 9.2.10. Is it something critical that requires version downgrade or I can just ignore that errors? -- Kind regards, Sergey Konoplev PostgreSQL Consultant and DBA http://www.linkedin.com/in/grayhemp +1 (415) 867-9984, +7 (499) 346-7196, +7 (988) 888-1979 gray...@gmail.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2015-01-05 18:49:06 +0900, Fujii Masao wrote: > On Mon, Jan 5, 2015 at 6:22 PM, Andres Freund wrote: > > On 2015-01-05 16:22:56 +0900, Fujii Masao wrote: > >> On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund > >> wrote: > >> > On 2015-01-03 16:03:36 +0100, Andres Freund wrote: > >> > pg_basebackup really > >> > changed heavily since it's introduction. And desparately needs some > >> > restructuring. > >> > >> The patch seems to break pg_receivexlog. I got the following error message > >> while running pg_receivexlog. > > > >> pg_receivexlog: could not create archive status file > >> "mmm/archive_status/00010003.done": No such file or > >> directory > > > > Dang. Stupid typo. And my tests didn't catch it, because I had > > archive_directory in the target directory :( > > > > At least it's only broken in master :/ I've pushed the trivial fix, and verified using my adapted testscript that it works on all branches. > > Thanks for the catch. Do you have some additional testsuite or did you > > catch it manually? > > Manually... I just tested the tools and options which the patch may affect... Thanks! Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Mon, Jan 5, 2015 at 6:22 PM, Andres Freund wrote: > On 2015-01-05 16:22:56 +0900, Fujii Masao wrote: >> On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund wrote: >> > On 2015-01-03 16:03:36 +0100, Andres Freund wrote: >> > pg_basebackup really >> > changed heavily since it's introduction. And desparately needs some >> > restructuring. >> >> The patch seems to break pg_receivexlog. I got the following error message >> while running pg_receivexlog. > >> pg_receivexlog: could not create archive status file >> "mmm/archive_status/00010003.done": No such file or >> directory > > Dang. Stupid typo. And my tests didn't catch it, because I had > archive_directory in the target directory :( > > At least it's only broken in master :/ > > Thanks for the catch. Do you have some additional testsuite or did you > catch it manually? Manually... I just tested the tools and options which the patch may affect... Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2015-01-05 16:22:56 +0900, Fujii Masao wrote: > On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund wrote: > > On 2015-01-03 16:03:36 +0100, Andres Freund wrote: > > pg_basebackup really > > changed heavily since it's introduction. And desparately needs some > > restructuring. > > The patch seems to break pg_receivexlog. I got the following error message > while running pg_receivexlog. > pg_receivexlog: could not create archive status file > "mmm/archive_status/00010003.done": No such file or > directory Dang. Stupid typo. And my tests didn't catch it, because I had archive_directory in the target directory :( At least it's only broken in master :/ Thanks for the catch. Do you have some additional testsuite or did you catch it manually? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Sun, Jan 4, 2015 at 5:47 AM, Andres Freund wrote: > On 2015-01-03 16:03:36 +0100, Andres Freund wrote: >> On 2014-12-31 16:32:19 +0100, Andres Freund wrote: >> > On 2014-12-05 16:18:02 +0900, Fujii Masao wrote: >> > > On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund >> > > wrote: >> > > > So I think we just need to make pg_basebackup create to .ready >> > > > files. >> > > >> > > s/.ready/.done? If yes, +1. >> > >> > That unfortunately requires changes to both backend and pg_basebackup to >> > support fetch and stream modes respectively. >> > >> > I've attached a preliminary patch for this. I'd appreciate feedback. I >> > plan to commit it in a couple of days, after some more >> > testing/rereading. >> >> Attached are two updated patches that I am starting to backport >> now. I've fixed a couple minor oversights. And tested the patches. > > Pushed this after some major pain with backporting. Thanks! > pg_basebackup really > changed heavily since it's introduction. And desparately needs some > restructuring. The patch seems to break pg_receivexlog. I got the following error message while running pg_receivexlog. pg_receivexlog: could not create archive status file "mmm/archive_status/00010003.done": No such file or directory Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2015-01-03 16:03:36 +0100, Andres Freund wrote: > On 2014-12-31 16:32:19 +0100, Andres Freund wrote: > > On 2014-12-05 16:18:02 +0900, Fujii Masao wrote: > > > On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund > > > wrote: > > > > So I think we just need to make pg_basebackup create to .ready > > > > files. > > > > > > s/.ready/.done? If yes, +1. > > > > That unfortunately requires changes to both backend and pg_basebackup to > > support fetch and stream modes respectively. > > > > I've attached a preliminary patch for this. I'd appreciate feedback. I > > plan to commit it in a couple of days, after some more > > testing/rereading. > > Attached are two updated patches that I am starting to backport > now. I've fixed a couple minor oversights. And tested the patches. Pushed this after some major pain with backporting. pg_basebackup really changed heavily since it's introduction. And desparately needs some restructuring. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On 2014-12-31 16:32:19 +0100, Andres Freund wrote:
> On 2014-12-05 16:18:02 +0900, Fujii Masao wrote:
> > On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund
> > wrote:
> > > So I think we just need to make pg_basebackup create to .ready
> > > files.
> >
> > s/.ready/.done? If yes, +1.
>
> That unfortunately requires changes to both backend and pg_basebackup to
> support fetch and stream modes respectively.
>
> I've attached a preliminary patch for this. I'd appreciate feedback. I
> plan to commit it in a couple of days, after some more
> testing/rereading.
Attached are two updated patches that I am starting to backport
now. I've fixed a couple minor oversights. And tested the patches.
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
>From 7140b41651a5a7a7bfdbe2ed7192bf0a2475e57f Mon Sep 17 00:00:00 2001
From: Andres Freund
Date: Sat, 3 Jan 2015 15:57:49 +0100
Subject: [PATCH 1/2] Add pg_string_endswith as the start of a string helper
library in src/common.
Backpatch to 9.3 where src/common was introduce, because a bugfix that
needs to be backpatched, requires the function. Earlier branches will
have to duplicate the code.
---
src/backend/replication/slot.c | 21 ++---
src/common/Makefile| 2 +-
src/common/string.c| 43 ++
src/include/common/string.h| 15 +++
src/tools/msvc/Mkvcbuild.pm| 2 +-
5 files changed, 62 insertions(+), 21 deletions(-)
create mode 100644 src/common/string.c
create mode 100644 src/include/common/string.h
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 937b669..8708616 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -40,6 +40,7 @@
#include
#include "access/transam.h"
+#include "common/string.h"
#include "miscadmin.h"
#include "replication/slot.h"
#include "storage/fd.h"
@@ -780,24 +781,6 @@ CheckSlotRequirements(void)
}
/*
- * Returns whether the string `str' has the postfix `end'.
- */
-static bool
-string_endswith(const char *str, const char *end)
-{
- size_t slen = strlen(str);
- size_t elen = strlen(end);
-
- /* can't be a postfix if longer */
- if (elen > slen)
- return false;
-
- /* compare the end of the strings */
- str += slen - elen;
- return strcmp(str, end) == 0;
-}
-
-/*
* Flush all replication slots to disk.
*
* This needn't actually be part of a checkpoint, but it's a convenient
@@ -864,7 +847,7 @@ StartupReplicationSlots(void)
continue;
/* we crashed while a slot was being setup or deleted, clean up */
- if (string_endswith(replication_de->d_name, ".tmp"))
+ if (pg_str_endswith(replication_de->d_name, ".tmp"))
{
if (!rmtree(path, true))
{
diff --git a/src/common/Makefile b/src/common/Makefile
index 7edbaaa..e5c345d 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -23,7 +23,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
LIBS += $(PTHREAD_LIBS)
-OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o username.o wait_error.o
+OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o string.o username.o wait_error.o
OBJS_FRONTEND = $(OBJS_COMMON) fe_memutils.o
diff --git a/src/common/string.c b/src/common/string.c
new file mode 100644
index 000..27e0743
--- /dev/null
+++ b/src/common/string.c
@@ -0,0 +1,43 @@
+/*-
+ *
+ * string.c
+ * string handling helpers
+ *
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/common/string.c
+ *
+ *-
+ */
+
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/string.h"
+
+
+/*
+ * Returns whether the string `str' has the postfix `end'.
+ */
+bool
+pg_str_endswith(const char *str, const char *end)
+{
+ size_t slen = strlen(str);
+ size_t elen = strlen(end);
+
+ /* can't be a postfix if longer */
+ if (elen > slen)
+ return false;
+
+ /* compare the end of the strings */
+ str += slen - elen;
+ return strcmp(str, end) == 0;
+}
diff --git a/src/include/common/string.h b/src/include/common/string.h
new file mode 100644
index 000..0233858
--- /dev/null
+++ b/src/include/common/string.h
@@ -0,0 +1,15 @@
+/*
+ * string.h
+ * string handling helpers
+ *
+ * Portions Copyright (c) 1996-2015, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/common/string.h
+ */
+#ifndef COMMON_STRING_H
+#define COMMON_STRING_H
+
+extern bool pg_str_endswith(const char *str, const char *end);
+
+#endif /* COMMON_STRING_H */
diff --git a/src/too
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
Hi,
On 2014-12-05 16:18:02 +0900, Fujii Masao wrote:
> On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund wrote:
> > So I think we just need to make pg_basebackup create to .ready
> > files.
>
> s/.ready/.done? If yes, +1.
That unfortunately requires changes to both backend and pg_basebackup to
support fetch and stream modes respectively.
I've attached a preliminary patch for this. I'd appreciate feedback. I
plan to commit it in a couple of days, after some more
testing/rereading.
> > Given that the walreceiver and restore_command already
> > unconditionally do XLogArchiveForceDone() I think we'd follow the
> > established precedent. Arguably it could make sense to archive files
> > again on the standby after a promotion as they aren't guaranteed to have
> > been on the then primary. But we don't have any infrastructure anyway
> > for that and walsender doesn't do so, so it doesn't seem to make any
> > sense to do that for pg_basebackup.
> >
> > Independent from this bug, there's also some debatable behaviour about
> > what happens if a node with a high wal_keep_segments turns on
> > archive_mode. Suddenly all those old files are archived... I think it
> > might be a good idea to simply always create .done files when
> > archive_mode is disabled while a wal segment is finished.
>
> +1
I tend to think that's a master only change. Agreed?
Greetings,
Andres Freund
>From 3db116bc5b9465f555957bb11ac6cb8b20c18405 Mon Sep 17 00:00:00 2001
From: Andres Freund
Date: Wed, 31 Dec 2014 14:50:57 +0100
Subject: [PATCH 1/2] Add pg_string_endswith as the start of a string helper
library in src/common.
Backpatch to 9.3 where src/common was introduce, because a bugfix that
needs to be backpatched, requires the function. Earlier branches will
have to duplicate the code.
---
src/backend/replication/slot.c | 21 ++---
src/common/Makefile| 2 +-
src/common/string.c| 43 ++
src/include/common/string.h| 15 +++
src/tools/msvc/Mkvcbuild.pm| 2 +-
5 files changed, 62 insertions(+), 21 deletions(-)
create mode 100644 src/common/string.c
create mode 100644 src/include/common/string.h
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 937b669..698ca6b 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -40,6 +40,7 @@
#include
#include "access/transam.h"
+#include "common/string.h"
#include "miscadmin.h"
#include "replication/slot.h"
#include "storage/fd.h"
@@ -780,24 +781,6 @@ CheckSlotRequirements(void)
}
/*
- * Returns whether the string `str' has the postfix `end'.
- */
-static bool
-string_endswith(const char *str, const char *end)
-{
- size_t slen = strlen(str);
- size_t elen = strlen(end);
-
- /* can't be a postfix if longer */
- if (elen > slen)
- return false;
-
- /* compare the end of the strings */
- str += slen - elen;
- return strcmp(str, end) == 0;
-}
-
-/*
* Flush all replication slots to disk.
*
* This needn't actually be part of a checkpoint, but it's a convenient
@@ -864,7 +847,7 @@ StartupReplicationSlots(void)
continue;
/* we crashed while a slot was being setup or deleted, clean up */
- if (string_endswith(replication_de->d_name, ".tmp"))
+ if (pg_string_endswith(replication_de->d_name, ".tmp"))
{
if (!rmtree(path, true))
{
diff --git a/src/common/Makefile b/src/common/Makefile
index 7edbaaa..e5c345d 100644
--- a/src/common/Makefile
+++ b/src/common/Makefile
@@ -23,7 +23,7 @@ include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -DFRONTEND $(CPPFLAGS)
LIBS += $(PTHREAD_LIBS)
-OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o username.o wait_error.o
+OBJS_COMMON = exec.o pgfnames.o psprintf.o relpath.o rmtree.o string.o username.o wait_error.o
OBJS_FRONTEND = $(OBJS_COMMON) fe_memutils.o
diff --git a/src/common/string.c b/src/common/string.c
new file mode 100644
index 000..07a2aaf
--- /dev/null
+++ b/src/common/string.c
@@ -0,0 +1,43 @@
+/*-
+ *
+ * string.c
+ * string handling helpers
+ *
+ *
+ * Portions Copyright (c) 1996-2014, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/common/string.c
+ *
+ *-
+ */
+
+
+#ifndef FRONTEND
+#include "postgres.h"
+#else
+#include "postgres_fe.h"
+#endif
+
+#include "common/string.h"
+
+
+/*
+ * Returns whether the string `str' has the postfix `end'.
+ */
+bool
+pg_string_endswith(const char *str, const char *end)
+{
+ size_t slen = strlen(str);
+ size_t elen = strlen(end);
+
+ /* can't be a postfix if longer */
+ if (elen > slen)
+ return false;
+
+ /* compare the end of the strings */
+ str += slen - elen;
+ return strcmp(str, end) == 0;
+}
diff --git a/src/include/common/string.h b/
Re: [HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
On Fri, Dec 5, 2014 at 9:28 AM, Andres Freund wrote: > Hi, > > We've recently observed a case where, after a promotion, a postgres > server suddenly started to archive a large amount of old WAL. > > After some digging the problem is this: > > pg_basebackup -X creates files in pg_xlog/ without creating the > corresponding .done file. Note that walreceiver *does* create them. The > standby in this case, just like the master, had a significant > wal_keep_segments. RemoveOldXlogFiles() then, during recovery restart > points, calls XLogArchiveCheckDone() which in turn does: > /* Retry creation of the .ready file */ > XLogArchiveNotify(xlog); > return false; > if there's neither a .done nor a .ready file present and archive_mode is > enabled. These segments then aren't removed because there's a .ready > present and they're never archived as long as the node is a standby > because we don't do archiving on standbys. > Once the node is promoted archiver will be started and suddenly archive > all these files - which might be months old. > > And additional, at first strange, nice detail is that a lot of the > .ready files had nearly the same timestamps. Turns out that's due to > wal_keep_segments. Initially RemoveOldXlogFiles() doesn't process the > files because they're newer than allowed due to wal_keep_segments. Then > every checkpoint a couple segments would be old enough to reach > XLogArchiveCheckDone() which then'd create the .ready marker... But not > all at once :) > > > So I think we just need to make pg_basebackup create to .ready > files. s/.ready/.done? If yes, +1. > Given that the walreceiver and restore_command already > unconditionally do XLogArchiveForceDone() I think we'd follow the > established precedent. Arguably it could make sense to archive files > again on the standby after a promotion as they aren't guaranteed to have > been on the then primary. But we don't have any infrastructure anyway > for that and walsender doesn't do so, so it doesn't seem to make any > sense to do that for pg_basebackup. > > Independent from this bug, there's also some debatable behaviour about > what happens if a node with a high wal_keep_segments turns on > archive_mode. Suddenly all those old files are archived... I think it > might be a good idea to simply always create .done files when > archive_mode is disabled while a wal segment is finished. +1 Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_basebackup -x/X doesn't play well with archive_mode & wal_keep_segments
Hi, We've recently observed a case where, after a promotion, a postgres server suddenly started to archive a large amount of old WAL. After some digging the problem is this: pg_basebackup -X creates files in pg_xlog/ without creating the corresponding .done file. Note that walreceiver *does* create them. The standby in this case, just like the master, had a significant wal_keep_segments. RemoveOldXlogFiles() then, during recovery restart points, calls XLogArchiveCheckDone() which in turn does: /* Retry creation of the .ready file */ XLogArchiveNotify(xlog); return false; if there's neither a .done nor a .ready file present and archive_mode is enabled. These segments then aren't removed because there's a .ready present and they're never archived as long as the node is a standby because we don't do archiving on standbys. Once the node is promoted archiver will be started and suddenly archive all these files - which might be months old. And additional, at first strange, nice detail is that a lot of the .ready files had nearly the same timestamps. Turns out that's due to wal_keep_segments. Initially RemoveOldXlogFiles() doesn't process the files because they're newer than allowed due to wal_keep_segments. Then every checkpoint a couple segments would be old enough to reach XLogArchiveCheckDone() which then'd create the .ready marker... But not all at once :) So I think we just need to make pg_basebackup create to .ready files. Given that the walreceiver and restore_command already unconditionally do XLogArchiveForceDone() I think we'd follow the established precedent. Arguably it could make sense to archive files again on the standby after a promotion as they aren't guaranteed to have been on the then primary. But we don't have any infrastructure anyway for that and walsender doesn't do so, so it doesn't seem to make any sense to do that for pg_basebackup. Independent from this bug, there's also some debatable behaviour about what happens if a node with a high wal_keep_segments turns on archive_mode. Suddenly all those old files are archived... I think it might be a good idea to simply always create .done files when archive_mode is disabled while a wal segment is finished. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
