2012-10-14 22:31 keltezéssel, Boszormenyi Zoltan írta:
2012-10-14 22:26 keltezéssel, Boszormenyi Zoltan írta:
2012-10-14 22:23 keltezéssel, Boszormenyi Zoltan írta:
Hi,
2012-10-14 18:41 keltezéssel, Boszormenyi Zoltan írta:
2012-10-14 18:02 keltezéssel, Fujii Masao írta:
Thanks for updating the patch!
On Sun, Oct 14, 2012 at 8:41 PM, Boszormenyi Zoltan <z...@cybertec.at> wrote:
Backing up a standby server without -R preserves the original recovery.conf
of the
standby, it points to the standby's source server.
Backing up a standby server with -R overwrites the original recovery.conf
with the new
one pointing to the standby instead of the standby's source server. Without
-Ft, it is
obvious. With -Ft, there are two recovery.conf files in the tar file and
upon extracting it,
the last written one (the one generated via -R) overwrites the original.
The tar file is always extracted such way in all platform which PostgreSQL
supports? I'm just concerned about that some tool in some platform might
prefer the original recovery.conf when extracting tar file. If the spec of tar
format specifies such behavior (i.e., the last written file of the same name
is always preferred), it's OK.
Since tar is a sequential archive format, I think this is the behaviour of
every tar extractor. But I will look at adding code to skip the original
recovery.conf if it exists in the tar file.
I found the bug that recovery.conf is included in the tar file of the tablespace
instead of base.tar, when there are tablespaces in the server.
You are right, I am looking into this. But I don't know how it got there,
I check for (rownum == 0 && writerecoveryconf) and rownum == 0
supposedly means that it's the base.tar. Looking again.
I made a mistake in the previous check, rownum is not reliable.
The tablespaces are sent first and base backup as the last.
Now recovery.conf is written into base.tar.
Maybe this is nitpicky problem,,,, but...
If port number is not explicitly specified in pg_basebackup, the port
number is not
included to primary_conninfo in recovery.conf which is created during
the backup.
That is, the standby server using such recovery.conf tries to connect
to the default
port number because the port number is not supplied in primary_conninfo. This
assumes that the default port number is the same between the master and standby.
But this is not true. The default port number can be changed in --with-pgport
configure option, so the default port number might be different
between the master
and standby. To avoid this uncertainty, pg_basebackup -R should always include
the port number in primary_conninfo?
I think you are right. But, I wouldn't restrict it only to the port setting.
Any of the values that are set and equal to the compiled-in default,
it should be written into recovery.conf.
Now all values that are set (even those being equal to the compiled-in default)
are put into recovery.conf.
When the password is required to connect to the server, pg_basebackup -R
always writes the password setting into primary_conninfo in recovery.conf.
But if the password is supplied from .pgpass, ISTM that the password setting
doesn't need to be written into primary_conninfo. Right?
How can you deduce it from the PQconninfoOption structure?
Also, if the machine you take the base backup on is different
from the one where you actually use the backup on, it can be
different not only in the --with-pgport compilation option but
in the presence of .pgpass or the PGPASSWORD envvar, too.
The administrator is there for a reason or there is no .pgpass
or PGPASSWORD at all.
+ The password written into recovery.conf is not escaped even if special
+ characters appear in it. The administrator must review recovery.conf
+ to ensure proper escaping.
Is it difficult to make pg_basebackup escape the special characters in the
password? It's better if we can remove this restriction.
It's not difficult. What other characters need to be escaped besides single
quotes?
All written values are escaped.
Other changes: the recovery.conf in base.tar is correctly skipped if it exists
and -R is given. The new recovery.conf is written with padding to round up to
512, the TAR chunk size.
Also, the check for conflict between -R and -x/-X is now removed.
The documentation for option -R has changed to reflect this and
there is no different error code 2 now: it would make the behaviour
inconsistent between -Fp and -Ft.
The PQconninfo patch is also attached but didn't change since the last mail.
Best regards,
Zoltán Böszörményi
--
----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
http://www.postgresql.at/
diff -durpN postgresql/doc/src/sgml/libpq.sgml postgresql.1/doc/src/sgml/libpq.sgml
--- postgresql/doc/src/sgml/libpq.sgml 2012-08-03 09:39:30.114266570 +0200
+++ postgresql.1/doc/src/sgml/libpq.sgml 2012-10-14 11:12:07.049196259 +0200
@@ -496,6 +496,37 @@ typedef struct
</listitem>
</varlistentry>
+ <varlistentry id="libpq-pqconninfo">
+ <term><function>PQconninfo</function><indexterm><primary>PQconninfo</></></term>
+ <listitem>
+ <para>
+ Returns the connection options used by a live connection.
+<synopsis>
+PQconninfoOption *PQconninfo(PGconn *conn, bool for_replication);
+</synopsis>
+ </para>
+
+ <para>
+ Returns a connection options array. This can be used to determine
+ all possible <function>PQconnectdb</function> options and their
+ current values that were used to connect to the server. The return
+ value points to an array of <structname>PQconninfoOption</structname>
+ structures, which ends with an entry having a null <structfield>keyword</>
+ pointer. Every notes above for <function>PQconndefaults</function> also apply.
+ </para>
+
+ <para>
+ The <literal>for_replication</> argument can be used to exclude some
+ options from the list which are used by the walreceiver module.
+ <application>pg_basebackup</application> uses this pre-filtered list
+ to construct <literal>primary_conninfo</> in the automatically generated
+ recovery.conf file.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
+
<varlistentry id="libpq-pqconninfoparse">
<term><function>PQconninfoParse</function><indexterm><primary>PQconninfoParse</></></term>
<listitem>
diff -durpN postgresql/src/interfaces/libpq/exports.txt postgresql.1/src/interfaces/libpq/exports.txt
--- postgresql/src/interfaces/libpq/exports.txt 2012-10-09 09:58:14.342782974 +0200
+++ postgresql.1/src/interfaces/libpq/exports.txt 2012-10-14 11:12:07.049196259 +0200
@@ -164,3 +164,4 @@ PQsetSingleRowMode 161
lo_lseek64 162
lo_tell64 163
lo_truncate64 164
+PQconninfo 165
diff -durpN postgresql/src/interfaces/libpq/fe-connect.c postgresql.1/src/interfaces/libpq/fe-connect.c
--- postgresql/src/interfaces/libpq/fe-connect.c 2012-09-09 08:11:09.470401480 +0200
+++ postgresql.1/src/interfaces/libpq/fe-connect.c 2012-10-14 11:12:07.053196284 +0200
@@ -137,6 +137,9 @@ static int ldapServiceLookup(const char
* PQconninfoOptions[] *must* be NULL. In a working copy, non-null "val"
* fields point to malloc'd strings that should be freed when the working
* array is freed (see PQconninfoFree).
+ *
+ * If you add a new connection option to this list, remember to add it to
+ * PQconninfoMappings[] below.
* ----------
*/
static const PQconninfoOption PQconninfoOptions[] = {
@@ -264,6 +267,66 @@ static const PQconninfoOption PQconninfo
NULL, NULL, 0}
};
+/*
+ * We need a mapping between the PQconninfoOptions[] array
+ * and PGconn members. We have to keep it separate from
+ * PQconninfoOptions[] to not leak info about PGconn members
+ * to clients.
+ */
+typedef struct PQconninfoMapping {
+ char *keyword;
+ size_t member_offset;
+ bool for_replication;
+ /*
+ * Special and simplistic value mapping between
+ * PQconninfoOption and PGconn. Only used by "requiressl".
+ */
+ char *conninfoValue;
+ char *connValue;
+} PQconninfoMapping;
+#define PGCONNMEMBERADDR(conn, mapping) ((char **)((char *)conn + mapping->member_offset))
+
+static const PQconninfoMapping PQconninfoMappings[] =
+{
+ /* "authtype" is not used anymore, there is no mapping to PGconn */
+ /* there is no mapping of "service" to PGconn */
+ { "user", offsetof(struct pg_conn, pguser), true, NULL, NULL },
+ { "password", offsetof(struct pg_conn, pgpass), true, NULL, NULL },
+ { "connect_timeout", offsetof(struct pg_conn, connect_timeout), true, NULL, NULL },
+ { "dbname", offsetof(struct pg_conn, dbName), false, NULL, NULL },
+ { "host", offsetof(struct pg_conn, pghost), true, NULL, NULL },
+ { "hostaddr", offsetof(struct pg_conn, pghostaddr), true, NULL, NULL },
+ { "port", offsetof(struct pg_conn, pgport), true, NULL, NULL },
+ { "client_encoding", offsetof(struct pg_conn, client_encoding_initial), false, NULL, NULL },
+ { "tty", offsetof(struct pg_conn, pgtty), false, NULL, NULL },
+ { "options", offsetof(struct pg_conn, pgoptions), true, NULL, NULL },
+ { "application_name", offsetof(struct pg_conn, appname), false, NULL, NULL },
+ { "fallback_application_name", offsetof(struct pg_conn, fbappname), false, NULL, NULL },
+ { "keepalives", offsetof(struct pg_conn, keepalives), true, NULL, NULL },
+ { "keepalives_idle", offsetof(struct pg_conn, keepalives_idle), true, NULL, NULL },
+ { "keepalives_interval", offsetof(struct pg_conn, keepalives_interval), true, NULL, NULL },
+ { "keepalives_count", offsetof(struct pg_conn, keepalives_count), true, NULL, NULL },
+#ifdef USE_SSL
+ { "requiressl", offsetof(struct pg_conn, sslmode), false, "1", "require" },
+#endif
+ { "sslmode", offsetof(struct pg_conn, sslmode), true, NULL, NULL },
+ { "sslcompression", offsetof(struct pg_conn, sslcompression), true, NULL, NULL },
+ { "sslcert", offsetof(struct pg_conn, sslcert), true, NULL, NULL },
+ { "sslkey", offsetof(struct pg_conn, sslkey), true, NULL, NULL },
+ { "sslrootcert", offsetof(struct pg_conn, sslrootcert), true, NULL, NULL },
+ { "sslcrl", offsetof(struct pg_conn, sslcrl), true, NULL, NULL },
+ { "requirepeer", offsetof(struct pg_conn, requirepeer), true, NULL, NULL },
+#if defined(KRB5) || defined(ENABLE_GSS) || defined(ENABLE_SSPI)
+ { "krbsrvname", offsetof(struct pg_conn, krbsrvname), true, NULL, NULL },
+#endif
+#if defined(ENABLE_GSS) && defined(ENABLE_SSPI)
+ { "gsslib", offsetof(struct pg_conn, requirepeer), true, NULL, NULL },
+#endif
+ { "replication", offsetof(struct pg_conn, replication), false, NULL, NULL },
+ /* Terminating entry --- MUST BE LAST */
+ { NULL, 0, false, NULL, NULL }
+};
+
static const PQEnvironmentOption EnvironmentOptions[] =
{
/* common user-interface settings */
@@ -295,7 +358,8 @@ static PGconn *makeEmptyPGconn(void);
static void fillPGconn(PGconn *conn, PQconninfoOption *connOptions);
static void freePGconn(PGconn *conn);
static void closePGconn(PGconn *conn);
-static PQconninfoOption *conninfo_init(PQExpBuffer errorMessage);
+static PQconninfoOption *conninfo_init(PQExpBuffer errorMessage,
+ bool for_replication);
static PQconninfoOption *parse_connection_string(const char *conninfo,
PQExpBuffer errorMessage, bool use_defaults);
static int uri_prefix_length(const char *connstr);
@@ -318,6 +382,8 @@ static char *conninfo_uri_decode(const c
static bool get_hexdigit(char digit, int *value);
static const char *conninfo_getval(PQconninfoOption *connOptions,
const char *keyword);
+static void conninfo_setval(PQconninfoOption *connOptions,
+ const char *keyword, const char *val);
static PQconninfoOption *conninfo_storeval(PQconninfoOption *connOptions,
const char *keyword, const char *value,
PQExpBuffer errorMessage, bool ignoreMissing, bool uri_decode);
@@ -627,7 +693,9 @@ PQconnectStart(const char *conninfo)
static void
fillPGconn(PGconn *conn, PQconninfoOption *connOptions)
{
+ const PQconninfoMapping *mapping;
const char *tmp;
+ char **memberaddr;
/*
* Move option values into conn structure
@@ -637,72 +705,24 @@ fillPGconn(PGconn *conn, PQconninfoOptio
*
* XXX: probably worth checking strdup() return value here...
*/
- tmp = conninfo_getval(connOptions, "hostaddr");
- conn->pghostaddr = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "host");
- conn->pghost = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "port");
- conn->pgport = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "tty");
- conn->pgtty = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "options");
- conn->pgoptions = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "application_name");
- conn->appname = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "fallback_application_name");
- conn->fbappname = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "dbname");
- conn->dbName = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "user");
- conn->pguser = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "password");
- conn->pgpass = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "connect_timeout");
- conn->connect_timeout = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "client_encoding");
- conn->client_encoding_initial = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "keepalives");
- conn->keepalives = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "keepalives_idle");
- conn->keepalives_idle = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "keepalives_interval");
- conn->keepalives_interval = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "keepalives_count");
- conn->keepalives_count = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "sslmode");
- conn->sslmode = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "sslcompression");
- conn->sslcompression = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "sslkey");
- conn->sslkey = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "sslcert");
- conn->sslcert = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "sslrootcert");
- conn->sslrootcert = tmp ? strdup(tmp) : NULL;
- tmp = conninfo_getval(connOptions, "sslcrl");
- conn->sslcrl = tmp ? strdup(tmp) : NULL;
-#ifdef USE_SSL
- tmp = conninfo_getval(connOptions, "requiressl");
- if (tmp && tmp[0] == '1')
+ for (mapping = PQconninfoMappings; mapping->keyword; mapping++)
{
- /* here warn that the requiressl option is deprecated? */
- if (conn->sslmode)
- free(conn->sslmode);
- conn->sslmode = strdup("require");
+ tmp = conninfo_getval(connOptions, mapping->keyword);
+ memberaddr = PGCONNMEMBERADDR(conn, mapping);
+
+ if (mapping->conninfoValue && mapping->connValue)
+ {
+ size_t len = strlen(mapping->conninfoValue);
+ if (tmp && strncmp(tmp, mapping->conninfoValue, len) == 0)
+ {
+ if (*memberaddr)
+ free(*memberaddr);
+ *memberaddr = strdup(mapping->connValue);
+ }
+ }
+ else
+ *memberaddr = tmp ? strdup(tmp) : NULL;
}
-#endif
- tmp = conninfo_getval(connOptions, "requirepeer");
- conn->requirepeer = tmp ? strdup(tmp) : NULL;
-#if defined(KRB5) || defined(ENABLE_GSS) || defined(ENABLE_SSPI)
- tmp = conninfo_getval(connOptions, "krbsrvname");
- conn->krbsrvname = tmp ? strdup(tmp) : NULL;
-#endif
-#if defined(ENABLE_GSS) && defined(ENABLE_SSPI)
- tmp = conninfo_getval(connOptions, "gsslib");
- conn->gsslib = tmp ? strdup(tmp) : NULL;
-#endif
- tmp = conninfo_getval(connOptions, "replication");
- conn->replication = tmp ? strdup(tmp) : NULL;
}
/*
@@ -884,7 +904,7 @@ PQconndefaults(void)
if (PQExpBufferDataBroken(errorBuf))
return NULL; /* out of memory already :-( */
- connOptions = conninfo_init(&errorBuf);
+ connOptions = conninfo_init(&errorBuf, false);
if (connOptions != NULL)
{
if (!conninfo_add_defaults(connOptions, &errorBuf))
@@ -4006,9 +4026,11 @@ PQconninfoParse(const char *conninfo, ch
/*
* Build a working copy of the constant PQconninfoOptions array.
+ * If for_replication is true, only return the options that are
+ * not added by libpqwalreceiver.
*/
static PQconninfoOption *
-conninfo_init(PQExpBuffer errorMessage)
+conninfo_init(PQExpBuffer errorMessage, bool for_replication)
{
PQconninfoOption *options;
@@ -4019,7 +4041,28 @@ conninfo_init(PQExpBuffer errorMessage)
libpq_gettext("out of memory\n"));
return NULL;
}
- memcpy(options, PQconninfoOptions, sizeof(PQconninfoOptions));
+ if (for_replication)
+ {
+ const PQconninfoMapping *mapping = PQconninfoMappings;
+ PQconninfoOption *opt_dest = options;
+
+ while (mapping->keyword)
+ {
+ PQconninfoOption *opt_src = conninfo_find((PQconninfoOption *)PQconninfoOptions, mapping->keyword);
+
+ if (opt_src && mapping->for_replication)
+ {
+ memcpy(opt_dest, opt_src, sizeof(PQconninfoOption));
+ opt_dest++;
+ }
+
+ opt_src++;
+ mapping++;
+ }
+ MemSet(opt_dest, 0, sizeof(PQconninfoOption));
+ }
+ else
+ memcpy(options, PQconninfoOptions, sizeof(PQconninfoOptions));
return options;
}
@@ -4095,7 +4138,7 @@ conninfo_parse(const char *conninfo, PQE
PQconninfoOption *options;
/* Make a working copy of PQconninfoOptions */
- options = conninfo_init(errorMessage);
+ options = conninfo_init(errorMessage, false);
if (options == NULL)
return NULL;
@@ -4295,7 +4338,7 @@ conninfo_array_parse(const char *const *
}
/* Make a working copy of PQconninfoOptions */
- options = conninfo_init(errorMessage);
+ options = conninfo_init(errorMessage, false);
if (options == NULL)
{
PQconninfoFree(dbname_options);
@@ -4485,7 +4528,7 @@ conninfo_uri_parse(const char *uri, PQEx
PQconninfoOption *options;
/* Make a working copy of PQconninfoOptions */
- options = conninfo_init(errorMessage);
+ options = conninfo_init(errorMessage, false);
if (options == NULL)
return NULL;
@@ -4985,6 +5028,24 @@ conninfo_getval(PQconninfoOption *connOp
}
/*
+ * Set an option value corresponding to the keyword in the connOptions array.
+ */
+static void
+conninfo_setval(PQconninfoOption *connOptions, const char *keyword,
+ const char *val)
+{
+ PQconninfoOption *option;
+
+ option = conninfo_find(connOptions, keyword);
+ if (option)
+ {
+ if (option->val)
+ free(option->val);
+ option->val = val ? strdup(val) : NULL;
+ }
+}
+
+/*
* Store a (new) value for an option corresponding to the keyword in
* connOptions array.
*
@@ -5066,6 +5127,50 @@ conninfo_find(PQconninfoOption *connOpti
}
+/*
+ * Return the connection options used for the connections
+ */
+PQconninfoOption *
+PQconninfo(PGconn *conn, bool for_replication)
+{
+ PQExpBufferData errorBuf;
+ PQconninfoOption *connOptions;
+
+ if (conn == NULL)
+ return NULL;
+
+ /* We don't actually report any errors here, but callees want a buffer */
+ initPQExpBuffer(&errorBuf);
+ if (PQExpBufferDataBroken(errorBuf))
+ return NULL; /* out of memory already :-( */
+
+ connOptions = conninfo_init(&errorBuf, for_replication);
+
+ termPQExpBuffer(&errorBuf);
+
+ if (connOptions != NULL)
+ {
+ const PQconninfoMapping *mapping;
+
+ for (mapping = PQconninfoMappings; mapping->keyword; mapping++)
+ {
+ char **memberaddr = PGCONNMEMBERADDR(conn, mapping);
+
+ if (mapping->conninfoValue && mapping->connValue)
+ {
+ size_t len = strlen(mapping->connValue);
+ if (*memberaddr && strncmp(*memberaddr, mapping->connValue, len) == 0)
+ conninfo_setval(connOptions, mapping->keyword, mapping->conninfoValue);
+ }
+ else
+ conninfo_setval(connOptions, mapping->keyword, *memberaddr);
+ }
+ }
+
+ return connOptions;
+}
+
+
void
PQconninfoFree(PQconninfoOption *connOptions)
{
diff -durpN postgresql/src/interfaces/libpq/libpq-fe.h postgresql.1/src/interfaces/libpq/libpq-fe.h
--- postgresql/src/interfaces/libpq/libpq-fe.h 2012-10-09 09:58:14.343782980 +0200
+++ postgresql.1/src/interfaces/libpq/libpq-fe.h 2012-10-14 11:12:07.054196291 +0200
@@ -262,6 +262,9 @@ extern PQconninfoOption *PQconndefaults(
/* parse connection options in same way as PQconnectdb */
extern PQconninfoOption *PQconninfoParse(const char *conninfo, char **errmsg);
+/* return the connection options used by a live connection */
+extern PQconninfoOption *PQconninfo(PGconn *conn, bool for_replication);
+
/* free the data structure returned by PQconndefaults() or PQconninfoParse() */
extern void PQconninfoFree(PQconninfoOption *connOptions);
diff -durpN postgresql.1/doc/src/sgml/ref/pg_basebackup.sgml postgresql.2/doc/src/sgml/ref/pg_basebackup.sgml
--- postgresql.1/doc/src/sgml/ref/pg_basebackup.sgml 2012-08-24 09:49:22.960530329 +0200
+++ postgresql.2/doc/src/sgml/ref/pg_basebackup.sgml 2012-10-15 09:42:35.293396090 +0200
@@ -189,6 +189,21 @@ PostgreSQL documentation
</varlistentry>
<varlistentry>
+ <term><option>-R</option></term>
+ <term><option>--write-recovery-conf</option></term>
+ <listitem>
+
+ <para>
+ Write a minimal recovery.conf into the output directory (or into
+ the base archive file if <option>--format=tar</option> was specified)
+ using the connection parameters from the command line to ease
+ setting up the standby.
+ </para>
+
+ </listitem>
+ </varlistentry>
+
+ <varlistentry>
<term><option>-x</option></term>
<term><option>--xlog</option></term>
<listitem>
diff -durpN postgresql.1/src/bin/pg_basebackup/pg_basebackup.c postgresql.2/src/bin/pg_basebackup/pg_basebackup.c
--- postgresql.1/src/bin/pg_basebackup/pg_basebackup.c 2012-10-03 10:40:48.297207389 +0200
+++ postgresql.2/src/bin/pg_basebackup/pg_basebackup.c 2012-10-15 09:42:39.078417914 +0200
@@ -25,6 +25,7 @@
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
+#include <time.h>
#ifdef HAVE_LIBZ
#include <zlib.h>
@@ -46,6 +47,7 @@ int compresslevel = 0;
bool includewal = false;
bool streamwal = false;
bool fastcheckpoint = false;
+bool writerecoveryconf = false;
int standby_message_timeout = 10 * 1000; /* 10 sec = default */
/* Progress counters */
@@ -70,6 +72,11 @@ static int has_xlogendptr = 0;
static volatile LONG has_xlogendptr = 0;
#endif
+#define TARCHUNKSZ (512)
+#define RCBUFSZ TARCHUNKSZ
+static char *recoveryconf = NULL;
+static size_t recoveryconflen = 0;
+
/* Function headers */
static void usage(void);
static void verify_dir_is_empty_or_create(char *dirname);
@@ -77,6 +84,15 @@ static void progress_report(int tablespa
static void ReceiveTarFile(PGconn *conn, PGresult *res, int rownum);
static void ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum);
+static void _reallocRCBuffer(char **buf, int *bufsz, int currsz,
+ char *keyword, char *value,
+ int extrasz);
+static void CreateRecoveryConf(PGconn *conn);
+static void WriteRecoveryConf(void);
+static int _tarChecksum(char *header);
+static void print_val(char *s, uint64 val, unsigned int base, size_t len);
+static void scan_val(char *s, uint64 *val, unsigned int base, size_t len);
+static void _tarCreateHeader(char *header);
static void BaseBackup(void);
static bool reached_end_position(XLogRecPtr segendpos, uint32 timeline,
@@ -107,6 +123,8 @@ usage(void)
printf(_("\nOptions controlling the output:\n"));
printf(_(" -D, --pgdata=DIRECTORY receive base backup into directory\n"));
printf(_(" -F, --format=p|t output format (plain (default), tar)\n"));
+ printf(_(" -R, --write-recovery-conf\n"
+ " write recovery.conf after backup\n"));
printf(_(" -x, --xlog include required WAL files in backup (fetch mode)\n"));
printf(_(" -X, --xlog-method=fetch|stream\n"
" include required WAL files with specified method\n"));
@@ -467,6 +485,11 @@ ReceiveTarFile(PGconn *conn, PGresult *r
char filename[MAXPGPATH];
char *copybuf = NULL;
FILE *tarfile = NULL;
+ char tarhdr[512];
+ bool basebackup;
+ bool in_tarhdr, skip_file;
+ size_t tarhdrsz;
+ uint64 filesz;
#ifdef HAVE_LIBZ
gzFile ztarfile = NULL;
@@ -519,6 +542,8 @@ ReceiveTarFile(PGconn *conn, PGresult *r
tarfile = fopen(filename, "wb");
}
}
+
+ basebackup = true;
}
else
{
@@ -547,6 +572,8 @@ ReceiveTarFile(PGconn *conn, PGresult *r
PQgetvalue(res, rownum, 0));
tarfile = fopen(filename, "wb");
}
+
+ basebackup = false;
}
#ifdef HAVE_LIBZ
@@ -584,6 +611,10 @@ ReceiveTarFile(PGconn *conn, PGresult *r
disconnect_and_exit(1);
}
+ in_tarhdr = true;
+ skip_file = false;
+ tarhdrsz = filesz = 0;
+
while (1)
{
int r;
@@ -598,13 +629,69 @@ ReceiveTarFile(PGconn *conn, PGresult *r
if (r == -1)
{
/*
- * End of chunk. Close file (but not stdout).
+ * End of chunk. Write recovery.conf into the tar file (if it
+ * was requested) and close file (but not stdout).
*
* Also, write two completely empty blocks at the end of the tar
* file, as required by some tar programs.
*/
char zerobuf[1024];
+ if (basebackup && writerecoveryconf)
+ {
+ char header[512];
+ int lastchunk;
+
+ _tarCreateHeader(header);
+
+ lastchunk = recoveryconflen % TARCHUNKSZ;
+ if (lastchunk > 0)
+ recoveryconflen += (TARCHUNKSZ - lastchunk);
+#ifdef HAVE_LIBZ
+ if (ztarfile != NULL)
+ {
+ if (gzwrite(ztarfile, header, sizeof(header)) !=
+ sizeof(header))
+ {
+ fprintf(stderr,
+ _("%s: could not write to compressed file \"%s\": %s\n"),
+ progname, filename, get_gz_error(ztarfile));
+ disconnect_and_exit(1);
+ }
+
+ if (gzwrite(ztarfile, recoveryconf, recoveryconflen) !=
+ recoveryconflen)
+ {
+ fprintf(stderr,
+ _("%s: could not write to compressed file \"%s\": %s\n"),
+ progname, filename, get_gz_error(ztarfile));
+ disconnect_and_exit(1);
+ }
+ }
+ else
+#endif
+ {
+ if (fwrite(header, sizeof(header), 1, tarfile) != 1)
+ {
+ fprintf(stderr,
+ _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
+
+ if (fwrite(recoveryconf, recoveryconflen, 1, tarfile) != 1)
+ {
+ fprintf(stderr,
+ _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ }
+ fprintf(stderr, _("%s: recovery.conf written into '%s'\n"), progname, filename);
+
+ free(recoveryconf);
+ }
+
MemSet(zerobuf, 0, sizeof(zerobuf));
#ifdef HAVE_LIBZ
if (ztarfile != NULL)
@@ -665,25 +752,130 @@ ReceiveTarFile(PGconn *conn, PGresult *r
disconnect_and_exit(1);
}
-#ifdef HAVE_LIBZ
- if (ztarfile != NULL)
+ if (basebackup && writerecoveryconf)
{
- if (gzwrite(ztarfile, copybuf, r) != r)
+ int rr = r;
+ int pos = 0;
+
+ while (rr > 0)
{
- fprintf(stderr,
- _("%s: could not write to compressed file \"%s\": %s\n"),
- progname, filename, get_gz_error(ztarfile));
- disconnect_and_exit(1);
+ if (in_tarhdr)
+ {
+ if (tarhdrsz < 512)
+ {
+ int hdrleft, bytes2copy;
+
+ hdrleft = TARCHUNKSZ - tarhdrsz;
+ bytes2copy = (rr > hdrleft ? hdrleft : rr);
+
+ memcpy(&tarhdr[tarhdrsz], copybuf + pos, bytes2copy);
+
+ rr -= bytes2copy;
+ pos += bytes2copy;
+ tarhdrsz += bytes2copy;
+ }
+ else
+ {
+ int lastchunk;
+
+ skip_file = (strcmp(&tarhdr[0], "recovery.conf") == 0);
+
+ scan_val(&tarhdr[124], &filesz, 8, 11);
+ lastchunk = filesz % TARCHUNKSZ;
+ if (lastchunk > 0)
+ filesz += (TARCHUNKSZ - lastchunk);
+
+ in_tarhdr = false;
+ tarhdrsz = 0;
+
+#ifdef HAVE_LIBZ
+ if (ztarfile != NULL)
+ {
+ if (!skip_file && gzwrite(ztarfile, tarhdr, TARCHUNKSZ) != TARCHUNKSZ)
+ {
+ fprintf(stderr,
+ _("%s: could not write to compressed file \"%s\": %s\n"),
+ progname, filename, get_gz_error(ztarfile));
+ disconnect_and_exit(1);
+ }
+ }
+ else
+#endif
+ {
+ if (!skip_file && fwrite(tarhdr, TARCHUNKSZ, 1, tarfile) != 1)
+ {
+ fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ }
+ }
+ }
+ else /* inside a file */
+ {
+ if (filesz > 0)
+ {
+ int bytes2write;
+
+ bytes2write = (filesz > rr ? rr : filesz);
+
+#ifdef HAVE_LIBZ
+ if (ztarfile != NULL)
+ {
+ if (!skip_file && gzwrite(ztarfile, copybuf + pos, bytes2write) != bytes2write)
+ {
+ fprintf(stderr,
+ _("%s: could not write to compressed file \"%s\": %s\n"),
+ progname, filename, get_gz_error(ztarfile));
+ disconnect_and_exit(1);
+ }
+ }
+ else
+#endif
+ {
+ if (!skip_file && fwrite(copybuf + pos, bytes2write, 1, tarfile) != 1)
+ {
+ fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ }
+
+ rr -= bytes2write;
+ pos += bytes2write;
+ filesz -= bytes2write;
+ }
+ else
+ {
+ in_tarhdr = true;
+ skip_file = false;
+ tarhdrsz = filesz = 0;
+ }
+ }
}
}
else
-#endif
{
- if (fwrite(copybuf, r, 1, tarfile) != 1)
+#ifdef HAVE_LIBZ
+ if (ztarfile != NULL)
{
- fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
- progname, filename, strerror(errno));
- disconnect_and_exit(1);
+ if (gzwrite(ztarfile, copybuf, r) != r)
+ {
+ fprintf(stderr,
+ _("%s: could not write to compressed file \"%s\": %s\n"),
+ progname, filename, get_gz_error(ztarfile));
+ disconnect_and_exit(1);
+ }
+ }
+ else
+#endif
+ {
+ if (fwrite(copybuf, r, 1, tarfile) != 1)
+ {
+ fprintf(stderr, _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
}
}
totaldone += r;
@@ -712,13 +904,20 @@ ReceiveAndUnpackTarFile(PGconn *conn, PG
char filename[MAXPGPATH];
int current_len_left;
int current_padding = 0;
+ bool basebackup;
char *copybuf = NULL;
FILE *file = NULL;
if (PQgetisnull(res, rownum, 0))
+ {
strcpy(current_path, basedir);
+ basebackup = true;
+ }
else
+ {
strcpy(current_path, PQgetvalue(res, rownum, 1));
+ basebackup = false;
+ }
/*
* Get the COPY data
@@ -937,6 +1136,330 @@ ReceiveAndUnpackTarFile(PGconn *conn, PG
if (copybuf != NULL)
PQfreemem(copybuf);
+
+ if (basebackup)
+ WriteRecoveryConf();
+}
+
+static int
+_tarChecksum(char *header)
+{
+ int i,
+ sum;
+
+ /*
+ * Per POSIX, the checksum is the simple sum of all bytes in the header,
+ * treating the bytes as unsigned, and treating the checksum field (at
+ * offset 148) as though it contained 8 spaces.
+ */
+ sum = 8 * ' '; /* presumed value for checksum field */
+ for (i = 0; i < 512; i++)
+ if (i < 148 || i >= 156)
+ sum += 0xFF & header[i];
+ return sum;
+}
+
+
+/*
+ * Utility routine to print possibly larger than 32 bit integers in a
+ * portable fashion. Filled with zeros.
+ */
+static void
+print_val(char *s, uint64 val, unsigned int base, size_t len)
+{
+ int i;
+
+ for (i = len; i > 0; i--)
+ {
+ int digit = val % base;
+
+ s[i - 1] = '0' + digit;
+ val = val / base;
+ }
+}
+
+
+/*
+ * Inverse for print_val()
+ */
+static void
+scan_val(char *s, uint64 *val, unsigned int base, size_t len)
+{
+ uint64 tmp = 0;
+ int i;
+
+ for (i = 0; i < len; i++)
+ {
+ int digit = s[i] - '0';
+
+ tmp = tmp * base + digit;
+ }
+
+ *val = tmp;
+}
+
+
+static void
+_tarCreateHeader(char *header)
+{
+ /*
+ * Note: most of the fields in a tar header are not supposed to be
+ * null-terminated. We use sprintf, which will write a null after the
+ * required bytes; that null goes into the first byte of the next field.
+ * This is okay as long as we fill the fields in order.
+ */
+ memset(header, 0, 512 /* sizeof the tar header */);
+
+ /* Name 100 */
+ sprintf(&header[0], "%.99s", "recovery.conf");
+
+ /* Mode 8 */
+ sprintf(&header[100], "0000600 ");
+
+ /* User ID 8 */
+ sprintf(&header[108], "0004000 ");
+
+ /* Group 8 */
+ sprintf(&header[116], "0002000 ");
+
+ /* File size 12 - 11 digits, 1 space; use print_val for 64 bit support */
+ print_val(&header[124], recoveryconflen, 8, 11);
+ sprintf(&header[135], " ");
+
+ /* Mod Time 12 */
+ sprintf(&header[136], "%011o ", (int) time(NULL));
+
+ /* Checksum 8 cannot be calculated until we've filled all other fields */
+
+ /* Type - regular file */
+ sprintf(&header[156], "0");
+
+ /* Link Name 100 (leave as nulls) */
+
+ /* Magic 6 */
+ sprintf(&header[257], "ustar");
+
+ /* Version 2 */
+ sprintf(&header[263], "00");
+
+ /* User 32 */
+ /* XXX: Do we need to care about setting correct username? */
+ sprintf(&header[265], "%.31s", "postgres");
+
+ /* Group 32 */
+ /* XXX: Do we need to care about setting correct group name? */
+ sprintf(&header[297], "%.31s", "postgres");
+
+ /* Major Dev 8 */
+ sprintf(&header[329], "%07o ", 0);
+
+ /* Minor Dev 8 */
+ sprintf(&header[337], "%07o ", 0);
+
+ /* Prefix 155 - not used, leave as nulls */
+
+ /*
+ * We mustn't overwrite the next field while inserting the checksum.
+ * Fortunately, the checksum can't exceed 6 octal digits, so we just write
+ * 6 digits, a space, and a null, which is legal per POSIX.
+ */
+ sprintf(&header[148], "%06o ", _tarChecksum(header));
+}
+
+/*
+ * Escape single quotes in a string
+ */
+static char *
+escape_string(char *value)
+{
+ char *tmp, *escaped, *esc;
+ int num_quotes;
+
+ tmp = value;
+ num_quotes = 0;
+ while (*tmp)
+ {
+ if (*tmp == '\'')
+ num_quotes++;
+ tmp++;
+ }
+
+ escaped = malloc((tmp - value) + num_quotes + 1);
+ if (escaped == NULL)
+ {
+ fprintf(stderr, _("%s: out of memory"), progname);
+ disconnect_and_exit(1)
+ }
+
+ tmp = value;
+ esc = escaped;
+ while (*tmp)
+ {
+ if (*tmp == '\'')
+ *esc++ = '\\';
+ *esc++ = *tmp++;
+ }
+ *esc = '\0';
+
+ return escaped;
+}
+
+/*
+ * Reallocate the buffer for recovery.conf if needed
+ */
+static void
+_reallocRCBuffer(char **buf, int *bufsz, int currsz, char *keyword, char *value, int extrasz)
+{
+ int len = extrasz;
+
+ if (keyword != NULL)
+ len += strlen(keyword);
+ if (value != NULL)
+ len += strlen(value);
+
+ if (*bufsz < currsz + len)
+ {
+ char *tmp;
+
+ tmp = realloc(*buf, *bufsz + RCBUFSZ);
+ if (tmp == NULL)
+ {
+ fprintf(stderr, _("%s: out of memory"), progname);
+ disconnect_and_exit(1)
+ }
+ *buf = tmp;
+ *bufsz += RCBUFSZ;
+ memset(tmp + currsz, 0, *bufsz - currsz);
+ }
+}
+
+/*
+ * Try to create recovery.conf in memory and set the length to write later.
+ */
+static void
+CreateRecoveryConf(PGconn *conn)
+{
+ char *buf = NULL;
+ PQconninfoOption *connOptions;
+ PQconninfoOption *option;
+ int bufsz;
+ int total = 0;
+ int written;
+
+ if (!writerecoveryconf)
+ return;
+
+ connOptions = PQconninfo(conn, true);
+ if (connOptions == NULL)
+ {
+ fprintf(stderr, _("%s: out of memory"), progname);
+ disconnect_and_exit(1);
+ }
+
+
+ bufsz = RCBUFSZ;
+ buf = malloc(bufsz);
+ if (buf == NULL)
+ {
+ fprintf(stderr, _("%s: out of memory"), progname);
+ disconnect_and_exit(1);
+ }
+
+ written = sprintf(&buf[total], "standby_mode = 'on'\n");
+ if (written < 0)
+ {
+ fprintf(stderr, _("%s: cannot write to string: %s"), progname, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ total += written;
+
+ written = sprintf(&buf[total], "primary_conninfo = '");
+ if (written < 0)
+ {
+ fprintf(stderr, _("%s: cannot write to string: %s"), progname, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ total += written;
+
+ for (option = connOptions; option && option->keyword; option++)
+ {
+ char *escaped;
+ /*
+ * Do not emit this setting if not set or empty.
+ * The list of options was already pre-filtered for options
+ * usable for replication with PQconninfo(conn, true).
+ */
+ if ((option->val == NULL) ||
+ (option->val != NULL && option->val[0] == '\0'))
+ continue;
+
+ /*
+ * Write "keyword='value'" pieces, the value string is escaped
+ * if necessary and doubled single quotes around the value string.
+ */
+ escaped = escape_string(option->val);
+
+ _reallocRCBuffer(&buf, &bufsz, total, option->keyword, escaped, 6);
+
+ written = sprintf(&buf[total], "%s=''%s'' ", option->keyword, option->val);
+
+ free(escaped);
+
+ if (written < 0)
+ {
+ fprintf(stderr, _("%s: cannot write to string: %s"), progname, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ total += written;
+ }
+
+ _reallocRCBuffer(&buf, &bufsz, total, NULL, NULL, 2);
+ written = sprintf(&buf[total], "'\n");
+ if (written < 0)
+ {
+ fprintf(stderr, _("%s: cannot write to string: %s"), progname, strerror(errno));
+ disconnect_and_exit(1);
+ }
+ total += written;
+
+ recoveryconf = buf;
+ recoveryconflen = total;
+
+ PQconninfoFree(connOptions);
+}
+
+
+static void
+WriteRecoveryConf(void)
+{
+ char filename[MAXPGPATH];
+ FILE *cf;
+
+ if (!writerecoveryconf)
+ return;
+
+ sprintf(filename, "%s/recovery.conf", basedir);
+
+ cf = fopen(filename, "w");
+ if (cf == NULL)
+ {
+ fprintf(stderr, _("%s: cannot create %s: %s"), progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
+
+ if (fwrite(recoveryconf, recoveryconflen, 1, cf) != 1)
+ {
+ fprintf(stderr,
+ _("%s: could not write to file \"%s\": %s\n"),
+ progname, filename, strerror(errno));
+ disconnect_and_exit(1);
+ }
+
+ fclose(cf);
+
+ free(recoveryconf);
+
+ fprintf(stderr, _("%s: recovery.conf written.\n"), progname);
}
@@ -960,6 +1483,8 @@ BaseBackup(void)
/* Error message already written in GetConnection() */
exit(1);
+ CreateRecoveryConf(conn);
+
/*
* Run IDENTIFY_SYSTEM so we can get the timeline
*/
@@ -1243,6 +1768,7 @@ main(int argc, char **argv)
{"pgdata", required_argument, NULL, 'D'},
{"format", required_argument, NULL, 'F'},
{"checkpoint", required_argument, NULL, 'c'},
+ {"write-recovery-conf", no_argument, NULL, 'R'},
{"xlog", no_argument, NULL, 'x'},
{"xlog-method", required_argument, NULL, 'X'},
{"gzip", no_argument, NULL, 'z'},
@@ -1280,7 +1806,7 @@ main(int argc, char **argv)
}
}
- while ((c = getopt_long(argc, argv, "D:F:xX:l:zZ:c:h:p:U:s:wWvP",
+ while ((c = getopt_long(argc, argv, "D:F:RxX:l:zZ:c:h:p:U:s:wWvP",
long_options, &option_index)) != -1)
{
switch (c)
@@ -1301,6 +1827,9 @@ main(int argc, char **argv)
exit(1);
}
break;
+ case 'R':
+ writerecoveryconf = true;
+ break;
case 'x':
if (includewal)
{
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
