Re: [HACKERS] pg_basebackup --progress output for batch execution

[Martin Marques]

Updated patch with documentation of the new option.


-- 
Martín Marquéshttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
>From ede201ed96d41d799dc3c83dfab1cdcc03e5ced4 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Mart=C3=ADn=20Marqu=C3=A9s?=
 
Date: Sun, 1 Oct 2017 16:39:41 -0300
Subject: [PATCH] Adding an option to pg_basebackup to output messages as if it
 were running in batch-mode, as opossed to running in a tty.

This is usefull when using --progress and redirecting the output to
a file for later inspection with tail.

New option --batch-mode with the short option -b added.
---
 doc/src/sgml/ref/pg_basebackup.sgml   | 16 
 src/bin/pg_basebackup/pg_basebackup.c | 19 +--
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml
index f790c56..db5160f 100644
--- a/doc/src/sgml/ref/pg_basebackup.sgml
+++ b/doc/src/sgml/ref/pg_basebackup.sgml
@@ -461,6 +461,22 @@ PostgreSQL documentation
  
 
  
+  -b
+  --batch-mode
+  
+   
+Runs pg_basebackup in batch mode. This is useful if
+the output is to be pipped so the other end of the pipe reads each line.
+   
+   
+Using this option with --progress will result in
+printing each progress output with a newline at the end, instead of a
+carrige return.
+   
+  
+ 
+
+ 
   -S slotname
   --slot=slotname
   
diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index dac7299..cf97ea3 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -82,6 +82,7 @@ static char format = 'p';		/* p(lain)/t(ar) */
 static char *label = "pg_basebackup base backup";
 static bool noclean = false;
 static bool showprogress = false;
+static bool batchmode = false;
 static int	verbose = 0;
 static int	compresslevel = 0;
 static IncludeWal includewal = STREAM_WAL;
@@ -355,6 +356,7 @@ usage(void)
 	printf(_("  -P, --progress show progress information\n"));
 	printf(_("  -S, --slot=SLOTNAMEreplication slot to use\n"));
 	printf(_("  --no-slot  prevent creation of temporary replication slot\n"));
+	printf(_("  -b, --batch-mode   run in batch mode\n"));
 	printf(_("  -v, --verbose  output verbose messages\n"));
 	printf(_("  -V, --version  output version information, then exit\n"));
 	printf(_("  -?, --help show this help, then exit\n"));
@@ -806,7 +808,10 @@ progress_report(int tablespacenum, const char *filename, bool force)
 totaldone_str, totalsize_str, percent,
 tablespacenum, tablespacecount);
 
-	fprintf(stderr, "\r");
+	if (batchmode)
+		fprintf(stderr, "\n");
+	else
+		fprintf(stderr, "\r");
 }
 
 static int32
@@ -1786,7 +1791,13 @@ BaseBackup(void)
 progname);
 
 	if (showprogress && !verbose)
+	{
 		fprintf(stderr, "waiting for checkpoint\r");
+		if (batchmode)
+			fprintf(stderr, "\n");
+		else
+			fprintf(stderr, "\r");
+	}
 
 	basebkp =
 		psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s",
@@ -2118,6 +2129,7 @@ main(int argc, char **argv)
 		{"status-interval", required_argument, NULL, 's'},
 		{"verbose", no_argument, NULL, 'v'},
 		{"progress", no_argument, NULL, 'P'},
+		{"batch-mode", no_argument, NULL, 'b'},
 		{"waldir", required_argument, NULL, 1},
 		{"no-slot", no_argument, NULL, 2},
 		{NULL, 0, NULL, 0}
@@ -2146,7 +2158,7 @@ main(int argc, char **argv)
 
 	atexit(cleanup_directories_atexit);
 
-	while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWvP",
+	while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWvPb",
 			long_options, _index)) != -1)
 	{
 		switch (c)
@@ -2288,6 +2300,9 @@ main(int argc, char **argv)
 			case 'P':
 showprogress = true;
 break;
+			case 'b':
+batchmode = true;
+break;
 			default:
 
 /*
-- 
2.9.5


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] pg_basebackup --progress output for batch execution

[Martin Marques]

Hi,

Some time ago I had to work on a system where I was cloning a standby
using pg_basebackup, that didn't have screen or tmux. For that reason I
redirected the output to a file and ran it with nohup.

I normally (always actually ;) ) run pg_basebackup with --progress and
--verbose so I can follow how much has been done. When done on a tty you
get a nice progress bar with the percentage that has been cloned.

The problem came with the execution and redirection of the output, as
the --progress option will write a *very* long line!

Back then I thought of writing a patch (actually someone suggested I do
so) to add a --batch-mode option which would change the behavior
pg_basebackup has when printing the output messages.

Attach is the patch. I'll be submitting it to the CF.

P.D.: I'm aware that there's a documentation patch missing. :)

Kind regards,

-- 
Martín Marquéshttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
>From f6e54c0d062d62daf70c0870f96032eb0c102e66 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Mart=C3=ADn=20Marqu=C3=A9s?=
 
Date: Fri, 29 Sep 2017 19:21:44 -0300
Subject: [PATCH] Adding an option to pg_basebackup to output messages as if it
 were running in batch-mode, as opossed to running in a tty.

This is usefull when using --progress and redirecting the output to
a file for later inspection with tail.

New option --batch-mode with the short option -b added.
---
 src/bin/pg_basebackup/pg_basebackup.c | 19 +--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index dac7299..cf97ea3 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -82,6 +82,7 @@ static char format = 'p';		/* p(lain)/t(ar) */
 static char *label = "pg_basebackup base backup";
 static bool noclean = false;
 static bool showprogress = false;
+static bool batchmode = false;
 static int	verbose = 0;
 static int	compresslevel = 0;
 static IncludeWal includewal = STREAM_WAL;
@@ -355,6 +356,7 @@ usage(void)
 	printf(_("  -P, --progress show progress information\n"));
 	printf(_("  -S, --slot=SLOTNAMEreplication slot to use\n"));
 	printf(_("  --no-slot  prevent creation of temporary replication slot\n"));
+	printf(_("  -b, --batch-mode   run in batch mode\n"));
 	printf(_("  -v, --verbose  output verbose messages\n"));
 	printf(_("  -V, --version  output version information, then exit\n"));
 	printf(_("  -?, --help show this help, then exit\n"));
@@ -806,7 +808,10 @@ progress_report(int tablespacenum, const char *filename, bool force)
 totaldone_str, totalsize_str, percent,
 tablespacenum, tablespacecount);
 
-	fprintf(stderr, "\r");
+	if (batchmode)
+		fprintf(stderr, "\n");
+	else
+		fprintf(stderr, "\r");
 }
 
 static int32
@@ -1786,7 +1791,13 @@ BaseBackup(void)
 progname);
 
 	if (showprogress && !verbose)
+	{
 		fprintf(stderr, "waiting for checkpoint\r");
+		if (batchmode)
+			fprintf(stderr, "\n");
+		else
+			fprintf(stderr, "\r");
+	}
 
 	basebkp =
 		psprintf("BASE_BACKUP LABEL '%s' %s %s %s %s %s %s",
@@ -2118,6 +2129,7 @@ main(int argc, char **argv)
 		{"status-interval", required_argument, NULL, 's'},
 		{"verbose", no_argument, NULL, 'v'},
 		{"progress", no_argument, NULL, 'P'},
+		{"batch-mode", no_argument, NULL, 'b'},
 		{"waldir", required_argument, NULL, 1},
 		{"no-slot", no_argument, NULL, 2},
 		{NULL, 0, NULL, 0}
@@ -2146,7 +2158,7 @@ main(int argc, char **argv)
 
 	atexit(cleanup_directories_atexit);
 
-	while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWvP",
+	while ((c = getopt_long(argc, argv, "CD:F:r:RS:T:X:l:nNzZ:d:c:h:p:U:s:wWvPb",
 			long_options, _index)) != -1)
 	{
 		switch (c)
@@ -2288,6 +2300,9 @@ main(int argc, char **argv)
 			case 'P':
 showprogress = true;
 break;
+			case 'b':
+batchmode = true;
+break;
 			default:
 
 /*
-- 
2.9.5


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] I need Help

[Martin Marques]

El Mar 04 May 2004 01:50, Christopher Kings-Lynne escribió:
  I installed postgresql 7.4 in my computer, I'm using
  redhat 9.0 .
  I installed pgadmin III but I can't to conecct to the
  server.
 
  The port 5432 is not open.

 You need to set tcpip_socket = true in your postgresql.conf.

And in the case of RH check that the firewall isn't blocking that port.

P.D.: Olivia, I would suggest you to use the pgsql-general list for this kind 
of questions, were you are likely to get more and maybe even better answers. 
If you have problems with your english, try one of the local speeking lists.

-- 
 08:30:02 up 5 days, 13:52,  2 users,  load average: 0.93, 0.86, 0.79
-
Martín Marqués| select 'mmarques' || '@' || 'unl.edu.ar'
Centro de Telematica  |  DBA, Programador, Administrador
 Universidad Nacional
  del Litoral
-

---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] 7.5 features

[Martin Marques]

El Tuesday 27 April 2004 10:27, Bruce Momjian escribió:
 Here are features that are being worked on, hopefully for 7.5:

   o  tablespaces (Gavin)
   o  nested transactions (Alvaro)
   o  two-phase commit (Heikki Linnakangas)
   o  integrated pg_autovacuum (O'Connor)
   o  PITR (Riggs)
   o  Win32 (Claudio, Magnus)

 If we get the majority of them, and I think we will, this will be a
 great release.

How's Jans' Slowny-I doing? Any chance of getting it at least in the contribs 
(depending on how stable it gets)?

-- 
 11:21:02 up 49 days, 15:45,  4 users,  load average: 0.53, 0.68, 0.71
-
Martín Marqués| select 'mmarques' || '@' || 'unl.edu.ar'
Centro de Telematica  |  DBA, Programador, Administrador
 Universidad Nacional
  del Litoral
-


---(end of broadcast)---
TIP 8: explain analyze is your friend

Re: [HACKERS] Remove MySQL Tools from Source?

[Martin Marques]

El Mié 14 Abr 2004 22:22, Christopher Kings-Lynne escribió:
  ... on projects.postgresql.org, or similar.They really aren't doing
  any good in /contrib.
 
  I've already set up a category conversion tools on pgFoundry, and my
  idea was one project per target system.

 I reckon that by far the best way to do a mysql2pgsql converter is to
 just modify mysqldump C source code to output in postgresql format!

I always ran one of the 2 scripts (can't remember which one) and after that 
started checking the dump file, because there were things that didn't get 
changed correctly[1].

[1]: I always remember the first conversion I did. I found out that MySQL 
accepted dates like 30/2/2000 or 0-0-.
Very odd.

-- 
 08:24:01 up 37 days, 12:51,  2 users,  load average: 0.91, 0.85, 0.98
-
Martín Marqués| select 'mmarques' || '@' || 'unl.edu.ar'
Centro de Telematica  |  DBA, Programador, Administrador
 Universidad Nacional
  del Litoral
-


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly

Re: [HACKERS] make == as = ?

[Martin Marques]

El Mié 07 Abr 2004 06:28, Fabien COELHO escribió:
=? as != is a synonum for , it would make sense.
  
   That was never such a terribly good idea, IMHO.
 
  Agreed. Compilers should give errors and not try to work around bad code.

 Is it bad code? Not for people who come from a C/C++/Java background.
 They are used to operators such as == !=  %  || !... Some of these
 are available from pg, some are not, so at the time it is incoherent.

I have such a background, and still don't use != to ask for inequality. The 
correct thing to use is , because thats what the SQL standards say.

-- 
 17:28:01 up 29 days, 21:55,  2 users,  load average: 0.34, 0.31, 0.29
-
Martín Marqués| select 'mmarques' || '@' || 'unl.edu.ar'
Centro de Telematica  |  DBA, Programador, Administrador
 Universidad Nacional
  del Litoral
-


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] lib for clients

[Martin Marques]

El Lun 01 Mar 2004 11:11, ivan escribió:
 hi,

 is there same packets (or sources to compile) only for client-systems
 ( headers and libs like libpq, and so on) ?

On a normal Linux distribution, you would have packages like this:

postgresql - PostgreSQL client programs and libraries.
postgresql-contrib - Contributed source and binaries distributed with 
PostgreSQL
postgresql-devel - PostgreSQL development header files and libraries.
postgresql-docs - Extra documentation for PostgreSQL
postgresql-jdbc - Files needed for Java programs to access a PostgreSQL 
database.
postgresql-libs - The shared libraries required for any PostgreSQL clients.
postgresql-odbc - The ODBC driver needed for accessing a PostgreSQL DB using 
ODBC.
postgresql-perl - Development module needed for Perl code to access a 
PostgreSQL DB.
postgresql-python - Development module for Python code to access a PostgreSQL 
DB.
postgresql-server - The programs needed to create and run a PostgreSQL server.
postgresql-tcl - A Tcl client library, and the PL/Tcl procedural language for 
PostgreSQL.
postgresql-test - The test suite distributed with PostgreSQL.
postgresql-tk - Tk shell and tk-based GUI for PostgreSQL.

Hope this helps you.

-- 
 11:19:02 up 96 days, 17:28,  7 users,  load average: 0.62, 0.65, 0.89
-
Martín Marqués| select 'mmarques' || '@' || 'unl.edu.ar'
Centro de Telematica  |  DBA, Programador, Administrador
 Universidad Nacional
  del Litoral
-


---(end of broadcast)---
TIP 6: Have you searched our list archives?

   http://archives.postgresql.org

Incremental Backup (Was: [HACKERS] And ppl complain about *our* beta cycles ...)

[Martin Marques]

Mensaje citado por David Garamond [EMAIL PROTECTED]:

 Marc G. Fournier wrote:
 From the Firebird FAQ:
  
  The first beta was released on January 29, 2003. We are hoping to be
  close to a full release some time around Easter 2003.
  
  They are at RC8 right now ... running a *wee* bit behind scheduale :)
 
 Yes, they're pretty late. Last time I read, the only major issues
 preventing their final release is around the installer. The 1.5 codebase
 itself has been stabilized for quite a while. Practically all work is
 now done to the 2.0 branch/HEAD. They have several goodies in store for
 the 2.0 release (e.g.: incremental backup).

Anyone working on incremental backups for PostgreSQL? I kind of miss them from my
Informix times (hey, I think it's the only thing I really miss :-) ).

Maybe when PITR is ready, making incremental backups will be trivial.

-- 
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
-
Martín Marqués  |   Programador, DBA
Centro de Telemática| Administrador
   Universidad Nacional
del Litoral
-

---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] What's planned for 7.5?

[Martin Marques]

Mensaje citado por Marc G. Fournier [EMAIL PROTECTED]:

 
 Native Win32 is planned for it (whether it makes it or not is another
 question, but it is the goal) ...

Replication wasn't another BIG one?

-- 
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
-
Martín Marqués  |   Programador, DBA
Centro de Telemática| Administrador
   Universidad Nacional
del Litoral
-

---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send unregister YourEmailAddressHere to [EMAIL PROTECTED])

Re: [HACKERS] What's planned for 7.5?

[Martin Marques]

Mensaje citado por ow [EMAIL PROTECTED]:

 Hi,
 
 Is this all that's planned for 7.5? (based on current TODO list)
 
 -Change factorial to return a numeric (Gavin)
 -COMMENT ON [ CAST | CONVERSION | OPERATOR CLASS | LARGE OBJECT | LANGUAGE ]
 (Christopher)
 -Have psql \dn show only visible temp schemas using current_schemas()
 -Have psql '\i ~/tabtab' actually load files it displays from home dir
 -Allow psql \du to show groups, and add \dg for groups
 -Allow pg_dump to dump CREATE CONVERSION (Christopher)
 -Use dependency information to dump data in proper order
 -Use background process to write dirty shared buffers to disk

For what I have just seen in Robert Treats PostgreSQL Weekly News most of
these issues are already solved in the CVS, and I guess that in a bit less than
a year that's left for the release of 7.5 many new things will appear.

Anyway, from http://developer.postgresql.org/todo.php I see in the URGENT part
this:

# Add replication of distributed databases [replication]

* Automatic failover
* Load balancing
* Master/slave replication
* Multi-master replication
* Partition data across servers
* Queries across databases or servers (two-phase commit)
* Allow replication over unreliable or non-persistent links
* http://gborg.postgresql.org/project/pgreplication/projdisplay.php 

# Point-in-time data recovery using backup and write-ahead log
# Create native Win32 port, 
*  http://momjian.postgresql.org/main/writings/pgsql/win32.html 

The first isn't new, but as for what I know, it's a new aproach that Jan has for
adding replication to the server.

The other 2 didn't get in the 7.4 release.

-- 
select 'mmarques' || '@' || 'unl.edu.ar' AS email;
-
Martín Marqués  |   Programador, DBA
Centro de Telemática| Administrador
   Universidad Nacional
del Litoral
-

---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

   http://www.postgresql.org/docs/faqs/FAQ.html

Re: [HACKERS] time format

[Martin Marques]

El Sáb 03 Ene 2004 18:20, ivan escribió:
 ok, bat each time where i want to do select .. a nie tu use to_char,
 but it should be in function timestamp_out to convert time to string
 it would be easer and faster.

Look deeper into what Christopher said and use casting to get the right 
output:

prueba= select now();
  now
---
 2004-01-04 09:53:41.131079-03
(1 row)

prueba= select now()::timestamp(0);
 now
-
 2004-01-04 09:53:43
(1 row)


Get the difference?

-- 
 09:52:01 up 39 days, 16:08,  2 users,  load average: 0.60, 0.71, 0.72
-
Martín Marqués| select 'mmarques' || '@' || 'unl.edu.ar'
Centro de Telematica  |  DBA, Programador, Administrador
 Universidad Nacional
  del Litoral
-


---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]

Re: [HACKERS] time format

[Martin Marques]

Quoting Tom Lane [EMAIL PROTECTED]:

 Martin Marques [EMAIL PROTECTED] writes:
  Look deeper into what Christopher said and use casting to get the right 
  output:
  prueba= select now()::timestamp(0);
 
 There's also current_timestamp(0), which is a more standards-compliant
 way of doing the same thing.

Didn´t know that existed. :-)

Anyway, my observation was not on the now() function, but at the casting. He
wants the output of th select over a timestamp field to come out without the
milliseconds, which is done with the casting.

Now, seeing your mail I realize that what he may want is this:

CREATE TABLE table_name (
  .
  time_field   timestamp(0)
);

Remember you will lose those milliseconds for ever with this table definition.

-- 
Porqué usar una base de datos relacional cualquiera,
si podés usar PostgreSQL?
-
Martín Marqués  |[EMAIL PROTECTED]
Programador, Administrador, DBA |   Centro de Telematica
   Universidad Nacional
del Litoral
-

---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster

Re: [HACKERS] Incremental backup

[Martin Marques]

On Jue 13 Feb 2003 16:38, Bruce Momjian wrote:
 Patrick Macdonald wrote:
  Bruce Momjian wrote:
   Someone at Red Hat is working on point-in-time recovery, also known as
   incremental backups.
 
  PITR and incremental backup are different beasts.  PITR deals with a
  backup + logs.  Incremental backup deals with a full backup + X
  smaller/incremental backups.
 
  So... it doesn't look like anyone is working on incremental backup at the
  moment.

 But why would someone want incremental backups compared to PITR?  The
 backup would be mixture of INSERTS, UPDATES, and DELETES, right?  Seems
 pretty weird.  :-)

Good backup systems, such as Informix (it's the one I used) doesn't do a query 
backup, but a pages backup. What I mean is that it looks for pages in the 
system that has changed from the las full backup and backs them up.

That's how an incremental backup works. PITR is another thing, which is even 
more important. :-)

-- 
Porqué usar una base de datos relacional cualquiera,
si podés usar PostgreSQL?
-
Martín Marqués  |[EMAIL PROTECTED]
Programador, Administrador, DBA |   Centro de Telematica
   Universidad Nacional
del Litoral
-


---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html

Re: [HACKERS] Incremental backup

[Martin Marques]

On Vie 14 Feb 2003 09:52, Bruce Momjian wrote:
 OK, once we have PITR, will anyone want incremental backups?

I will probably not need it, but I know of people how have databases which 
build dumps of more then 20GB.
They are interested in live incremental backups.

-- 
Porqué usar una base de datos relacional cualquiera,
si podés usar PostgreSQL?
-
Martín Marqués  |[EMAIL PROTECTED]
Programador, Administrador, DBA |   Centro de Telematica
   Universidad Nacional
del Litoral
-


---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly

[HACKERS] Incremental backup

[Martin Marques]

How's this issue going on the 7.4 development tree?
I saw it on the TODO list, but didn't find much on the archives of this 
mailing list.

-- 
Porqué usar una base de datos relacional cualquiera,
si podés usar PostgreSQL?
-
Martín Marqués  |[EMAIL PROTECTED]
Programador, Administrador, DBA |   Centro de Telematica
   Universidad Nacional
del Litoral
-


---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster