[HACKERS] pg_basebackup failed to back up large file

[Fujii Masao]

Hi,

I got the bug report of pg_basebackup off-list that it causes an error
when there is large file (e.g., 4GB) in the database cluster. It's easy
to reproduce this problem.

$ dd if=/dev/zero of=$PGDATA/test bs=1G count=4
$ pg_basebackup -D hoge -c fast
pg_basebackup: invalid tar block header size: 32768

2014-06-03 22:56:50 JST data LOG:  could not send data to client: Broken pipe
2014-06-03 22:56:50 JST data ERROR:  base backup could not send data,
aborting backup
2014-06-03 22:56:50 JST data FATAL:  connection to client lost

The cause of this problem is that pg_basebackup uses an integer to
store the size of the file to receive from the server and an integer
overflow can happen when the file is very large. I think that
pg_basebackup should be able to handle even such large file properly
because it can exist in the database cluster, for example,
the server log file under $PGDATA/pg_log can be such large one.
Attached patch changes pg_basebackup so that it uses uint64 to store
the file size and doesn't cause an integer overflow.

Thought?

Regards,

-- 
Fujii Masao

From d42fb3ebdd144e7302196ba02a8aab0c51094f24 Mon Sep 17 00:00:00 2001
From: MasaoFujii <masao.fu...@gmail.com>
Date: Tue, 3 Jun 2014 22:29:36 +0900
Subject: [PATCH] Fix pg_basebackup so that it can back up even large file.

So far, pg_basebackup used an integer variable to store the size of
the file to receive from the server. If the file size was too large
to fall within the range of an integer, an integer overflow would
happen and then pg_basebackup failed to back up that large file.

pg_basebackup should be able to handle even such large file properly
because it can exist in the database cluster, for example,
the server log file under $PGDATA/pg_log can be such large one.
This commit changes pg_basebackup so that it uses uint64 to store
the file size and doesn't cause an integer overflow.

Back-patch to 9.1.
---
 src/bin/pg_basebackup/pg_basebackup.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_basebackup/pg_basebackup.c b/src/bin/pg_basebackup/pg_basebackup.c
index b119fc0..959f0c0 100644
--- a/src/bin/pg_basebackup/pg_basebackup.c
+++ b/src/bin/pg_basebackup/pg_basebackup.c
@@ -1150,7 +1150,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
 {
 	char		current_path[MAXPGPATH];
 	char		filename[MAXPGPATH];
-	int			current_len_left;
+	uint64		current_len_left;
 	int			current_padding = 0;
 	bool		basetablespace = PQgetisnull(res, rownum, 0);
 	char	   *copybuf = NULL;
@@ -1216,7 +1216,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
 			}
 			totaldone += 512;
 
-			if (sscanf(copybuf + 124, "%11o", &current_len_left) != 1)
+			if (sscanf(copybuf + 124, "%11lo", &current_len_left) != 1)
 			{
 				fprintf(stderr, _("%s: could not parse file size\n"),
 						progname);
-- 
1.7.1


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers