Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
Hi Marco, On 16/01/15 16:55, Marco Nenciarini wrote: > On 14/01/15 17:22, Gabriele Bartolini wrote: > > > > My opinion, Marco, is that for version 5 of this patch, you: > > > > 1) update the information on the wiki (it is outdated - I know you have > > been busy with LSN map optimisation) > > Done. > > > 2) modify pg_basebackup in order to accept a directory (or tar file) and > > automatically detect the LSN from the backup profile > > New version of patch attached. The -I parameter now requires a backup > profile from a previous backup. I've added a sanity check that forbid > incremental file level backups if the base timeline is different from > the current one. > > > 3) add the documentation regarding the backup profile and pg_basebackup > > > > Next on my TODO list. > > > Once we have all of this, we can continue trying the patch. Some > > unexplored paths are: > > > > * tablespace usage > > I've improved my pg_restorebackup python PoC. It now supports tablespaces. About tablespaces, I noticed that any pointing to tablespace locations is lost during the recovery of an incremental backup changing the tablespace mapping (-T option). Here the steps I followed: - creating and filling a test database obtained through pgbench psql -c "CREATE DATABASE pgbench" pgbench -U postgres -i -s 5 -F 80 pgbench - a first base backup with pg_basebackup: mkdir -p backups/$(date '+%d%m%y%H%M')/data && pg_basebackup -v -F p -D backups/$(date '+%d%m%y%H%M')/data -x - creation of a new tablespace, alter the table "pgbench_accounts" to set the new tablespace: mkdir -p /home/gbroccolo/pgsql/tbls psql -c "CREATE TABLESPACE tbls LOCATION '/home/gbroccolo/pgsql/tbls'" psql -c "ALTER TABLE pgbench_accounts SET TABLESPACE tbls" pgbench - Doing some work on the database: pgbench -U postgres -T 120 pgbench - a second incremental backup with pg_basebackup specifying the new location for the tablespace through the tablespace mapping: mkdir -p backups/$(date '+%d%m%y%H%M')/data backups/$(date '+%d%m%y%H%M')/tbls && pg_basebackup -v -F p -D backups/$(date '+%d%m%y%H%M')/data -x -I backups/2601151641/data/backup_profile -T /home/gbroccolo/pgsql/tbls=/home/gbroccolo/pgsql/backups/$(date '+%d%m%y%H%M')/tbls - a recovery based on the tool pg_restorebackup.py attached in http://www.postgresql.org/message-id/54b9428e.9020...@2ndquadrant.it ./pg_restorebackup.py backups/2601151641/data backups/2601151707/data /tmp/data -T /home/gbroccolo/pgsql/backups/2601151707/tbls=/tmp/tbls In the last step, I obtained the following stack trace: Traceback (most recent call last): File "./pg_restorebackup.py", line 74, in shutil.copy2(base_file, dest_file) File "/home/gbroccolo/.pyenv/versions/2.7.5/lib/python2.7/shutil.py", line 130, in copy2 copyfile(src, dst) File "/home/gbroccolo/.pyenv/versions/2.7.5/lib/python2.7/shutil.py", line 82, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: 'backups/2601151641/data/base/16384/16406_fsm' Any idea on what's going wrong? Thanks, Giuseppe. -- Giuseppe Broccolo - 2ndQuadrant Italy PostgreSQL Training, Services and Support giuseppe.brocc...@2ndquadrant.it | www.2ndQuadrant.it
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
On 14/01/15 17:22, Gabriele Bartolini wrote: > > My opinion, Marco, is that for version 5 of this patch, you: > > 1) update the information on the wiki (it is outdated - I know you have > been busy with LSN map optimisation) Done. > 2) modify pg_basebackup in order to accept a directory (or tar file) and > automatically detect the LSN from the backup profile New version of patch attached. The -I parameter now requires a backup profile from a previous backup. I've added a sanity check that forbid incremental file level backups if the base timeline is different from the current one. > 3) add the documentation regarding the backup profile and pg_basebackup > Next on my TODO list. > Once we have all of this, we can continue trying the patch. Some > unexplored paths are: > > * tablespace usage I've improved my pg_restorebackup python PoC. It now supports tablespaces. > * tar format > * performance impact (in both "read-only" and heavily updated contexts) From the server point of view, the current code generates a load similar to normal backup. It only adds an initial scan of any data file to decide whether it has to send it. One it found a single newer page it immediately stop scanning and start sending the file. The IO impact should not be that big due to the filesystem cache, but I agree with you that it has to be measured. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it From f7cf8b9dd7d32f64a30dafaeeaeb56cbcd2eafff Mon Sep 17 00:00:00 2001 From: Marco Nenciarini Date: Tue, 14 Oct 2014 14:31:28 +0100 Subject: [PATCH] File-based incremental backup v5 Add backup profile to pg_basebackup INCREMENTAL option implementaion --- src/backend/access/transam/xlog.c | 7 +- src/backend/access/transam/xlogfuncs.c | 2 +- src/backend/replication/basebackup.c | 335 +++-- src/backend/replication/repl_gram.y| 6 + src/backend/replication/repl_scanner.l | 1 + src/bin/pg_basebackup/pg_basebackup.c | 147 +-- src/include/access/xlog.h | 3 +- src/include/replication/basebackup.h | 4 + 8 files changed, 473 insertions(+), 32 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 629a457..1e50625 100644 *** a/src/backend/access/transam/xlog.c --- b/src/backend/access/transam/xlog.c *** XLogFileNameP(TimeLineID tli, XLogSegNo *** 9249,9255 * permissions of the calling user! */ XLogRecPtr ! do_pg_start_backup(const char *backupidstr, bool fast, TimeLineID *starttli_p, char **labelfile) { boolexclusive = (labelfile == NULL); --- 9249,9256 * permissions of the calling user! */ XLogRecPtr ! do_pg_start_backup(const char *backupidstr, bool fast, ! XLogRecPtr incremental_startpoint, TimeLineID *starttli_p, char **labelfile) { boolexclusive = (labelfile == NULL); *** do_pg_start_backup(const char *backupids *** 9468,9473 --- 9469,9478 (uint32) (startpoint >> 32), (uint32) startpoint, xlogfilename); appendStringInfo(&labelfbuf, "CHECKPOINT LOCATION: %X/%X\n", (uint32) (checkpointloc >> 32), (uint32) checkpointloc); + if (incremental_startpoint > 0) + appendStringInfo(&labelfbuf, "INCREMENTAL FROM LOCATION: %X/%X\n", +(uint32) (incremental_startpoint >> 32), +(uint32) incremental_startpoint); appendStringInfo(&labelfbuf, "BACKUP METHOD: %s\n", exclusive ? "pg_start_backup" : "streamed"); appendStringInfo(&labelfbuf, "BACKUP FROM: %s\n", diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 2179bf7..ace84d8 100644 *** a/src/backend/access/transam/xlogfuncs.c --- b/src/backend/access/transam/xlogfuncs.c *** pg_start_backup(PG_FUNCTION_ARGS) *** 59,65 (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), errmsg("must be superuser or replication role to run a backup"))); ! startpoint = do_pg_start_backup(backupidstr, fast, NULL, NULL); PG_RETURN_LSN(startpoint); } --- 59,65 (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), errmsg("must be superuser or replication role to run a backup"))); ! startpoint = do_pg_start_backup(backupidstr, fast, 0, NULL, NULL); PG_RETURN_LSN(startpoint); } diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 07030a2..05
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
Hi Marco, thank you for sending an updated patch. I am writing down a report of this initial (and partial) review. IMPORTANT: This patch is not complete, as stated by Marco. See the "Conclusions" section for my proposed TODO list. == Patch application I have been able to successfully apply your patch and compile it. Regression tests passed. == Initial run I have created a fresh new instance of PostgreSQL and activated streaming replication to be used by pg_basebackup. I have done a pgbench run with scale 100. I have taken a full consistent backup with pg_basebackup (in plain format): pg_basebackup -v -F p -D $BACKUPDIR/backup-$(date '+%s') -x I have been able to verify that the backup_profile is correctly placed in the destination PGDATA directory. Here is an excerpt: POSTGRESQL BACKUP PROFILE 1 START WAL LOCATION: 0/358 (file 00010003) CHECKPOINT LOCATION: 0/38C BACKUP METHOD: streamed BACKUP FROM: master START TIME: 2015-01-14 10:07:07 CET LABEL: pg_basebackup base backup FILE LIST \N \N t 1421226427 206 backup_label \N \N t 1421225508 88 postgresql.auto.conf ... As suggested by Marco, I have manually taken the LSN from this file (next version must do this automatically). I have then executed pg_basebackup and activated the incremental feature by using the LSN from the previous backup, as follows: LSN=$(awk '/^START WAL/{print $4}' backup_profile) pg_basebackup -v -F p -D $BACKUPDIR/backup-$(date '+%s') -I $LSN -x The time taken by this operation has been much lower than the previous one and the size is much lower (I have not done any operation in the meantime): du -hs backup-1421226* 1,5Gbackup-1421226427 17M backup-1421226427 I have done some checks on the file system and then used the prototype of recovery script in Python written by Marco. ./recover.py backup-1421226427 backup-1421226427 new-data The cluster started successfully. I have then run a pg_dump of the pgbench database and were able to reload it on the initial cluster. == Conclusions The first run of this patch seems promising. While the discussion on the LSN map continues (which is mainly an optimisation of this patch), I would really like to see this patch progress as it would be a killer feature in several contexts (not in every context). Just in this period we are releasing file based incremental backup for Barman and customers using the alpha version are experiencing on average a deduplication ratio between 50% to 70%. This is for example an excerpt of "barman show-backup" from one of our customers (a daily saving of 550GB is not bad): Base backup information: Disk usage : 1.1 TiB (1.1 TiB with WALs) Incremental size : 564.6 GiB (-50.60%) ... My opinion, Marco, is that for version 5 of this patch, you: 1) update the information on the wiki (it is outdated - I know you have been busy with LSN map optimisation) 2) modify pg_basebackup in order to accept a directory (or tar file) and automatically detect the LSN from the backup profile 3) add the documentation regarding the backup profile and pg_basebackup Once we have all of this, we can continue trying the patch. Some unexplored paths are: * tablespace usage * tar format * performance impact (in both "read-only" and heavily updated contexts) * consistency checks I would then leave for version 6 the pg_restorebackup utility (unless you want to do everything at once). One limitation of the current recovery script is that it cannot accept multiple incremental backups (it just accepts three parameters: base backup, incremental backup and merge destination). Maybe you can change the syntax as follows: ./recover.py DESTINATION BACKUP_1 BACKUP_2 [BACKUP_3, ...] Thanks a lot for working on this. I am looking forward to continuing the review. Ciao, Gabriele -- Gabriele Bartolini - 2ndQuadrant Italia - Managing Director PostgreSQL Training, Services and Support gabriele.bartol...@2ndquadrant.it | www.2ndQuadrant.it 2015-01-13 17:21 GMT+01:00 Marco Nenciarini : > Il 13/01/15 12:53, Gabriele Bartolini ha scritto: > > Hi Marco, > > > > could you please send an updated version the patch against the current > > HEAD in order to facilitate reviewers? > > > > Here is the updated patch for incremental file based backup. > > It is based on the current HEAD. > > I'm now working to the client tool to rebuild a full backup starting > from a file based incremental backup. > > Regards, > Marco > > -- > Marco Nenciarini - 2ndQuadrant Italy > PostgreSQL Training, Services and Support > marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it >
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
Il 13/01/15 12:53, Gabriele Bartolini ha scritto: > Hi Marco, > > could you please send an updated version the patch against the current > HEAD in order to facilitate reviewers? > Here is the updated patch for incremental file based backup. It is based on the current HEAD. I'm now working to the client tool to rebuild a full backup starting from a file based incremental backup. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it From 50ff0872d3901a30b6742900170052eabe0e06dd Mon Sep 17 00:00:00 2001 From: Marco Nenciarini Date: Tue, 14 Oct 2014 14:31:28 +0100 Subject: [PATCH] File-based incremental backup v4 Add backup profile to pg_basebackup INCREMENTAL option implementaion --- src/backend/access/transam/xlog.c | 7 +- src/backend/access/transam/xlogfuncs.c | 2 +- src/backend/replication/basebackup.c | 335 +++-- src/backend/replication/repl_gram.y| 6 + src/backend/replication/repl_scanner.l | 1 + src/bin/pg_basebackup/pg_basebackup.c | 83 ++-- src/include/access/xlog.h | 3 +- src/include/replication/basebackup.h | 4 + 8 files changed, 409 insertions(+), 32 deletions(-) diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 839ea7c..625a5df 100644 *** a/src/backend/access/transam/xlog.c --- b/src/backend/access/transam/xlog.c *** XLogFileNameP(TimeLineID tli, XLogSegNo *** 9249,9255 * permissions of the calling user! */ XLogRecPtr ! do_pg_start_backup(const char *backupidstr, bool fast, TimeLineID *starttli_p, char **labelfile) { boolexclusive = (labelfile == NULL); --- 9249,9256 * permissions of the calling user! */ XLogRecPtr ! do_pg_start_backup(const char *backupidstr, bool fast, ! XLogRecPtr incremental_startpoint, TimeLineID *starttli_p, char **labelfile) { boolexclusive = (labelfile == NULL); *** do_pg_start_backup(const char *backupids *** 9468,9473 --- 9469,9478 (uint32) (startpoint >> 32), (uint32) startpoint, xlogfilename); appendStringInfo(&labelfbuf, "CHECKPOINT LOCATION: %X/%X\n", (uint32) (checkpointloc >> 32), (uint32) checkpointloc); + if (incremental_startpoint > 0) + appendStringInfo(&labelfbuf, "INCREMENTAL FROM LOCATION: %X/%X\n", +(uint32) (incremental_startpoint >> 32), +(uint32) incremental_startpoint); appendStringInfo(&labelfbuf, "BACKUP METHOD: %s\n", exclusive ? "pg_start_backup" : "streamed"); appendStringInfo(&labelfbuf, "BACKUP FROM: %s\n", diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index 2179bf7..ace84d8 100644 *** a/src/backend/access/transam/xlogfuncs.c --- b/src/backend/access/transam/xlogfuncs.c *** pg_start_backup(PG_FUNCTION_ARGS) *** 59,65 (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), errmsg("must be superuser or replication role to run a backup"))); ! startpoint = do_pg_start_backup(backupidstr, fast, NULL, NULL); PG_RETURN_LSN(startpoint); } --- 59,65 (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), errmsg("must be superuser or replication role to run a backup"))); ! startpoint = do_pg_start_backup(backupidstr, fast, 0, NULL, NULL); PG_RETURN_LSN(startpoint); } diff --git a/src/backend/replication/basebackup.c b/src/backend/replication/basebackup.c index 07030a2..05b19c5 100644 *** a/src/backend/replication/basebackup.c --- b/src/backend/replication/basebackup.c *** *** 30,40 --- 30,42 #include "replication/basebackup.h" #include "replication/walsender.h" #include "replication/walsender_private.h" + #include "storage/bufpage.h" #include "storage/fd.h" #include "storage/ipc.h" #include "utils/builtins.h" #include "utils/elog.h" #include "utils/ps_status.h" + #include "utils/pg_lsn.h" #include "utils/timestamp.h" *** typedef struct *** 46,56 boolnowait; boolincludewal; uint32 maxrate; } basebackup_options; ! static int64 sendDir(char *path, int basepathlen, bool sizeonly, List *tablespaces); ! static int64 sendTablespace(char *path, bool sizeonly); static bool sendFile(char *readfilename, char *tarfilename, struct stat * statbuf, bool missing_ok); static void sendFileWithConte
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
Hi Marco, could you please send an updated version the patch against the current HEAD in order to facilitate reviewers? Thanks, Gabriele -- Gabriele Bartolini - 2ndQuadrant Italia - Managing Director PostgreSQL Training, Services and Support gabriele.bartol...@2ndquadrant.it | www.2ndQuadrant.it 2015-01-07 11:00 GMT+01:00 Marco Nenciarini : > Il 06/01/15 14:26, Robert Haas ha scritto: > > I suggest leaving this out altogether for the first version. I can > > think of three possible ways that we can determine which blocks need > > to be backed up. One, just read every block in the database and look > > at the LSN of each one. Two, maintain a cache of LSN information on a > > per-segment (or smaller) basis, as you suggest here. Three, scan the > > WAL generated since the incremental backup and summarize it into a > > list of blocks that need to be backed up. This last idea could either > > be done when the backup is requested, or it could be done as the WAL > > is generated and used to populate the LSN cache. In the long run, I > > think some variant of approach #3 is likely best, but in the short > > run, approach #1 (scan everything) is certainly easiest. While it > > doesn't optimize I/O, it still gives you the benefit of reducing the > > amount of data that needs to be transferred and stored, and that's not > > nothing. If we get that much working, we can improve things more > > later. > > > > Hi, > The patch now uses the approach #1, but I've just sent a patch that uses > the #2 approach. > > 54ad016e.9020...@2ndquadrant.it > > Regards, > Marco > > -- > Marco Nenciarini - 2ndQuadrant Italy > PostgreSQL Training, Services and Support > marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it > >
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
Il 06/01/15 14:26, Robert Haas ha scritto: > I suggest leaving this out altogether for the first version. I can > think of three possible ways that we can determine which blocks need > to be backed up. One, just read every block in the database and look > at the LSN of each one. Two, maintain a cache of LSN information on a > per-segment (or smaller) basis, as you suggest here. Three, scan the > WAL generated since the incremental backup and summarize it into a > list of blocks that need to be backed up. This last idea could either > be done when the backup is requested, or it could be done as the WAL > is generated and used to populate the LSN cache. In the long run, I > think some variant of approach #3 is likely best, but in the short > run, approach #1 (scan everything) is certainly easiest. While it > doesn't optimize I/O, it still gives you the benefit of reducing the > amount of data that needs to be transferred and stored, and that's not > nothing. If we get that much working, we can improve things more > later. > Hi, The patch now uses the approach #1, but I've just sent a patch that uses the #2 approach. 54ad016e.9020...@2ndquadrant.it Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it signature.asc Description: OpenPGP digital signature
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
On Tue, 6 Jan 2015 08:26:22 -0500 Robert Haas wrote: > Three, scan the WAL generated since the incremental backup and summarize it > into a list of blocks that need to be backed up. This can be done from the archive side. I was talking about some months ago now: http://www.postgresql.org/message-id/51c4dd20.3000...@free.fr One of the traps I could think of it that it requires "full_page_write=on" so we can forge each block correctly. So collar is that we need to start a diff backup right after a checkpoints then. And even without "full_page_write=on", maybe we could add a function, say "pg_start_backupdiff()", which would force to log full pages right after it only, the same way "full_page_write" does after a checkpoint. Diff backups would be possible from each LSN where we pg_start_backupdiff'ed till whenever. Building this backup by merging versions of blocks from WAL is on big step. But then, there is a file format to define, how to restore it and to decide what tools/functions/GUCs to expose to admins. After discussing with Magnus, he adviced me to wait for a diff backup file format to emerge from online tools, like discussed here (by the time, that was Michael's proposal based on pg_basebackup that was discussed). But I wonder how easier it would be to do this the opposite way? If this idea of building diff backup offline from archives is possible, wouldn't it remove a lot of trouble you are discussing here? Regards, -- Jehan-Guillaume de Rorthais Dalibo http://www.dalibo.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
On Tue, Oct 14, 2014 at 1:17 PM, Marco Nenciarini wrote: > I would to replace the getMaxLSN function with a more-or-less persistent > structure which contains the maxLSN for each data segment. > > To make it work I would hook into the ForwardFsyncRequest() function in > src/backend/postmaster/checkpointer.c and update an in memory hash every > time a block is going to be fsynced. The structure could be persisted on > disk at some time (probably on checkpoint). > > I think a good key for the hash would be a BufferTag with blocknum > "rounded" to the start of the segment. > > I'm here asking for comments and advices on how to implement it in an > acceptable way. I'm afraid this is going to be quite tricky to implement. There's no way to make the in-memory hash table large enough that it can definitely contain all of the entries for the entire database. Even if it's big enough at a certain point in time, somebody can create 100,000 new tables and now it's not big enough any more. This is not unlike the problem we had with the visibility map and free space map before 8.4 (and you probably remember how much fun that was). I suggest leaving this out altogether for the first version. I can think of three possible ways that we can determine which blocks need to be backed up. One, just read every block in the database and look at the LSN of each one. Two, maintain a cache of LSN information on a per-segment (or smaller) basis, as you suggest here. Three, scan the WAL generated since the incremental backup and summarize it into a list of blocks that need to be backed up. This last idea could either be done when the backup is requested, or it could be done as the WAL is generated and used to populate the LSN cache. In the long run, I think some variant of approach #3 is likely best, but in the short run, approach #1 (scan everything) is certainly easiest. While it doesn't optimize I/O, it still gives you the benefit of reducing the amount of data that needs to be transferred and stored, and that's not nothing. If we get that much working, we can improve things more later. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
On Mon, Jan 5, 2015 at 7:56 PM, Marco Nenciarini wrote: > I've noticed that I missed to add this to the commitfest. > > I've just added it. > > It is not meant to end up in a committable state, but at this point I'm > searching for some code review and more discusison. > > I'm also about to send an additional patch to implement an LSN map as an > additional fork for heap files. Moved to CF 2015-02. -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
I've noticed that I missed to add this to the commitfest. I've just added it. It is not meant to end up in a committable state, but at this point I'm searching for some code review and more discusison. I'm also about to send an additional patch to implement an LSN map as an additional fork for heap files. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it signature.asc Description: OpenPGP digital signature
Re: [HACKERS] [RFC] Incremental backup v3: incremental PoC
I've noticed that I missed to add this to the commitfest. I've just added it. It is not meant to end up in a committable state, but at this point I'm searching for some code review and more discusison. I'm also about to send an additional patch to implement an LSN map as an additional fork for heap files. Regards, Marco -- Marco Nenciarini - 2ndQuadrant Italy PostgreSQL Training, Services and Support marco.nenciar...@2ndquadrant.it | www.2ndQuadrant.it signature.asc Description: OpenPGP digital signature