Re: [HACKERS] pg_dump --exclude-table-data
On Dec 14, 2011, at 6:31 AM, Andrew Dunstan wrote: > Thanks. Committed with that changed, although we seem to be getting > altogether too obsessive about white space, IMNSHO. If that’s all there is to complain about, I think it’s a pretty good sign. ;-P David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On 14 December 2011 14:31, Andrew Dunstan wrote: > Thanks. Committed with that changed, although we seem to be getting > altogether too obsessive about white space, IMNSHO. I agree, but I think it's important that we judge patches by a consistent standard. Right now, for better or worse, that standard includes being obsessed with white space. -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On 12/14/2011 06:28 AM, Peter Geoghegan wrote: Not sure that I have a lot to add here, but I am officially listed as a reviewer, which is a responsibility that I don't want to shirk. In my opinion, this patch is obviously useful. I don't find the asymmetry that it will create with pg_restore to be troubling, so I'd favour committing it as-is. Extremely minor problem noted: There are two spaces at the start of one sentence in your SGML doc updates. Thanks. Committed with that changed, although we seem to be getting altogether too obsessive about white space, IMNSHO. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
Not sure that I have a lot to add here, but I am officially listed as a reviewer, which is a responsibility that I don't want to shirk. In my opinion, this patch is obviously useful. I don't find the asymmetry that it will create with pg_restore to be troubling, so I'd favour committing it as-is. Extremely minor problem noted: There are two spaces at the start of one sentence in your SGML doc updates. -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On 12/08/2011 11:36 AM, Andrew Dunstan wrote:
On 12/08/2011 11:13 AM, Robert Haas wrote:
Ah, hmm. Well, maybe it's fine the way that you have it. But I'd be
tempted to at least add a sentence to the sgml documentation for each
option referring to the other, e.g. "To dump only schema for all
tables in the database, see --schema-only." and "To exclude table data
for only a subset of tables in the database, see
--exclude-table-data.", or something along those lines.
Sure, no problem with that.
Revised patch attached.
cheers
andrew
*** a/doc/src/sgml/ref/pg_dump.sgml
--- b/doc/src/sgml/ref/pg_dump.sgml
***
*** 404,409 PostgreSQL documentation
--- 404,413
Dump only the object definitions (schema), not data.
+
+ To exclude table data for only a subset of tables in the database,
+ see --exclude-table-data.
+
***
*** 612,617 PostgreSQL documentation
--- 616,639
+ --exclude-table-data=table
+
+
+ Do not dump data for any tables matching the table pattern. The pattern is
+ interpreted according to the same rules as for -t.
+ --exclude-table-data can be given more than once to
+ exclude tables matching any of several patterns. This option is
+ useful when you need the definition of a particular table even
+ though you do not need the data in it.
+
+
+ To exclude data for all tables in the database, see --schema-only.
+
+
+
+
+
--inserts
*** a/src/bin/pg_dump/pg_dump.c
--- b/src/bin/pg_dump/pg_dump.c
***
*** 115,120 static SimpleStringList table_include_patterns = {NULL, NULL};
--- 115,122
static SimpleOidList table_include_oids = {NULL, NULL};
static SimpleStringList table_exclude_patterns = {NULL, NULL};
static SimpleOidList table_exclude_oids = {NULL, NULL};
+ static SimpleStringList tabledata_exclude_patterns = {NULL, NULL};
+ static SimpleOidList tabledata_exclude_oids = {NULL, NULL};
/* default, if no "inclusion" switches appear, is to dump everything */
static bool include_everything = true;
***
*** 324,329 main(int argc, char **argv)
--- 326,332
{"column-inserts", no_argument, &column_inserts, 1},
{"disable-dollar-quoting", no_argument, &disable_dollar_quoting, 1},
{"disable-triggers", no_argument, &disable_triggers, 1},
+ {"exclude-table-data", required_argument, NULL, 4},
{"inserts", no_argument, &dump_inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
***
*** 487,492 main(int argc, char **argv)
--- 490,499
use_role = optarg;
break;
+ case 4: /* exclude table(s) data */
+ simple_string_list_append(&tabledata_exclude_patterns, optarg);
+ break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
***
*** 715,720 main(int argc, char **argv)
--- 722,731
}
expand_table_name_patterns(&table_exclude_patterns,
&table_exclude_oids);
+
+ expand_table_name_patterns(&tabledata_exclude_patterns,
+ &tabledata_exclude_oids);
+
/* non-matching exclusion patterns aren't an error */
/*
***
*** 854,859 help(const char *progname)
--- 865,871
printf(_(" --column-insertsdump data as INSERT commands with column names\n"));
printf(_(" --disable-dollar-quotingdisable dollar quoting, use SQL standard quoting\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
+ printf(_(" --exclude-table-data=TABLE do NOT dump data for the named table(s)\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --no-security-labelsdo not dump security label assignments\n"));
printf(_(" --no-tablespacesdo not dump tablespace assignments\n"));
***
*** 1087,1092 selectDumpableTable(TableInfo *tbinfo)
--- 1099,1113
simple_oid_list_member(&table_exclude_oids,
tbinfo->dobj.catId.oid))
tbinfo->dobj.dump = false;
+
+ /* If table is to be dumped, check that the data is not excluded */
+ if (tbinfo->dobj.dump && !
+ simple_oid_list_member(&tabledata_exclude_oids,
+ tbinfo->dobj.catId.oid))
+ tbinfo->dobj.dumpdata = true;
+ else
+ tbinfo->dobj.dumpdata = false;
+
}
/*
***
*** 1518,1523 dumpTableData(Archive *fout, TableDataInfo *tdinfo)
--- 1539,1548
DataDumperPtr dumpFn;
char *copyStmt;
+ /* don't do anything if the data isn't wanted */
+ if (!tbinfo->dobj.dumpdata)
+ return;
+
if (!dump_inserts)
{
/* Dump/restore using COPY */
*** a/src/b
Re: [HACKERS] pg_dump --exclude-table-data
On 12/08/2011 11:13 AM, Robert Haas wrote: On Wed, Dec 7, 2011 at 10:19 PM, Andrew Dunstan wrote: I'm also a bit concerned about the relationship between this and the existing -s option. It seems odd that you use --schema-only to get the behavior database-wide, and --exclude-table-data to get it for just one table. Is there some way we can make that a bit more consistent? It's consistent, and was designed to be, with the --exclude-table option. I'm not sure what you want it to look like instead. But TBH I'm more interested in getting the functionality than in how it's spelled. Ah, hmm. Well, maybe it's fine the way that you have it. But I'd be tempted to at least add a sentence to the sgml documentation for each option referring to the other, e.g. "To dump only schema for all tables in the database, see --schema-only." and "To exclude table data for only a subset of tables in the database, see --exclude-table-data.", or something along those lines. Sure, no problem with that. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On Wed, Dec 7, 2011 at 10:19 PM, Andrew Dunstan wrote: >> I'm also a bit concerned about the relationship between this and the >> existing -s option. It seems odd that you use --schema-only to get >> the behavior database-wide, and --exclude-table-data to get it for >> just one table. Is there some way we can make that a bit more >> consistent? > > It's consistent, and was designed to be, with the --exclude-table option. > I'm not sure what you want it to look like instead. But TBH I'm more > interested in getting the functionality than in how it's spelled. Ah, hmm. Well, maybe it's fine the way that you have it. But I'd be tempted to at least add a sentence to the sgml documentation for each option referring to the other, e.g. "To dump only schema for all tables in the database, see --schema-only." and "To exclude table data for only a subset of tables in the database, see --exclude-table-data.", or something along those lines. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On 11/04/2011 10:21 AM, Robert Haas wrote: A slightly updated patch is attached, the main change being that I removed use of a short option and only support the long name option. "-D" didn't seem sufficiently mnemonic to me. I'll add this to the November commitfest, but I'd like to get it committed ASAP as it will simplify setting up the -pre and -post data patches. Instead of: do NOT dump data for the named table(s) How about: dump only schema for the named table(s) I have no great objection to the wording change. I'm also a bit concerned about the relationship between this and the existing -s option. It seems odd that you use --schema-only to get the behavior database-wide, and --exclude-table-data to get it for just one table. Is there some way we can make that a bit more consistent? It's consistent, and was designed to be, with the --exclude-table option. I'm not sure what you want it to look like instead. But TBH I'm more interested in getting the functionality than in how it's spelled. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On Wed, Nov 2, 2011 at 6:02 PM, Andrew Dunstan wrote: > On 09/02/2011 03:15 PM, Josh Berkus wrote: >>> >>> OK, this seems to have some pluses and no negative comments, so it seems >>> worth going forward. Do we want an equivalent pg_restore option? >> >> I'm not sure it's *as* important for pg_restore, since I can easily use >> a manifest to avoid restoring data for a single table. So I guess it's >> a question of "how hard is it to add it?" >> > > The short answer is "more work than I want to put in to this." pg_restore > doesn't have any of pg_dump's infrastructure for handling table name > patterns, nor for excluding tables. So I think all that would remain a TODO. > (A good beginner project, maybe). > > A slightly updated patch is attached, the main change being that I removed > use of a short option and only support the long name option. "-D" didn't > seem sufficiently mnemonic to me. I'll add this to the November commitfest, > but I'd like to get it committed ASAP as it will simplify setting up the > -pre and -post data patches. Instead of: do NOT dump data for the named table(s) How about: dump only schema for the named table(s) I'm also a bit concerned about the relationship between this and the existing -s option. It seems odd that you use --schema-only to get the behavior database-wide, and --exclude-table-data to get it for just one table. Is there some way we can make that a bit more consistent? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On 09/02/2011 03:15 PM, Josh Berkus wrote:
OK, this seems to have some pluses and no negative comments, so it seems
worth going forward. Do we want an equivalent pg_restore option?
I'm not sure it's *as* important for pg_restore, since I can easily use
a manifest to avoid restoring data for a single table. So I guess it's
a question of "how hard is it to add it?"
The short answer is "more work than I want to put in to this."
pg_restore doesn't have any of pg_dump's infrastructure for handling
table name patterns, nor for excluding tables. So I think all that would
remain a TODO. (A good beginner project, maybe).
A slightly updated patch is attached, the main change being that I
removed use of a short option and only support the long name option.
"-D" didn't seem sufficiently mnemonic to me. I'll add this to the
November commitfest, but I'd like to get it committed ASAP as it will
simplify setting up the -pre and -post data patches.
cheers
andrew
diff --git a/doc/src/sgml/ref/pg_dump.sgml b/doc/src/sgml/ref/pg_dump.sgml
index f6f33de..a5443b7 100644
--- a/doc/src/sgml/ref/pg_dump.sgml
+++ b/doc/src/sgml/ref/pg_dump.sgml
@@ -612,6 +612,21 @@ PostgreSQL documentation
+ --exclude-table-data=table
+
+
+Do not dump data for any tables matching the table pattern. The pattern is
+interpreted according to the same rules as for -t.
+--exclude-table-data can be given more than once to
+exclude tables matching any of several patterns. This option is
+useful when you need the definition of a particular table even
+though you do not need the data in it.
+
+
+
+
+
--inserts
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c17b52c..6048e2c 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -114,6 +114,8 @@ static SimpleStringList table_include_patterns = {NULL, NULL};
static SimpleOidList table_include_oids = {NULL, NULL};
static SimpleStringList table_exclude_patterns = {NULL, NULL};
static SimpleOidList table_exclude_oids = {NULL, NULL};
+static SimpleStringList tabledata_exclude_patterns = {NULL, NULL};
+static SimpleOidList tabledata_exclude_oids = {NULL, NULL};
/* default, if no "inclusion" switches appear, is to dump everything */
static bool include_everything = true;
@@ -322,6 +324,7 @@ main(int argc, char **argv)
{"column-inserts", no_argument, &column_inserts, 1},
{"disable-dollar-quoting", no_argument, &disable_dollar_quoting, 1},
{"disable-triggers", no_argument, &disable_triggers, 1},
+ {"exclude-table-data", required_argument, NULL, 4},
{"inserts", no_argument, &dump_inserts, 1},
{"lock-wait-timeout", required_argument, NULL, 2},
{"no-tablespaces", no_argument, &outputNoTablespaces, 1},
@@ -485,6 +488,10 @@ main(int argc, char **argv)
use_role = optarg;
break;
+ case 4: /* exclude table(s) data */
+simple_string_list_append(&tabledata_exclude_patterns, optarg);
+break;
+
default:
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
exit(1);
@@ -712,6 +719,10 @@ main(int argc, char **argv)
}
expand_table_name_patterns(&table_exclude_patterns,
&table_exclude_oids);
+
+ expand_table_name_patterns(&tabledata_exclude_patterns,
+ &tabledata_exclude_oids);
+
/* non-matching exclusion patterns aren't an error */
/*
@@ -851,6 +862,7 @@ help(const char *progname)
printf(_(" --column-insertsdump data as INSERT commands with column names\n"));
printf(_(" --disable-dollar-quotingdisable dollar quoting, use SQL standard quoting\n"));
printf(_(" --disable-triggers disable triggers during data-only restore\n"));
+ printf(_(" --exclude-table-data=TABLE do NOT dump data for the named table(s)\n"));
printf(_(" --inserts dump data as INSERT commands, rather than COPY\n"));
printf(_(" --no-security-labelsdo not dump security label assignments\n"));
printf(_(" --no-tablespacesdo not dump tablespace assignments\n"));
@@ -1084,6 +1096,15 @@ selectDumpableTable(TableInfo *tbinfo)
simple_oid_list_member(&table_exclude_oids,
tbinfo->dobj.catId.oid))
tbinfo->dobj.dump = false;
+
+ /* If table is to be dumped, check that the data is not excluded */
+ if (tbinfo->dobj.dump && !
+ simple_oid_list_member(&tabledata_exclude_oids,
+ tbinfo->dobj.catId.oid))
+ tbinfo->dobj.dumpdata = true;
+ else
+ tbinfo->dobj.dumpdata = false;
+
}
/*
@@ -1515,6 +1536,10 @@ dumpTableData(Archive *fout, TableDataInfo *tdinfo)
DataDumperPtr dumpFn;
char *copyStmt;
+ /* don't do anything if the data isn't wanted */
+ if (!tbinfo->dobj.dumpdata)
+ return;
+
if (!dump_inserts)
{
/* Dump/restore using COPY */
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 3d5d534..d6b5dd4 100644
--- a/src/bin/
Re: [HACKERS] pg_dump --exclude-table-data
> OK, this seems to have some pluses and no negative comments, so it seems > worth going forward. Do we want an equivalent pg_restore option? I'm not sure it's *as* important for pg_restore, since I can easily use a manifest to avoid restoring data for a single table. So I guess it's a question of "how hard is it to add it?" -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On 08/24/2011 05:01 PM, Josh Berkus wrote: For those who are (like my clients :-) ) anxious to get their hands on this immediately, a backport patch is also attached which applies to 9.0 sources, and applies with offsets to 8.4 sources. FWIW, I have immediate use for this in creating cut-down versions of databases for testing purposes. It'll eliminate a couple pages of shell scripts for me. OK, this seems to have some pluses and no negative comments, so it seems worth going forward. Do we want an equivalent pg_restore option? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
On Wed, Aug 24, 2011 at 2:01 PM, Josh Berkus wrote: > > FWIW, I have immediate use for this in creating cut-down versions of > databases for testing purposes. It'll eliminate a couple pages of shell > scripts for me. Speaking of "cut-down versions", I have recently been using pg_sample, and been happy with the resulting subset database. I created a db <10 GB in size from a source db ~600Gb in a few minutes. https://github.com/mla/pg_sample -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_dump --exclude-table-data
> For those who are (like my clients :-) ) anxious to get their hands on > this immediately, a backport patch is also attached which applies to 9.0 > sources, and applies with offsets to 8.4 sources. FWIW, I have immediate use for this in creating cut-down versions of databases for testing purposes. It'll eliminate a couple pages of shell scripts for me. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
