Re: [HACKERS] Debian readline/libedit breakage
Hello all, thanks Markus for CC'ing me, I'm not on -hackers@. Markus Wanner [2011-02-14 13:37 +0100]: On 02/10/2011 11:34 PM, Joshua D. Drake wrote: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=607109 Note that the recent discussions happened on bug 608442, in particular http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=608442#30 and the following comments. Personally, I'm a bit suspicious about that solution (technically as well as from a licensing perspective), [...] For the record, so am I (see comment 30 in the link above), as it uses the very same ld.so in both cases. However, Andreas Barth pointed out that with LD_PRELOAD it's guaranteed that we do not import any code from the libreadline header files, which guarantees that psql doesn't become something that can be considered a derived work. Technically, this is a bit fragile, of course, as there might be some subtle ABI differences which lead to crashes. However, the preloading workaround already makes the situation so much better than before, so IMHO it's better than the previous status quo. I don't really like this situation, and personally I'd rather move back to libreadline until OpenSSL or readline or PostgreSQL threatens Debian with a legal case for license violation (I daresay that the chances of this happening are very close to zero..). But oh well.. Thanks, and sorry for all the madness this created! Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) signature.asc Description: Digital signature
Re: [HACKERS] Versioned mo file installation
Hi Alvaro, Alvaro Herrera [2008-12-05 14:06 -0300]: Of course, as is it's not acceptable; the version number should be obtained from PG_VERSION or some such. Right, it has actually been on my long-term wishlist to replace that patch with an autoconfiscated one with a non-hardcoded version number. I just didn't do it yet because it was generally rejected when I proposed it upstream some years ago. I am not sure the libpq5 versioning is best; maybe we don't change the API but we could certainly change error messages between versions. I think we should keep it as libpq-8.3, like the rest. Doesn't make much of a difference indeed, I just made it match the package name. Also, libpq5 built from -8.3 is used with postgresql-8.2 as well. Martin -- Martin Pitt| http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_dump: add option to ignore TABLE DATA for failed TABLE object creation
Hi PostgreSQL developers,
some time ago I started a discussion [1] here about modifying pg_dump
to not restore TABLE DATA objects if the corresponding TABLE oject
failed to be created (usually because it already exists, but it might
fail due to a different error like a nonexisting data type). We need
this to provide automatic major version upgrades for databases with
extensions like PostGIS. Tom's reply [3] seemed to indicate that this
was not entirely crackful, so I implemented his approach, and after
some feedback I now have a fairly clean patch that works very well.
The patch was scheduled for review and inclusion [4], and indeed the
page had the patch for a while, but after some time it vanished.
Can you please reconsider this? If there is still a problem with the
patch, I'd like to work on it until it meets your standards.
For your convenience I attach the current patch version; a test script
[5] is also available (the ML kills shell script attachments, so I put
it on a Debian server). It does not alter the default behaviour, it
just adds a new option -X no-data-for-failed-tables. If you think this
mode should be the default, I'm happy to change it that way.
Thank you a lot!
Martin
[1] http://archives.postgresql.org/pgsql-hackers/2006-02/msg00694.php
[2] http://bugs.debian.org/351571
[3] http://archives.postgresql.org/pgsql-hackers/2006-02/msg00716.php
[4] http://archives.postgresql.org/pgsql-hackers/2006-02/msg01253.php
[5] http://people.debian.org/~mpitt/test-pg_restore-existing.sh
--
Martin Pitthttp://www.piware.de
Ubuntu Developer http://www.ubuntu.com
Debian Developer http://www.debian.org
In a world without walls and fences, who needs Windows and Gates?
diff -ruN postgresql-8.1.3-old/doc/src/sgml/ref/pg_restore.sgml
postgresql-8.1.3/doc/src/sgml/ref/pg_restore.sgml
--- postgresql-8.1.3-old/doc/src/sgml/ref/pg_restore.sgml 2005-11-01
22:09:50.0 +0100
+++ postgresql-8.1.3/doc/src/sgml/ref/pg_restore.sgml 2006-03-03
19:13:50.0 +0100
@@ -395,6 +395,20 @@
/listitem
/varlistentry
+ varlistentry
+ termoption-X no-data-for-failed-tables//term
+ listitem
+ para
+ By default, table data objects are restored even if the
+ associated table could not be successfully created (e. g.
+ because it already exists). With this option, such table
+ data is silently ignored. This is useful for dumping and
+ restoring databases with tables which contain auxiliary data
+ for PostgreSQL extensions (e. g. PostGIS).
+ /para
+ /listitem
+ /varlistentry
+
/variablelist
/para
diff -ruN postgresql-8.1.3-old/src/bin/pg_dump/pg_backup_archiver.c
postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c
--- postgresql-8.1.3-old/src/bin/pg_dump/pg_backup_archiver.c 2006-02-05
21:58:57.0 +0100
+++ postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c 2006-03-03
19:14:03.0 +0100
@@ -268,6 +268,23 @@
_printTocEntry(AH, te, ropt, false, false);
defnDumped = true;
+ /* If we could not create a table, ignore the
respective TABLE DATA if
+* -X no-data-for-failed-tables is given */
+ if (ropt-noDataForFailedTables AH-lastErrorTE ==
te strcmp (te-desc, TABLE) == 0) {
+ TocEntry *tes, *last;
+
+ ahlog (AH, 1, table %s could not be created,
will not restore its data\n, te-tag);
+
+ for (last = te, tes = te-next; tes != AH-toc;
last = tes, tes = tes-next) {
+ if (strcmp (tes-desc, TABLE DATA) ==
0 strcmp (tes-tag, te-tag) == 0
+ strcmp (tes-namespace ?
tes-namespace : , te-namespace ? te-namespace : ) == 0) {
+ /* remove this node */
+ last-next = tes-next;
+break;
+ }
+ }
+ }
+
/* If we created a DB, connect to it... */
if (strcmp(te-desc, DATABASE) == 0)
{
diff -ruN postgresql-8.1.3-old/src/bin/pg_dump/pg_backup.h
postgresql-8.1.3/src/bin/pg_dump/pg_backup.h
--- postgresql-8.1.3-old/src/bin/pg_dump/pg_backup.h2005-10-15
04:49:38.0 +0200
+++ postgresql-8.1.3/src/bin/pg_dump/pg_backup.h2006-03-03
19:13:50.0 +0100
@@ -106,6 +106,7 @@
char *pghost;
char *username;
int ignoreVersion;
+ int noDataForFailedTables;
int requirePassword;
int exit_on_error;
diff -ruN postgresql-8.1.3-old/src/bin/pg_dump
Re: [HACKERS] Please help, pgAdmin3 on Debian!
Hi lmyho, lmyho [2006-03-28 0:17 -0800]: I am totally new to the PostgreSQL, and pgAdmin. I really need your help. I just installed the PostgreSQL8.1 and pgAdmin3 on a Debian system, using the apt-get install command. Apparently, the initial database and the user postgres have also been automatically created during the installation. Please feel free to mail me directly ([EMAIL PROTECTED]) for questions related to the Debian packages. It might be regarded as noise on the upstream lists. Which is good. But I've got big trouble to login to this initial db by using this auto-created username postgres through pgAdmin:((( The first try failed due to Ident authentication failed The 'postgres' user in Debian is a system user with a locked password, since it is not recommended to use it for normal work with the database. As /usr/share/postgresql-common/README.Debian describes, you should first create your own database user and work with that. Then the default 'ident' authentication scheme will work, and you are free to set a password for your db user as well (so that connecting from remote computer over TCP works as well). If you really need to connect as user postgres to do administrative tasks, then the easiest solution is to set a password for the user postgres, as already mentioned in the previous reply. HTH, Martin -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? signature.asc Description: Digital signature
Re: [HACKERS] Adding an ignore list to pg_restore, patch take #3
Hi all,
thanks for the feedback. I updated the patch now.
Alvaro Herrera [2006-02-25 13:47 -0300]:
I improved the patch now to only ignore TABLE DATA for existing tables
if '-X ignore-existing-tables' is specified. I also updated the
documentation.
Is this really an appropiate description for the behavior? What happens
if the table is not created for some other reason? Consider for example
a table using a datatype that couldn't be created.
Right. However, if the table is not present at all, then it makes even
less sense to attempt to restore its data. Therefore I consider this
mainly a documentation issue. I changed the option to
-X no-data-for-failed-tables and described it as
By default, table data objects are restored even if the associated
table could not be successfully created (e. g. because it already
exists). [...]
Tom Lane [2006-02-25 12:18 -0500]:
Martin Pitt [EMAIL PROTECTED] writes:
Martin Pitt [2006-02-19 14:39 +0100]:
Since this changes the behaviour of pg_restore, this should probably
become an option, e. g. -D / --ignore-existing-table-data. I'll do
this if you agree to the principle of the current patch.
I improved the patch now to only ignore TABLE DATA for existing tables
if '-X ignore-existing-tables' is specified. I also updated the
documentation.
This patch is unbelievably ugly and probably vulnerable to coredumps.
Please use a cleaner way of disabling the subsequent load than tromping
all over the TOC datastructure, ie, not this:
+strcpy (tes-desc,
IGNOREDATA);
It should not segfault, but I agree that this is a bit hackish. The
updated patch completely removes the TABLE DATA node from the linked
list. It does not free its memory, though; I did not find a
free_tocentry() or similar function. However, pg_restore is no daemon,
and without the new option the memory would be allocated, too, so it
does not make much difference. Can anyone give me a hint how to
properly free the struct?
BTW, I'm pretty sure it fails for tables with same names in different
schemas, too.
Right, sorry for that. I fixed that, too.
Bruce Momjian [2006-02-28 19:54 -0500]:
I will clean it up before applying.
Thank you. I hope the updated patch makes that a little bit easier.
Your patch has been added to the PostgreSQL unapplied patches list at:
http://momjian.postgresql.org/cgi-bin/pgpatches
It will be applied as soon as one of the PostgreSQL committers reviews
and approves it.
Great, thanks!
Martin
P.S. I also updated the test script to create two namespaces with
identidal table names.
http://people.debian.org/~mpitt/test-pg_restore-existing.sh
--
Martin Pitthttp://www.piware.de
Ubuntu Developer http://www.ubuntu.com
Debian Developer http://www.debian.org
In a world without walls and fences, who needs Windows and Gates?
diff -ruN postgresql-8.1.3-old/doc/src/sgml/ref/pg_restore.sgml
postgresql-8.1.3/doc/src/sgml/ref/pg_restore.sgml
--- postgresql-8.1.3-old/doc/src/sgml/ref/pg_restore.sgml 2005-11-01
22:09:50.0 +0100
+++ postgresql-8.1.3/doc/src/sgml/ref/pg_restore.sgml 2006-03-03
19:13:50.0 +0100
@@ -395,6 +395,20 @@
/listitem
/varlistentry
+ varlistentry
+ termoption-X no-data-for-failed-tables//term
+ listitem
+ para
+ By default, table data objects are restored even if the
+ associated table could not be successfully created (e. g.
+ because it already exists). With this option, such table
+ data is silently ignored. This is useful for dumping and
+ restoring databases with tables which contain auxiliary data
+ for PostgreSQL extensions (e. g. PostGIS).
+ /para
+ /listitem
+ /varlistentry
+
/variablelist
/para
diff -ruN postgresql-8.1.3-old/src/bin/pg_dump/pg_backup_archiver.c
postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c
--- postgresql-8.1.3-old/src/bin/pg_dump/pg_backup_archiver.c 2006-02-05
21:58:57.0 +0100
+++ postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c 2006-03-03
19:14:03.0 +0100
@@ -268,6 +268,23 @@
_printTocEntry(AH, te, ropt, false, false);
defnDumped = true;
+ /* If we could not create a table, ignore the
respective TABLE DATA if
+* -X no-data-for-failed-tables is given */
+ if (ropt-noDataForFailedTables AH-lastErrorTE ==
te strcmp (te-desc, TABLE) == 0) {
+ TocEntry *tes, *last;
+
+ ahlog (AH, 1, table %s could not be created,
will not restore its data\n, te-tag);
+
+ for (last = te, tes = te-next; tes != AH-toc;
last = tes, tes = tes-next) {
+ if (strcmp (tes-desc, TABLE DATA) ==
0 strcmp (tes-tag, te-tag
Re: [HACKERS] Adding an ignore list to pg_restore, prototype patch #1
Hi again,
Martin Pitt [2006-02-19 14:39 +0100]:
Since this changes the behaviour of pg_restore, this should probably
become an option, e. g. -D / --ignore-existing-table-data. I'll do
this if you agree to the principle of the current patch.
I improved the patch now to only ignore TABLE DATA for existing tables
if '-X ignore-existing-tables' is specified. I also updated the
documentation.
Since this doesn't change the default behaviour now any more, I would
like to put this patch into the Debian packages to provide automatic
upgrades for PostGIS-enabled databases (see [1]). Does anyone object
to this?
Do you consider to adopt this upstream?
Thanks in advance, and have a nice weekend!
Martin
[1] http://bugs.debian.org/351571
--
Martin Pitthttp://www.piware.de
Ubuntu Developer http://www.ubuntu.com
Debian Developer http://www.debian.org
In a world without walls and fences, who needs Windows and Gates?
--- postgresql-8.1.3/doc/src/sgml/ref/pg_restore.sgml
+++ postgresql-8.1.3/doc/src/sgml/ref/pg_restore.sgml
@@ -395,6 +395,19 @@
/listitem
/varlistentry
+ varlistentry
+ termoption-X ignore-existing-tables//term
+ listitem
+ para
+ By default, table data objects are restored even if the
+ associated table already exists. With this option, such table
+ data is silently ignored. This is useful for dumping and
+ restoring databases with tables which contain auxiliary data
+ for PostgreSQL extensions (e. g. PostGIS).
+ /para
+ /listitem
+ /varlistentry
+
/variablelist
/para
--- postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c
+++ postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c
@@ -268,6 +268,21 @@
_printTocEntry(AH, te, ropt, false, false);
defnDumped = true;
+ /* If we could not create a table, ignore the
respective TABLE DATA if
+* -X ignore-existing-tables is given */
+ if (ropt-ignoreExistingTables AH-lastErrorTE == te
strcmp (te-desc, TABLE) == 0) {
+ TocEntry *tes;
+
+ ahlog (AH, 1, table %s could not be created,
will not restore its data\n, te-tag);
+
+ for (tes = te-next; tes != AH-toc; tes =
tes-next) {
+ if (strcmp (tes-desc, TABLE DATA) ==
0 strcmp (tes-tag, te-tag) == 0) {
+strcpy (tes-desc, IGNOREDATA);
+break;
+ }
+ }
+ }
+
/* If we created a DB, connect to it... */
if (strcmp(te-desc, DATABASE) == 0)
{
@@ -1876,6 +1891,10 @@
if (strcmp(te-desc, ENCODING) == 0)
return 0;
+ /* IGNOREDATA is a TABLE DATA which should not be restored */
+ if (strcmp (te-desc, IGNOREDATA) == 0)
+ return 0;
+
/* If it's an ACL, maybe ignore it */
if ((!include_acls || ropt-aclsSkip) strcmp(te-desc, ACL) == 0)
return 0;
--- postgresql-8.1.3/src/bin/pg_dump/pg_backup.h
+++ postgresql-8.1.3/src/bin/pg_dump/pg_backup.h
@@ -106,6 +106,7 @@
char *pghost;
char *username;
int ignoreVersion;
+ int ignoreExistingTables;
int requirePassword;
int exit_on_error;
--- postgresql-8.1.3/src/bin/pg_dump/pg_restore.c
+++ postgresql-8.1.3/src/bin/pg_dump/pg_restore.c
@@ -254,6 +254,8 @@
use_setsessauth = 1;
else if (strcmp(optarg, disable-triggers) ==
0)
disable_triggers = 1;
+ else if (strcmp(optarg,
ignore-existing-tables) == 0)
+ opts-ignoreExistingTables = 1;
else
{
fprintf(stderr,
@@ -394,6 +396,8 @@
printf(_( -X use-set-session-authorization,
--use-set-session-authorization\n
use SESSION AUTHORIZATION
commands instead of\n
OWNER TO commands\n));
+ printf(_( -X ignore-existing-tables\n
+ skip restoration of data for
already existing tables\n));
printf(_(\nConnection options:\n));
printf(_( -h, --host=HOSTNAME database server host or socket
directory\n));
signature.asc
Description: Digital signature
Re: [HACKERS] pg_config, pg_service.conf, postgresql.conf ....
Hi Mark, hi Martijn! Martijn van Oosterhout [2006-02-23 12:10 +0100]: If you're talking about standards perhaps you should consider how Debian does it. All configuration is stored in /etc/postgresql/version/clustername/ It provides wrapper scripts to run pg_ctl (pg_ctlcluster) on any particular cluster which can be run by either the system user owning the db, or root. Within those files is all the information required to find the cluster (base directory, etc). Right. But although this fits well for Debian, this is not necessarily the case for other Linux distributions, let alone non-Linux systems. Even less when using the upstream tarball (where all configuration is usually kept in the data directory itself). However, Debian has a tool 'pg_lsclusters' which comes pretty close to what Mark wants, AFAICS: $ pg_lsclusters Version Cluster Port Status OwnerData directory Log file 7.4 main 5432 online postgres /var/lib/postgresql/7.4/main /var/log/postgresql/postgresql-7.4-main.log 8.1 main 5434 down postgres /var/lib/postgresql/8.1/main /var/log/postgresql/postgresql-8.1-main.log 8.1 test 5433 online martin /home/martin/psql /var/log/postgresql/postgresql-8.1-test.log So, even in the current Debian system we don't have a fixed location for the data directories, log files, etc. There are defaults, but they can be customized, for good reasons (like keeping my test cluster in my home directory, and so on). I think the default config is to start all clusters on bootup. Right; this is customizable in a file start.conf (auto|manual|disabled) in the cluster configuration directory. The main downside of this system is that some sysadmin pretty much needs to create the clusters for everyone. What do you mean in particular? The packages install a default cluster (e. g. postgresql-8.1 creates a cluster 8.1/main), more clusters can be created with pg_createcluster. I doubt something like this would ever make it into the standard distribution though. What you're asking is more a distributer issue than an issue for postgres. I agree. Also, FYI, a while ago I proposed some patches which allow several major versions to be installed in parallel, but they were deemed too specific for upstream inclusion (which is fine; it's the distributor's task to integrate an application into a distribution's file layout/configuration handling/other quirks). Thanks, Martin -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? signature.asc Description: Digital signature
Re: [HACKERS] pg_config, pg_service.conf, postgresql.conf ....
Hi Martijn! Martijn van Oosterhout [2006-02-23 13:33 +0100]: What I mean is that only root can run pg_createcluster (either via package installation or directly). At least, that's what my reading of the code tells me. Uless you have an pg_adoptcluster somewhere :) Ah, right, now I know what you mean. This is necessary since non-root users can't (or rather shouldn't) write into /etc. If there is a desire for it, we can certainly discuss a way to manage clusters entirely as an user (with some missing features like autostart at boot, of course). But that should happen in the Debian BTS, not here. I really like the way Debian does it, you do a good job. Thanks :) Have you considered autogenerating entries for pg_service.conf? Not yet. TBH I didn't know about the pg_service.conf feature until just recently (in Debian you can select a cluster with --cluster version/name, that worked pretty well so far). Sounds interesting, though. :) (Again, let's discuss that in the Debian BTS). Thanks for the suggestions, Martin -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? signature.asc Description: Digital signature
Re: [HACKERS] Adding an ignore list to pg_restore
Hi Tom! Tom Lane [2006-02-18 14:34 -0500]: Hm. Rather than a variant of the -L facility (which is hard to use, and I don't see your proposal being much easier), maybe what's wanted is just a flag saying don't try to restore data into any table whose creation command fails. Maybe that should even be the default ... and you could extend it to indexes and constraints on such tables too, as those would likely end up being duplicated as well. This comes close to my alternative proposal, it sounds fine to me. I'll try to come up with a reasonably clean implementation and report back then. Thank you, and have a nice Sunday, Martin -- Martin Pitt http://www.piware.de Ubuntu Developer http://www.ubuntulinux.org Debian Developerhttp://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] Adding an ignore list to pg_restore, prototype patch #1
Hi again, Tom Lane [2006-02-18 14:34 -0500]: The core problem is that we want to not restore objects (mainly tables) in the destination database which already exist. Why is this a problem? It's already the default behavior --- the creation commands fail but pg_restore keeps going. The problem is that pg_restore would restore the TABLE DATA object, although we don't want that (the postgis specific tables are pre-populated by PostGIS itself, and should not be altered by the upgrade. Hm. Rather than a variant of the -L facility (which is hard to use, and I don't see your proposal being much easier), maybe what's wanted is just a flag saying don't try to restore data into any table whose creation command fails. Maybe that should even be the default ... and you could extend it to indexes and constraints on such tables too, as those would likely end up being duplicated as well. My first stab at this is a patch which only does the minimal changes, just to get me going. If the restoration of a TABLE object fails, it marks the corresponding TABLE DATA object as to be ignored. Do you think the current patch is a valid approach? Since this changes the behaviour of pg_restore, this should probably become an option, e. g. -D / --ignore-existing-table-data. I'll do this if you agree to the principle of the current patch. For convenience, I wrote a small test script which demonstrates the behaviour. The table 'userdata' should be restored, while the table 'auxdata' is already present in the destination db, and its contents should not be modified. Output with pg_restore from 8.1.3: --- snip $ LC_ALL=C sudo -u postgres ./test-pg_restore-existing.sh === create empty databases === === populating old database === === pre-creating auxdata in new database === === restoring old to new === pg_restore: connecting to database for restore pg_restore: creating SCHEMA public pg_restore: creating COMMENT SCHEMA public pg_restore: creating TABLE auxdata pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 1183; 1259 17184 TABLE auxdata postgres pg_restore: [archiver (db)] could not execute query: FEHLER: Relation »auxdata« existiert bereits Command was: CREATE TABLE auxdata ( x integer ); pg_restore: creating TABLE userdata pg_restore: restoring data for table auxdata pg_restore: restoring data for table userdata pg_restore: setting owner and privileges for SCHEMA public pg_restore: setting owner and privileges for COMMENT SCHEMA public pg_restore: setting owner and privileges for ACL public pg_restore: setting owner and privileges for TABLE auxdata pg_restore: setting owner and privileges for TABLE userdata WARNING: errors ignored on restore: 1 pg_restore failed with 1 === new/userdata: === 42 256 === new/auxdata: === -1 -2 1 2 --- snip Output with patched pg_restore: --- snip $ LC_ALL=C sudo -u postgres ./test-pg_restore-existing.sh === create empty databases === === populating old database === === pre-creating auxdata in new database === === restoring old to new === pg_restore: connecting to database for restore pg_restore: creating SCHEMA public pg_restore: creating COMMENT SCHEMA public pg_restore: creating TABLE auxdata pg_restore: [archiver (db)] Error while PROCESSING TOC: pg_restore: [archiver (db)] Error from TOC entry 1183; 1259 17194 TABLE auxdata postgres pg_restore: [archiver (db)] could not execute query: FEHLER: Relation »auxdata« existiert bereits Command was: CREATE TABLE auxdata ( x integer ); pg_restore: table auxdata could not be created, will not restore its data pg_restore: creating TABLE userdata pg_restore: restoring data for table userdata pg_restore: setting owner and privileges for SCHEMA public pg_restore: setting owner and privileges for COMMENT SCHEMA public pg_restore: setting owner and privileges for ACL public pg_restore: setting owner and privileges for TABLE auxdata pg_restore: setting owner and privileges for TABLE userdata WARNING: errors ignored on restore: 1 pg_restore failed with 1 === new/userdata: === 42 256 === new/auxdata: === -1 -2 --- snip Thus, with the patch, auxdata is not restored (which produced the additional entries '1' and '2'). Thanks, Martin -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? test-pg_restore-existing.sh Description: Bourne shell script diff -ruN postgresql-8.1.3-old/src/bin/pg_dump/pg_backup_archiver.c postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c --- postgresql-8.1.3-old/src/bin/pg_dump/pg_backup_archiver.c 2006-02-05 21:58:57.0 +0100 +++ postgresql-8.1.3/src/bin/pg_dump/pg_backup_archiver.c 2006-02-19 14:20:36.0
Re: [HACKERS] Adding an ignore list to pg_restore, prototype patch #1
Hi again, Meh, the list server didn't like the attached test script, so I put it here: http://people.debian.org/~mpitt/test-pg_restore-existing.sh Martin -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? signature.asc Description: Digital signature
[HACKERS] Adding an ignore list to pg_restore
Hi PostgreSQL developers! On [1], Stephen and I are currently discussing how to provide seamless automatic version upgrades of PostgreSQL databases with third party modules like PostGIS. The core problem is that we want to not restore objects (mainly tables) in the destination database which already exist. pg_restore currently offers the -L option to selectively restore only particular objects, so our original idea was to automatically create this list based on the output of pg_dump -Fc --schema-only $db | pg_restore -l for the original and target databases. However, there is a fundamental problem with this approach. When using --schema-only, the generated list naturally does not contain TABLE DATA entries. This means that we cannot figure out the catalog id of the DATA object. Of course there are workarounds [2], but both of them are inefficient. So my current idea is to add a new option --ignore-list to pg_restore which specifies a file with objects that should not be restored. This file should not contain catalog IDs, but human readable data: type schema tag e. g. if we know that we don't want to restore the public.foo table, then this file would contain TABLE public foo TABLE DATA public foo I would be willing to provide an implementation, but before I wanted to ask if this feature is something you would generally accept, or regard as totally crackful? Another way would be to add an option which specifically ignores objects that are already present in the target database. This would be more robust and probably easier to implement, but less general than the ignore list. Thank you in advance for any comment, Martin [1] http://bugs.debian.org/351571 [2] (1) run pg_dump without --schema-only twice (once for pg_restore -l, second time for actual restoration), or (2) save pg_dump output into a temporary file -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? signature.asc Description: Digital signature
Re: [HACKERS] Adding an ignore list to pg_restore
Hi Tom! Tom Lane [2006-02-18 13:32 -0500]: Martin Pitt [EMAIL PROTECTED] writes: The core problem is that we want to not restore objects (mainly tables) in the destination database which already exist. Why is this a problem? It's already the default behavior --- the creation commands fail but pg_restore keeps going. The problem is that pg_restore would restore the TABLE DATA object, although we don't want that (the postgis specific tables are pre-populated by PostGIS itself, and should not be altered by the upgrade. (Stephen knows the details of PostGIS). Thanks, Martin -- Martin Pitthttp://www.piware.de Ubuntu Developer http://www.ubuntu.com Debian Developer http://www.debian.org In a world without walls and fences, who needs Windows and Gates? signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi! Bruce Momjian [2005-06-06 21:23 -0400]: Is this a direction we want to explore --- using the SONAME as part of the translation domain? If that would go upstream, so much the better. I already do it in the Debian and Ubuntu packages since I don't have any choice anyway, and it's not really hard to maintain. But if it would go upstream, other vendors might benefit from that as well. However, as Alvaro already said, the patch should probably be refined to take the SONAME from the Makefile somehow instead of hardcoding it. Thanks and have a nice day! Martin -- Martin Pitt http://www.piware.de Ubuntu Developer http://www.ubuntulinux.org Debian Developerhttp://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi!
Bruce Momjian [2005-02-09 18:05 -0500]:
However, I just stumbled across another problem: libpq3 and the new
libpq4 use the same translation domain libpq4, thus they cannot be
installed in parallel. Can you please change the domain to libpq4 as
well? This should generally be done anyway, but at least we can live
with one breakage of this rule (for libpq3), so that we don't need to
mess with the old libpq3 package.
Uh, what is the translation domain?
In short, that is the unique name for a bunch of translations for a
particular application or group of related applications. The key point
is that the name of the file in
/usr/share/locale/LOCALE/LC_MESSAGES/DOMAIN.mo and the call
bindtextdomain (DOMAIN, NULL)
must use the same DOMAIN so that libintl can find the mo file.
However, that means that all applications that are installed in
parallel need a distinct domain. Since the whole point of SONAMes is
to allow several different library API versions to be installed in
parallel, every library API (i. e. SONAME) should have an unique
domain, which can be achieved easiest by just appending the SONAME to
the translation domain.
In my test packages I used the following patch:
diff -ruN postgresql-8.0.1-old/src/interfaces/libpq/fe-misc.c
postgresql-8.0.1/src/interfaces/libpq/fe-misc.c
--- postgresql-8.0.1-old/src/interfaces/libpq/fe-misc.c 2004-12-31
23:03:50.0 +0100
+++ postgresql-8.0.1/src/interfaces/libpq/fe-misc.c 2005-02-07
22:55:13.177419296 +0100
@@ -1133,7 +1133,7 @@
{
already_bound = 1;
/* No relocatable lookup here because the binary could be
anywhere */
- bindtextdomain(libpq, getenv(PGLOCALEDIR) ?
getenv(PGLOCALEDIR) : LOCALEDIR);
+ bindtextdomain(libpq4, getenv(PGLOCALEDIR) ?
getenv(PGLOCALEDIR) : LOCALEDIR);
}
return dgettext(libpq, msgid);
diff -ruN postgresql-8.0.1-old/src/interfaces/libpq/nls.mk
postgresql-8.0.1/src/interfaces/libpq/nls.mk
--- postgresql-8.0.1-old/src/interfaces/libpq/nls.mk2005-01-14
09:57:06.0 +0100
+++ postgresql-8.0.1/src/interfaces/libpq/nls.mk2005-02-07
22:54:54.770217616 +0100
@@ -1,5 +1,5 @@
# $PostgreSQL: pgsql/src/interfaces/libpq/nls.mk,v 1.20 2005/01/14 08:57:06
petere Exp $
-CATALOG_NAME := libpq
+CATALOG_NAME := libpq4
AVAIL_LANGUAGES:= af cs de es fr hr it ko nb pl pt_BR ru sk sl sv tr
zh_CN zh_TW
GETTEXT_FILES := fe-auth.c fe-connect.c fe-exec.c fe-lobj.c fe-misc.c
fe-protocol2.c fe-protocol3.c fe-secure.c
GETTEXT_TRIGGERS:= libpq_gettext pqInternalNotice:2
Compared to the SONAME, changing the translation domain is relatively
uncritical, so if you don't want to change this upstream, I can
maintain this patch for Debian/Ubuntu. However, I heard that some RPM
guys plan a infrastructure similar to mine, and at that point they
will have precisely the same problems :-)
Thanks for considering and have a nice day!
Martin
--
Martin Pitt http://www.piware.de
Ubuntu Developerhttp://www.ubuntulinux.org
Debian GNU/Linux Developer http://www.debian.org
signature.asc
Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi! Tom Lane [2005-02-04 10:27 -0500]: This problem isn't worth spending more development time on than it takes to change SO_MAJOR_VERSION (we have lots of higher-priority issues). I just did that: --- postgresql-8.0.1-old/src/interfaces/libpq/Makefile 2005-01-26 20:24:19.0 +0100 +++ postgresql-8.0.1/src/interfaces/libpq/Makefile 2005-02-07 20:09:38.719686432 +0100 @@ -16,8 +16,8 @@ # shared library parameters NAME= pq -SO_MAJOR_VERSION= 3 -SO_MINOR_VERSION= 2 +SO_MAJOR_VERSION= 4 +SO_MINOR_VERSION= 0 DLTYPE= library override CPPFLAGS := -DFRONTEND -I$(srcdir) $(CPPFLAGS) -I$(top_builddir)/src/port Is that right? Or will you use SO_MINOR_VERSION=1? However, I just stumbled across another problem: libpq3 and the new libpq4 use the same translation domain libpq4, thus they cannot be installed in parallel. Can you please change the domain to libpq4 as well? This should generally be done anyway, but at least we can live with one breakage of this rule (for libpq3), so that we don't need to mess with the old libpq3 package. Thanks, Martin -- Martin Pitt http://www.piware.de Ubuntu Developerhttp://www.ubuntulinux.org Debian GNU/Linux Developer http://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi! Tom Lane [2005-02-03 11:12 -0500]: Martin Pitt [EMAIL PROTECTED] writes: I am thinking the easiest solution will be to re-add get_progname() to 8.0.X and bump the major for 8.1. Seconded. Then we don't need another library version, and it is still not necessary to drag this get_progname accident forever. We're going to stop dragging the accident now, not a year from now. Putting get_progname back into libpq would require re-adding all of path.c, which creates many more issues than just adjusting SO_MAJOR_VERSION would do. The consensus seems to be to just bump the SONAME. The next one will definitively be 4 then? I would like to fix this in my packages before I announce them for public testing, but this issue is the only blocker to it. Will there be a new bugfix release soon? Thanks for your work and have a nice day! Martin -- Martin Pitt http://www.piware.de Ubuntu Developerhttp://www.ubuntulinux.org Debian GNU/Linux Developer http://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi! Bruce Momjian [2005-02-03 9:42 -0500]: Peter Eisentraut wrote: Bruce Momjian wrote: Uh, if we bump up the major library version in 8.0.X, will that require 8.0.0 user applications to be recompiled? No, they just keep using the old library. That assumes the old libraries stay around. Will they? I am thinking the easiest solution will be to re-add get_progname() to 8.0.X and bump the major for 8.1. Seconded. Then we don't need another library version, and it is still not necessary to drag this get_progname accident forever. Thanks, Martin -- Martin Pitt http://www.piware.de Ubuntu Developerhttp://www.ubuntulinux.org Debian GNU/Linux Developer http://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi! Andrew Dunstan [2005-02-03 11:24 -0500]: Maybe I'm dense, but I don't understand why this is causing anyone a headache. Why would one install the 8.0 libs without the 8.0 clients? That's not the point. The point is that this breakage makes it impossible to install _both_ 7.4 and 8.0 server/client at the same time. In addition, this breaks partial upgrades: if the package libpq3 (as it is called in Debian) is upgraded from 7.4.x to 8.0.x (which is perfectly valid and does not break any dependencies, since it is the same library with the same API), but the postgresql-client package isn't (because it is on hold, or whatever), then this breaks the old postgresql-client as well. I'm fine with immediately bumping the SONAME too, but as I said, it's important that this happens upstream, not in the distributions. Thanks, Martin -- Martin Pitt http://www.piware.de Ubuntu Developerhttp://www.ubuntulinux.org Debian GNU/Linux Developer http://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi! (sorry for the additional addresses; I'm not subscribed to -hackers, so my mail will last a while until it arrives there). Tom Lane [2005-02-02 11:07 -0500]: Peter Eisentraut [EMAIL PROTECTED] writes: Martin Pitt has detected that the libpq API has changed incompatibly between 7.4 and 8.0. This has the effect, for example, that 7.4's psql cannot run with 8.0's libpq. [ shrug... ] I don't think we've ever guaranteed that anyway. If you don't guarantee a backwards compatible API, then you should give libpq a new SONAME. By keeping the SONAME 3 you did promise backwards compatibility. The problem with this issue is that it is a real pain for distributors to handle broken SONAMEs. If every distributor invents his own one, then there will be conflicts at some point. This gets worse if you do release a new libpq SONAME later, which might conflict with any fake SONAME a distributor might have invented. I will resist putting get_progname back into libpq, because it should never have been there, at least not with that name: it's an undocumented infringement on application namespace. I understand such concerns, but you cannot undo the history. libpq3 is out there and installed on maybe hundreds of thousands of machines, and your own psql frontend uses it. The next best option would be to fix the 7.4 version of psql to not use this symbol any more. However, this is still a pain since then you cannot upgrade from earlier versions to 8.0 any more. What would you propose as a solution? There can only be one /usr/lib/libpq.so.3. Thanks, Martin -- Martin Pitt http://www.piware.de Ubuntu Developerhttp://www.ubuntulinux.org Debian GNU/Linux Developer http://www.debian.org signature.asc Description: Digital signature
Re: [HACKERS] libpq API incompatibility between 7.4 and 8.0
Hi Tom! Tom Lane [2005-02-02 12:01 -0500]: Martin Pitt [EMAIL PROTECTED] writes: What would you propose as a solution? Do nothing. That's unfortunately not an option. The problem you are raising isn't very serious since RPM-style installations don't support concurrent installation of multiple PG versions anyway. That being the case, it doesn't really matter whether 8.0 psql can use a 7.4 library or vice versa. That's exactly the point I want to change for Debian, I work on a structure which permits to run several clusters of different versions in parallel (see [1]). We now have a /usr/lib/libpq.so.3 for 7.4, and we can't have a _second_ incompatible /usr/lib/libpq.so.3 for 8.0. Sorry, but that's just the way how shared libraries work. SONAMEs are for API compatibility and it becomes _very_ hard to workaround broken ones. To do otherwise would essentially amount to deciding that get_progname is part of the exported API of libpq forevermore. That's not true. It must stay part of the exported API for SONAME 3 of libpq, not for anything else. That's not something I'm willing to buy into. It was a mistake that it was done that way in 7.4, and I want to rectify that mistake before it gets any more entrenched. Then please release 8.0.2 with a SONAME 4. Would you consider this? Thanks a lot and have a nice day! Martin [1] http://people.debian.org/~mpitt/postgresql-ng.html -- Martin Pitt http://www.piware.de Ubuntu Developerhttp://www.ubuntulinux.org Debian GNU/Linux Developer http://www.debian.org signature.asc Description: Digital signature
