Re: [HACKERS] pg_dump and dependencies and --section ... it's a mess
Andrew Dunstan and...@dunslane.net writes: On 06/22/2012 04:43 PM, Tom Lane wrote: A possible objection to it is that there are now three different ways in which the pg_dump code knows which DO_XXX object types go in which dump section: the new addBoundaryDependencies() function knows this, the SECTION_xxx arguments to ArchiveEntry calls know it, and the sort ordering constants in pg_dump_sort.c have to agree too. My original idea was to add an explicit section field to DumpableObject to reduce the number of places that know this, but that would increase pg_dump's memory consumption still more, and yet still not give us a single point of knowledge. Has anybody got a better idea? Not off hand. I still don't have a great idea for eliminating the duplicativeness, but it occurred to me that we could add a bit of code to check that the finished TOC list is in correct section order. This should catch most cases where one value is out of sync with the others, and it seems like a good idea in any case. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] random failing builds on spoonbill - backends not exiting...
Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: On 06/22/2012 11:47 PM, Tom Lane wrote: Could you gdb each of these processes and get a stack trace? [ unsurprising stack traces ] OK, so they're waiting exactly where they should be. So what we know is that the shutdown failure is caused by the child processes having all blockable signals blocked. What we don't know is how they could have gotten that way. I do not see any way that the code in Postgres would ever block all signals, which makes this look a lot like a BSD libc bug. But that doesn't help much. The only way I can think of to narrow it down is to run the postmaster under strace -f or local equivalent, and look for sigprocmask calls that set all the bits. That could be pretty tedious though, if the occurrence of the bug is as low-probability as it seems to be. Given your earlier comment about a new threading library, I wonder whether this has something to do with confusion between process-wide and per-thread signal masks ... regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL format changes
Ok, committed all the WAL format changes now. On 19.06.2012 18:57, Robert Haas wrote: Should we keep the old representation in the replication protocol messages? That would make it simpler to write a client that works with different server versions (like pg_receivexlog). Or, while we're at it, perhaps we should mandate network-byte order for all the integer and XLogRecPtr fields in the replication protocol. That would make it easier to write a client that works across different architectures, in= 9.3. The contents of the WAL would of course be architecture-dependent, but it would be nice if pg_receivexlog and similar tools could nevertheless be architecture-independent. I share Andres' question about how we're doing this already. I think if we're going to break this, I'd rather do it in 9.3 than 5 years from now. At this point it's just a minor annoyance, but it'll probably get worse as people write more tools that understand WAL. I didn't touch the replication protocol yet, but I think we should do it some time during 9.3. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Preferred way to define 64-bit constants?
I just committed the patch to change XLogRecPtr into a 64-bit constant, and I did this in the patch: #define XLogSegmentsPerXLogId (0x1LL / XLOG_SEG_SIZE) But I started to wonder, is that LL representation the preferred way to define 64-bit integer constants? I thought it is, but now that I grep around, I don't see any constants like that in the source tree. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_prewarm
Le samedi 23 juin 2012 02:47:15, Josh Berkus a écrit : The biggest problem with pgfincore from my point of view is that it only works under Linux, whereas I use a MacOS X machine for my development, and there is also Windows to think about. Even if that were fixed, though, I feel we ought to have something in the core distribution. This patch got more +1s than 95% of what gets proposed on hackers. Fincore is only a blocker to this patch if we think pgfincore is ready to be proposed for the core distribution. Do we? I'll make it ready for. (not a huge task). -- Cédric Villemain +33 (0)6 20 30 22 52 http://2ndQuadrant.fr/ PostgreSQL: Support 24x7 - Développement, Expertise et Formation signature.asc Description: This is a digitally signed message part.
Re: [HACKERS] Preferred way to define 64-bit constants?
On 24 June 2012 18:23, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I just committed the patch to change XLogRecPtr into a 64-bit constant, and I did this in the patch: #define XLogSegmentsPerXLogId (0x1LL / XLOG_SEG_SIZE) But I started to wonder, is that LL representation the preferred way to define 64-bit integer constants? I thought it is, but now that I grep around, I don't see any constants like that in the source tree. This looks to be a long long int literal. That's only specified in the C99 standard, as well as GNU C. It may very well not be a problem in practice, but I'm told that some very esoteric compilers could baulk at things like that. http://gcc.gnu.org/onlinedocs/gcc/Long-Long.html -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WAL format changes
On 24 June 2012 17:24, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok, committed all the WAL format changes now. Nice! -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Preferred way to define 64-bit constants?
On sön, 2012-06-24 at 20:23 +0300, Heikki Linnakangas wrote: I just committed the patch to change XLogRecPtr into a 64-bit constant, and I did this in the patch: #define XLogSegmentsPerXLogId (0x1LL / XLOG_SEG_SIZE) But I started to wonder, is that LL representation the preferred way to define 64-bit integer constants? I thought it is, but now that I grep around, I don't see any constants like that in the source tree. See INT64CONST, UINT64CONST. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] warning handling in Perl scripts
Every time I make a change to the structure of the catalog files,
genbki.pl produces a bunch of warnings (like Use of uninitialized value
in string eq at genbki.pl line ...), and produces corrupted output
files, that are then (possibly) detected later by the compiler. Also,
getting out of that is difficult because due to the complicated
dependency relationship between the involved files, you need to remove a
bunch of files manually, or clean everything. So error handling could
be better.
It seems that adding
diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
index ebc4825..7d66da9 100644
--- a/src/backend/catalog/genbki.pl
+++ b/src/backend/catalog/genbki.pl
@@ -19,6 +19,8 @@
use strict;
use warnings;
+local $SIG{__WARN__} = sub { die $_[0] };
+
my @input_files;
our @include_path;
my $output_path = '';
would address that.
Could that cause any other problems? Should it be added to all Perl
scripts?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Preferred way to define 64-bit constants?
On 24.06.2012 21:34, Peter Eisentraut wrote: On sön, 2012-06-24 at 20:23 +0300, Heikki Linnakangas wrote: I just committed the patch to change XLogRecPtr into a 64-bit constant, and I did this in the patch: #define XLogSegmentsPerXLogId (0x1LL / XLOG_SEG_SIZE) But I started to wonder, is that LL representation the preferred way to define 64-bit integer constants? I thought it is, but now that I grep around, I don't see any constants like that in the source tree. See INT64CONST, UINT64CONST. Thanks, fixed. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] warning handling in Perl scripts
On Sun, Jun 24, 2012 at 2:40 PM, Peter Eisentraut pete...@gmx.net wrote:
Every time I make a change to the structure of the catalog files,
genbki.pl produces a bunch of warnings (like Use of uninitialized value
in string eq at genbki.pl line ...), and produces corrupted output
files, that are then (possibly) detected later by the compiler. Also,
getting out of that is difficult because due to the complicated
dependency relationship between the involved files, you need to remove a
bunch of files manually, or clean everything. So error handling could
be better.
It seems that adding
diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl
index ebc4825..7d66da9 100644
--- a/src/backend/catalog/genbki.pl
+++ b/src/backend/catalog/genbki.pl
@@ -19,6 +19,8 @@
use strict;
use warnings;
+local $SIG{__WARN__} = sub { die $_[0] };
+
my @input_files;
our @include_path;
my $output_path = '';
would address that.
Could that cause any other problems? Should it be added to all Perl
scripts?
This seems like a band-aid. How about if we instead add whatever
error-handling the script is missing, so that it produces an
appropriate, human-readable error message?
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
On Thursday, June 21, 2012 01:41:25 PM Andres Freund wrote: Below are two possible implementation strategies for that concept Advantages: * Decoding is done on the master in an asynchronous fashion * low overhead during normal DML execution, not much additional code in that path * can be very efficient if architecture/version are the same * version/architecture compatibility can be done transparently by falling back to textual versions on mismatch Disadvantages: * decoding probably has to happen on the master which might not be what people want performancewise 3b) Ensure that enough information in the catalog remains by fudging the xmin horizon. Then reassemble an appropriate snapshot to read the catalog as the tuple in question has seen it. Advantages: * should be implementable with low impact to general code Disadvantages: * requires some complex code for assembling snapshots * it might be hard to guarantee that we always have enough information to reassemble a snapshot (subxid overflows ...) * impacts vacuum if replication to some site is slow There are some interesting problems related to locking and snapshots here. Not sure if they are resolvable: We need to restrict SnapshotNow to represent to the view it had back when the wal record were currently decoding had. Otherwise we would possibly get wrong column types and similar. As were working in the past locking doesn't protect us against much here. I have that (mostly and inefficiently). One interesting problem are table rewrites (truncate, cluster, some ALTER TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it had back when the wal record was created we get the old relfilenode. Which might have been dropped in part of the transaction cleanup... With most types thats not a problem. Even things like records and arrays aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. pg_enum) and vacuum full'ing a table which is used in the *_out function of a type (like a user level pg_enum implementation). The only theoretical way I see against that problem would be to postpone all relation unlinks untill everything that could possibly read them has finished. Doesn't seem to alluring although it would be needed if we ever move more things of SnapshotNow. Input/Ideas/Opinions? Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] empty backup_label
Howdy, We're using NetApp's flexclone's whenever we need to move our DB between machines. One specific case where we do that is when we're creating a new streaming replication target. The basic steps we're using are: pg_start_backup(); flex clone within the netapp pg_stop_backup(); The problem i'm seeing is that periodically the backup_label is empty, which means I can't start the new standby. I believe that since the NetApp stuff is all happening within the SAN this file hasn't been fsynced to disk by the time we take the snapshot. One option would be to do a sync prior to the clone, however that seems kind of like a heavy operation, and it's slightly more complicated to script. (having to have a user account on the system to sudo rather than just connecting to the db to issue the pg_start_backup(...) ) Another option is to add pg_fsync(fileno(fp)) after the fflush() when creating the file (I'm not sure if fsync implies fflush or not, if it does you could just replace it.) I think this type of snapshot is fairly common, I've been doing them since 2000 with EMC, i'm sure that most SAN vendors support it. I also suspect that this type of problem could show up on AWS if you tried to use their EBS snapshots Attached is a patch for the suggested change. Dave *** a/src/backend/access/transam/xlog.c --- b/src/backend/access/transam/xlog.c *** *** 9311,9316 do_pg_start_backup(const char *backupidstr, bool fast, char **labelfile) --- 9311,9317 BACKUP_LABEL_FILE))); if (fwrite(labelfbuf.data, labelfbuf.len, 1, fp) != 1 || fflush(fp) != 0 || + pg_fsync(fileno(fp)) != 0 || ferror(fp) || FreeFile(fp)) ereport(ERROR, -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
On 24 June 2012 22:11, Andres Freund and...@2ndquadrant.com wrote: One interesting problem are table rewrites (truncate, cluster, some ALTER TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it had back when the wal record was created we get the old relfilenode. Which might have been dropped in part of the transaction cleanup... With most types thats not a problem. Even things like records and arrays aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. pg_enum) and vacuum full'ing a table which is used in the *_out function of a type (like a user level pg_enum implementation). That's only a problem if you are generating changes to the relfilenode rather than the relid. ISTM that this step differs depending upon whether we are generating portable SQL, or whether we are generating changes for immediate apply. If it is the latter, then it should never actually happen because if a table rewrite occurred and then committed we would never need to re-read earlier WAL. So treating this as a generic problem leads to some weird results that we don't need to worry about cos they can't actually happen. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
On 22 June 2012 20:30, Andres Freund and...@2ndquadrant.com wrote: The problem making replacement of SnapshotNow.satisfies useful is that there is no convenient way to represent subtransactions of the current transaction which already have committed according to the TransactionLog but aren't yet visible at the current lsn because they only started afterwards. Its relatively easy to fake this in an mvcc snapshot but way harder for SnapshotNow because you cannot mark transactions as in-progress. I'm starting to like these ideas now. We can solve many things by emitting a new WAL record type in any subtransaction that issues catalog changes. That wasn't possible in Hot Standby for performance reasons, but since we only care about catalog changes those things are much rarer and wouldn't be a problem. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
On Sunday, June 24, 2012 11:37:26 PM Simon Riggs wrote: On 24 June 2012 22:11, Andres Freund and...@2ndquadrant.com wrote: One interesting problem are table rewrites (truncate, cluster, some ALTER TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it had back when the wal record was created we get the old relfilenode. Which might have been dropped in part of the transaction cleanup... With most types thats not a problem. Even things like records and arrays aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. pg_enum) and vacuum full'ing a table which is used in the *_out function of a type (like a user level pg_enum implementation). That's only a problem if you are generating changes to the relfilenode rather than the relid. Hm. I can't follow so far. Could you paraphrase? ISTM that this step differs depending upon whether we are generating portable SQL, or whether we are generating changes for immediate apply. I fear only generating changes for immediate, low-level apply is going to fly given the various interests people have voiced. If it is the latter, then it should never actually happen because if a table rewrite occurred and then committed we would never need to re-read earlier WAL. So treating this as a generic problem leads to some weird results that we don't need to worry about cos they can't actually happen. Well, even if it were true that we don't need to worry about the state before a full-table rewritte - I don't think it is - we still need to be able to cope with CLUSTER or VACUUM FULL... Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
On 24 June 2012 22:50, Andres Freund and...@2ndquadrant.com wrote: On Sunday, June 24, 2012 11:37:26 PM Simon Riggs wrote: On 24 June 2012 22:11, Andres Freund and...@2ndquadrant.com wrote: One interesting problem are table rewrites (truncate, cluster, some ALTER TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it had back when the wal record was created we get the old relfilenode. Which might have been dropped in part of the transaction cleanup... With most types thats not a problem. Even things like records and arrays aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. pg_enum) and vacuum full'ing a table which is used in the *_out function of a type (like a user level pg_enum implementation). That's only a problem if you are generating changes to the relfilenode rather than the relid. Hm. I can't follow so far. Could you paraphrase? Basically, whether we generate SQL or not. ISTM that this step differs depending upon whether we are generating portable SQL, or whether we are generating changes for immediate apply. I fear only generating changes for immediate, low-level apply is going to fly given the various interests people have voiced. I'm OK with that, just checking what the objectives are. If it is the latter, then it should never actually happen because if a table rewrite occurred and then committed we would never need to re-read earlier WAL. So treating this as a generic problem leads to some weird results that we don't need to worry about cos they can't actually happen. Well, even if it were true that we don't need to worry about the state before a full-table rewritte - I don't think it is - we still need to be able to cope with CLUSTER or VACUUM FULL... If you have a WAL record for a new relfilenode, then you don't need to read the catalog at all. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
On Sun, Jun 24, 2012 at 5:11 PM, Andres Freund and...@2ndquadrant.com wrote: There are some interesting problems related to locking and snapshots here. Not sure if they are resolvable: We need to restrict SnapshotNow to represent to the view it had back when the wal record were currently decoding had. Otherwise we would possibly get wrong column types and similar. As were working in the past locking doesn't protect us against much here. I have that (mostly and inefficiently). One interesting problem are table rewrites (truncate, cluster, some ALTER TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it had back when the wal record was created we get the old relfilenode. Which might have been dropped in part of the transaction cleanup... With most types thats not a problem. Even things like records and arrays aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. pg_enum) and vacuum full'ing a table which is used in the *_out function of a type (like a user level pg_enum implementation). The only theoretical way I see against that problem would be to postpone all relation unlinks untill everything that could possibly read them has finished. Doesn't seem to alluring although it would be needed if we ever move more things of SnapshotNow. Input/Ideas/Opinions? Yeah, this is slightly nasty. I'm not sure whether or not there's a way to make it work. I had another idea. Suppose decoding happens directly on the primary, because I'm still hoping there's a way to swing that. Suppose further that we handle DDL by insisting that (1) any backend which wants to add columns or change the types of existing columns must first wait for logical replication to catch up and (2) if a backend which has added columns or changed the types of existing columns then writes to the modified table, decoding of those writes will be postponed until transaction commit. I think that's enough to guarantee that the decoding process can just use the catalogs as they stand, with plain old SnapshotNow. The downside of this approach is that it makes certain kinds of DDL suck worse if logical replication is in use and behind. But I don't necessarily see that as prohibitive because (1) logical replication being behind is likely to suck for a lot of other reasons too and (2) adding or retyping columns isn't a terribly frequent operation and people already expect a hit when they do it. Also, I suspect that we could find ways to loosen those restrictions at least in common cases in some future version; meanwhile, less work now. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] libpq compression
On Fri, Jun 22, 2012 at 12:38 PM, Euler Taveira eu...@timbira.com wrote: On 20-06-2012 17:40, Marko Kreen wrote: On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug f...@phlo.org wrote: I'm starting to think that relying on SSL/TLS for compression of unencrypted connections might not be such a good idea after all. We'd be using the protocol in a way it quite clearly never was intended to be used... Maybe, but what is the argument that we should avoid on encryption+compression at the same time? AES is quite lightweight compared to compression, so should be no problem in situations where you care about compression. If we could solve compression problem without AES that will turn things easier. Compression-only via encryption is a weird manner to solve the problem in the user's POV. RSA is noticeable, but only for short connections. Thus easily solvable with connection pooling. RSA overhead is not the main problem. SSL/TLS setup is. And for really special compression needs you can always create a UDF that does custom compression for you. You have to own the code to modify it; it is not always an option. So what exactly is the situation we need to solve with postgres-specific protocol compression? Compression only support. Why do I need to set up SSL/TLS just for compression? IMHO SSL/TLS use is no different from relying in another library to handle compression for the protocol and more it is compression-specific. That way, we could implement another algorithms in such library without needing to modify libpq code. Using SSL/TLS you are bounded by what SSL/TLS software products decide to use as compression algorithms. I'll be happy to maintain the code iif it is postgres-specific or even as close as possible to core. I guess my feeling on this is that, so far as I can see, supporting compression via OpenSSL involves work and trade-offs, and supporting it without depending on OpenSSL also involves work, and trade-offs. So it's not real evident to me that we should prefer one to the other on general principle. It seems to me that a lot might come down to performance. If someone can demonstrate that using an external library involves gets significantly better compression, chews up significantly less CPU time, and/or is significantly less code than supporting this via OpenSSL, then maybe we ought to consider it. OpenSSL is kind of an ugly piece of code, and all things being equal depending on it more heavily would not be my first choice. On the other hand, this does not seem to me to be a situation where we should accept a patch to use an external library just because someone takes the time to write one, because there is also a non-trivial cost for the entire project to depending on more things; or if the compression code gets put into the backend, then there's a cost to us to maintain that code inside our source base. So I think we really need someone to try this both ways and compare. Right now it seems like we're mostly speculating on how well either approach would work, which is not as good as having some experimental results. If, for example, someone can demonstrate that an awesomebsdlz compresses 10x as fast as OpenSSL... that'd be pretty compelling. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: Replace XLogRecPtr struct with a 64-bit integer.
Excerpts from Heikki Linnakangas's message of dom jun 24 12:22:30 -0400 2012: Replace XLogRecPtr struct with a 64-bit integer. This simplifies code that needs to do arithmetic on XLogRecPtrs. To avoid changing on-disk format of data pages, the LSN on data pages is still stored in the old format. That should keep pg_upgrade happy. However, we have XLogRecPtrs embedded in the control file, and in the structs that are sent over the replication protocol, so this changes breaks compatibility of pg_basebackup and server. I didn't do anything about this in this patch, per discussion on -hackers, the right thing to do would to be to change the replication protocol to be architecture-independent, so that you could use a newer version of pg_receivexlog, for example, against an older server version. One commit in this series broke pg_upgrade, which depended on pg_resetxlog output: First log file id after reset. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] foreign key locks
Excerpts from Kevin Grittner's message of sáb jun 23 18:38:10 -0400 2012: Alvaro Herrera wrote: So here's the rebased version. I found a couple problems on `make check-world`. Attached is a patch to fix one of them. The other is on pg_upgrade, pasted below. Ah, I mismerged pg_upgrade. The fix is trivial, AFAICS -- a function call is missing a log file name to be specified. Strange that the compiler didn't warn me of a broken printf format specifier. However, pg_upgrade itself has been broken by an unrelated commit and that fix is not so trivial, so I'm stuck waiting for that to be fixed. (We really need automated pg_upgrade testing.) Thanks for the patch for the other problem, I'll push it out. I'm marking it as Waiting for Author since this seems like a must fix, but I'll keep looking at it, Great, thanks. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas Sent: Monday, June 25, 2012 6:39 AM To: Andres Freund Cc: pgsql-hackers@postgresql.org; Florian Pflug; Simon Riggs Subject: Re: [HACKERS] Catalog/Metadata consistency during changeset extraction from wal On Sun, Jun 24, 2012 at 5:11 PM, Andres Freund and...@2ndquadrant.com wrote: There are some interesting problems related to locking and snapshots here. Not sure if they are resolvable: We need to restrict SnapshotNow to represent to the view it had back when the wal record were currently decoding had. Otherwise we would possibly get wrong column types and similar. As were working in the past locking doesn't protect us against much here. I have that (mostly and inefficiently). One interesting problem are table rewrites (truncate, cluster, some ALTER TABLE's) and dropping tables. Because we nudge SnapshotNow to the past view it had back when the wal record was created we get the old relfilenode. Which might have been dropped in part of the transaction cleanup... With most types thats not a problem. Even things like records and arrays aren't problematic. More interesting cases include VACUUM FULL $systable (e.g. pg_enum) and vacuum full'ing a table which is used in the *_out function of a type (like a user level pg_enum implementation). The only theoretical way I see against that problem would be to postpone all relation unlinks untill everything that could possibly read them has finished. Doesn't seem to alluring although it would be needed if we ever move more things of SnapshotNow. Input/Ideas/Opinions? Yeah, this is slightly nasty. I'm not sure whether or not there's a way to make it work. I had another idea. Suppose decoding happens directly on the primary, because I'm still hoping there's a way to swing that. Suppose further that we handle DDL by insisting that (1) any backend which wants to add columns or change the types of existing columns must first wait for logical replication to catch up and (2) if a backend which has added columns or changed the types of existing columns then writes to the modified table, decoding of those writes will be postponed until transaction commit. I think that's enough to guarantee that the decoding process can just use the catalogs as they stand, with plain old SnapshotNow. What will be the way to make the DDL's wait for logical replication to catch up? One way can be that by looking WAL location till what it has generated logical records. If we do this by holding table level locks (which will be required for DDL's), then other DML's will also get blocked. And If we do this without holding locks and decided to go ahead then there can be other WAL generated by the meantime and can create a problem. Or if there is another way to make DDL's wait? With Regards, Amit Kapila. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_tablespace.spclocation column removed in 9.2
Hello, Pgsql-bugs. According to the Moving tablespaces thread started by Bruce http://archives.postgresql.org/pgsql-docs/2011-12/msg3.php pg_tablespace.spclocation column is removed in the 9.2beta. However this breaks backward compatibility for a bunch of products, e.g. pgAdmin, phpPgAdmin, PgMDD etc. I'm not sure this is the best choice. Because each application with tablespace support will need additional check now to determine what way to use for obtaining tablespace location: pg_get_tablespace_location(oid) or tablespace.spclocation I'm aware of problems caused by this hard coded column. My proposal is to convert pg_tablespace to system view may be? -- With best wishes, Pavel mailto:pa...@gf.microolap.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
