[HACKERS] Clarifications of licences on pgfoundry
I notice that there are more than a few projects on pgfoundry that are marked as BSD licence but then the project files don't contain any mention of the licence details. In some cases, projects are also clearly marked Copyright of people or organizations. For example, pg_batch is clearly marked BSD licence, yet the docs and many of the files are marked Copyright (c) 2010, NIPPON TELEGRAPH AND TELEPHONE CORPORATION. pg_lesslog does contain a BSD-looking licence in the COPYRIGHT file, but is also marked with copyrights. My understanding is that we had a policy of copyright novation to the PGDG. Is that not followed up for pgfoundry projects? I think we should move to a policy of explicit licencing. In the absence of a licence file, when a project is marked BSD licence on pgfoundry I think it is safe to presume that the licence for those files is the same as PostgreSQL's licence. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, May 18, 2010 at 2:26 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 12:02 +0900, Fujii Masao wrote: On Mon, May 17, 2010 at 9:01 PM, Simon Riggs si...@2ndquadrant.com wrote: (1) Smart or fast shutdown requested in PM_STARTUP state always removes the backup_label file if it exists. But it might be still required for subsequent recovery. I changed your patch so that additionally the postmaster skips deleting the backup_label in that case. Don't like the name NeedBackupLabel seems too specific. That really corresponds to we were in recovery. We should have a couple of super-states that correspond to am in recovery/am not in recovery so we can drive it from that. ISTM that we can use XLogCtl-SharedRecoveryInProgress for that. Is this OK? That can change state at any time. Would that work? Yes. XLogCtl-SharedRecoveryInProgress is set to TRUE only when XLogCtl structure is initialized (i.e., XLOGShmemInit), and it's set to FALSE only at the end of recovery. Here is the updated patch (fix_smart_shutdown_in_recovery_v3_fujii.patch). I used XLogCtl-SharedRecoveryInProgress instead of NeedBackupLabel, which made the patch very simple. And I prevented the postmaster from invoking walreceiver after shutdown or recovery. (2) pg_ctl -ms stop emits the following warning whenever there is the backup_label file in $PGDATA. WARNING: online backup mode is active Shutdown will not complete until pg_stop_backup() is called. This warning doesn't fit in with the shutdown during recovery case. Since smart shutdown might be requested by other than pg_ctl, the warning should be emitted in server side rather than client, I think. How about moving the warning to the server side? Though I'm not sure if this should be fixed for 9.0, I attached the patch (move_bkp_cancel_warning_v1.patch). Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center fix_smart_shutdown_in_recovery_v3_fujii.patch Description: Binary data move_bkp_cancel_warning_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, 2010-05-18 at 15:09 +0900, Fujii Masao wrote: On Tue, May 18, 2010 at 2:26 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 12:02 +0900, Fujii Masao wrote: On Mon, May 17, 2010 at 9:01 PM, Simon Riggs si...@2ndquadrant.com wrote: (1) Smart or fast shutdown requested in PM_STARTUP state always removes the backup_label file if it exists. But it might be still required for subsequent recovery. I changed your patch so that additionally the postmaster skips deleting the backup_label in that case. Don't like the name NeedBackupLabel seems too specific. That really corresponds to we were in recovery. We should have a couple of super-states that correspond to am in recovery/am not in recovery so we can drive it from that. ISTM that we can use XLogCtl-SharedRecoveryInProgress for that. Is this OK? That can change state at any time. Would that work? Yes. XLogCtl-SharedRecoveryInProgress is set to TRUE only when XLogCtl structure is initialized (i.e., XLOGShmemInit), and it's set to FALSE only at the end of recovery. You should be using RecoveryInProgress() Here is the updated patch (fix_smart_shutdown_in_recovery_v3_fujii.patch). I used XLogCtl-SharedRecoveryInProgress instead of NeedBackupLabel, which made the patch very simple. And I prevented the postmaster from invoking walreceiver after shutdown or recovery. (2) pg_ctl -ms stop emits the following warning whenever there is the backup_label file in $PGDATA. WARNING: online backup mode is active Shutdown will not complete until pg_stop_backup() is called. This warning doesn't fit in with the shutdown during recovery case. Since smart shutdown might be requested by other than pg_ctl, the warning should be emitted in server side rather than client, I think. How about moving the warning to the server side? Though I'm not sure if this should be fixed for 9.0, I attached the patch (move_bkp_cancel_warning_v1.patch). Regards, -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, May 18, 2010 at 3:24 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 15:09 +0900, Fujii Masao wrote: On Tue, May 18, 2010 at 2:26 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 12:02 +0900, Fujii Masao wrote: On Mon, May 17, 2010 at 9:01 PM, Simon Riggs si...@2ndquadrant.com wrote: (1) Smart or fast shutdown requested in PM_STARTUP state always removes the backup_label file if it exists. But it might be still required for subsequent recovery. I changed your patch so that additionally the postmaster skips deleting the backup_label in that case. Don't like the name NeedBackupLabel seems too specific. That really corresponds to we were in recovery. We should have a couple of super-states that correspond to am in recovery/am not in recovery so we can drive it from that. ISTM that we can use XLogCtl-SharedRecoveryInProgress for that. Is this OK? That can change state at any time. Would that work? Yes. XLogCtl-SharedRecoveryInProgress is set to TRUE only when XLogCtl structure is initialized (i.e., XLOGShmemInit), and it's set to FALSE only at the end of recovery. You should be using RecoveryInProgress() Isn't access to a bool variable atomic? And I think that postmaster should not take any locks since its deadlock would cause a fatal situation. No? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, May 18, 2010 at 6:57 AM, Simon Riggs si...@2ndquadrant.com wrote: I notice that there are more than a few projects on pgfoundry that are marked as BSD licence but then the project files don't contain any mention of the licence details. In some cases, projects are also clearly marked Copyright of people or organizations. I agree that projects should make their licence clear. Gurjeet and I were just talking about this in relation to Slony, which has only a copy of the PostgreSQL licence tucked away in an SGML file in the guts of the tarball, with no text at all to say it's the licence used for Slony, and not just for PG. For example, pg_batch is clearly marked BSD licence, yet the docs and many of the files are marked Copyright (c) 2010, NIPPON TELEGRAPH AND TELEPHONE CORPORATION. Don't mix up copyright and licence. They are not the same thing at all. pg_lesslog does contain a BSD-looking licence in the COPYRIGHT file, but is also marked with copyrights. My understanding is that we had a policy of copyright novation to the PGDG. Is that not followed up for pgfoundry projects? I think we should move to a policy of explicit licencing. No - pgFoundry projects are licenced and copyright-attributed as their authors see fit (as long as it's an open source licence of course). In the absence of a licence file, when a project is marked BSD licence on pgfoundry I think it is safe to presume that the licence for those files is the same as PostgreSQL's licence. The PostgreSQL Licence is not the same as any of the BSD variants, so that is not a safe presumption to make. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, May 18, 2010 at 3:45 PM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, May 18, 2010 at 3:24 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 15:09 +0900, Fujii Masao wrote: On Tue, May 18, 2010 at 2:26 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 12:02 +0900, Fujii Masao wrote: On Mon, May 17, 2010 at 9:01 PM, Simon Riggs si...@2ndquadrant.com wrote: (1) Smart or fast shutdown requested in PM_STARTUP state always removes the backup_label file if it exists. But it might be still required for subsequent recovery. I changed your patch so that additionally the postmaster skips deleting the backup_label in that case. Don't like the name NeedBackupLabel seems too specific. That really corresponds to we were in recovery. We should have a couple of super-states that correspond to am in recovery/am not in recovery so we can drive it from that. ISTM that we can use XLogCtl-SharedRecoveryInProgress for that. Is this OK? That can change state at any time. Would that work? Yes. XLogCtl-SharedRecoveryInProgress is set to TRUE only when XLogCtl structure is initialized (i.e., XLOGShmemInit), and it's set to FALSE only at the end of recovery. You should be using RecoveryInProgress() Isn't access to a bool variable atomic? If it's not atomic, I'll add the following comment into CancelBackup(): Don't bother with lock to access XLogCtl-SharedRecoveryInProgress, because there should be no other processes running when this code is reached. Thought? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, 2010-05-18 at 07:53 +0100, Dave Page wrote: For example, pg_batch is clearly marked BSD licence, yet the docs and many of the files are marked Copyright (c) 2010, NIPPON TELEGRAPH AND TELEPHONE CORPORATION. Don't mix up copyright and licence. They are not the same thing at all. I didn't mix those things up, I just used them in the same sentence. They are two aspects of ownership and appear to offer conflicting messages, which is a concern to some users. pg_lesslog does contain a BSD-looking licence in the COPYRIGHT file, but is also marked with copyrights. My understanding is that we had a policy of copyright novation to the PGDG. Is that not followed up for pgfoundry projects? I think we should move to a policy of explicit licencing. No - pgFoundry projects are licenced and copyright-attributed as their authors see fit (as long as it's an open source licence of course). Yes, are they open source licences? In the absence of a licence file, when a project is marked BSD licence on pgfoundry I think it is safe to presume that the licence for those files is the same as PostgreSQL's licence. The PostgreSQL Licence is not the same as any of the BSD variants, so that is not a safe presumption to make. If, as you say, the licence is unclear then whether-or-not it is an open source licence must also be unclear. The copyright holders can change the licence in future as they see fit, as we've witnessed on other formerly open source projects. Since the licence is unclear now and the future is subject to change, I think its safe to say that those projects are fairly unsafe for open source users. I'm sure the various other Telco companies out there don't want to hear that they are using software that NTT might decide in the future to contest as to whether it was open source or not. Nothing against NTT, though the principle is clear and effects everything on pgfoundry. That puts a fairly large hole in recommending that people visit pgFoundry. That either needs to fixed or users will no longer be able to trust PgFoundry. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, 2010-05-18 at 16:05 +0900, Fujii Masao wrote: ISTM that we can use XLogCtl-SharedRecoveryInProgress for that. Is this OK? That can change state at any time. Would that work? Yes. XLogCtl-SharedRecoveryInProgress is set to TRUE only when XLogCtl structure is initialized (i.e., XLOGShmemInit), and it's set to FALSE only at the end of recovery. You should be using RecoveryInProgress() Isn't access to a bool variable atomic? If it's not atomic, I'll add the following comment into CancelBackup(): Don't bother with lock to access XLogCtl-SharedRecoveryInProgress, because there should be no other processes running when this code is reached. Call it via a function. There is no need for postmaster to know the innards of xlog.c, which could change in future. Modularity. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, May 18, 2010 at 9:06 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 07:53 +0100, Dave Page wrote: For example, pg_batch is clearly marked BSD licence, yet the docs and many of the files are marked Copyright (c) 2010, NIPPON TELEGRAPH AND TELEPHONE CORPORATION. Don't mix up copyright and licence. They are not the same thing at all. I didn't mix those things up, I just used them in the same sentence. They are two aspects of ownership and appear to offer conflicting messages, which is a concern to some users. No, copyright is about ownership. The licence is a right granted by the copyright holders to other to govern their *use* of the code. No - pgFoundry projects are licenced and copyright-attributed as their authors see fit (as long as it's an open source licence of course). Yes, are they open source licences? All the options on pgFoundry are, yes. The PostgreSQL Licence is not the same as any of the BSD variants, so that is not a safe presumption to make. If, as you say, the licence is unclear then whether-or-not it is an open source licence must also be unclear. Not at all. If it's listed on www.opensource.org, then a licence is open source. Why do you think I busted a gut to get the PostgreSQL licence approved when we realised it wasn't BSD? The copyright holders can change the licence in future as they see fit, as we've witnessed on other formerly open source projects. Since the licence is unclear now and the future is subject to change, I think its safe to say that those projects are fairly unsafe for open source users. That is the case for *anything*. We could change the PostgreSQL licence if we wanted, but it would take a huge amount of effort and approval of every contributor ever whose work could be considered an artistic contribution. With PostgreSQL we rely on the sheer number of contributors to ensure the licence will never actually change. We cannot have such a guarantee for most smaller projects of course - simply attributing copyright to a non-existent legal entity such as PGDG (or as I understand it, even an actual entity) doesn't actually change who legally owns the copyright. To get the protection I think you seek, I believe we'd need to create a legal entity to own the copyright and then have every contributor to anything on pgFoundry sign a copyright assignment agreement that grants the legal entity copyright on the current and all future versions of that work, as hosted on there. And even then, there's no guarantee that the legal entity couldn't be bought or change it's charter, unless there's some way to irrevocably build things into its statutes. Of course, as you know I'm not a lawyer but have spent a fair bit of^W^W^Wfar too much time talking to them about this sort of stuff, so I at least *think* I know what I'm talking about :-) -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, May 18, 2010 at 5:10 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, 2010-05-18 at 16:05 +0900, Fujii Masao wrote: ISTM that we can use XLogCtl-SharedRecoveryInProgress for that. Is this OK? That can change state at any time. Would that work? Yes. XLogCtl-SharedRecoveryInProgress is set to TRUE only when XLogCtl structure is initialized (i.e., XLOGShmemInit), and it's set to FALSE only at the end of recovery. You should be using RecoveryInProgress() Isn't access to a bool variable atomic? If it's not atomic, I'll add the following comment into CancelBackup(): Don't bother with lock to access XLogCtl-SharedRecoveryInProgress, because there should be no other processes running when this code is reached. Call it via a function. There is no need for postmaster to know the innards of xlog.c, which could change in future. Modularity. In the patch, it's accessed in CancelBackup() which is in xlog.c. CancelBackup() should call further wrapping function? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, 2010-05-18 at 09:33 +0100, Dave Page wrote: No - pgFoundry projects are licenced and copyright-attributed as their authors see fit (as long as it's an open source licence of course). Yes, are they open source licences? All the options on pgFoundry are, yes. The PostgreSQL Licence is not the same as any of the BSD variants, so that is not a safe presumption to make. If, as you say, the licence is unclear then whether-or-not it is an open source licence must also be unclear. Not at all. If it's listed on www.opensource.org, then a licence is open source. Why do you think I busted a gut to get the PostgreSQL licence approved when we realised it wasn't BSD? Dave, this is important and so this thread must have a clear resolution, so we must stick to a single point and be clear about our logic and our statements. You're saying these two things, I think, or if you or anybody else disagrees, please so clearly. * When project realised that the PostgreSQL licence wasn't actually a BSD licence, that PostgreSQL was clarified to be the TPL, yet pgfoundry was not covered by that clarification for some reason. * In the absence of any licence text in any of the files of a project on a certain date, then if the project is advertised on PgFoundry on that date as having a BSD licence then the software will be covered by http://www.opensource.org/licenses/bsd-license.php -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Partitioning/inherited tables vs FKs
On Thu, May 6, 2010 at 10:38 AM, Tom Lane t...@sss.pgh.pa.us wrote: * the index grows as the size of the total data set, it's not limited by partition size * can't cheaply drop one partition any more, you have to vacuum the (big) index first So I wholeheartedly agree with the general sentiment that if you need global indexes then partitioning just isn't really the right tool for you. But it occurs to me that we could defer the vacuum safely. I'm assuming a index heap-tid pointer for a global index would include a relid or some other identifier to specify which partition the tuple is in. If you drop that partition those can all just be left as dangling pointers as long as we don't reuse that id. So all we would need is some way to leave a catalog entry reserving that id. The data files can be truncated and deleted normally and whenever vacuum does run against the index it can clean up the catalog entries for the deleted partitions. But I would rather work on having unique and foreign key constraints that work on keys which include the partition key than work on global indexes. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, May 18, 2010 at 4:06 AM, Simon Riggs si...@2ndquadrant.com wrote: If, as you say, the licence is unclear then whether-or-not it is an open source licence must also be unclear. I would suggest you, or anyone else who notices, open bugs on any packages you want to use for which you find no LICENSE file matching the license asserted in pgfoundry. Are there so many that we need a more organized mass effort? Do we need automated checks for this? The copyright holders can change the licence in future as they see fit, as we've witnessed on other formerly open source projects. This is always true. The protection open source licenses have for this is that they're irrevocable. So while NTT could stop releasing future work under an open source license, the code which was already released would still be available under the license it was released under and anyone who wants to could pay anyone willing to support it without asking NTT for permission. The question that arises then is whether pgfoundry archives the source it has in a way that the project maintainer can't delete. If an author decides to stop releasing a package and deletes the source from pgfoundry can we get the last version they released from pgfoundry and put it back up as an orphaned project or with a new set of maintainers? As long as we have the infrastructure to do that conveniently I think we're protected against this danger. -- greg -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, May 18, 2010 at 9:59 AM, Simon Riggs si...@2ndquadrant.com wrote: Dave, this is important and so this thread must have a clear resolution, so we must stick to a single point and be clear about our logic and our statements. OK. I thought you were talking about copyright and licences though. You're saying these two things, I think, or if you or anybody else disagrees, please so clearly. * When project realised that the PostgreSQL licence wasn't actually a BSD licence, that PostgreSQL was clarified to be the TPL, yet pgfoundry was not covered by that clarification for some reason. No. The licences never changed on anything - all we did was get it approved by the OSI, and clarify our *naming* of the licence in PostgreSQL (and pgAdmin). It's entirely up to the maintainers of each project on pgFoundry to decide whether the licence text or the licence name is what they intend, and to carify accordingly for their projects. * In the absence of any licence text in any of the files of a project on a certain date, then if the project is advertised on PgFoundry on that date as having a BSD licence then the software will be covered by http://www.opensource.org/licenses/bsd-license.php Yes, I believe that is a fair and safe assumption. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, 2010-05-18 at 06:32 -0400, Greg Stark wrote: On Tue, May 18, 2010 at 4:06 AM, Simon Riggs si...@2ndquadrant.com wrote: If, as you say, the licence is unclear then whether-or-not it is an open source licence must also be unclear. I would suggest you, or anyone else who notices, open bugs on any packages you want to use for which you find no LICENSE file matching the license asserted in pgfoundry. I'm not personally going to do this, though I will point out to people the dangers of imprecisely licenced software when they ask. Are there so many that we need a more organized mass effort? Do we need automated checks for this? I would say so. The copyright holders can change the licence in future as they see fit, as we've witnessed on other formerly open source projects. This is always true. The protection open source licenses have for this is that they're irrevocable. So while NTT could stop releasing future work under an open source license, the code which was already released would still be available under the license it was released under and anyone who wants to could pay anyone willing to support it without asking NTT for permission. The question that arises then is whether pgfoundry archives the source it has in a way that the project maintainer can't delete. If an author decides to stop releasing a package and deletes the source from pgfoundry can we get the last version they released from pgfoundry and put it back up as an orphaned project or with a new set of maintainers? As long as we have the infrastructure to do that conveniently I think we're protected against this danger. Well, whoever runs pgfoundry.org gets to make that decision. They may choose how they respond if someone says I request X, in the name of PostgreSQL and open source, There may or may not keep archived copies. If they just keep a latest backup, then once the developer quietly deletes stuff then its gone forever. Who could monitor that to make sure it never takes place?? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
That is the case for *anything*. We could change the PostgreSQL licence if we wanted, but it would take a huge amount of effort and approval of every contributor ever whose work could be considered an artistic contribution. I doubt it. Do you think that every single contributor is contactable? Haven't some died? My guess is that it would be completely impossible. With PostgreSQL we rely on the sheer number of contributors to ensure the licence will never actually change. We cannot have such a guarantee for most smaller projects of course - simply attributing copyright to a non-existent legal entity such as PGDG (or as I understand it, even an actual entity) doesn't actually change who legally owns the copyright. To get the protection I think you seek, I believe we'd need to create a legal entity to own the copyright and then have every contributor to anything on pgFoundry sign a copyright assignment agreement that grants the legal entity copyright on the current and all future versions of that work, as hosted on there. And even then, there's no guarantee that the legal entity couldn't be bought or change it's charter, unless there's some way to irrevocably build things into its statutes. IANAL, but I know that there was a similar situation when Trolltech still existed and controlled the Qt framework. It was dual licensed GPL2/proprietary (it is now dual LGPL/proprietary). Contributors were required to sign reams of paperwork, which had to be sent out by fax (I'm not sure why), to assign the copyright to Trolltech. Thankfully, that situation has changed under Nokia - contributors retain the copyright, and there is minimal red tape. Contributors are now asked to grant Qt Software a non-exclusive right to re-use code as a part of Qt, the first time they submit code for inclusion. Regards, Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication patch built on SR
Thanks for your reply! On Fri, May 14, 2010 at 10:33 PM, Boszormenyi Zoltan z...@cybertec.at wrote: In your design, the transaction commit on the master waits for its XID to be read from the XLOG_XACT_COMMIT record and replied by the standby. Right? This design seems not to be extensible to #2 and #3 since walreceiver cannot read XID from the XLOG_XACT_COMMIT record. Yes, this was my problem, too. I would have had to implement a custom interpreter into walreceiver to process the WAL records and extract the XIDs. Isn't reading the same WAL twice (by walreceiver and startup process) inefficient? In synchronous replication, the overhead of walreceiver directly affects the performance of the master. We should not assign such a hard work to walreceiver, I think. But at least the supporting details, i.e. not opening another connection, instead being able to do duplex COPY operations in a server-acknowledged way is acceptable, no? :-) Though I might not understand your point (sorry), it's OK for the standby to send the reply to the master by using CopyData message. Currently PQputCopyData() cannot be executed in COPY OUT, but we can relax that. How about using LSN instead of XID? That is, the transaction commit waits until the standby has reached its LSN. LSN is more easy-used for walreceiver and startup process, I think. Indeed, using the LSN seems to be more appropriate for the walreceiver, but how would you extract the information that a certain LSN means a COMMITted transaction? Or we could release a locked transaction in case the master receives an LSN greater than or equal to the transaction's own LSN? Yep, we can ensure that the transaction has been replicated by comparing its own LSN with the smallest LSN in the latest LSNs of each connected synchronous standby. Sending back all the LSNs in case of long transactions would increase the network traffic compared to sending back only the XIDs, but the amount is not clear for me. What I am more worried about is the contention on the ProcArrayLock. XIDs are rarer then LSNs, no? No. For example, when WAL data sent by walsender at a time has two XLOG_XACT_COMMIT records, in XID approach, walreceiver would need to send two replies. OTOH, in LSN approach, only one reply which indicates the last received location would need to be sent. What if the synchronous standby starts up from the very old backup? The transaction on the master needs to wait until a large amount of outstanding WAL has been applied? I think that synchronous replication should start with *asynchronous* replication, and should switch to the sync level after the gap between servers has become enough small. What's your opinion? It's certainly one option, which I think partly addressed with the strict_sync_replication knob below. If strict_sync_replication = off, then the master doesn't make its transactions wait for the synchronous reports, and the client(s) can work through their WALs. IIRC, the walreceiver connects to the master only very late in the recovery process, no? No, the master might have a large number of WAL files which the standby doesn't have. I have added 3 new options, two GUCs in postgresql.conf and one setting in recovery.conf. These options are: 1. min_sync_replication_clients = N where N is the number of reports for a given transaction before it's released as committed synchronously. 0 means completely asynchronous, the value is maximized by the value of max_wal_senders. Anything in between 0 and max_wal_senders means different levels of partially synchronous replication. 2. strict_sync_replication = boolean where the expected number of synchronous reports from standby servers is further limited to the actual number of connected synchronous standby servers if the value of this GUC is false. This means that if no standby servers are connected yet then the replication is asynchronous and transactions are allowed to finish without waiting for synchronous reports. If the value of this GUC is true, then transactions wait until enough synchronous standbys connect and report back. Why are these options necessary? Can these options cover more than three synchronization levels? I think I explained it in my mail. If min_sync_replication_clients == 0, then the replication is async. If min_sync_replication_clients == max_wal_senders then the replication is fully synchronous. If 0 min_sync_replication_clients max_wal_senders then the replication is partially synchronous, i.e. the master can wait only for say, 50% of the clients to report back before it's considered synchronous and the relevant transactions get released from the wait. Seems s/min_sync_replication_clients/max_sync_replication_clients min_sync_replication_clients is required to prevent outside attacker from connecting to the master as synchronous standby, and degrading the performance on the master? Other usecase? Regards, -- Fujii
Re: [HACKERS] Clarifications of licences on pgfoundry
Simon Riggs wrote: That puts a fairly large hole in recommending that people visit pgFoundry. That either needs to fixed or users will no longer be able to trust PgFoundry. pgFoundry is a resource we provide the community. The projects there are the responsibility of their individual owners. We are not going to start being the license police. I at least have neither the time to do that nor any interest in doing it. If people want to use what is on pgFoundry then it is up to them to make sure it has whatever licence meets their requirements. What we should do is add the PostgreSQL license to the list of available licenses and make sure it is the default for new projects. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Mon, May 17, 2010 at 10:40 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, May 17, 2010 at 10:20 PM, Robert Haas robertmh...@gmail.com wrote: OK, I think I understand now. But, the SIGTERM sent by the postmaster doesn't kill the recovery process unconditionally. It will invoke StartupProcShutdownHandler(), which will set set shutdown_requested = true. That gets checked by RestoreArchivedFile() and HandleStartupProcInterrupts(), and I think that neither of those can get invoked until after the control file has been updated. Do you see a way it can happen? Yeah, the way is: StartupXLOG() -- ReadCheckpointRecord() -- ReadRecord() -- XLogPageRead() -- XLogFileReadAnyTLI() -- XLogFileRead() -- RestoreArchivedFile() ReadCheckpointRecord() is called before pg_control is updated. OK. In that case, I'm wondering if we should reverse course and rejigger the logic so that the shutdown gets processed when we transition to PM_RECOVERY. Seems like that might be simpler. ISTM that walreceiver might be invoked even after shutdown is requested. We should prevent the postmaster from starting up walreceiver if Shutdown NoShutdown? Well, when we did the previous shutdown patch, we decided it was not right to kill walreceiver until all backends had exited, so it seems inconsistent to make the opposite decision here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication patch built on SR
On Sat, May 15, 2010 at 4:59 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: BTW, What I'd like to see as a very first patch first is to change the current poll loops in walreceiver and walsender to, well, not poll. That's a requirement for synchronous replication, is very useful on its own, and requires a some design and implementation effort to get right. It would be nice to get that out of the way before/during we discuss the more user-visible behavior. Yeah, we should wake up the walesender from sleep to send WAL data as soon as it's flushed. But why do we need to change the loop of walreceiver? Or you mean changing the poll loop in the startup process? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
Hi I tried running pg_upgrade from the current snapshot of postgresql and upgrading from 8.4.4 to the snapshot version. Everything seem to look fine in the process and all that came out was only ok's but when I tried a simple query on the databse it keeps throwing these message out of the back side. DETAIL: You might have already suffered transaction-wraparound data loss. WARNING: some databases have not been vacuumed in over 2 billion transactions The database was around 600GB and it took a couple of minutes to run pg_upgrade after I had all the binaries in the correct place. It is not really an easy task to throw around 600GB of data, so I cannot gaurantee that the above is reproducible, but I'll see if I can get time and try to reproduce it. Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On 05/18/2010 07:32 AM, Andrew Dunstan wrote: Simon Riggs wrote: That puts a fairly large hole in recommending that people visit pgFoundry. That either needs to fixed or users will no longer be able to trust PgFoundry. pgFoundry is a resource we provide the community. The projects there are the responsibility of their individual owners. We are not going to start being the license police. I at least have neither the time to do that nor any interest in doing it. If people want to use what is on pgFoundry then it is up to them to make sure it has whatever licence meets their requirements. I agree there - pgfoundry is just the resource provider, we are not a licence police (and given that none of the pgf admins is an actual lawyer there is no sense in even trying). People wanting to get some sort of indemnification or whatever need to look into commercial providers (or use distribution provided packages for stuff because those are usually very well checked for licence stuff in all major linux distributions). What we should do is add the PostgreSQL license to the list of available licenses and make sure it is the default for new projects. I can look into that... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Clarifications of licences on pgfoundry
On Tue, 2010-05-18 at 07:32 -0400, Andrew Dunstan wrote: Simon Riggs wrote: That puts a fairly large hole in recommending that people visit pgFoundry. That either needs to fixed or users will no longer be able to trust PgFoundry. pgFoundry is a resource we provide the community. The projects there are the responsibility of their individual owners. We are not going to start being the license police. I at least have neither the time to do that nor any interest in doing it. If people want to use what is on pgFoundry then it is up to them to make sure it has whatever licence meets their requirements. Agreed, though that significantly lessens the value of that resource for everybody. If somebody would like to try to improve that by attempting to improve or police the licencing, it would be appreciated. What we should do is add the PostgreSQL license to the list of available licenses and make sure it is the default for new projects. Good idea. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Bug with ordering aggregates?
Greetings, This doesn't seem right to me: postgres=# select postgres-# string_agg(column1::text order by column1 asc,',') postgres-# from (values (3),(4),(1),(2)) a; string_agg 1234 (1 row) I'm thinking we should toss a syntax error here and force the 'order by' to be at the end of any arguments to the aggregate. Alternatively, we should actually make this work like this one does: postgres=# select postgres-# string_agg(column1::text,',' order by column1 asc) postgres-# from (values (3),(4),(1),(2)) a; string_agg 1,2,3,4 (1 row) Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Bug with ordering aggregates?
Stephen Frost sfr...@snowman.net writes: This doesn't seem right to me: postgres=# select postgres-# string_agg(column1::text order by column1 asc,',') postgres-# from (values (3),(4),(1),(2)) a; string_agg 1234 (1 row) Looks fine to me: you have two ordering columns (the second rather useless, but that's no matter). regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug with ordering aggregates?
On 18 May 2010 16:37, Stephen Frost sfr...@snowman.net wrote: Greetings, This doesn't seem right to me: postgres=# select postgres-# string_agg(column1::text order by column1 asc,',') postgres-# from (values (3),(4),(1),(2)) a; string_agg 1234 (1 row) I'm thinking we should toss a syntax error here and force the 'order by' to be at the end of any arguments to the aggregate. Alternatively, we should actually make this work like this one does: postgres=# select postgres-# string_agg(column1::text,',' order by column1 asc) postgres-# from (values (3),(4),(1),(2)) a; string_agg 1,2,3,4 (1 row) I find that 2nd example confusing. It suggests the delimiter is being ordered as the order by clause appears in its parameter. But I can see why the first one is returning the wrong result. The order by clause conflicts with the delimiter parameter as obviously the order by clause prevents you specifying a 2nd parameter in the aggregate function. The delimiter would either need to be the first parameter, or the order by clause would require a way to terminate it's order by list. Thom -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug with ordering aggregates?
* Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: This doesn't seem right to me: postgres=# select postgres-# string_agg(column1::text order by column1 asc,',') postgres-# from (values (3),(4),(1),(2)) a; string_agg 1234 (1 row) Looks fine to me: you have two ordering columns (the second rather useless, but that's no matter). Ah, yeah, guess I'll just complain that having the order by look like it's an argument to an aggregate makes things confusing. Not much to be done about it though. Thanks, Stephen signature.asc Description: Digital signature
[HACKERS] Documentation Bug/Misnomer?
Greetings, Under: http://developer.postgresql.org/pgdocs/postgres/runtime-config-file-locations.html We have: ident_file (string) Specifies the configuration file for ident authentication (customarily called pg_ident.conf). This parameter can only be set at server start. That's not really accurate anymore, is it? It's referring to the username maps now, which are used for Ident, GSSAPI, SSL, etc.. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
jes...@krogh.cc wrote: Hi I tried running pg_upgrade from the current snapshot of postgresql and upgrading from 8.4.4 to the snapshot version. Everything seem to look fine in the process and all that came out was only ok's but when I tried a simple query on the databse it keeps throwing these message out of the back side. DETAIL: You might have already suffered transaction-wraparound data loss. WARNING: some databases have not been vacuumed in over 2 billion transactions The database was around 600GB and it took a couple of minutes to run pg_upgrade after I had all the binaries in the correct place. It is not really an easy task to throw around 600GB of data, so I cannot gaurantee that the above is reproducible, but I'll see if I can get time and try to reproduce it. This certainly should never have happened, so I am guessing it is a bug. pg_upgrade tries hard to make sure all your datfrozenxid and relfrozenxid are properly migrated from the old server, and the transaction id is set properly. Unfortunately this is the first time I have heard of such a problem, so I am unclear on its cause. The warning is issued from vacuum.c::vac_truncate_clog(). Can you run this query and show us the output: SELECT datname, datfrozenxid FROM pg_database; It would be good to see these numbers on both the old and new servers. I would also like to see: SELECT txid_current(); on the old and new servers, but if you can only provide these values on one of the two servers, it is still useful. Thanks. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
jes...@krogh.cc wrote: Hi I tried running pg_upgrade from the current snapshot of postgresql and upgrading from 8.4.4 to the snapshot version. Everything seem to look fine in the process and all that came out was only ok's but when I tried a simple query on the databse it keeps throwing these message out of the back side. DETAIL: You might have already suffered transaction-wraparound data loss. WARNING: some databases have not been vacuumed in over 2 billion transactions The database was around 600GB and it took a couple of minutes to run pg_upgrade after I had all the binaries in the correct place. It is not really an easy task to throw around 600GB of data, so I cannot gaurantee that the above is reproducible, but I'll see if I can get time and try to reproduce it. This certainly should never have happened, so I am guessing it is a bug. pg_upgrade tries hard to make sure all your datfrozenxid and relfrozenxid are properly migrated from the old server, and the transaction id is set properly. Unfortunately this is the first time I have heard of such a problem, so I am unclear on its cause. The warning is issued from vacuum.c::vac_truncate_clog(). Can you run this query and show us the output: SELECT datname, datfrozenxid FROM pg_database; It would be good to see these numbers on both the old and new servers. I would also like to see: SELECT txid_current(); on the old and new servers, but if you can only provide these values on one of the two servers, it is still useful. Thanks. Hi Bruce, thanks for your prompt response. First the new one.. j...@pal:~$ psql -p 5433 psql (9.0beta1) Type help for help. data=# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 654 postgres | 2374592801 data | 2023782337 jk| 2023822188 template1 | 2374592801 workqueue | 2023822188 (6 rows) data=# SELECT txid_current(); txid_current -- 2375384556 (1 row) data=# \q Then the old one. j...@pal:~$ psql data psql (9.0beta1, server 8.4.1) WARNING: psql version 9.0, server version 8.4. Some psql features might not work. Type help for help. data# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 2073823552 postgres | 2023820521 data | 2023782337 jk| 2023822188 template1 | 2073823552 workqueue | 2023822188 (6 rows) data=# SELECT txid_current(); txid_current -- 2390524243 (1 row) The old database has been copied over using rsync and pg_start_backup()/pg_stop_backup() procecures and started up using a recovery.conf file. Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Unexpected data beyond EOF during heavy writes
Hello everyone, We are seeing the following error message occasionally in the postgres logs: 2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation custom_discoveryprofile 2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system. I have read several threads on this, including: http://archives.postgresql.org/pgsql-general/2007-03/msg01535.php and http://archives.postgresql.org/pgsql-general/2009-07/msg01011.php I am wondering if anyone has ever placed a bug report to any of the linux vendors or open source sites? I could not find a bug report that matched this at http://bugzilla.redhat.com/ and am trying to track down what is needed to resolve the error. We are running a an IBM Blade Center server 64 bit Red Hat 5.3 kernel with 16 cores doing heavy I/O to the database. uname -a Linux 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux x86_64 x86_64 x86_64 GNU/Linux Thanks for any pointers, Tony Sullivan
Re: [HACKERS] Clarifications of licences on pgfoundry
On 05/18/2010 09:22 AM, Simon Riggs wrote: On Tue, 2010-05-18 at 07:32 -0400, Andrew Dunstan wrote: Simon Riggs wrote: That puts a fairly large hole in recommending that people visit pgFoundry. That either needs to fixed or users will no longer be able to trust PgFoundry. pgFoundry is a resource we provide the community. The projects there are the responsibility of their individual owners. We are not going to start being the license police. I at least have neither the time to do that nor any interest in doing it. If people want to use what is on pgFoundry then it is up to them to make sure it has whatever licence meets their requirements. Agreed, though that significantly lessens the value of that resource for everybody. If somebody would like to try to improve that by attempting to improve or police the licencing, it would be appreciated. even if somebody steps up and tries to to that - we have hundreds of projects on pgf and I think it is impossible to do anything that would actually provide some sort of guarantee that the licence stuff is properly done fore every project which is the only thing that would prevent you to do your own research or evaluation. However it makes sense to the the projects you where you ran into an issue about so it can be fixed (technically this is simply a bug that needs to be reported). What we should do is add the PostgreSQL license to the list of available licenses and make sure it is the default for new projects. Good idea. done Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation Bug/Misnomer?
On Tue, May 18, 2010 at 12:49 PM, Stephen Frost sfr...@snowman.net wrote: Greetings, Under: http://developer.postgresql.org/pgdocs/postgres/runtime-config-file-locations.html We have: ident_file (string) Specifies the configuration file for ident authentication (customarily called pg_ident.conf). This parameter can only be set at server start. That's not really accurate anymore, is it? It's referring to the username maps now, which are used for Ident, GSSAPI, SSL, etc.. That definitely looks wrong. How about this? diff -c -r1.276 config.sgml *** config.sgml 3 May 2010 10:31:29 - 1.276 --- config.sgml 18 May 2010 18:44:30 - *** *** 242,249 /indexterm listitem para ! Specifies the configuration file for ! applicationident/ authentication (customarily called filenamepg_ident.conf/). This parameter can only be set at server start. /para --- 242,248 /indexterm listitem para ! Specifies the configuration file for username mapping (customarily called filenamepg_ident.conf/). This parameter can only be set at server start. /para -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation Bug/Misnomer?
* Magnus Hagander (mag...@hagander.net) wrote: That definitely looks wrong. How about this? Looks good to me, but I'd add a link to the Username Maps page. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
jes...@krogh.cc wrote: on the old and new servers, but if you can only provide these values on one of the two servers, it is still useful. Thanks. Hi Bruce, thanks for your prompt response. First the new one.. Great. j...@pal:~$ psql -p 5433 psql (9.0beta1) Type help for help. data=# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 654 postgres | 2374592801 data | 2023782337 jk| 2023822188 template1 | 2374592801 workqueue | 2023822188 (6 rows) data=# SELECT txid_current(); txid_current -- 2375384556 (1 row) data=# \q Then the old one. j...@pal:~$ psql data psql (9.0beta1, server 8.4.1) WARNING: psql version 9.0, server version 8.4. Some psql features might not work. Type help for help. data# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 2073823552 This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid, datallowconn FROM pg_database; I am wondering if you set datallowconn for template0 to 'true'. postgres | 2023820521 data | 2023782337 jk| 2023822188 template1 | 2073823552 workqueue | 2023822188 (6 rows) data=# SELECT txid_current(); txid_current -- 2390524243 (1 row) The old database has been copied over using rsync and pg_start_backup()/pg_stop_backup() procecures and started up using a recovery.conf file. My other idea is that somehow recovery touches datallowconn for template0. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Documentation Bug/Misnomer?
On Tue, May 18, 2010 at 2:46 PM, Stephen Frost sfr...@snowman.net wrote: * Magnus Hagander (mag...@hagander.net) wrote: That definitely looks wrong. How about this? Looks good to me, but I'd add a link to the Username Maps page. Done and applied. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
On 2010-05-18 20:52, Bruce Momjian wrote: This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid, datallowconn FROM pg_database; Only from the old database: data=# SELECT datname, datfrozenxid, datallowconn FROM pg_database; datname | datfrozenxid | datallowconn ---+--+-- template0 | 2073823552 | f postgres | 2023820521 | t data | 2023782337 | t jk| 2023822188 | t template1 | 2073823552 | t workqueue | 2023822188 | t (6 rows) I am wondering if you set datallowconn for template0 to 'true'. From this database, I cannot give any more results, I ran some other queries and then restarted postgres, subsequently it seemed totally broken. I'm in the process of running the test over again, but it'll take a while before data is in. I'll report back. -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
Jesper Krogh wrote: On 2010-05-18 20:52, Bruce Momjian wrote: This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid, datallowconn FROM pg_database; Only from the old database: data=# SELECT datname, datfrozenxid, datallowconn FROM pg_database; datname | datfrozenxid | datallowconn ---+--+-- template0 | 2073823552 | f postgres | 2023820521 | t data | 2023782337 | t jk| 2023822188 | t template1 | 2073823552 | t workqueue | 2023822188 | t (6 rows) OK, datallowconn = false is right for template0, but I am still confused how it got set to that high value. I am wondering if you set datallowconn for template0 to 'true'. From this database, I cannot give any more results, I ran some other queries and then restarted postgres, subsequently it seemed totally broken. I'm in the process of running the test over again, but it'll take a while before data is in. I'll report back. OK, thanks. This does seem odd. Frankly, having template0's datfrozenxid be wrong would not cause any kind of instability because template0 is used only by pg_dump, so I am wondering if something else is seriously wrong. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Synchronous replication patch built on SR
On 18/05/10 07:41, Fujii Masao wrote: On Sat, May 15, 2010 at 4:59 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: BTW, What I'd like to see as a very first patch first is to change the current poll loops in walreceiver and walsender to, well, not poll. That's a requirement for synchronous replication, is very useful on its own, and requires a some design and implementation effort to get right. It would be nice to get that out of the way before/during we discuss the more user-visible behavior. Yeah, we should wake up the walesender from sleep to send WAL data as soon as it's flushed. But why do we need to change the loop of walreceiver? Or you mean changing the poll loop in the startup process? Yeah, changing the poll loop in the startup process is what I meant. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
jes...@krogh.cc wrote: First the new one.. j...@pal:~$ psql -p 5433 psql (9.0beta1) Type help for help. data=# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 654 postgres | 2374592801 data | 2023782337 jk| 2023822188 template1 | 2374592801 workqueue | 2023822188 (6 rows) data=# SELECT txid_current(); txid_current -- 2375384556 (1 row) I just ran a test and all the datfrozenxids are less than the current xid, so the only database that could be generating a wraparound warning is 'template0'. But, again, I though that template0 was not touched for wraparound protection --- I am starting to think I am wrong. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
Excerpts from jesper's message of mar may 18 13:22:12 -0400 2010: j...@pal:~$ psql data psql (9.0beta1, server 8.4.1) WARNING: psql version 9.0, server version 8.4. Some psql features might not work. Type help for help. data# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 2073823552 postgres | 2023820521 data | 2023782337 jk| 2023822188 template1 | 2073823552 workqueue | 2023822188 (6 rows) Does the old server have pg_database.datallowconn = true for template0? -- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
Alvaro Herrera wrote: Excerpts from jesper's message of mar may 18 13:22:12 -0400 2010: j...@pal:~$ psql data psql (9.0beta1, server 8.4.1) WARNING: psql version 9.0, server version 8.4. Some psql features might not work. Type help for help. data# SELECT datname, datfrozenxid FROM pg_database; datname | datfrozenxid ---+-- template0 | 2073823552 postgres | 2023820521 data | 2023782337 jk| 2023822188 template1 | 2073823552 workqueue | 2023822188 (6 rows) Does the old server have pg_database.datallowconn = true for template0? The user reported back that it did not: Only from the old database: data=# SELECT datname, datfrozenxid, datallowconn FROM pg_database; datname | datfrozenxid | datallowconn ---+--+-- template0 | 2073823552 | f postgres | 2023820521 | t data | 2023782337 | t jk| 2023822188 | t template1 | 2073823552 | t workqueue | 2023822188 | t (6 rows) -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Unexpected data beyond EOF during heavy writes
Excerpts from Tony Sullivan's message of mar may 18 13:19:13 -0400 2010: Hello everyone, We are seeing the following error message occasionally in the postgres logs: 2010-05-13 23:49:03 PDT ERROR: unexpected data beyond EOF in block 4106698 of relation custom_discoveryprofile 2010-05-13 23:49:03 PDT HINT: This has been seen to occur with buggy kernels; consider updating your system. What's your storage? -- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] fillfactor gets set to zero for toast tables
Excerpts from Tom Lane's message of vie may 14 15:03:57 -0400 2010: Maybe a better solution is to have some kind of notion of a default-only entry, which is sufficient to insert the default into the struct but isn't accepted as a user-settable item. This patch (for 8.4, but applies fuzzily to 9.0) implements this idea. Note that there's no explicit check that every heap option has a corresponding toast option; that's left to the developer's judgement to add. I added the new member to relopt_gen struct so that existing entries did not require changes in initializers. (I'll leave the error as ERRCODE_CANT_CHANGE_RUNTIME_PARAM unless someone thinks ERRCODE_INVALID_PARAMETER_VALUE is better) I checked and this is the only heap setting that did not have a toast setting. I'll leave this open to comments for a bit more time than usual, to give room for pgcon beers and such ... -- relopt-toast-ff.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] BYTEA / DBD::Pg change in 9.0 beta
Hi.
I'm trying to do a test move of one of our applications onto 9.0beta1.
We use storable and serializes data into a bytea column in the database.
This script uses that:
#!/usr/bin/perl
use strict;
use warnings;
use Storable;
use DBI;
use DBD::Pg;
use Data::Dumper;
my $dbh = DBI-connect(dbi:Pg:dbname=testdb,,,{AutoCommit = 1});
my $sql = END
create table testtable (id serial, testbytea bytea);
END
;
eval {
$dbh-do($sql);
};
$dbh-do(delete from testtable);
my $href = { this = 1, that = 2};
print Before: . Dumper($href) . \n;
my $sth = $dbh-prepare(insert into testtable (testbytea) values (?));
my $frozen = Storable::nfreeze($href);
$sth-bind_param(1, $frozen, { pg_type=DBD::Pg::PG_BYTEA });
$sth-execute;
$sth = $dbh-prepare(select testbytea from testtable);
$sth-execute();
my $row = $sth-fetchrow_hashref();
my $href2 = Storable::thaw($row-{testbytea});
print Dumper($href2);
Running it against 8.4 gives:
$ perl bin/test-bytea
NOTICE: CREATE TABLE will create implicit sequence testtable_id_seq1
for serial column testtable.id
DBD::Pg::db do failed: ERROR: relation testtable already exists at
bin/efam/test-bytea line 16.
Before: $VAR1 = {
'that' = '2',
'this' = '1'
};
$VAR1 = {
'that' = '2',
'this' = '1'
};
Whereas 9.0beta1 gives:
$ perl bin/test-bytea
NOTICE: CREATE TABLE will create implicit sequence testtable_id_seq1
for serial column testtable.id
DBD::Pg::db do failed: ERROR: relation testtable already exists at
bin/efam/test-bytea line 16.
Before: $VAR1 = {
'that' = '2',
'this' = '1'
};
Storable binary image v60.48 more recent than I am (v2.7) at
../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al)
line 366, at bin/test-bytea line 28
Inspecting the data seems that it is the insert that does something to
the data:
8.4
id | testbytea
+
9 |
\005\007\003\000\000\000\002\012\0012\000\000\000\004that\012\0011\000\000\000\004this
(1 row)
9.0beta1
id | testbytea
+--
3 | \x05070300020a01320004746861740a0131000474686973
(1 row)
Jesper
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalive for max_standby_delay
On 17/05/10 04:40, Simon Riggs wrote: On Sun, 2010-05-16 at 16:53 +0100, Simon Riggs wrote: Attached patch rearranges the walsender loops slightly to fix the above. XLogSend() now only sends up to MAX_SEND_SIZE bytes (== XLOG_SEG_SIZE / 2) in one round and returns to the main loop after that even if there's unsent WAL, and the main loop no longer sleeps if there's unsent WAL. That way the main loop gets to respond to signals quickly, and we also get to update the shared memory status and PS display more often when there's a lot of catching up to do. Comments 8MB at a time still seems like a large batch to me. libpq is going to send it in smaller chunks anyway, so I don't see the importance of trying to keep the batch too large. It just introduces delay into the sending process. We should be sending chunks that matches libpq better. More to the point the logic will fail if XLOG_BLCKSZ PQ_BUFFER_SIZE because it will send partial pages. I don't see a failure. We rely on not splitting WAL records across messages, but we're talking about libpq-level CopyData messages, not TCP messages. Having MAX_SEND_SIZE PQ_BUFFER_SIZE is pointless, as libpq currently stands. Well, it does affect the size of the read() in walsender, and I'm sure there's some overhead in setting the ps display and the other small stuff we do once per message. But you're probably right that we could easily make MAX_SEND_SIZE much smaller with no noticeable affect on performance, while making walsender more responsive to signals. I'll decrease it to, say, 512 kB. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
On 2010-05-18 21:56, Bruce Momjian wrote: Jesper Krogh wrote: On 2010-05-18 20:52, Bruce Momjian wrote: This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid, datallowconn FROM pg_database; Only from the old database: data=# SELECT datname, datfrozenxid, datallowconn FROM pg_database; datname | datfrozenxid | datallowconn ---+--+-- template0 | 2073823552 | f postgres | 2023820521 | t data | 2023782337 | t jk| 2023822188 | t template1 | 2073823552 | t workqueue | 2023822188 | t (6 rows) OK, datallowconn = false is right for template0, but I am still confused how it got set to that high value. This is the production system. I have absolutely no indications that anything should be wrong in there. It has run rock-solid since it got migrated (dump/restore) to 8.4 for about 7 months now. So I am a bit scared about you telling that it seems wrong. (but that cannot be attributed to pg_upgrade) OK, thanks. This does seem odd. Frankly, having template0's datfrozenxid be wrong would not cause any kind of instability because template0 is used only by pg_dump, so I am wondering if something else is seriously wrong. I also think that something was seriously wrong with the pg_upgrade'd version. I'll try to reproduce and be a bit more carefull in tracking the steps this time. -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalive for max_standby_delay
On 17/05/10 12:36, Jim Nasby wrote: On May 15, 2010, at 12:05 PM, Heikki Linnakangas wrote: What exactly is the user trying to monitor? If it's how far behind is the standby, the difference between pg_current_xlog_insert_location() in the master and pg_last_xlog_replay_location() in the standby seems more robust and well-defined to me. It's a measure of XLOG location (ie. bytes) instead of time, but time is a complicated concept. I can tell you that end users *will* want a time-based indication of how far behind we are. DBAs will understand we're this many transactions behind, but managers and end users won't. Unless it's unreasonable to provide that info, we should do so. No doubt about that, the problem is that it's hard to provide a reliable time-based indication. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] BYTEA / DBD::Pg change in 9.0 beta
On Tue, May 18, 2010 at 14:54, Jesper Krogh jes...@krogh.cc wrote: Hi. I'm trying to do a test move of one of our applications onto 9.0beta1. We use storable and serializes data into a bytea column in the database. [ snip insert/select using bytea ] 8.4 id | testbytea + 9 | \005\007\003\000\000\000\002\012\0012\000\000\000\004that\012\0011\000\000\000\004this (1 row) 9.0beta1 id | testbytea +-- 3 | \x05070300020a01320004746861740a0131000474686973 (1 row) Try adding $db-do(set bytea_output 'escape';); as 9.0 defaults to hex encoding. Id bet DBD::Pg does not account for that yet. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Bug with ordering aggregates?
This is an area that the SQL standard didn't think through very clearly (IMHO). They actually have two ways of specifying functions like this, one is the ordered aggregate section that this syntax is modeled on, which is indeed very confusing for multi-parameter aggregates. The other is the hypothetical set function syntax which is actually much clearer for this sort of operation, though I haven't dug deep enough into the standard to be sure this wouldn't include any gotchas: SELECT agg(parameter1, parameter2) WITHIN GROUP (ORDER BY column1 asc) (See section 10.9 on aggregate function syntax) Supporting the hypthothetical set functions could give a preferable syntax. Regards, Caleb On 5/18/10 9:42 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: This doesn't seem right to me: postgres=# select postgres-# string_agg(column1::text order by column1 asc,',') postgres-# from (values (3),(4),(1),(2)) a; string_agg 1234 (1 row) Looks fine to me: you have two ordering columns (the second rather useless, but that's no matter). Ah, yeah, guess I'll just complain that having the order by look like it's an argument to an aggregate makes things confusing. Not much to be done about it though. Thanks, Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade - link mode and transaction-wraparound data loss
Jesper Krogh wrote: On 2010-05-18 21:56, Bruce Momjian wrote: Jesper Krogh wrote: On 2010-05-18 20:52, Bruce Momjian wrote: This line above looks very odd because I didn't think the template0 datfrozenxid could be advanced. Can I see the output of this query: SELECT datname, datfrozenxid, datallowconn FROM pg_database; Only from the old database: data=# SELECT datname, datfrozenxid, datallowconn FROM pg_database; datname | datfrozenxid | datallowconn ---+--+-- template0 | 2073823552 | f postgres | 2023820521 | t data | 2023782337 | t jk| 2023822188 | t template1 | 2073823552 | t workqueue | 2023822188 | t (6 rows) OK, datallowconn = false is right for template0, but I am still confused how it got set to that high value. This is the production system. I have absolutely no indications that anything should be wrong in there. It has run rock-solid since it got migrated (dump/restore) to 8.4 for about 7 months now. So I am a bit scared about you telling that it seems wrong. (but that cannot be attributed to pg_upgrade) I am on chat with Alvaro now and it seems we do somehow connect to template0 for transaction id wraparound. I think Alvaro will post shortly on this. OK, thanks. This does seem odd. Frankly, having template0's datfrozenxid be wrong would not cause any kind of instability because template0 is used only by pg_dump, so I am wondering if something else is seriously wrong. I also think that something was seriously wrong with the pg_upgrade'd version. I'll try to reproduce and be a bit more carefull in tracking the steps this time. Thanks, but I think the entire problem might be this template0 xid issue that Alvaro and I are researching. I can now see how invalid template0 xids could cause the instability you saw in the new database. Odd no one has seen this bug before. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalive for max_standby_delay
On Tue, 2010-05-18 at 17:06 -0400, Heikki Linnakangas wrote: On 17/05/10 04:40, Simon Riggs wrote: On Sun, 2010-05-16 at 16:53 +0100, Simon Riggs wrote: Attached patch rearranges the walsender loops slightly to fix the above. XLogSend() now only sends up to MAX_SEND_SIZE bytes (== XLOG_SEG_SIZE / 2) in one round and returns to the main loop after that even if there's unsent WAL, and the main loop no longer sleeps if there's unsent WAL. That way the main loop gets to respond to signals quickly, and we also get to update the shared memory status and PS display more often when there's a lot of catching up to do. Comments 8MB at a time still seems like a large batch to me. libpq is going to send it in smaller chunks anyway, so I don't see the importance of trying to keep the batch too large. It just introduces delay into the sending process. We should be sending chunks that matches libpq better. More to the point the logic will fail if XLOG_BLCKSZ PQ_BUFFER_SIZE because it will send partial pages. I don't see a failure. We rely on not splitting WAL records across messages, but we're talking about libpq-level CopyData messages, not TCP messages. OK Having MAX_SEND_SIZE PQ_BUFFER_SIZE is pointless, as libpq currently stands. Well, it does affect the size of the read() in walsender, and I'm sure there's some overhead in setting the ps display and the other small stuff we do once per message. But you're probably right that we could easily make MAX_SEND_SIZE much smaller with no noticeable affect on performance, while making walsender more responsive to signals. I'll decrease it to, say, 512 kB. I'm pretty certain we don't need to set the ps display once per message. ps doesn't need an update 5 times per second on average. There's no reason that the buffer size we use for XLogRead() should be the same as the send buffer, if you're worried about that. My point is that pq_putmessage contains internal flushes so at the libpq level you gain nothing by big batches. The read() will be buffered anyway with readahead so not sure what the issue is. We'll have to do this for sync rep anyway, so what's the big deal? Just do it now, once. Do we really want 9.1 code to differ here? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] BYTEA / DBD::Pg change in 9.0 beta
On 2010-05-18 23:12, Alex Hunsaker wrote: set bytea_output 'escape'; That was it. Knowing what the problem was I had no problem finding it in the release notes. May I ask whats the reason is for breaking the compatibillity? -- Jesper -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalive for max_standby_delay
On Tue, 2010-05-18 at 17:08 -0400, Heikki Linnakangas wrote: On 17/05/10 12:36, Jim Nasby wrote: On May 15, 2010, at 12:05 PM, Heikki Linnakangas wrote: What exactly is the user trying to monitor? If it's how far behind is the standby, the difference between pg_current_xlog_insert_location() in the master and pg_last_xlog_replay_location() in the standby seems more robust and well-defined to me. It's a measure of XLOG location (ie. bytes) instead of time, but time is a complicated concept. I can tell you that end users *will* want a time-based indication of how far behind we are. DBAs will understand we're this many transactions behind, but managers and end users won't. Unless it's unreasonable to provide that info, we should do so. No doubt about that, the problem is that it's hard to provide a reliable time-based indication. I think I have one now. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalive for max_standby_delay
On 18/05/10 17:17, Simon Riggs wrote: There's no reason that the buffer size we use for XLogRead() should be the same as the send buffer, if you're worried about that. My point is that pq_putmessage contains internal flushes so at the libpq level you gain nothing by big batches. The read() will be buffered anyway with readahead so not sure what the issue is. We'll have to do this for sync rep anyway, so what's the big deal? Just do it now, once. Do we really want 9.1 code to differ here? Do what? What exactly is it that you want instead, then? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] BYTEA / DBD::Pg change in 9.0 beta
On Tue, May 18, 2010 at 15:20, Jesper Krogh jes...@krogh.cc wrote: On 2010-05-18 23:12, Alex Hunsaker wrote: set bytea_output 'escape'; That was it. Knowing what the problem was I had no problem finding it in the release notes. May I ask whats the reason is for breaking the compatibillity? There were a couple IIRC, the big ones being speed and size. Id look at the archives for more. I imagine at some point DBD::Pg will handle this transparently. I also imagine Greg would happily accept patches :-) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] BYTEA / DBD::Pg change in 9.0 beta
--On 18. Mai 2010 23:20:26 +0200 Jesper Krogh jes...@krogh.cc wrote: That was it. Knowing what the problem was I had no problem finding it in the release notes. May I ask whats the reason is for breaking the compatibillity? Efficency, if i am allowed to call it this way. The new hex representation should be more efficient to retrieve and to handle than the old one. I think bytea_output was set to hex for testing purposes on the first hand, but not sure wether there was a consensus to leave it there finally later. -- Thanks Bernd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] BYTEA / DBD::Pg change in 9.0 beta
On Tue, May 18, 2010 at 03:26:17PM -0600, Alex Hunsaker wrote: On Tue, May 18, 2010 at 15:20, Jesper Krogh jes...@krogh.cc wrote: On 2010-05-18 23:12, Alex Hunsaker wrote: set bytea_output 'escape'; That was it. Knowing what the problem was I had no problem finding it in the release notes. May I ask whats the reason is for breaking the compatibillity? There were a couple IIRC, the big ones being speed and size. Id look at the archives for more. I imagine at some point DBD::Pg will handle this transparently. I also imagine Greg would happily accept patches :-) Yes, the new format is much faster, more space efficient, and uses less CPU to do the encoding. The older format caused the COPY for bytea to be CPU limited in many more situations. Regards, Ken -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Keepalive for max_standby_delay
On Tue, 2010-05-18 at 17:25 -0400, Heikki Linnakangas wrote: On 18/05/10 17:17, Simon Riggs wrote: There's no reason that the buffer size we use for XLogRead() should be the same as the send buffer, if you're worried about that. My point is that pq_putmessage contains internal flushes so at the libpq level you gain nothing by big batches. The read() will be buffered anyway with readahead so not sure what the issue is. We'll have to do this for sync rep anyway, so what's the big deal? Just do it now, once. Do we really want 9.1 code to differ here? Do what? What exactly is it that you want instead, then? Read and write smaller messages, so the latency is minimised. Libpq will send in 8192 byte packets, so writing anything larger gains nothing when the WAL data is also chunked at exactly the same size. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] ecmascript 5 DATESTYLE
hi, i mistakenly had posted this to pgsql-bugs already and got a response (see below - edited). i'm posting here since afaik it is the way i should be requesting new features. my suggestion is to add a DATESTYLE format to match the format specified for date time strings in ecmascript 5. the following is from the ecmascript 5 specification at http://www.ecmascript.org/docs/tc39-2009-043.pdf page 168: 15.9.1.15 Date Time String Format ECMAScript defines a string interchange format for date-times based upon a simplification of the ISO 8601 Extended Format. The format is as follows: -MM-DDTHH:mm:ss.sssZ ecmascript 5 is the most recent specification for JavaScript and i would think that having a DATESTYLE format to simplify interoperability with JavaScript applications would be highly desirable. thanks, ben... On May 16, 2010, at 2:22 AM, Pavel Stehule wrote: I have nothing against some new datestyles - xml, ecma5 and I am able to add to pg when hackers will agree
Re: [HACKERS] Row-level Locks SERIALIZABLE transactions, postgres vs. Oracle
On May 17, 2010, at 3:30 , Robert Haas wrote: On Sun, May 16, 2010 at 9:07 PM, Florian Pflug f...@phlo.org wrote: On May 14, 2010, at 22:54 , Robert Haas wrote: On Thu, May 13, 2010 at 5:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Florian Pflug f...@phlo.org writes: All in all, I believe that SHARE and UPDATE row-level locks should be changed to cause concurrent UPDATEs to fail with a serialization error. I don't see an argument for doing that for FOR SHARE locks, and it already happens for FOR UPDATE (at least if the row actually gets updated). AFAICS this proposal mainly breaks things, in pursuit of an unnecessary and probably-impossible-anyway goal of making FK locking work with only user-level snapshots. After giving this considerable thought and testing the behavior at some length, I think the OP has it right. One thing I sometimes need to do is denormalize a copy of a field, e.g. snipped example I've whipped up a quick and still rather dirty patch that implements the behavior I proposed, at least for the case of conflicts between FOR UPDATE locks and updates. With the patch, any attempt to UPDATE or FOR UPDATE lock a row that has concurrently been FOR UPDATE locked will cause a serialization error. (The same for an actually updated row of course, but that happened before too). While this part of the patch was fairly straight forward, make FOR SHARE conflict too seems to be much harder. The assumption that a lock becomes irrelevant after the transaction(s) that held it completely is built deeply into the multi xact machinery that powers SHARE locks. That machinery therefore assumes that once all members of a multi xact have completed the multi xact is dead also. But my proposal depends on a SERIALIZABLE transaction being able to find if any of the lockers of a row are invisible under it's snapshot - for which it'd need any multi xact containing invisible xids to outlive its snapshot. Thanks for putting this together. I suggest adding it to the open CommitFest - even if we decide to go forward with this, I don't imagine anyone is going to be excited about changing it during beta. https://commitfest.postgresql.org/action/commitfest_view/open Will do. Thanks for the link. Here is an updated version that works for SHARE locks too. (This message mainly serves as a way to link the updated patch to the commit fest entry. Is this how I'm supposed to do that, or am I doing something wrong?) best regards, Florian Pflug serializable_lock_consistency.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, May 18, 2010 at 8:35 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, May 17, 2010 at 10:40 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, May 17, 2010 at 10:20 PM, Robert Haas robertmh...@gmail.com wrote: OK, I think I understand now. But, the SIGTERM sent by the postmaster doesn't kill the recovery process unconditionally. It will invoke StartupProcShutdownHandler(), which will set set shutdown_requested = true. That gets checked by RestoreArchivedFile() and HandleStartupProcInterrupts(), and I think that neither of those can get invoked until after the control file has been updated. Do you see a way it can happen? Yeah, the way is: StartupXLOG() -- ReadCheckpointRecord() -- ReadRecord() -- XLogPageRead() -- XLogFileReadAnyTLI() -- XLogFileRead() -- RestoreArchivedFile() ReadCheckpointRecord() is called before pg_control is updated. OK. In that case, I'm wondering if we should reverse course and rejigger the logic so that the shutdown gets processed when we transition to PM_RECOVERY. Seems like that might be simpler. You mean keeping shutdown waiting until the postmaster has reached PM_RECOVERY, i.e., the startup process has sent PMSIGNAL_RECOVERY_STARTED? The startup process must call ReadCheckpointRecord() before sending that signal. ReadCheckpointRecord() might get stuck for some reasons, e.g., neither master nor standby might have the recovery starting checkpoint WAL record. So that signal might not be sent forever, in this case, shutdown would get stuck. ISTM that walreceiver might be invoked even after shutdown is requested. We should prevent the postmaster from starting up walreceiver if Shutdown NoShutdown? Well, when we did the previous shutdown patch, we decided it was not right to kill walreceiver until all backends had exited, so it seems inconsistent to make the opposite decision here. Oh, right. How about allowing the postmaster only in PM_STARTUP, PM_RECOVERY, PM_HOT_STANDBY or PM_WAIT_READONLY state to invoke walreceiver? We can keep walreceiver alive until all read only backends have gone, and prevent unexpected startup of walreceiver. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Tue, May 18, 2010 at 10:40 PM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, May 18, 2010 at 8:35 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, May 17, 2010 at 10:40 PM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, May 17, 2010 at 10:20 PM, Robert Haas robertmh...@gmail.com wrote: OK, I think I understand now. But, the SIGTERM sent by the postmaster doesn't kill the recovery process unconditionally. It will invoke StartupProcShutdownHandler(), which will set set shutdown_requested = true. That gets checked by RestoreArchivedFile() and HandleStartupProcInterrupts(), and I think that neither of those can get invoked until after the control file has been updated. Do you see a way it can happen? Yeah, the way is: StartupXLOG() -- ReadCheckpointRecord() -- ReadRecord() -- XLogPageRead() -- XLogFileReadAnyTLI() -- XLogFileRead() -- RestoreArchivedFile() ReadCheckpointRecord() is called before pg_control is updated. OK. In that case, I'm wondering if we should reverse course and rejigger the logic so that the shutdown gets processed when we transition to PM_RECOVERY. Seems like that might be simpler. You mean keeping shutdown waiting until the postmaster has reached PM_RECOVERY, i.e., the startup process has sent PMSIGNAL_RECOVERY_STARTED? The startup process must call ReadCheckpointRecord() before sending that signal. ReadCheckpointRecord() might get stuck for some reasons, e.g., neither master nor standby might have the recovery starting checkpoint WAL record. So that signal might not be sent forever, in this case, shutdown would get stuck. Ah, OK. In terms of removing the backup label file, can we simply have an additional boolean in the postmaster that indicates whether we've ever reached PM_RUN, and only consider removing the backup file if so? ISTM that walreceiver might be invoked even after shutdown is requested. We should prevent the postmaster from starting up walreceiver if Shutdown NoShutdown? Well, when we did the previous shutdown patch, we decided it was not right to kill walreceiver until all backends had exited, so it seems inconsistent to make the opposite decision here. Oh, right. How about allowing the postmaster only in PM_STARTUP, PM_RECOVERY, PM_HOT_STANDBY or PM_WAIT_READONLY state to invoke walreceiver? We can keep walreceiver alive until all read only backends have gone, and prevent unexpected startup of walreceiver. Yes, that seems like something we should be checking, if we aren't already. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Row-level Locks SERIALIZABLE transactions, postgres vs. Oracle
On Tue, May 18, 2010 at 8:15 PM, Florian Pflug f...@phlo.org wrote: Will do. Thanks for the link. Here is an updated version that works for SHARE locks too. (This message mainly serves as a way to link the updated patch to the commit fest entry. Is this how I'm supposed to do that, or am I doing something wrong?) Yeah - just go to the existing CF entry and say New Comment then select type Patch. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ecmascript 5 DATESTYLE
On Tue, May 18, 2010 at 6:26 PM, Ben Hockey neonstalw...@gmail.com wrote: hi, i mistakenly had posted this to pgsql-bugs already and got a response (see below - edited). i'm posting here since afaik it is the way i should be requesting new features. my suggestion is to add a DATESTYLE format to match the format specified for date time strings in ecmascript 5. the following is from the ecmascript 5 specification at http://www.ecmascript.org/docs/tc39-2009-043.pdf page 168: 15.9.1.15 Date Time String Format ECMAScript defines a string interchange format for date-times based upon a simplification of the ISO 8601 Extended Format. The format is as follows: -MM-DDTHH:mm:ss.sssZ ecmascript 5 is the most recent specification for JavaScript and i would think that having a DATESTYLE format to simplify interoperability with JavaScript applications would be highly desirable. I don't object, if someone wants to write a patch. I guess the question is whether to keep adding named formats, or try to create a general mechanism to allow the user to specify an arbitrary format, as we do with to_char(). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ecmascript 5 DATESTYLE
2010/5/19 Robert Haas robertmh...@gmail.com: On Tue, May 18, 2010 at 6:26 PM, Ben Hockey neonstalw...@gmail.com wrote: hi, i mistakenly had posted this to pgsql-bugs already and got a response (see below - edited). i'm posting here since afaik it is the way i should be requesting new features. my suggestion is to add a DATESTYLE format to match the format specified for date time strings in ecmascript 5. the following is from the ecmascript 5 specification at http://www.ecmascript.org/docs/tc39-2009-043.pdf page 168: 15.9.1.15 Date Time String Format ECMAScript defines a string interchange format for date-times based upon a simplification of the ISO 8601 Extended Format. The format is as follows: -MM-DDTHH:mm:ss.sssZ ecmascript 5 is the most recent specification for JavaScript and i would think that having a DATESTYLE format to simplify interoperability with JavaScript applications would be highly desirable. I don't object, if someone wants to write a patch. I guess the question is whether to keep adding named formats, or try to create a general mechanism to allow the user to specify an arbitrary format, as we do with to_char(). I can write patch. I am against to general solution - It can be new way for SQL injection. Regards Pavel -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Stefan's bug (was: max_standby_delay considered harmful)
On Wed, May 19, 2010 at 12:59 PM, Robert Haas robertmh...@gmail.com wrote: In terms of removing the backup label file, can we simply have an additional boolean in the postmaster that indicates whether we've ever reached PM_RUN, and only consider removing the backup file if so? Yes, but I prefer XLogCtl-SharedRecoveryInProgress, which is the almost same indicator as the boolean you suggested. Thought? ISTM that walreceiver might be invoked even after shutdown is requested. We should prevent the postmaster from starting up walreceiver if Shutdown NoShutdown? Well, when we did the previous shutdown patch, we decided it was not right to kill walreceiver until all backends had exited, so it seems inconsistent to make the opposite decision here. Oh, right. How about allowing the postmaster only in PM_STARTUP, PM_RECOVERY, PM_HOT_STANDBY or PM_WAIT_READONLY state to invoke walreceiver? We can keep walreceiver alive until all read only backends have gone, and prevent unexpected startup of walreceiver. Yes, that seems like something we should be checking, if we aren't already. I'll do that. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
