Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 9:41 AM, Simon Riggs si...@2ndquadrant.com wrote: On 27 May 2013 15:36, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: That said, many discussions and ideas do get shut down, perhaps too early, because of pg_upgrade considerations. If we had a plan to have an incompatible release in the future, those ideas and discussions might be able to progress to a point where we determine it's worth it to take the pain of a non-pg_upgrade-supported release. That's a bit of a stretch, in my view, but I suppose it's possible. Even so though, I would suggest that we put together a wiki page to list out those items and encourage people to add to such a list; perhaps having an item on that list would make discussion about it progress beyond it breaks pg_upgrade. Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. Precisely. We've said right along that we reserve the right to have a non-upgradable disk format change whenever sufficiently many reasons accumulate to do that. Here's one that's come up a few times: being able to tweak the out-of-line storage strategy, e.g. change the compression format used. I think some folks were lamenting the lack of a convenient byte in the right place for that one. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sat, May 25, 2013 at 11:27 AM, Merlin Moncure mmonc...@gmail.com wrote: On Sat, May 25, 2013 at 4:39 AM, Simon Riggs si...@2ndquadrant.com wrote: There are a number of changes we'd probably like to make to the way things work in Postgres. This thread is not about discussing what those are, just to say that requirements exist and have been discussed in various threads over time. The constraint on such changes is that we've decided that we must have an upgrade path from release to release. So I'd like to make a formal suggestion of a plan for how we cope with this: 1. Implement online upgrade in 9.4 via the various facilities we have in-progress. That looks completely possible. 2. Name the next release after that 10.0 (would have been 9.5). We declare now that a) 10.0 will support on-line upgrade from 9.4 (only) b) various major incompatibilities will be introduced in 10.0 - the change in release number will indicate to everybody that is the case c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so that we will not be constrained by that This plan doesn't presume any particular change. Each change would need to be discussed on a separate thread, with a separate case for each. All I'm suggesting is that we have a coherent plan for the timing of such changes, so we can bundle them together into one release. By doing this now we give ourselves lots of time to plan changes that will see us good for another decade. If we don't do this, then we simply risk losing the iniative by continuing to support legacy formats and approaches. Huh. I don't think that bumping the version number to 10.0 vs 9.5 is justification to introduce breaking changes. In fact, I would rather see 10.0 be the version where we formally stop doing that. I understand that some stuff needs to be improved but it often doesn't seem to be worth the cost in the long run. Please disregard this comment -- I didn't realize the topic was regarding on disk format -- I mistakenly though it was opening the door for user level feature changes. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 06:13 AM, Joshua D. Drake wrote: On 05/27/2013 06:53 PM, Craig Ringer wrote: On 05/28/2013 09:39 AM, Gavin Flower wrote: Yes, I hate the Firefox style number inflation. I was arguing *for* it ;-) I don't like it much either, but (a) we do about one release a year, not one every few weeks and (b) it's very clear from a quick look at Stack Overflow or first-posts to pgsql-general how confusing two-part major versions are to users. If it's a bit less aesthetically pleasing I'm OK with that. This argument comes up every couple of years and the people that are trying to solve the problem by changing the versioning are ignoring the fact that there is no problem to solve. Consider the following exchange: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 9.2.4 or at least 9.0.13 If the problem has the at least part, then the first part is superfluous. If somebody wants to figure out how to run streaming CTE-s on postgresql 8 then you need to ask for exact major version which is two first digits if they want to run streaming replication there you can skip on e-mail exchange and tell right away that SR was added in version 9.0 ... The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. At least some of the real-world problems with PostgreSQL comes from We are developer software mentality. Yes, We are developer software, but we are also a DBA/maintainer/infrastructure manager software which needs to live a long time after the development is finished. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
This argument comes up every couple of years and the people that are trying to solve the problem by changing the versioning are ignoring the fact that there is no problem to solve. We just had this discussion on -advocacy (where it belongs, frankly) a couple months ago: http://www.postgresql.org/message-id/512e8ef8.3000...@agliodbs.com To sum up: the negatives of changing our version numbering scheme outweighed the positives. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 11:56 AM, Josh Berkus j...@agliodbs.com wrote: This argument comes up every couple of years and the people that are trying to solve the problem by changing the versioning are ignoring the fact that there is no problem to solve. We just had this discussion on -advocacy (where it belongs, frankly) +1. a couple months ago: http://www.postgresql.org/message-id/512e8ef8.3000...@agliodbs.com To sum up: the negatives of changing our version numbering scheme outweighed the positives. And +1 to that, too. FWIW, I think we may want to consider retitling 9.4 as 10.0, not because of any binary compatibility break (which, for the record, I oppose) but because of features. It's a little early to make that call just yet, of course, but I have a good feeling about this cycle. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 08:36 AM, Hannu Krosing wrote: The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. At least some of the real-world problems with PostgreSQL comes from We are developer software mentality. Yes, We are developer software, but we are also a DBA/maintainer/infrastructure manager I would not hire any of those three that weren't smart enough to understand our versioning scheme or had the wits to open a web browser and google: PostgreSQL versioning The answer is link #1 on Google. That said, I won't raise a stink. I am not really of a strong opinion either way except to say we are not solving a problem. We are just tickling each other's fancies. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 07:58:33AM +0800, Craig Ringer wrote: On 05/28/2013 12:41 AM, Simon Riggs wrote: I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. Any wire protocol or syntax changes? I can't seem to find a things we want to do in wire protocol v4 doc in the wiki but I know I've seen occasional discussion of things that can't be done without protocol changes. Anyone with a better memory than me able to pitch in? Sure, it is on the TODO list: https://wiki.postgresql.org/wiki/Todo#.2Fcontrib.2Fpg_upgrade I can only get a link to pg_upgrade from there, so look two sections below that for Wire Protocol Changes. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 05:21:16PM -0700, Joshua D. Drake wrote: On 05/27/2013 04:58 PM, Craig Ringer wrote: On 05/28/2013 12:41 AM, Simon Riggs wrote: I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. Any wire protocol or syntax changes? I can't seem to find a things we want to do in wire protocol v4 doc in the wiki but I know I've seen occasional discussion of things that can't be done without protocol changes. Anyone with a better memory than me able to pitch in? What'd be required to support in-band query cancellation? Sending per-statement GUCs (to allow true statement timeout)? I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution? Could it be optional? What about functions that modify the database --- isn't that only known at execution time? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 02:09:05PM -0400, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote: Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. OK, I have added a section to the TODO list for this: Desired changes that would prevent upgrades with pg_upgrade 32-bit page checksums Are there any others? GiST indexes really oughta have a metapage so there can be a version number in them. Also, if we are going to unify hstore and json, it'd be nice if we could change the existing binary representation of hstore (per discussions at Oleg and Teodor's talk --- this will be moot if we invent a new core type, but it'd be better not to have to). There are probably some other data-type-specific cleanups we could make, but I have to go get on an airplane so no time to think about it. OK, GiST and hstore added to TODO list. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 02:18 PM, Bruce Momjian wrote: I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution? Could it be optional? What about functions that modify the database --- isn't that only known at execution time? I can't speak to the actual C code that would be required but from a user space, I could see something like this: con = psycopg2.connect(database='testdb', user='test', transaction-type='r') Thus when the connection is made, before anything else is done, we know it is a read only connection and therefore any load balancer speaking libpq would also know it is a read only. The default of course would be r/w and you would use a different connection handler for r/w or w queries. The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Bruce Momjian wrote: On Mon, May 27, 2013 at 05:21:16PM -0700, Joshua D. Drake wrote: I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution? Could it be optional? What about functions that modify the database --- isn't that only known at execution time? Well, if you hit anything that tries to acquire an Xid, and you're in a context that said only read-only was acceptable, just raise an error. In a similar vein, I vaguely recall we discussed (after some security vulnerability involving SQL injection) a mode where we only accept only one command per PQexec() call, i.e. reject execution of commands that contain multiple queries. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 02:26:06PM -0700, Joshua D. Drake wrote: Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution? Could it be optional? What about functions that modify the database --- isn't that only known at execution time? I can't speak to the actual C code that would be required but from a user space, I could see something like this: con = psycopg2.connect(database='testdb', user='test', transaction-type='r') Thus when the connection is made, before anything else is done, we know it is a read only connection and therefore any load balancer speaking libpq would also know it is a read only. The default of course would be r/w and you would use a different connection handler for r/w or w queries. The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 03:06:13PM -0400, Alvaro Herrera wrote: Bruce Momjian wrote: OK, I have added a section to the TODO list for this: Desired changes that would prevent upgrades with pg_upgrade 32-bit page checksums Are there any others? I would have each data segment be self-identifying, i.e. have a magic number at the beginning of the file and the relation OID, some fork identification and the segment number somewhere -- probably the special space of the first page. Is this something we want on the TODO? I was not clear how to do with without making the first page format special or wasting space on all the other pages. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. Oh, that is an interesting idea. The application is indicating it is read-only via the protocol, and poolers can optimize that. Don't we have the ability to pass arbitrary GUC values back through the protocol, e.g. transaction_read_only? If not, that might be a way to do this cleanly. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 04:05 PM, Bruce Momjian wrote: On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. Oh, that is an interesting idea. The application is indicating it is read-only via the protocol, and poolers can optimize that. Don't we have the ability to pass arbitrary GUC values back through the protocol, e.g. transaction_read_only? If not, that might be a way to do this cleanly. I don't know but I don't think so. Anything that is calling SET is going to run through the parser. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/29/2013 05:11 AM, Bruce Momjian wrote: Sure, it is on the TODO list: https://wiki.postgresql.org/wiki/Todo#.2Fcontrib.2Fpg_upgrade I can only get a link to pg_upgrade from there, so look two sections below that for Wire Protocol Changes. Thanks. The direct link is https://wiki.postgresql.org/wiki/Todo#Wire_Protocol_Changes for anyone looking for it later. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 4:26 PM, Joshua D. Drake j...@commandprompt.com wrote: On 05/28/2013 02:18 PM, Bruce Momjian wrote: I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. Sounds nice, but how would we do that? That would require libpq to know it, right? Do we pass anything back after parsing but before execution? Could it be optional? What about functions that modify the database --- isn't that only known at execution time? I can't speak to the actual C code that would be required but from a user space, I could see something like this: con = psycopg2.connect(database='testdb', user='test', transaction-type='r') Thus when the connection is made, before anything else is done, we know it is a read only connection and therefore any load balancer speaking libpq would also know it is a read only. The default of course would be r/w and you would use a different connection handler for r/w or w queries. you can do that today already, kind-of create an entry in pgbouncer that connect to host=read-only.servers.dns and make read-only.servers.dns to point to more than 1 ip. then when the application wants to do load balancing, just connect to the entry that points to read-only.servers.dns and let the magic happens which would be great is this to happen transparently to the application The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. another idea, as someone else mentioned, and i think has been discussed bedore is a function that says if the query is r-o or not... maybe even exporting the plan so we don't need to replan again... Not sure if that is possible, just hand waving... -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación Phone: +593 4 5107566 Cell: +593 987171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 04:05 PM, Bruce Momjian wrote: On Tue, May 28, 2013 at 03:39:10PM -0700, Joshua D. Drake wrote: On 05/28/2013 03:36 PM, Bruce Momjian wrote: The other option would be to do it on query execute but that doesn't seem as efficient as it would have to be parsed each time. Although it would still be better than reading the actual SQL. Well, you could do SET TRANSACTION READ ONLY, and that would prevent any write transactions. You could assume it is a read query, and get the error and resubmit on the master if that happens, but that sounds inefficient. I thought you were asking for something where you could submit a query and it would report back as read/write or read-only. No I am suggesting something that before anything happens with the parser, the protocol knows what is up. So things like pgpool-ii don't even need a parser, it just knows it is a read only query because the protocol says so. Oh, that is an interesting idea. The application is indicating it is read-only via the protocol, and poolers can optimize that. Don't we have the ability to pass arbitrary GUC values back through the protocol, e.g. transaction_read_only? If not, that might be a way to do this cleanly. I don't know but I don't think so. Anything that is calling SET is going to run through the parser. Right. SET command needs to be parsed by the parser. However, we already have embedded parameters in the start up packet, which needs to be recognized by pooler anyway. See StartupMessage section in: http://www.postgresql.org/docs/9.3/static/protocol-message-formats.html I am not sure backend currently permits to have default_transaction_read_only = on in the startup packet or not though. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Bruce Momjian wrote: On Mon, May 27, 2013 at 03:06:13PM -0400, Alvaro Herrera wrote: Bruce Momjian wrote: I would have each data segment be self-identifying, i.e. have a magic number at the beginning of the file and the relation OID, some fork identification and the segment number somewhere -- probably the special space of the first page. Is this something we want on the TODO? I was not clear how to do with without making the first page format special or wasting space on all the other pages. I don't think the special space has to necessarily be identically sized in all the pages -- I admit I haven't looked closely, but the special size is part of the page header. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 2:01 PM, Craig Ringer cr...@2ndquadrant.com wrote: On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. In this case you don't even need the 2nd digit... Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series. -- Michael
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sun, May 26, 2013 at 09:18:41PM -0400, Stephen Frost wrote: * Josh Berkus (j...@agliodbs.com) wrote: and it's entirely possible that we'll be able to implement SMs without breaking pgupgrade. I'd certainly hope so.. It's certainly not obvious, to me at least, why a new SM or supporting any of those features would require breaking pg_upgrade. Perhaps there's something I'm not seeing there, but it had better be a *really* good reason.. If I had to _guess_, I would say users who are using the default storage manager would still be able to use pg_upgrade, and those using non-default storage managers perhaps can't. But, again, this is all so hypothetical that it doesn't seem worth talking about. My big point is that someone came to me at PGCon asking if I knew anything about why Simon thought we needed to break pg_upgrade in 2 years, and I said no, so I had go digging into my email to find out what was going on. Simon has a very visible position in the community, so when he suggests something, people take it seriously, which means I have to address it. I would prefer if there was more thought put into the ideas before they are posted. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
* Bruce Momjian (br...@momjian.us) wrote: If I had to _guess_, I would say users who are using the default storage manager would still be able to use pg_upgrade, and those using non-default storage managers perhaps can't. That would make sense. But, again, this is all so hypothetical that it doesn't seem worth talking about. Having a specific list of these are the things we want to change, and why, and here is why pg_upgrade can't support it would be much more useful to work from, I agree. That said, many discussions and ideas do get shut down, perhaps too early, because of pg_upgrade considerations. If we had a plan to have an incompatible release in the future, those ideas and discussions might be able to progress to a point where we determine it's worth it to take the pain of a non-pg_upgrade-supported release. That's a bit of a stretch, in my view, but I suppose it's possible. Even so though, I would suggest that we put together a wiki page to list out those items and encourage people to add to such a list; perhaps having an item on that list would make discussion about it progress beyond it breaks pg_upgrade. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: If I had to _guess_, I would say users who are using the default storage manager would still be able to use pg_upgrade, and those using non-default storage managers perhaps can't. That would make sense. But, again, this is all so hypothetical that it doesn't seem worth talking about. Having a specific list of these are the things we want to change, and why, and here is why pg_upgrade can't support it would be much more useful to work from, I agree. That said, many discussions and ideas do get shut down, perhaps too early, because of pg_upgrade considerations. If we had a plan to have an incompatible release in the future, those ideas and discussions might be able to progress to a point where we determine it's worth it to take the pain of a non-pg_upgrade-supported release. That's a bit of a stretch, in my view, but I suppose it's possible. Even so though, I would suggest that we put together a wiki page to list out those items and encourage people to add to such a list; perhaps having an item on that list would make discussion about it progress beyond it breaks pg_upgrade. Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/26/2013 06:18 PM, Josh Berkus wrote: Not sure which ones Simon meant, but at least any new/better storage manager would seem to me to be requiring a non-pg_upgrade upgrade path unless we require the storage manager to also include a parallel implementation of pg_upgrade. Isn't this a bit of horse-cart inversion here? We just hashed out a tentative, incomplete pseudo-spec for storage managers *yesterday*. Many people have been *thinking* about pluggable storage / storage managers for much longer time. We don't have a complete spec at this point, let alone a development plan, I think we will have a development plan *before* complete spec anyway :) and it's entirely possible that we'll be able to implement SMs without breaking pgupgrade. My point was exactly to not spend majority of new storage manager discussion on does it break pg_upgrade, maybe we can find a way to do it without breaking pg_upgrade, etc... It's also not at all clear that we can develop SMs in less than 2 years. I tend to think it unlikely. I think the important part of Simons message was not two years First, let's have a few features for which breaking binary compatibility is a necessity or a clear benefit. Then we'll schedule when to break them. But rather than it breaks pg_upgrade not being a complete stopper for proposed useful features that might break it. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Bruce Momjian br...@momjian.us writes: On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: That said, many discussions and ideas do get shut down, perhaps too early, because of pg_upgrade considerations. If we had a plan to have an incompatible release in the future, those ideas and discussions might be able to progress to a point where we determine it's worth it to take the pain of a non-pg_upgrade-supported release. That's a bit of a stretch, in my view, but I suppose it's possible. Even so though, I would suggest that we put together a wiki page to list out those items and encourage people to add to such a list; perhaps having an item on that list would make discussion about it progress beyond it breaks pg_upgrade. Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. Precisely. We've said right along that we reserve the right to have a non-upgradable disk format change whenever sufficiently many reasons accumulate to do that. The way to go about that is to collect projects that need to be kept on hold for such a release --- not to say we're going to have such a release and then look for reasons. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Michael Paquier escribió: On Mon, May 27, 2013 at 2:01 PM, Craig Ringer cr...@2ndquadrant.com wrote: On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. In this case you don't even need the 2nd digit... You do -- they are used for minor releases, i.e. 10.1 would be a bugfix release for 10.0. If we continue using the current numbering scheme, 10.1 would be the major version after 10.0. Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series. All major releases bring lots of enhancements. Streaming replication might be great for some people, but I'm sure there are things in 8.4 and 9.1 that are equally great for some other people. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Tom Lane wrote: Bruce Momjian br...@momjian.us writes: Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. Precisely. We've said right along that we reserve the right to have a non-upgradable disk format change whenever sufficiently many reasons accumulate to do that. Do we have a wiki page about this? -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote: That said, many discussions and ideas do get shut down, perhaps too early, because of pg_upgrade considerations. If we had a plan to have an incompatible release in the future, those ideas and discussions might be able to progress to a point where we determine it's worth it to take the pain of a non-pg_upgrade-supported release. That's a bit of a stretch, in my view, but I suppose it's possible. Even so though, I would suggest that we put together a wiki page to list out those items and encourage people to add to such a list; perhaps having an item on that list would make discussion about it progress beyond it breaks pg_upgrade. Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. OK, I have added a section to the TODO list for this: Desired changes that would prevent upgrades with pg_upgrade 32-bit page checksums Are there any others? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 27 May 2013 15:36, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: On Mon, May 27, 2013 at 08:26:48AM -0400, Stephen Frost wrote: That said, many discussions and ideas do get shut down, perhaps too early, because of pg_upgrade considerations. If we had a plan to have an incompatible release in the future, those ideas and discussions might be able to progress to a point where we determine it's worth it to take the pain of a non-pg_upgrade-supported release. That's a bit of a stretch, in my view, but I suppose it's possible. Even so though, I would suggest that we put together a wiki page to list out those items and encourage people to add to such a list; perhaps having an item on that list would make discussion about it progress beyond it breaks pg_upgrade. Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. Precisely. We've said right along that we reserve the right to have a non-upgradable disk format change whenever sufficiently many reasons accumulate to do that. I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. The way to go about that is to collect projects that need to be kept on hold for such a release --- not to say we're going to have such a release and then look for reasons. Agreed. I was trying to establish a realistic timeline for such events, so that the planning was able to be taken seriously. Yes, it wass a work backwards or what if type of planning. But now we have a rough plan of how it might look, collecting ideas can begin. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Bruce Momjian br...@momjian.us writes: On Mon, May 27, 2013 at 09:17:50AM -0400, Bruce Momjian wrote: Yes, we should be collecting things we want to do for a pg_upgrade break so we can see the list all in one place. OK, I have added a section to the TODO list for this: Desired changes that would prevent upgrades with pg_upgrade 32-bit page checksums Are there any others? GiST indexes really oughta have a metapage so there can be a version number in them. Also, if we are going to unify hstore and json, it'd be nice if we could change the existing binary representation of hstore (per discussions at Oleg and Teodor's talk --- this will be moot if we invent a new core type, but it'd be better not to have to). There are probably some other data-type-specific cleanups we could make, but I have to go get on an airplane so no time to think about it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Bruce Momjian wrote: OK, I have added a section to the TODO list for this: Desired changes that would prevent upgrades with pg_upgrade 32-bit page checksums Are there any others? I would have each data segment be self-identifying, i.e. have a magic number at the beginning of the file and the relation OID, some fork identification and the segment number somewhere -- probably the special space of the first page. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Michael Paquier escribió: On Mon, May 27, 2013 at 2:01 PM, Craig Ringer cr...@2ndquadrant.com wrote: On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. In this case you don't even need the 2nd digit... You do -- they are used for minor releases, i.e. 10.1 would be a bugfix release for 10.0. If we continue using the current numbering scheme, 10.1 would be the major version after 10.0. Sorry for the confusion. I meant that the 2nd digit would not be necessary when identifying a given major release, so I just didn't get the meaning of what Craig said. As you say, you would still need the 2nd digit for minor releases. -- Michael
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 07:39:35AM +0900, Michael Paquier wrote: On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: Michael Paquier escribió: On Mon, May 27, 2013 at 2:01 PM, Craig Ringer cr...@2ndquadrant.com wrote: On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. In this case you don't even need the 2nd digit... You do -- they are used for minor releases, i.e. 10.1 would be a bugfix release for 10.0. If we continue using the current numbering scheme, 10.1 would be the major version after 10.0. Sorry for the confusion. I meant that the 2nd digit would not be necessary when identifying a given major release, so I just didn't get the meaning of what Craig said. As you say, you would still need the 2nd digit for minor releases. What's been proposed before that wouldn't break previous applications is a numbering system like this: 10.0.0 10.0.1 10.0.2 10.0.3 ... 11.0.0 11.0.1 i.e. only change the most-major version number and always leave the less-major number as zero. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Michael Paquier escribió: On Tue, May 28, 2013 at 12:36 AM, Alvaro Herrera alvhe...@2ndquadrant.comwrote: You do -- they are used for minor releases, i.e. 10.1 would be a bugfix release for 10.0. If we continue using the current numbering scheme, 10.1 would be the major version after 10.0. Sorry for the confusion. I meant that the 2nd digit would not be necessary when identifying a given major release, so I just didn't get the meaning of what Craig said. As you say, you would still need the 2nd digit for minor releases. Well, that seems okay to me. We used to see a lot of people talking about Postgres 8.x when they meant, say, 8.3; and now we have people talking about Postgres 9 when in reality they mean 9.1 or some other specific major version. Having the second digit be part of the major version number is a difficult idea to convey to external people. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Tue, May 28, 2013 at 7:52 AM, David Fetter da...@fetter.org wrote: What's been proposed before that wouldn't break previous applications is a numbering system like this: 10.0.0 10.0.1 10.0.2 10.0.3 ... 11.0.0 11.0.1 i.e. only change the most-major version number and always leave the less-major number as zero. Thanks for the clarification. Firefox did exactly the same from 4.0. -- Michael
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/27/2013 05:45 PM, Michael Paquier wrote: On Mon, May 27, 2013 at 2:01 PM, Craig Ringer cr...@2ndquadrant.com wrote: On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. In this case you don't even need the 2nd digit... Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series. I don't find bumping the major to be particularly helpful. Every release brings major features - and some introduce major incompatibilities. 8.4 introduced CTEs. 8.3 broke tons of client code with the removal of implicit casts to text. It really depends on what features you consider more major/significant. Personally I don't think it makes sense to try to say this release is bigger in Pg - at least not in terms of enhancement. I can see value in using this-release-is-bigger for this brings more breakage - but would strongly prefer a smooth and continuous release numbering that doesn't confuse the heck out of users. I'm extremely tired of being told I'm running PostgreSQL 8.x or I'm running PostgreSQL 9.x and having to point out the version policy, the fact that there are four years and huge fixes/enhancements between 8.0 and 8.4, etc. The version policy makes _no distinction_ between which digit changes in a major release: PostgreSQL major releases include new features and occur roughly once every year. A major release is numbered by increasing either the first or second part of the version number, e.g. 8.2 to 8.3. Major releases usually change the internal format of system tables and data files. These changes are often complex, so we do not maintain backward compatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for major upgrades. and I strongly believe that we should drop the notion entirely. ... -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 07:22 AM, Michael Paquier wrote: On Tue, May 28, 2013 at 7:52 AM, David Fetter da...@fetter.org wrote: What's been proposed before that wouldn't break previous applications is a numbering system like this: 10.0.0 10.0.1 10.0.2 10.0.3 ... 11.0.0 11.0.1 i.e. only change the most-major version number and always leave the less-major number as zero. Thanks for the clarification. Firefox did exactly the same from 4.0. Yeah... I was more meaning 10.0, 10.1, 10.2 etc for minor releases, but I can imagine people coding logic to check major version using the first two digits, so you're quite right that it'd need to be grandfathered into 10.0.1, 10.0.2, etc. Sigh. The upside of that is that it'd reinforce the idea that we sometimes struggle to get across to people - that minor patch releases are *minor* and *safe* to just upgrade to without jumping through change-approval hoops, vendor approval for updates, two-year-long QA and all the other baggage many IT departments seem to have. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 12:41 AM, Simon Riggs wrote: I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. Any wire protocol or syntax changes? I can't seem to find a things we want to do in wire protocol v4 doc in the wiki but I know I've seen occasional discussion of things that can't be done without protocol changes. Anyone with a better memory than me able to pitch in? What'd be required to support in-band query cancellation? Sending per-statement GUCs (to allow true statement timeout)? I can't think of any major syntax warts and grandfathered quirks that'd be really great to get rid of if we had the freedom to break things. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/27/2013 04:58 PM, Craig Ringer wrote: On 05/28/2013 12:41 AM, Simon Riggs wrote: I'm happy with that. I was also thinking about collecting changes not related just to disk format, if any exist. Any wire protocol or syntax changes? I can't seem to find a things we want to do in wire protocol v4 doc in the wiki but I know I've seen occasional discussion of things that can't be done without protocol changes. Anyone with a better memory than me able to pitch in? What'd be required to support in-band query cancellation? Sending per-statement GUCs (to allow true statement timeout)? I would like to see the ability to define if a query is read only at the protocol level, so that load balances that speak libpq can know what to do with the query without parsing it. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC, @cmdpromptinc For my dreams of your image that blossoms a rose in the deeps of my heart. - W.B. Yeats -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 28/05/13 11:48, Craig Ringer wrote: On 05/27/2013 05:45 PM, Michael Paquier wrote: On Mon, May 27, 2013 at 2:01 PM, Craig Ringercr...@2ndquadrant.com wrote: On 05/25/2013 05:39 PM, Simon Riggs wrote: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. In this case you don't even need the 2nd digit... Btw, -1 for the idea, as it would remove the possibility to tell that a new major release incrementing the 1st digit of version number brings more enhancement than normal major releases incrementing the 1st digit. This was the case for 9.0, helping people in remembering that streaming replication has been introduced from 9.x series. I don't find bumping the major to be particularly helpful. Every release brings major features - and some introduce major incompatibilities. 8.4 introduced CTEs. 8.3 broke tons of client code with the removal of implicit casts to text. It really depends on what features you consider more major/significant. Personally I don't think it makes sense to try to say this release is bigger in Pg - at least not in terms of enhancement. I can see value in using this-release-is-bigger for this brings more breakage - but would strongly prefer a smooth and continuous release numbering that doesn't confuse the heck out of users. I'm extremely tired of being told I'm running PostgreSQL 8.x or I'm running PostgreSQL 9.x and having to point out the version policy, the fact that there are four years and huge fixes/enhancements between 8.0 and 8.4, etc. The version policy makes _no distinction_ between which digit changes in a major release: PostgreSQL major releases include new features and occur roughly once every year. A major release is numbered by increasing either the first or second part of the version number, e.g. 8.2 to 8.3. Major releases usually change the internal format of system tables and data files. These changes are often complex, so we do not maintain backward compatibility of all stored data. A dump/reload of the database or use of the pg_upgrade module is required for major upgrades. and I strongly believe that we should drop the notion entirely. ... -- Craig Ringerhttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services Yes, I hate the Firefox style number inflation. Cheers, Gavin
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/28/2013 09:39 AM, Gavin Flower wrote: Yes, I hate the Firefox style number inflation. I was arguing *for* it ;-) I don't like it much either, but (a) we do about one release a year, not one every few weeks and (b) it's very clear from a quick look at Stack Overflow or first-posts to pgsql-general how confusing two-part major versions are to users. If it's a bit less aesthetically pleasing I'm OK with that. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/27/2013 06:53 PM, Craig Ringer wrote: On 05/28/2013 09:39 AM, Gavin Flower wrote: Yes, I hate the Firefox style number inflation. I was arguing *for* it ;-) I don't like it much either, but (a) we do about one release a year, not one every few weeks and (b) it's very clear from a quick look at Stack Overflow or first-posts to pgsql-general how confusing two-part major versions are to users. If it's a bit less aesthetically pleasing I'm OK with that. This argument comes up every couple of years and the people that are trying to solve the problem by changing the versioning are ignoring the fact that there is no problem to solve. Consider the following exchange: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 9.2.4 or at least 9.0.13 Now, if we change the version numbers: Client: I have X problem with PostgreSQL CMD: What version? Client: 9 CMD: Which version of 9? Client: 9.0.2 CMD: You should be running 10.0.5 or at least 9.0.13 The conversation does not change. Further, we are not Firefox. We are not user software. We are developer software. Sincerely, Joshua D. Drake -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 25 May 2013 21:44, Bruce Momjian br...@momjian.us wrote: On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote: There are a number of changes we'd probably like to make to the way things work in Postgres. This thread is not about discussing what those are, just to say that requirements exist and have been discussed in various threads over time. The constraint on such changes is that we've decided that we must have an upgrade path from release to release. So I'd like to make a formal suggestion of a plan for how we cope with this: 1. Implement online upgrade in 9.4 via the various facilities we have in-progress. That looks completely possible. 2. Name the next release after that 10.0 (would have been 9.5). We declare now that a) 10.0 will support on-line upgrade from 9.4 (only) b) various major incompatibilities will be introduced in 10.0 - the change in release number will indicate to everybody that is the case c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so that we will not be constrained by that Assuming online upgrade is going to require logical replication, you are also assuming 2x storage as you need to have a second cluster to perform the upgrade. The people that want online upgrade already have 1+ other systems to do this with. pg_upgrade would still be needed to upgrade a cluster in-place. This sounds like, I created a new tool which does some of what the old tool does. Let's break the old tool to allow some unspecified changes I might want to make. I haven't argued against pg_upgrade in general, nor said anything about breaking it. I proposed that we don't support a pg_upgrade path between two near-future releases, as a way of introducing incompatibilities. After that, we would continue to use pg_upgrade for later releases. Logical replication is being developed, which gives us a complete code path for doing what we'd need to do. The most important thing is we wouldn't need to develop any other code that exists just for upgrade. Writing special code just for pg_upgrade will take a lot of work. Running that code would mean pg_upgrade would touch the actual database, which would be down for a long time while it runs. And if it hits a bug during or after, you're hosed. So you'd need to take a full backup before you started the process, probably storing it on disk somewhere and so you would need x2 disk space with this route also. Specialised code is less well tested, which means bugs are more likely to occur and tends to perform more poorly. Not only that, but the first person to want an incompatibility gets to write all the code needed and take responsibility for the bugs. I can't comment for others, but I can say I would not personally choose that route - it looks both expensive and risky. I consider this thread to be not thought-through, obviously. My proposal has had lots of serious consideration, but that is not the topic of this thread. The title of the thread is a general one, with a clear objective. I'm looking for a way forwards that allows us to introduce the changes that many have proposed and which regrettably result in incompatibilities. If we have no plan I think its likely it will never happen and it is currently blocking useful change. Please explain what you consider to be a better plan, so we can judge all proposals together. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote: I consider this thread to be not thought-through, obviously. My proposal has had lots of serious consideration, but that is not the topic of this thread. The title of the thread is a general one, with a clear objective. I'm looking for a way forwards that allows us to introduce the changes that many have proposed and which regrettably result in incompatibilities. If we have no plan I think its likely it will never happen and it is currently blocking useful change. Please explain what you consider to be a better plan, so we can judge all proposals together. I agree with the idea of using logical replication as a way to do pg_upgrade version-breaking releases. What I don't know is what incompatible changes are pending that would require this. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote: On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote: I consider this thread to be not thought-through, obviously. My proposal has had lots of serious consideration, but that is not the topic of this thread. The title of the thread is a general one, with a clear objective. I'm looking for a way forwards that allows us to introduce the changes that many have proposed and which regrettably result in incompatibilities. If we have no plan I think its likely it will never happen and it is currently blocking useful change. Please explain what you consider to be a better plan, so we can judge all proposals together. I agree with the idea of using logical replication as a way to do pg_upgrade version-breaking releases. What I don't know is what incompatible changes are pending that would require this. Sorry I was unclear. When I said not thought-through, I meant, you need to start with the _reason_ we need to break pg_upgrade in an upcoming version, then we can start to plan how to do it. The logical replication idea is a good one for getting us through pg_upgrade version-breaking releases. I am fine with breaking pg_upgrade, but I just don't see the pending reason at this point. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/26/2013 04:22 PM, Bruce Momjian wrote: On Sun, May 26, 2013 at 09:18:11AM -0400, Bruce Momjian wrote: On Sun, May 26, 2013 at 10:53:37AM +0100, Simon Riggs wrote: I consider this thread to be not thought-through, obviously. My proposal has had lots of serious consideration, but that is not the topic of this thread. The title of the thread is a general one, with a clear objective. I'm looking for a way forwards that allows us to introduce the changes that many have proposed and which regrettably result in incompatibilities. If we have no plan I think its likely it will never happen and it is currently blocking useful change. Please explain what you consider to be a better plan, so we can judge all proposals together. I agree with the idea of using logical replication as a way to do pg_upgrade version-breaking releases. What I don't know is what incompatible changes are pending that would require this. Sorry I was unclear. When I said not thought-through, I meant, you need to start with the _reason_ we need to break pg_upgrade in an upcoming version, then we can start to plan how to do it. The logical replication idea is a good one for getting us through pg_upgrade version-breaking releases. I am fine with breaking pg_upgrade, but I just don't see the pending reason at this point. Not sure which ones Simon meant, but at least any new/better storage manager would seem to me to be requiring a non-pg_upgrade upgrade path unless we require the storage manager to also include a parallel implementation of pg_upgrade. The family of possible storage magers here would include column stores, distributed / partitioned / replicated memory-only / index-structured / ... storages which all could have advantages in certain situations and whic all need an upgrade path. While you could do this using sequance of first pg_upgrading and then doing some internal data migration to new storage manager doing this in one go would be much smoother. -- Hannu Krosing PostgreSQL Consultant Performance, Scalability and High Availability 2ndQuadrant Nordic OÜ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Not sure which ones Simon meant, but at least any new/better storage manager would seem to me to be requiring a non-pg_upgrade upgrade path unless we require the storage manager to also include a parallel implementation of pg_upgrade. Isn't this a bit of horse-cart inversion here? We just hashed out a tentative, incomplete pseudo-spec for storage managers *yesterday*. We don't have a complete spec at this point, let alone a development plan, and it's entirely possible that we'll be able to implement SMs without breaking pgupgrade. It's also not at all clear that we can develop SMs in less than 2 years. I tend to think it unlikely. First, let's have a few features for which breaking binary compatibility is a necessity or a clear benefit. Then we'll schedule when to break them. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
* Josh Berkus (j...@agliodbs.com) wrote: and it's entirely possible that we'll be able to implement SMs without breaking pgupgrade. I'd certainly hope so.. It's certainly not obvious, to me at least, why a new SM or supporting any of those features would require breaking pg_upgrade. Perhaps there's something I'm not seeing there, but it had better be a *really* good reason.. btw, has anyone posted the SM API proposal..? Unfortunately, I think I had to leave before that was hashed out.. First, let's have a few features for which breaking binary compatibility is a necessity or a clear benefit. Then we'll schedule when to break them. Agreed. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
Stephen Frost sfr...@snowman.net writes: btw, has anyone posted the SM API proposal..? Unfortunately, I think I had to leave before that was hashed out.. There isn't one yet. We think we understand where the pain points are, but there's still a long way to go to have a proposal. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
The assumption that we ought to plan expressly for an incompatibility that essentially discards pg_upgrade seems premature, particularly in advance of would-be solutions that, in some cases, mightn't actually work. If pg_upgrade doesn't work, then, at present, the plausible solutions are to either dump and restore, which might take way too long, or use one of the logical replication systems (e.g. - Slony, Londiste, or similar, in the absence of the would-be built-in logical replication). Unfortunately, there are significant scenarios where none of these work, particularly for data warehouse-like systems where the database size is so large that the users cannot afford the disk space to construct a replica. It sure seems premature to intentionally leave that set of users out in the cold.
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 05/25/2013 05:39 PM, Simon Riggs wrote: 2. Name the next release after that 10.0 (would have been 9.5). We declare now that a) 10.0 will support on-line upgrade from 9.4 (only) b) various major incompatibilities will be introduced in 10.0 - the change in release number will indicate to everybody that is the case c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so that we will not be constrained by that While we're talking about changing things, what about: - Switching to single-major-version release numbering. The number of people who say PostgreSQL 9.x is amazing; even *packagers* get this wrong and produce postgresql-9 packages. Witness Amazon Linux's awful PostgreSQL packages for example. Going to PostgreSQL 10.0, 11.0, 12.0, etc with a typical major/minor scheme might be worth considering. - s/cluster/server/g . Just because cluster is historical usage doesn't make it any less confusing for users. *dives for asbestos fire suit* -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sat, May 25, 2013 at 4:39 AM, Simon Riggs si...@2ndquadrant.com wrote: There are a number of changes we'd probably like to make to the way things work in Postgres. This thread is not about discussing what those are, just to say that requirements exist and have been discussed in various threads over time. The constraint on such changes is that we've decided that we must have an upgrade path from release to release. So I'd like to make a formal suggestion of a plan for how we cope with this: 1. Implement online upgrade in 9.4 via the various facilities we have in-progress. That looks completely possible. 2. Name the next release after that 10.0 (would have been 9.5). We declare now that a) 10.0 will support on-line upgrade from 9.4 (only) b) various major incompatibilities will be introduced in 10.0 - the change in release number will indicate to everybody that is the case c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so that we will not be constrained by that This plan doesn't presume any particular change. Each change would need to be discussed on a separate thread, with a separate case for each. All I'm suggesting is that we have a coherent plan for the timing of such changes, so we can bundle them together into one release. By doing this now we give ourselves lots of time to plan changes that will see us good for another decade. If we don't do this, then we simply risk losing the iniative by continuing to support legacy formats and approaches. Huh. I don't think that bumping the version number to 10.0 vs 9.5 is justification to introduce breaking changes. In fact, I would rather see 10.0 be the version where we formally stop doing that. I understand that some stuff needs to be improved but it often doesn't seem to be worth the cost in the long run. merlin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote: The constraint on such changes is that we've decided that we must have an upgrade path from release to release. Is this proposal only relaxing the binary upgrade requirement, or would it also relax other compatibility requirements, such as language and API compatibility? We need a couple major drivers of the incompatibility that really show users some value for going through the upgrade pain. Preferably, at least one would be a serious performance boost, because the users that encounter the most logical upgrade pain are also the ones that need a performance boost the most. Before we set a specific schedule, I think it would be a good idea to start prototyping some performance improvements that involve breaking the data format. Then, depending on how achievable it is, we can plan for however many more 9.X releases we think we need. That being said, I agree with you that planning in advance is important here, so that everyone knows when they need to get format-breaking changes in by. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On 25 May 2013 18:13, Jeff Davis pg...@j-davis.com wrote: On Sat, 2013-05-25 at 10:39 +0100, Simon Riggs wrote: The constraint on such changes is that we've decided that we must have an upgrade path from release to release. Is this proposal only relaxing the binary upgrade requirement, or would it also relax other compatibility requirements, such as language and API compatibility? I'm suggesting that as many as possible changes we would like to make can happen in one release. This is for the benefit of users, so we dont make every release a source of incompatibilities. And that release should be the first one where we have online upgrade possible, which is one after next. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Planning incompatibilities for Postgres 10.0
On Sat, May 25, 2013 at 10:39:30AM +0100, Simon Riggs wrote: There are a number of changes we'd probably like to make to the way things work in Postgres. This thread is not about discussing what those are, just to say that requirements exist and have been discussed in various threads over time. The constraint on such changes is that we've decided that we must have an upgrade path from release to release. So I'd like to make a formal suggestion of a plan for how we cope with this: 1. Implement online upgrade in 9.4 via the various facilities we have in-progress. That looks completely possible. 2. Name the next release after that 10.0 (would have been 9.5). We declare now that a) 10.0 will support on-line upgrade from 9.4 (only) b) various major incompatibilities will be introduced in 10.0 - the change in release number will indicate to everybody that is the case c) agree that there will be no pg_upgrade patch from 9.4 to 10.0, so that we will not be constrained by that Assuming online upgrade is going to require logical replication, you are also assuming 2x storage as you need to have a second cluster to perform the upgrade. pg_upgrade would still be needed to upgrade a cluster in-place. This sounds like, I created a new tool which does some of what the old tool does. Let's break the old tool to allow some unspecified changes I might want to make. I consider this thread to be not thought-through, obviously. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
