Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Tom, > That sounds good, but there are corner cases where it wouldn't work --- > consider a page containing a single maximum-length tuple. Certainly any mature upgrade-in-place tool will require a "checker" which you run first which determines if you have a prohibitive corner case. Besides, I thought we didn't allow tuples to grow to more than 1/2 page size? Or am I thinking indexes here? > In general I don't see us accepting changes that would increase data > size, but there's at least one troubling exception on the horizon: > per-column or per-value locale support. Another problem is that a strict > rule of "no data size increase ever" might forbid acceptance of changes > that achieve average space savings at the cost of increasing the size of > some lesser-used cases. Our attitude at the meeting was "let's burn that bridge when we come to it". If we can develop a solid in-place upgrade too which will work for 8.1->8.2 and 8.2->8.3, then we've done something worthwhile even if we break it in 8.4 or 8.5. It's possible that we won't implement anything that breaks it for five years or that someone will invent another solution before then, in which case we'd feel pretty dumb for having kept it on the drawing board all that time. Or to put it another way, the page-grows-on-upgrade problem is hard enough that it will probably take as much effort as the whole rest of the upgrade process to solve. So let's tackle one problem at a time. -- Josh Berkus PostgreSQL @ Sun San Francisco ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Heikki Linnakangas wrote: Since we're discussing upgrades, let me summarize the discussions we had over dinner in Ottawa for the benefit of all: Thanks for summary. As before, someone just needs to step up and do it. I'm now working on proposal. I hope that it will ready soon. Zdenek ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Heikki Linnakangas <[EMAIL PROTECTED]> writes: > * Page format changes that grow data size are problematic, because there > can be pages that can't be expanded to the new format because there's > not enough space. However, it would be possible to write a pre-upgrade > program to look for the problematic pages, and do a dummy UPDATE on some > tuples to move them away from such pages, to make room. The pre-upgrade > program could be run while the old database is still up, so it doesn't > cause downtime. That sounds good, but there are corner cases where it wouldn't work --- consider a page containing a single maximum-length tuple. In general I don't see us accepting changes that would increase data size, but there's at least one troubling exception on the horizon: per-column or per-value locale support. Another problem is that a strict rule of "no data size increase ever" might forbid acceptance of changes that achieve average space savings at the cost of increasing the size of some lesser-used cases. In short, this point seems to need more thought. regards, tom lane ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Since we're discussing upgrades, let me summarize the discussions we had over dinner in Ottawa for the benefit of all: * pg_migrator is a sound approach to handling catalog changes. * Handling any page format change that doesn't grow the space taken by data is straightforward. Note that all 8.2 -> 8.3 changes fall in this category. * Page format changes that grow data size are problematic, because there can be pages that can't be expanded to the new format because there's not enough space. However, it would be possible to write a pre-upgrade program to look for the problematic pages, and do a dummy UPDATE on some tuples to move them away from such pages, to make room. The pre-upgrade program could be run while the old database is still up, so it doesn't cause downtime. * Page format conversion can be done at upgrade time, scanning all relations and rewriting them. Or it can be done at runtime in ReadBuffer whenever a page in the old format is accessed. We don't know what changes upcoming releases will bring, but 8.2 -> 8.3 upgrade is feasible. Any future changes will need to be considered on a case-by-case basis, but we have a pretty good basic strategy that doesn't impose any strict restrictions on future development. Even if we can't guarantee that we can do a non dump/restore upgrade between every release, if we can do it say between every other release, that's *much* better than none. As before, someone just needs to step up and do it. Zdenek Kotala wrote: Andrew Sullivan wrote: On Wed, Jun 20, 2007 at 12:34:21PM -0400, Robert Treat wrote: FWIW pg_migrator is a pretty good swing at an in-place upgrade tool for 8.1->8.2. Unfortunately until the PGDG decides that in-place upgrade is a constraint their willing to place on development, I see them a good chicken/egg away from making it a continually usefull tool. Or maybe cart/horse. It seems to me that the rule more likely needs to be that the migrator follow the development of the database than that the database engine be strongly constrained by the needs of an upgrade tool. I agree that some commitment is needed, though. I don't think that upgrade will strongly constrain a database engine. But some constrains we can expect. Any new development in some area will have to take care about upgrade. If you break some "upgrade coding rules" you will not to able create upgrade procedure or upgrade process will be risky. For example upgrade between 8.1 -> 8.2 is risky, because you are not able easy determine what format of cidr/inet is used on the page. If something fails during table structure upgrade, then it is not easy to recovery from this situation. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Andrew Sullivan wrote: On Wed, Jun 20, 2007 at 12:34:21PM -0400, Robert Treat wrote: FWIW pg_migrator is a pretty good swing at an in-place upgrade tool for 8.1->8.2. Unfortunately until the PGDG decides that in-place upgrade is a constraint their willing to place on development, I see them a good chicken/egg away from making it a continually usefull tool. Or maybe cart/horse. It seems to me that the rule more likely needs to be that the migrator follow the development of the database than that the database engine be strongly constrained by the needs of an upgrade tool. I agree that some commitment is needed, though. I don't think that upgrade will strongly constrain a database engine. But some constrains we can expect. Any new development in some area will have to take care about upgrade. If you break some "upgrade coding rules" you will not to able create upgrade procedure or upgrade process will be risky. For example upgrade between 8.1 -> 8.2 is risky, because you are not able easy determine what format of cidr/inet is used on the page. If something fails during table structure upgrade, then it is not easy to recovery from this situation. Zdenek ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
On Wed, Jun 20, 2007 at 12:34:21PM -0400, Robert Treat wrote: > FWIW pg_migrator is a pretty good swing at an in-place upgrade tool for > 8.1->8.2. Unfortunately until the PGDG decides that in-place upgrade is a > constraint their willing to place on development, I see them a good > chicken/egg away from making it a continually usefull tool. Or maybe cart/horse. It seems to me that the rule more likely needs to be that the migrator follow the development of the database than that the database engine be strongly constrained by the needs of an upgrade tool. I agree that some commitment is needed, though. A -- Andrew Sullivan | [EMAIL PROTECTED] The whole tendency of modern prose is away from concreteness. --George Orwell ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
On Tuesday 19 June 2007 10:15, Tom Lane wrote:
> Zdenek Kotala <[EMAIL PROTECTED]> writes:
> > I'm little bit confused when we introduce new page layout version? I
> > expect that new version become with changes with pageheader, tuple
> > header or data encoding (varlen/TOAST ...). But in case when there is
> > new data type internal implementation, there was not reason to update
> > version (see inet/cidr between 8.1 -> 8.2). Can me somebody clarify this?
>
> Well, we've changed it when there was a benefit to an existing tool to
> do so. So far that's meant page header and tuple header changes. If
> we ever had a working in-place upgrade solution, I think we'd be willing
> to make the page version account for datatype format changes too.
>
FWIW pg_migrator is a pretty good swing at an in-place upgrade tool for
8.1->8.2. Unfortunately until the PGDG decides that in-place upgrade is a
constraint their willing to place on development, I see them a good
chicken/egg away from making it a continually usefull tool.
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
Re: What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Zdenek Kotala <[EMAIL PROTECTED]> writes: > I'm little bit confused when we introduce new page layout version? I > expect that new version become with changes with pageheader, tuple > header or data encoding (varlen/TOAST ...). But in case when there is > new data type internal implementation, there was not reason to update > version (see inet/cidr between 8.1 -> 8.2). Can me somebody clarify this? Well, we've changed it when there was a benefit to an existing tool to do so. So far that's meant page header and tuple header changes. If we ever had a working in-place upgrade solution, I think we'd be willing to make the page version account for datatype format changes too. regards, tom lane ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
What does Page Layout version mean? (Was: Re: [HACKERS] Reducing NUMERIC size for 8.3)
Simon Riggs wrote: The objections to applying this patch originally were: 1. it changes on-disk format (we've done this, so argument is void) I'm little bit confused when we introduce new page layout version? I expect that new version become with changes with pageheader, tuple header or data encoding (varlen/TOAST ...). But in case when there is new data type internal implementation, there was not reason to update version (see inet/cidr between 8.1 -> 8.2). Can me somebody clarify this? However, In upgrade point of view. It will be necessary to keep information which postgreSQL version creates page to correctly track data type implementation (or special structure). Information in pg_control file and PG_VERSION files are unusable for upgrade purpose. Zdenek ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
