Re: [HACKERS] [COMMITTERS] pgsql: Fix PGXS conventions so that extensions can be built against

2007-07-03 Thread Fabien COELHO
Dear Robert, Fix PGXS conventions so that extensions can be built against Postgres installations whose pg_config program does not appear first in the PATH. Per gripe from Eddie Stanley and subsequent discussions with Fabien Coelho and others. Is there any chance of this being backpatched? I

Re: [HACKERS] what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL

2007-07-03 Thread Gregory Stark
Pavel Stehule [EMAIL PROTECTED] writes: Global temp table can be created from template only when is used. It's has not negative efect on app which doesn't use it. The benefit of g.t.t. is simplifycation of stored procedures. And if it's used in 200 txns/s? Imagine the earlier poster who was

[HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
I attach In-Place upgrade project concept. Any technical details about implementation of each part will be sent later (after concept acceptance). Please, let me know your comments. thanks Zdenek In-place Upgrade project --- Overview PostgreSQL

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Michael Paesold
Joshua D. Drake wrote: Alvaro Herrera wrote: Joshua D. Drake wrote: Did we change the default autovac parameters for 8.3 (beyond turning it on?) because on any reasonably used database, they are way to conservative. We're still on time to change them ... Any concrete proposals? I could

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Heikki Linnakangas
Zdenek Kotala wrote: Project Goals - ... 3) Do not require an old version of PostgreSQL to be installed. Why not? Having two versions installed at the same time doesn't seem like a problem to me. You can remove the old version as soon as the upgrade is done. 4) Allow a fallback to

Re: [HACKERS] what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL

2007-07-03 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: Alvaro Herrera [EMAIL PROTECTED] writes: Tom Lane escribió: I rather doubt that. The most likely implementation would involve cloning a template entry into pg_class. How about a new relkind which causes the table to be located in

Re: [HACKERS] what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL

2007-07-03 Thread Pavel Stehule
Global temp table can be created from template only when is used. It's has not negative efect on app which doesn't use it. The benefit of g.t.t. is simplifycation of stored procedures. And if it's used in 200 txns/s? Imagine the earlier poster who was looking for a way to display the count of

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Gregory Stark
Heikki Linnakangas [EMAIL PROTECTED] writes: As before, upgrade can be done, it's just a matter of someone scratching the itch. pg_migrator can handle the catalog changes. Doing the page conversion from 8.2 - 8.3 is possible, and it could be done on-the-fly inside PostgreSQL the first time a

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Heikki Linnakangas wrote: Zdenek Kotala wrote: Project Goals - ... 3) Do not require an old version of PostgreSQL to be installed. Why not? Having two versions installed at the same time doesn't seem like a problem to me. You can remove the old version as soon as the upgrade is done.

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Heikki Linnakangas
Gregory Stark wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: As before, upgrade can be done, it's just a matter of someone scratching the itch. pg_migrator can handle the catalog changes. Doing the page conversion from 8.2 - 8.3 is possible, and it could be done on-the-fly inside

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Heikki Linnakangas wrote: Gregory Stark wrote: Conceivably we could grab another infomask bit to indicate uses new-style varlenas and then have heaptuple.c understand how to convert them in place. But that leads to a ton of memory management or page locking problems. My thinking is that

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 12:05:07PM +0100, Heikki Linnakangas wrote: This is actually a bit of a problem. We would need to know when we read in a page what the tupledescriptor for that relation looks like to know which fields are varlena. I'm not sure how easy it would be to arrange for the

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Heikki Linnakangas
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 12:05:07PM +0100, Heikki Linnakangas wrote: This is actually a bit of a problem. We would need to know when we read in a page what the tupledescriptor for that relation looks like to know which fields are varlena. I'm not sure how easy it

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 12:05:07PM +0100, Heikki Linnakangas wrote: This is actually a bit of a problem. We would need to know when we read in a page what the tupledescriptor for that relation looks like to know which fields are varlena. I'm not sure how easy it

Re: [HACKERS] Postgresql.conf cleanup

2007-07-03 Thread Peter Eisentraut
Am Montag, 2. Juli 2007 13:03 schrieb Josh Berkus: (change requires restart): this phrase appears over 20 times in the notes.  This is enough times to be really repetitive and take up a lot of scrolling space, while not actually covering all startup-time parameters. Which ones are missing?

Re: [HACKERS] what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL

2007-07-03 Thread Bruce Momjian
Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Tom Lane escribi?: I rather doubt that. The most likely implementation would involve cloning a template entry into pg_class. How about a new relkind which causes the table to be located in

Re: [HACKERS] todo: Hash index creation

2007-07-03 Thread Naz Gassiep
Wow... not sure how I missed that. I *did* create this schema ages ago, perhaps it wasn't there, or at the time I had no idea what the implications were. *shrug* Regards, - Naz. Tom Lane wrote: Naz Gassiep [EMAIL PROTECTED] writes: As a result, when creating tables containing

[HACKERS] how to pg_dump, based in select command

2007-07-03 Thread Jeferson Kasper
Hello folks. I want to know how to dump some parts of a database? I need to extract the records in a select * from table and the pg_dump or other tool will create a file with the records found in this select command, and after this, i will restore this file in another database with the same

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: My thinking is that when a page in the old format is read in, it's converted to the new format before doing anything else with it. Yeah, I'm with Heikki on this. What I see as a sane project definition is: * pg_migrator or

Re: [HACKERS] how to pg_dump, based in select command

2007-07-03 Thread Josh Tolley
On 7/3/07, Jeferson Kasper [EMAIL PROTECTED] wrote: Hello folks. I want to know how to dump some parts of a database? I need to extract the records in a select * from table and the pg_dump or other tool will create a file with the records found in this select command, and after this, i will

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 11:36:03AM -0400, Tom Lane wrote: I'm not sure it's feasible to expect that we can change representations of user-defined types, either. I don't see how you would do that without catalog access (to look up the UDT), and the page conversion procedure is going to have to

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Martijn van Oosterhout
On Tue, Jul 03, 2007 at 07:13:29PM +0200, Zdenek Kotala wrote: pg_migrator is separate tool which requires old postgres version and I would like to have solution in postgres binary without old version presence. Very often new postgres version is store in same location (e.g. /usr/bin) and

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: Tom Lane wrote: Yeah, I'm with Heikki on this. What I see as a sane project definition is: * pg_migrator or equivalent to convert the system catalogs * a hook in ReadBuffer to allow a data page conversion procedure to be applied, on the basis of

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Martijn van Oosterhout wrote: On Tue, Jul 03, 2007 at 07:13:29PM +0200, Zdenek Kotala wrote: pg_migrator is separate tool which requires old postgres version and I would like to have solution in postgres binary without old version presence. Very often new postgres version is store in same

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Zdenek Kotala [EMAIL PROTECTED] writes: It is not downgrade. It is about keep old structure until user says convert to the new data structure. As Martijn already pointed out, the odds of problems surfacing only after that conversion starts seem high enough to render the whole idea a bit

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: My thinking is that when a page in the old format is read in, it's converted to the new format before doing anything else with it. Yeah, I'm with Heikki on this. What I see as a sane project definition is: * pg_migrator or equivalent to convert

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Zdenek Kotala
Tom Lane wrote: Again, you are setting yourself up for complete failure if you insist on having every possible nicety in the first version. An incremental approach is far more likely to succeed than a big bang. Yes, I know. I don't want to solve everything in one patch. I just looking

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Richard Huxton
Tom Lane wrote: Keep in mind that if your proposal involves any serious limitation on the developers' freedom to refactor internal backend APIs or change catalog representations around, it *will be rejected*. Do not have any illusions on that point. It'll be a tough enough sell freezing

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Martijn van Oosterhout [EMAIL PROTECTED] writes: On Tue, Jul 03, 2007 at 11:36:03AM -0400, Tom Lane wrote: ... (Thought experiment: a page is read in during crash recovery or PITR slave operation, and discovered to have the old format.) Hmm, actually, what's the problem with PITR restoring a

[HACKERS] ACM Paper relevant to our buffer algorithm

2007-07-03 Thread Gregory Stark
Incidentally I found this paper in ACM SIGMETRICS 1992 covering more or less precisely the same algorithm we're using for our clock sweep. I haven't quite digested it yet myself so I'm not sure what the conclusions about weights tell us to do with our buffer usage counter. I put a copy up for

Re: [HACKERS] ACM Paper relevant to our buffer algorithm

2007-07-03 Thread Greg Smith
Here are some more recent papers that also give good insight into research in this area: http://www.cs.usask.ca/~wew036/comprehensive.pdf http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-05-3.pdf -- * Greg Smith [EMAIL PROTECTED] http://www.gregsmith.com Baltimore, MD

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Jim C. Nasby
On Tue, Jul 03, 2007 at 11:31:08AM +0200, Michael Paesold wrote: Joshua D. Drake wrote: Alvaro Herrera wrote: Joshua D. Drake wrote: Did we change the default autovac parameters for 8.3 (beyond turning it on?) because on any reasonably used database, they are way to conservative. We're

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Jim C. Nasby
On Mon, Jul 02, 2007 at 11:19:12PM -0400, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: http://developer.postgresql.org/pgdocs/postgres/routine-vacuuming.html : Well, with autovac defaulting to ON in 8.3, that's certainly obsolete text now. Is there a reason to say anything

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: On Mon, Jul 02, 2007 at 11:19:12PM -0400, Tom Lane wrote: Is there a reason to say anything beyond use autovac? There is; I know that things like web session tables aren't handled very well by autovacuum if there are any moderately large tables (anything

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Alvaro Herrera
Jim C. Nasby wrote: On Tue, Jul 03, 2007 at 11:31:08AM +0200, Michael Paesold wrote: Joshua D. Drake wrote: Alvaro Herrera wrote: Joshua D. Drake wrote: Did we change the default autovac parameters for 8.3 (beyond turning it on?) because on any reasonably used database, they are way to

Re: [HACKERS] what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL

2007-07-03 Thread Jim C. Nasby
On Tue, Jul 03, 2007 at 11:49:05AM -0400, Bruce Momjian wrote: Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Tom Lane escribi?: I rather doubt that. The most likely implementation would involve cloning a template entry into pg_class. How about a new relkind which

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Kevin Grittner
On Tue, Jul 3, 2007 at 3:36 PM, in message [EMAIL PROTECTED], Tom Lane [EMAIL PROTECTED] wrote: Jim C. Nasby [EMAIL PROTECTED] writes: On Mon, Jul 02, 2007 at 11:19:12PM -0400, Tom Lane wrote: Is there a reason to say anything beyond use autovac? There is; I know that things like web

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: 2. Page format conversion is WAL-logged as a complete page replacement It seems we get that for free. By definition any modification to a page after conversion will be the first record for that page since a checkpoint. It just means we have to force

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Kevin Grittner
On Tue, Jul 3, 2007 at 5:17 PM, in message [EMAIL PROTECTED], Kevin Grittner [EMAIL PROTECTED] wrote: We have a 406GB table where 304GB is in one table. The next two tables It's probably obvious, but I meant a 406GB database. Sorry. ---(end of

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Alvaro Herrera
Kevin Grittner wrote: We have a 406GB table where 304GB is in one table. The next two tables are 57GB and 40GB. Inserts to these three tables are constant during the business day, along with inserts, updates, and very few deletes to the other tables. Database modifications are few and

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: 2. Page format conversion is WAL-logged as a complete page replacement It seems we get that for free. By definition any modification to a page after conversion will be the first record for that page since a checkpoint.

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Richard Huxton [EMAIL PROTECTED] writes: One thing no-one's mentioned is how we're going to deal with definitive incompatibilities. I don't really think that in-place update changes that story at all. The advice has always been read the release notes and test your applications before updating.

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: Gregory Stark [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] writes: 2. Page format conversion is WAL-logged as a complete page replacement It seems we get that for free. By definition any modification to a page after conversion will be the first

Re: [HACKERS] Proposal: In-Place upgrade concept

2007-07-03 Thread Tom Lane
Gregory Stark [EMAIL PROTECTED] writes: I do have an objection though: TOAST data. Even were the bufmgr to be provided with a mechanism to find a data type conversion function I don't see what it would do about a toasted datum. Urgh, that *is* a nasty thought :-( Out-of-line datums aren't the

[HACKERS] Why so many out-of-disk-space failures on buildfarm machines?

2007-07-03 Thread Tom Lane
It seems like we see a remarkable number of occurrences of $subject. For instance, right now we have these members failing on various branches: echidna No space left on device asp No space left on device herring No space left on device (icc seems particularly unable

Re: [HACKERS] Still recommending daily vacuum...

2007-07-03 Thread Matthew T. O'Connor
Alvaro Herrera wrote: Jim C. Nasby wrote: FWIW, I normally go with the 8.2 defaults, though I could see dropping vacuum_scale_factor down to 0.1 or 0.15. I also think the thresholds could be decreased further, maybe divide by 10. How about pushing thresholds all the way down to 0? As long

Re: [HACKERS] Why so many out-of-disk-space failures on buildfarm machines?

2007-07-03 Thread Kris Jurka
On Tue, 3 Jul 2007, Tom Lane wrote: I realize that a lot of these members are running on old underpowered machines with not so much disk, but is it possible that the buildfarm itself is leaking disk space? Not cleaning up log files for instance? No, the buildfarm does not leak disk space.

Re: [HACKERS] what is difference between LOCAL and GLOBAL TEMP TABLES in PostgreSQL

2007-07-03 Thread Bruce Momjian
The use case is any system that uses temp tables in an OLTP setting, which certainly isn't uncommon. The problem is that today (and as well with a global temp table that is still writing to the catalogs) is that every OLTP operation that creates or drops a temp table is doing DDL. At best,