Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Feb 26, 2007 at 01:39:40PM -0500, Tom Lane wrote: [...] Or were you speaking of the pg_class.reltuples count? Yes (modulo my warning, that is) Regards - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux)

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 13:53 +0900, Galy Lee wrote: Tom Lane wrote: Huh? There is no extra cost in what I suggested; it'll perform exactly the same number of index scans that it would do anyway. The things I wanted to say is that: If we can stop at any point, we can make maintenance

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: On Wed, 2007-02-28 at 13:53 +0900, Galy Lee wrote: In the current implementation of concurrent vacuum, the third is not satisfied obviously, the first issue comes to my mind is the lazy_truncate_heap, it takes AccessExclusiveLock for a long time, that is

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Heikki Linnakangas
Tom Lane wrote: Galy Lee [EMAIL PROTECTED] writes: If we can stop at any point, we can make maintenance memory large sufficient to contain all of the dead tuples, then we only need to clean index for once. No matter how many times vacuum stops, indexes are cleaned for once. I beg your pardon?

Re: [HACKERS] Dead Space Map version 2

2007-02-28 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas [EMAIL PROTECTED] writes: Tom Lane wrote: I'd be happier if the DSM content could be treated as just a hint. If we don't have a frozen state, we can't use the DSM to implement index-only scans. To implement index-only scans, the DSM would have to be

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Galy Lee
Simon Riggs wrote: You haven't explained how saving the dead-tuple-list could be done in a safe mannner and it seems risky to me. The files are placed in a new directory $PGDATA/pg_vacuum with the name: spcNode.dbNode.relNode for each relations which have been interrupted during vacuum. It

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Zeugswetter Andreas ADI SD
The things I wanted to say is that: If we can stop at any point, we can make maintenance memory large sufficient to contain all of the dead tuples, then we only need to clean index for once. No matter how many times vacuum stops, indexes are cleaned for once. I agree that the

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Zeugswetter Andreas ADI SD
You haven't explained how saving the dead-tuple-list could be done in a safe mannner and it seems risky to me. The files are placed in a new directory $PGDATA/pg_vacuum with the name: spcNode.dbNode.relNode for each relations which have been interrupted during vacuum. It has the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Zeugswetter Andreas ADI SD
But we do don't we? fsync = off, full_page_writes = off? BTW, our testing seems to indicate that full_page_writes = off is safe on Solaris 10 on good hardware. At least, we haven't been able to break it yet. Is that an OS-dependent parameter? I always assumed it depended

[HACKERS] VACUUM and spoiling the buffer manager cache

2007-02-28 Thread Simon Riggs
Round about v.161 of src/storage/buffer/bufmgr.c, during the development of 8.0 a change was introduced to prevent VACUUM from changing the state of the Adaptive Replacement Cache buffer management strategy. At the time that change made lots of sense. Since then we have changed the buffer

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: First, rather than using 16-bytes per page and having to deal with handling the non-contiguous space, why not just use a page-level checksum like everyone else? Most of the systems I've seen seem to employ a simple CRC16 or CRC32. I think a CRC

Re: [HACKERS] Packed short varlenas, what next?

2007-02-28 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: I've committed this, but in testing with a hack that does ntohl() in the VARSIZE macro and vice-versa in SET_VARSIZE, I find that core passes regression but several contrib modules do not. It looks like the contrib modules were depending on various random

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 11:19 +0100, Zeugswetter Andreas ADI SD wrote: The things I wanted to say is that: If we can stop at any point, we can make maintenance memory large sufficient to contain all of the dead tuples, then we only need to clean index for once. No matter how many times

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 09:38 +, Heikki Linnakangas wrote: Tom Lane wrote: Galy Lee [EMAIL PROTECTED] writes: If we can stop at any point, we can make maintenance memory large sufficient to contain all of the dead tuples, then we only need to clean index for once. No matter how many

[HACKERS] psql problem querying relations

2007-02-28 Thread Zoltan Boszormenyi
Hi, this is with current CVS code: # \dt ERROR: did not find '}' at end of input node Server log: ERROR: did not find '}' at end of input node STATEMENT: SELECT n.nspname as Schema, c.relname as Name, CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i'

Re: [HACKERS] Implicit casts with generic arrays

2007-02-28 Thread Peter Eisentraut
Am Dienstag, 27. Februar 2007 19:50 schrieb Tom Lane: Seems basically we'd want to not cast unknown to anyarray unless there is some additional bit of context suggesting that that's the right thing. But what should that extra requirement be? Can we go as far as not doing this cast implicitly

Re: [HACKERS] psql problem querying relations

2007-02-28 Thread Andrew Dunstan
Zoltan Boszormenyi wrote: Hi, this is with current CVS code: # \dt ERROR: did not find '}' at end of input node Server log: ERROR: did not find '}' at end of input node It's working for me. Have you tried with a fresh checkout or after running make clean before you build? cheers

[HACKERS] Major Feature Interactions

2007-02-28 Thread Simon Riggs
There's a number of major features all coming together over the next weeks, so I wanted to raise some awareness of possible areas of interaction. My concern is more than just will multiple patches apply together?, but extends into whether the features interact with each other with synergy, ignore

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Gregory Stark
Simon Riggs [EMAIL PROTECTED] writes: How much memory would it save during VACUUM on a 1 billion row table with 200 million dead rows? Would that reduce the number of cycles a normal non-interrupted VACUUM would perform? It would depend on how many dead tuples you have per-page. If you have a

Re: [HACKERS] psql problem querying relations

2007-02-28 Thread Zoltan Boszormenyi
Andrew Dunstan írta: Zoltan Boszormenyi wrote: Hi, this is with current CVS code: # \dt ERROR: did not find '}' at end of input node Server log: ERROR: did not find '}' at end of input node It's working for me. Have you tried with a fresh checkout or after running make clean before you

[HACKERS] Compilation errors

2007-02-28 Thread Gregory Stark
Does anyone happen to know what it is about my build environment that causes these errors? I get a similar slew of errors from each flex lexer we built including the plpgsql lexer, the ecpg lexer, etc. Do I have the wrong version of flex installed? Something else? It's quite annoying, especially

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: But we've already seen that CRC checks can be expensive. Not everyone will want to take the cpu hit. Storing a byte counter in every block is cheap. CRC checking a page is most certainly the simplest. And, I disagree that it would be worse

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Gregory Stark wrote: Does anyone happen to know what it is about my build environment that causes these errors? I get a similar slew of errors from each flex lexer we built including the plpgsql lexer, the ecpg lexer, etc. Do I have the wrong version of flex installed? Something else? It's quite

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: This proposed design is overcomplicated and a waste of space. I mean, we reduce storage overhead using phantom command id and variable varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage overhead using phantom

Re: [HACKERS] Compilation errors

2007-02-28 Thread Peter Eisentraut
Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing. Everybody gets them. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Stefan Kaltenbrunner
Gregory Stark wrote: Jonah H. Harris [EMAIL PROTECTED] writes: This proposed design is overcomplicated and a waste of space. I mean, we reduce storage overhead using phantom command id and variable varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Peter Eisentraut wrote: Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing. Everybody gets them. That's not what the buildfarm shows. example:

Re: [HACKERS] VACUUM and spoiling the buffer manager cache

2007-02-28 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: VACUUM's current behaviour is to take blocks it has touched and place them on the head of the freelist, allowing them to be reused. No, it puts them at the tail of the freelist. So I am unconvinced by the rest of your argument.

Re: [HACKERS] [PATCHES]

2007-02-28 Thread korryd
Not that I think that anyone owning both a law degree and a computer in 2007 should legitimately be able to plead innocence here. FAST Australia's lawyers are making themselves look like idiots, and the same for every other company tacking on such notices. I think the real bottom line

Re: [HACKERS] Packed short varlenas, what next?

2007-02-28 Thread Tom Lane
Gregory Stark [EMAIL PROTECTED] writes: I just noticed that last night myself. In particular the GIST modules seems to be a major problem. they define dozens of new objects, many of which are just passing around C data structures internally but some of which are objects which get stored in the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: We reduced storage overhead using phantom command id by 8 bytes *per tuple*. I hardly think 8 bytes per page is much of a concern. You're already losing an average of 1/2 a tuple per page to rounding and that's a minimum of 16 bytes for the

[HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
Hello hackers, I was starting to think about next SOC and the project for it. And for a long time I wanted to implement the user quotas in PG. So, I'll try to explain my understanding of the implementation, and I'll be happy to hear any comments, objections, or pointings to my

Re: [HACKERS] VACUUM and spoiling the buffer manager cache

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 11:09 -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: VACUUM's current behaviour is to take blocks it has touched and place them on the head of the freelist, allowing them to be reused. No, it puts them at the tail of the freelist. That's a minor point

Re: [HACKERS] Compilation errors

2007-02-28 Thread Tom Lane
Peter Eisentraut [EMAIL PROTECTED] writes: Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing. Everybody gets them. I don't. What version of flex are you guys using?

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Do you really doubt that memcpy is faster than CRC32 checking? Especially when you're already doing memcpy anyways and the only overhead is the few unaligned bytes at the end and the 8 one-byte copies? I'm saying the complexity and implementation of it is going to get you a bit more than

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: 1) The main idea is to implement the per-user quota (not per tablespace for example). So, during the creation of the new user some quota can be specified, and after that the size of all the relations *owned* by that user should be limited by that

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joshua D. Drake
Sergey E. Koposov wrote: Hello hackers, I was starting to think about next SOC and the project for it. And for a long time I wanted to implement the user quotas in PG. So, I'll try to explain my understanding of the implementation, and I'll be happy to hear any comments, objections, or

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Tom Lane wrote: Sergey E. Koposov [EMAIL PROTECTED] writes: 1) The main idea is to implement the per-user quota (not per tablespace for example). So, during the creation of the new user some quota can be specified, and after that the size of all the relations *owned* by

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and checked. The good news is that both of these are non-critical paths.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Gregory Stark [EMAIL PROTECTED] writes: We've already seen wal CRC checking show up at the top of profiles. Do you really doubt that memcpy is faster than CRC32 checking? Especially when you're already doing memcpy anyways and the only overhead is the few unaligned bytes at the end and the 8

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: I could see this being useful per database, maybe. It seems like kind of an odd feature. Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Joshua D. Drake [EMAIL PROTECTED] writes: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Darn, I wish I had seen this post before I posted the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Heh, I like it :) -- Jonah H. Harris, Software

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Bruce Momjian
I have added this to the developer's FAQ to clarify the situtation of posting a patch: liPostgreSQL is licensed under a BSD license. By posting a patch to the public PostgreSQL mailling lists, you are giving the PostgreSQL Global Development Group the non-revokable right to

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Bruce Momjian [EMAIL PROTECTED] writes: I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and checked. The good news is

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Gregory Stark wrote: Joshua D. Drake [EMAIL PROTECTED] writes: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Darn, I wish I had seen this post

Re: [HACKERS] Compilation errors

2007-02-28 Thread Gregory Stark
Tom Lane [EMAIL PROTECTED] writes: Peter Eisentraut [EMAIL PROTECTED] writes: Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing. Everybody gets them. I don't. What version of flex

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joshua D. Drake
Sergey E. Koposov wrote: On Wed, 28 Feb 2007, Joshua D. Drake wrote: I could see this being useful per database, maybe. It seems like kind of an odd feature. Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Joshua D. Drake
Bruce Momjian wrote: I have added this to the developer's FAQ to clarify the situtation of posting a patch: liPostgreSQL is licensed under a BSD license. By posting a patch to the public PostgreSQL mailling lists, you are giving the PostgreSQL Global Development Group the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Gregory Stark wrote: Bruce Momjian [EMAIL PROTECTED] writes: I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Bruce Momjian
Joshua D. Drake wrote: Bruce Momjian wrote: I have added this to the developer's FAQ to clarify the situtation of posting a patch: liPostgreSQL is licensed under a BSD license. By posting a patch to the public PostgreSQL mailling lists, you are giving the PostgreSQL

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: Sergey E. Koposov wrote: On Wed, 28 Feb 2007, Joshua D. Drake wrote: Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I remember user

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joshua D. Drake
Then you cannot run any queries that extend the size of your relations (for example INSERT, UPDATE etc.). Unless you drop your tables or DELETE something Interesting. Well my two cents is don't go any deeper than database. I.e; don't try and track to the individual relation. Joshua D. Drake

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Tom Lane
Sergey E. Koposov [EMAIL PROTECTED] writes: Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I remember user requests for quotas in the mailing lists ... It hasn't ever made it onto

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joshua D. Drake
Tom Lane wrote: Sergey E. Koposov [EMAIL PROTECTED] writes: Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I remember user requests for quotas in the mailing lists ... It hasn't

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Tom Lane
Bruce Momjian [EMAIL PROTECTED] writes: Joshua D. Drake wrote: We should add this to the mailing list signup pages and the welcome pages to the lists. Yep, good idea. Marc? For -patches and -hackers, I agree. It seems a bit legalistic and off-putting for the general lists, though.

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Tom Lane wrote: 2) I looked into the code, and from my understanding, the main part of the code which should be affected by the quotas is storage/smgr/md.c. md.c is too low level to do catalog accesses and thus too low level to know who owns what. That's probably a dumb

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Gregory Stark wrote: Tom Lane [EMAIL PROTECTED] writes: Peter Eisentraut [EMAIL PROTECTED] writes: Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing.

[HACKERS] Possible Bug: high CPU usage for stats collector in 8.2

2007-02-28 Thread Joshua D. Drake
Darcy Buskermolen wrote: I'm observing high CPU usage (95%) of a 2.6GHz opteron by the stats collector on an 8.2.3 box investigation has lead me to belive that the stats file is written a lot more often that once every 500ms the following shows this behavior. PostgreSQL 8.2.3 on

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Martijn van Oosterhout
On Wed, Feb 28, 2007 at 09:58:52AM -0800, Joshua D. Drake wrote: E.g; if user foo then pg_database_size may not be than X? I guess the big question would be when do we check though? At each transaction seems like it would add significant overhead, especially if we had to rollback the

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joshua D. Drake
Martijn van Oosterhout wrote: On Wed, Feb 28, 2007 at 09:58:52AM -0800, Joshua D. Drake wrote: E.g; if user foo then pg_database_size may not be than X? I guess the big question would be when do we check though? At each transaction seems like it would add significant overhead, especially if

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the system up as soon as possible as long as they know any corrupt pages

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the system up as soon as possible as

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Oleg Bartunov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: Tom Lane wrote: Sergey E. Koposov [EMAIL PROTECTED] writes: Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I remember user requests for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: We have several methods suggested to check the blocks, like CRC. My point was that, whatever check method we use, we should be prepared to check on startup, or at least make it the default for a crash restart. Sounds like it should be a guc.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: We have several methods suggested to check the blocks, like CRC. My point was that, whatever check method we use, we should be prepared to check on startup, or at least make it the default for a crash restart.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:10 -0500, Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Keep in mind if you don't check it on startup, you will be checking it for every read, which for rare crashes, might not be wise. Well understood. That's how most everyone configures their database systems; they certainly don't optimize for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. -- Jonah H. Harris, Software Architect | phone: 732.331.1324 EnterpriseDB

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Heikki Linnakangas
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: But if the system was shut down uncleanly as the result of a Postgres crash or fast shutdown of Postgres then that isn't an issue. And many users may prefer to bring the system up as soon as possible as long as they

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. Uhmm... most do because most don't ever touch the

[HACKERS] 8.1.8 Installer Fails on Win2k Server

2007-02-28 Thread Saqib Awan
I am installing on a Cisco Media Server 7800 running Win2k Server and am noticing that the installer fails every time with the error dialog saying Failed to create process for initdb: Access is denied. It looks like that I need to change some permission in the registry allowing other users spawn

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:54 -0500, Jonah H. Harris wrote: On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. It's got a hard

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis [EMAIL PROTECTED] wrote: I'm not saying there's no cost, but the extra recovery cost seems lower to me than the CRC cost on every data page read during operation. I agree, I just think it should be configurable. Also, if we find an error, do we even have the ability to

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. Uhmm... most do because most don't ever touch the postgresql.conf and those that do, don't touch checkpoints because they don't

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. Uhmm... most do because most don't ever touch the postgresql.conf and those that do, don't touch checkpoints

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake [EMAIL PROTECTED] wrote: I am not trying to pick on the issue but I do think it is important to recognize that literally only those in the know, are going to ever touch the postgresql.conf. I agree. -- Jonah H. Harris, Software Architect | phone: 732.331.1324

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Tom Lane
Joshua D. Drake [EMAIL PROTECTED] writes: Martijn van Oosterhout wrote: Generally, rolling back a transaction doesn't reduce the amount of disk used. Only VACUUM FULL actually shrinks relations. Right, but what I mean was -- if we rollback because we hit quota we could potentially cause even

Re: [HACKERS] Proposal for Implenting read-only queries during wal replay (SoC 2007)

2007-02-28 Thread plabrh1
Thanks Josh, I'll look for the earlier one and try to add it there... -Paul -Original Message- From: Joshua D. Drake [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 28, 2007 12:09 AM To: Paul Silveira Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Proposal for Implenting

Re: [HACKERS] Compilation errors

2007-02-28 Thread Tom Lane
Andrew Dunstan [EMAIL PROTECTED] writes: Gregory Stark wrote: flex 2.5.33 Aha! Known to be broken, iirc. Use flex 2.5.4a No, the known breakages with flex were years ago; 2.5.33 has only been out a year. I think 2.5.31 might have been the one we saw big problems with (there's a note warning

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people off and lose people if you say things like that rather

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Oleg Bartunov
On Wed, 28 Feb 2007, Tom Lane wrote: Joshua D. Drake [EMAIL PROTECTED] writes: Martijn van Oosterhout wrote: Generally, rolling back a transaction doesn't reduce the amount of disk used. Only VACUUM FULL actually shrinks relations. Right, but what I mean was -- if we rollback because we

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Tom Lane wrote: Joshua D. Drake [EMAIL PROTECTED] writes: Martijn van Oosterhout wrote: Generally, rolling back a transaction doesn't reduce the amount of disk used. Only VACUUM FULL actually shrinks relations. Right, but what I mean was -- if we rollback because we

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: Jonah H. Harris [EMAIL PROTECTED] writes: Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people off and

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency check? How other databases do

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Gregory Stark wrote: flex 2.5.33 Aha! Known to be broken, iirc. Use flex 2.5.4a No, the known breakages with flex were years ago; 2.5.33 has only been out a year. I think 2.5.31 might have been the one we saw

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
I wrote: I guess we could conditionally add prototypes for those functions to all the .l files if you really want to move to 2.5.33. Kinda yucky, though. Actually, we couldn't.The definition section from the .l file gets included after these functions. So we'd need to include something

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Heikki Linnakangas
Gregory Stark wrote: Simon Riggs [EMAIL PROTECTED] writes: How much memory would it save during VACUUM on a 1 billion row table with 200 million dead rows? Would that reduce the number of cycles a normal non-interrupted VACUUM would perform? It would depend on how many dead tuples you have

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Martijn van Oosterhout
On Wed, Feb 28, 2007 at 03:57:56PM -0500, Tom Lane wrote: It's worse than that, because VACUUM FULL will actually bloat the indexes on the way to being able to reduce the table size (since it has to make new index entries for rows it moves). If the limit is strictly I was thinking that

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joshua D. Drake
I don't know, but in my opinion, I don't see anything bad in requiring dropping the data if the quota is full. That's what usually occurs in the case of normal filesystem quota... If you don't have a space there, you cannot edit files, copy them etc... And that solution should be definitely

Re: [HACKERS] Possible Bug: high CPU usage for stats collector in 8.2

2007-02-28 Thread Joshua D. Drake
Joshua D. Drake wrote: Darcy Buskermolen wrote: I'm observing high CPU usage (95%) of a 2.6GHz opteron by the stats collector on an 8.2.3 box investigation has lead me to belive that the stats file is written a lot more often that once every 500ms the following shows this behavior. I

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 21:13 +, Gregory Stark wrote: Hm that's an interesting thought. We only really have to check pages that would have received a full page write since the last checkpoint. So if we made Do we ever do a partial page write, or is what you're saying equivalent to we only

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Joachim Wieland
On Wed, Feb 28, 2007 at 12:56:13PM -0500, Tom Lane wrote: It hasn't ever made it onto the TODO list, which means there's not a consensus that we need it. Such a patch could improve the acceptance of PostgreSQL in shared hosting environments. Note that a database without quotas can be filled up

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: The bad point is not that we would rollback the transaction. The bad point is what happens when you need to rollback a transaction and in your scenario it is quite plausible that a large rollback could occur, more than once, causing the requirement of

Re: [HACKERS] SOC user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joachim Wieland wrote: On Wed, Feb 28, 2007 at 12:56:13PM -0500, Tom Lane wrote: It hasn't ever made it onto the TODO list, which means there's not a consensus that we need it. Such a patch could improve the acceptance of PostgreSQL in shared hosting environments. Note

Re: [HACKERS] Compilation errors

2007-02-28 Thread Tom Lane
Andrew Dunstan [EMAIL PROTECTED] writes: Actually, we couldn't.The definition section from the .l file gets included after these functions. So we'd need to include something in gram.y before including scan.c. Actually, since we don't use any of those functions, the following advice from the

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Simon Riggs
On Tue, 2007-02-27 at 09:32 -0800, Josh Berkus wrote: Simon, I think if you address me in a mail, it would be best not to explicitly *remove* my name from the address list. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian [EMAIL PROTECTED] wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
Jonah H. Harris [EMAIL PROTECTED] writes: On 2/28/07, Gregory Stark [EMAIL PROTECTED] wrote: Jonah H. Harris [EMAIL PROTECTED] writes: Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in

  1   2   >