Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread tomas
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, Feb 26, 2007 at 01:39:40PM -0500, Tom Lane wrote: [...] > Or were you speaking of the pg_class.reltuples count? Yes (modulo my warning, that is) Regards - -- tomás -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFF5T2SB

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 13:53 +0900, Galy Lee wrote: > > Tom Lane wrote: > > Huh? There is no extra cost in what I suggested; it'll perform > > exactly the same number of index scans that it would do anyway. > > The things I wanted to say is that: > If we can stop at any point, we can make mainten

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > On Wed, 2007-02-28 at 13:53 +0900, Galy Lee wrote: >> In the current implementation of concurrent vacuum, the third is not >> satisfied obviously, the first issue comes to my mind is the >> lazy_truncate_heap, it takes AccessExclusiveLock for a long time,

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Heikki Linnakangas
Tom Lane wrote: Galy Lee <[EMAIL PROTECTED]> writes: If we can stop at any point, we can make maintenance memory large sufficient to contain all of the dead tuples, then we only need to clean index for once. No matter how many times vacuum stops, indexes are cleaned for once. I beg your pardon

Re: [HACKERS] Dead Space Map version 2

2007-02-28 Thread Heikki Linnakangas
Tom Lane wrote: Heikki Linnakangas <[EMAIL PROTECTED]> writes: Tom Lane wrote: I'd be happier if the DSM content could be treated as just a hint. If we don't have a frozen state, we can't use the DSM to implement index-only scans. To implement index-only scans, the DSM would have to be exp

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Galy Lee
Simon Riggs wrote: > You haven't explained how saving the dead-tuple-list could be done > in a safe mannner and it seems risky to me. The files are placed in a new directory $PGDATA/pg_vacuum with the name: spcNode.dbNode.relNode for each relations which have been interrupted during vacuum. It

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Zeugswetter Andreas ADI SD
> > The things I wanted to say is that: > > If we can stop at any point, we can make maintenance memory large > > sufficient to contain all of the dead tuples, then we only need to > > clean index for once. No matter how many times vacuum > stops, indexes > > are cleaned for once. > > I agree

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Zeugswetter Andreas ADI SD
> > You haven't explained how saving the dead-tuple-list could be done in > > a safe mannner and it seems risky to me. > > The files are placed in a new directory $PGDATA/pg_vacuum > with the name: spcNode.dbNode.relNode for each relations > which have been interrupted during vacuum. > > It h

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Zeugswetter Andreas ADI SD
> > > > But we do don't we? fsync = off, full_page_writes = off? > > > > BTW, our testing seems to indicate that full_page_writes = off is safe > > on Solaris 10 on good hardware. At least, we haven't been able to break it yet. > > > > Is that an OS-dependent parameter? I always assumed it de

[HACKERS] VACUUM and spoiling the buffer manager cache

2007-02-28 Thread Simon Riggs
Round about v.161 of src/storage/buffer/bufmgr.c, during the development of 8.0 a change was introduced to prevent VACUUM from changing the state of the Adaptive Replacement Cache buffer management strategy. At the time that change made lots of sense. Since then we have changed the buffer managemen

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > First, rather than using 16-bytes per page and having to deal with > handling the non-contiguous space, why not just use a page-level > checksum like everyone else? Most of the systems I've seen seem to > employ a simple CRC16 or CRC32. I think a C

Re: [HACKERS] Packed short varlenas, what next?

2007-02-28 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > I've committed this, but in testing with a hack that does ntohl() in the > VARSIZE macro and vice-versa in SET_VARSIZE, I find that core passes > regression but several contrib modules do not. It looks like the > contrib modules were depending on various r

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 11:19 +0100, Zeugswetter Andreas ADI SD wrote: > > > The things I wanted to say is that: > > > If we can stop at any point, we can make maintenance memory large > > > sufficient to contain all of the dead tuples, then we only need to > > > clean index for once. No matter how

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 09:38 +, Heikki Linnakangas wrote: > Tom Lane wrote: > > Galy Lee <[EMAIL PROTECTED]> writes: > >> If we can stop at any point, we can make maintenance memory large > >> sufficient to contain all of the dead tuples, then we only need to > >> clean index for once. No matter

[HACKERS] psql problem querying relations

2007-02-28 Thread Zoltan Boszormenyi
Hi, this is with current CVS code: # \dt ERROR: did not find '}' at end of input node Server log: ERROR: did not find '}' at end of input node STATEMENT: SELECT n.nspname as "Schema", c.relname as "Name", CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i'

Re: [HACKERS] Implicit casts with generic arrays

2007-02-28 Thread Peter Eisentraut
Am Dienstag, 27. Februar 2007 19:50 schrieb Tom Lane: > Seems basically we'd want to not cast unknown to anyarray unless there > is some additional bit of context suggesting that that's the right thing. > But what should that extra requirement be? Can we go as far as not > doing this cast implicit

Re: [HACKERS] psql problem querying relations

2007-02-28 Thread Andrew Dunstan
Zoltan Boszormenyi wrote: Hi, this is with current CVS code: # \dt ERROR: did not find '}' at end of input node Server log: ERROR: did not find '}' at end of input node It's working for me. Have you tried with a fresh checkout or after running make clean before you build? cheers andre

[HACKERS] Major Feature Interactions

2007-02-28 Thread Simon Riggs
There's a number of major features all coming together over the next weeks, so I wanted to raise some awareness of possible areas of interaction. My concern is more than just "will multiple patches apply together?", but extends into whether the features interact with each other with synergy, ignore

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Gregory Stark
"Simon Riggs" <[EMAIL PROTECTED]> writes: > How much memory would it save during VACUUM on a 1 billion row table > with 200 million dead rows? Would that reduce the number of cycles a > normal non-interrupted VACUUM would perform? It would depend on how many dead tuples you have per-page. If you

Re: [HACKERS] psql problem querying relations

2007-02-28 Thread Zoltan Boszormenyi
Andrew Dunstan írta: Zoltan Boszormenyi wrote: Hi, this is with current CVS code: # \dt ERROR: did not find '}' at end of input node Server log: ERROR: did not find '}' at end of input node It's working for me. Have you tried with a fresh checkout or after running make clean before you

[HACKERS] Compilation errors

2007-02-28 Thread Gregory Stark
Does anyone happen to know what it is about my build environment that causes these errors? I get a similar slew of errors from each flex lexer we built including the plpgsql lexer, the ecpg lexer, etc. Do I have the wrong version of flex installed? Something else? It's quite annoying, especially a

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: But we've already seen that CRC checks can be expensive. Not everyone will want to take the cpu hit. Storing a byte counter in every block is cheap. CRC checking a page is most certainly the simplest. And, I disagree that it would be worse t

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Gregory Stark wrote: Does anyone happen to know what it is about my build environment that causes these errors? I get a similar slew of errors from each flex lexer we built including the plpgsql lexer, the ecpg lexer, etc. Do I have the wrong version of flex installed? Something else? It's quite

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > This proposed design is overcomplicated and a waste of space. I mean, > we reduce storage overhead using phantom command id and variable > varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage overhead using phan

Re: [HACKERS] Compilation errors

2007-02-28 Thread Peter Eisentraut
Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: > Does anyone happen to know what it is about my build environment that > causes these errors? Nothing. Everybody gets them. -- Peter Eisentraut http://developer.postgresql.org/~petere/ ---(end of broadcast)

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Stefan Kaltenbrunner
Gregory Stark wrote: "Jonah H. Harris" <[EMAIL PROTECTED]> writes: This proposed design is overcomplicated and a waste of space. I mean, we reduce storage overhead using phantom command id and variable varlena, but let's just fill it up again with unnecessary junk bytes. We reduced storage o

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Peter Eisentraut wrote: Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing. Everybody gets them. That's not what the buildfarm shows. example: http://www.pgbuildfarm.org/

Re: [HACKERS] VACUUM and spoiling the buffer manager cache

2007-02-28 Thread Tom Lane
"Simon Riggs" <[EMAIL PROTECTED]> writes: > VACUUM's current behaviour is to take blocks it has touched and place > them on the head of the freelist, allowing them to be reused. No, it puts them at the tail of the freelist. So I am unconvinced by the rest of your argument.

Re: [HACKERS] [PATCHES]

2007-02-28 Thread korryd
> > Not that I think that anyone owning both a law degree and a computer > > in 2007 should legitimately be able to plead innocence here. FAST > > Australia's lawyers are making themselves look like idiots, and the > > same for every other company tacking on such notices. I think the > > real bot

Re: [HACKERS] Packed short varlenas, what next?

2007-02-28 Thread Tom Lane
Gregory Stark <[EMAIL PROTECTED]> writes: > I just noticed that last night myself. In particular the GIST modules seems to > be a major problem. they define dozens of new objects, many of which are just > passing around C data structures internally but some of which are objects > which get stored i

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: We reduced storage overhead using phantom command id by 8 bytes *per tuple*. I hardly think 8 bytes per page is much of a concern. You're already losing an average of 1/2 a tuple per page to rounding and that's a minimum of 16 bytes for the nar

[HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
Hello hackers, I was starting to think about next SOC and the project for it. And for a long time I wanted to implement the user quotas in PG. So, I'll try to explain my understanding of the implementation, and I'll be happy to hear any comments, objections, or pointings to my misunderstanding

Re: [HACKERS] VACUUM and spoiling the buffer manager cache

2007-02-28 Thread Simon Riggs
On Wed, 2007-02-28 at 11:09 -0500, Tom Lane wrote: > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > VACUUM's current behaviour is to take blocks it has touched and place > > them on the head of the freelist, allowing them to be reused. > > No, it puts them at the tail of the freelist. That's a min

Re: [HACKERS] Compilation errors

2007-02-28 Thread Tom Lane
Peter Eisentraut <[EMAIL PROTECTED]> writes: > Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: >> Does anyone happen to know what it is about my build environment that >> causes these errors? > Nothing. Everybody gets them. I don't. What version of flex are you guys using?

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
>> Do you really doubt that memcpy is faster than CRC32 checking? >> Especially when >> you're already doing memcpy anyways and the only overhead is the few >> unaligned >> bytes at the end and the 8 one-byte copies? > > I'm saying the complexity and implementation of it is going to get you > a b

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Tom Lane
"Sergey E. Koposov" <[EMAIL PROTECTED]> writes: > 1) The main idea is to implement the per-user quota (not per tablespace > for example). So, during the creation of the new user some quota can be > specified, and after that the size of all the relations *owned* by that > user should be limited b

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joshua D. Drake
Sergey E. Koposov wrote: > Hello hackers, > > I was starting to think about next SOC and the project for it. And for a > long time I wanted to implement the user quotas in PG. > So, I'll try to explain my understanding of the implementation, and I'll > be happy to hear any comments, objections, or

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Tom Lane wrote: "Sergey E. Koposov" <[EMAIL PROTECTED]> writes: 1) The main idea is to implement the per-user quota (not per tablespace for example). So, during the creation of the new user some quota can be specified, and after that the size of all the relations *owned* by

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
I think we need to think about when these CRCs would be read and written. It would be written when it hits the disk, hopefully by the background writer, and I think after a server crash, all pages would have to be read and checked. The good news is that both of these are non-critical paths. ---

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Gregory Stark" <[EMAIL PROTECTED]> writes: > We've already seen wal CRC checking show up at the top of profiles. > > Do you really doubt that memcpy is faster than CRC32 checking? Especially when > you're already doing memcpy anyways and the only overhead is the few unaligned > bytes at the end a

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: I could see this being useful per database, maybe. It seems like kind of an odd feature. Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > LIVE FROM THE WWE, CAGE MATCH! > > Jonah (the Theorist) Harris versus Greg (the Brain) Stark. > > What is going to happen between these two brothers in arms when they > must both prove their theory! Darn, I wish I had seen this post before I posted

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: LIVE FROM THE WWE, CAGE MATCH! Jonah (the Theorist) Harris versus Greg (the Brain) Stark. What is going to happen between these two brothers in arms when they must both prove their theory! Heh, I like it :) -- Jonah H. Harris, Software A

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Bruce Momjian
I have added this to the developer's FAQ to clarify the situtation of posting a patch: PostgreSQL is licensed under a BSD license. By posting a patch to the public PostgreSQL mailling lists, you are giving the PostgreSQL Global Development Group the non-revokable right to distribute

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Bruce Momjian" <[EMAIL PROTECTED]> writes: > I think we need to think about when these CRCs would be read and > written. It would be written when it hits the disk, hopefully by the > background writer, and I think after a server crash, all pages would > have to be read and checked. The good new

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Gregory Stark wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > >> LIVE FROM THE WWE, CAGE MATCH! >> >> Jonah (the Theorist) Harris versus Greg (the Brain) Stark. >> >> What is going to happen between these two brothers in arms when they >> must both prove their theory! > > Darn, I wish I

Re: [HACKERS] Compilation errors

2007-02-28 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > Peter Eisentraut <[EMAIL PROTECTED]> writes: >> Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: >>> Does anyone happen to know what it is about my build environment that >>> causes these errors? > >> Nothing. Everybody gets them. > > I don't. W

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joshua D. Drake
Sergey E. Koposov wrote: > On Wed, 28 Feb 2007, Joshua D. Drake wrote: > >> I could see this being useful per database, maybe. It seems like kind of >> an odd feature. > > Per user AND per database (as Tom noted). But I dont see what's odd in > it... It exists in Oracle, and I need quotas in the

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Joshua D. Drake
Bruce Momjian wrote: > I have added this to the developer's FAQ to clarify the situtation of > posting a patch: > > PostgreSQL is licensed under a BSD license. By posting a patch > to the public PostgreSQL mailling lists, you are giving the PostgreSQL > Global Development Group the no

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Gregory Stark wrote: > "Bruce Momjian" <[EMAIL PROTECTED]> writes: > > > I think we need to think about when these CRCs would be read and > > written. It would be written when it hits the disk, hopefully by the > > background writer, and I think after a server crash, all pages would > > have to b

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Bruce Momjian
Joshua D. Drake wrote: > Bruce Momjian wrote: > > I have added this to the developer's FAQ to clarify the situtation of > > posting a patch: > > > > PostgreSQL is licensed under a BSD license. By posting a patch > > to the public PostgreSQL mailling lists, you are giving the PostgreSQL >

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: Sergey E. Koposov wrote: On Wed, 28 Feb 2007, Joshua D. Drake wrote: Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I remember user requests

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joshua D. Drake
> Then you cannot run any queries that extend the size of your relations > (for example INSERT, UPDATE etc.). Unless you drop your tables or DELETE > something Interesting. Well my two cents is don't go any deeper than database. I.e; don't try and track to the individual relation. Joshua D. Drak

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Tom Lane
"Sergey E. Koposov" <[EMAIL PROTECTED]> writes: > Per user AND per database (as Tom noted). But I dont see what's odd in > it... It exists in Oracle, and I need quotas in the project on which I'm > working. And I remember user requests for quotas in the mailing lists ... It hasn't ever made it

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joshua D. Drake
Tom Lane wrote: > "Sergey E. Koposov" <[EMAIL PROTECTED]> writes: >> Per user AND per database (as Tom noted). But I dont see what's odd in >> it... It exists in Oracle, and I need quotas in the project on which I'm >> working. And I remember user requests for quotas in the mailing lists ... >

Re: [HACKERS] [PATCHES]

2007-02-28 Thread Tom Lane
Bruce Momjian <[EMAIL PROTECTED]> writes: > Joshua D. Drake wrote: >> We should add this to the mailing list signup pages and the welcome >> pages to the lists. > Yep, good idea. Marc? For -patches and -hackers, I agree. It seems a bit legalistic and off-putting for the general lists, though.

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Tom Lane wrote: 2) I looked into the code, and from my understanding, the main part of the code which should be affected by the quotas is storage/smgr/md.c. md.c is too low level to do catalog accesses and thus too low level to know who owns what. That's probably a dumb q

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Gregory Stark wrote: "Tom Lane" <[EMAIL PROTECTED]> writes: Peter Eisentraut <[EMAIL PROTECTED]> writes: Am Mittwoch, 28. Februar 2007 16:23 schrieb Gregory Stark: Does anyone happen to know what it is about my build environment that causes these errors? Nothing. Eve

[HACKERS] Possible Bug: high CPU usage for stats collector in 8.2

2007-02-28 Thread Joshua D. Drake
Darcy Buskermolen wrote: > I'm observing high CPU usage (95%) of a 2.6GHz opteron by the stats collector > on an 8.2.3 box investigation has lead me to belive that the stats file is > written a lot more often that once every 500ms the following shows this > behavior. > > PostgreSQL 8.2.3 on x

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Martijn van Oosterhout
On Wed, Feb 28, 2007 at 09:58:52AM -0800, Joshua D. Drake wrote: > E.g; if user foo then pg_database_size may not be > than X? > > I guess the big question would be when do we check though? At each > transaction seems like it would add significant overhead, especially if > we had to rollback the t

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joshua D. Drake
Martijn van Oosterhout wrote: > On Wed, Feb 28, 2007 at 09:58:52AM -0800, Joshua D. Drake wrote: >> E.g; if user foo then pg_database_size may not be > than X? >> >> I guess the big question would be when do we check though? At each >> transaction seems like it would add significant overhead, espec

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > But if the system was shut down uncleanly as the result of a Postgres crash or > fast shutdown of Postgres then that isn't an issue. And many users may prefer > to bring the system up as soon as possible as long as they know any corrupt > pag

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > But if the system was shut down uncleanly as the result of a Postgres > > > crash or > > > fast shutdown of Postgres then that isn't an issue. And many users may > > > prefer > > > to bring the system up as soon a

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Oleg Bartunov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: Tom Lane wrote: "Sergey E. Koposov" <[EMAIL PROTECTED]> writes: Per user AND per database (as Tom noted). But I dont see what's odd in it... It exists in Oracle, and I need quotas in the project on which I'm working. And I remember user requests for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > Am am not comfortable starting and having something fail later. > > Then do you have some other idea for protecting pages from being torn > without storing an entire backup copy or performing a block-level > consiste

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: We have several methods suggested to check the blocks, like CRC. My point was that, whatever check method we use, we should be prepared to check on startup, or at least make it the default for a crash restart. Sounds like it should be a guc.

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > We have several methods suggested to check the blocks, like CRC. My > > point was that, whatever check method we use, we should be prepared to > > check on startup, or at least make it the default for a crash restart

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:10 -0500, Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > > But if the system was shut down uncleanly as the result of a Postgres > > > crash or > > > fast shutdown of Postgres then that isn't an issue. And many users may > > > prefer > >

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: Keep in mind if you don't check it on startup, you will be checking it for every read, which for rare crashes, might not be wise. Well understood. That's how most everyone configures their database systems; they certainly don't optimize for

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: That's 5 minutes of data, in the default configuration. Right, but I don't know anyone that keeps checkpoints at 5 minutes. At least not on OLTP configurations. -- Jonah H. Harris, Software Architect | phone: 732.331.1324 EnterpriseDB Corporati

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Heikki Linnakangas
Jonah H. Harris wrote: On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > But if the system was shut down uncleanly as the result of a Postgres crash or > fast shutdown of Postgres then that isn't an issue. And many users may prefer > to bring the system up as soon as possible as long as th

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: > On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: >> That's 5 minutes of data, in the default configuration. > > Right, but I don't know anyone that keeps checkpoints at 5 minutes. > At least not on OLTP configurations. Uhmm... most do because most don't ever touch the po

[HACKERS] 8.1.8 Installer Fails on Win2k Server

2007-02-28 Thread Saqib Awan
I am installing on a Cisco Media Server 7800 running Win2k Server and am noticing that the installer fails every time with the error dialog saying "Failed to create process for initdb: Access is denied". It looks like that I need to change some permission in the registry allowing other users spawn

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 14:54 -0500, Jonah H. Harris wrote: > On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: > > That's 5 minutes of data, in the default configuration. > > Right, but I don't know anyone that keeps checkpoints at 5 minutes. > At least not on OLTP configurations. > It's got a ha

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Jeff Davis <[EMAIL PROTECTED]> wrote: I'm not saying there's no cost, but the extra recovery cost seems lower to me than the CRC cost on every data page read during operation. I agree, I just think it should be configurable. Also, if we find an error, do we even have the ability t

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: > Right, but I don't know anyone that keeps checkpoints at 5 minutes. > At least not on OLTP configurations. Uhmm... most do because most don't ever touch the postgresql.conf and those that do, don't touch checkpoints because they don't unde

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Joshua D. Drake
Jonah H. Harris wrote: > On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: >> > Right, but I don't know anyone that keeps checkpoints at 5 minutes. >> > At least not on OLTP configurations. >> >> Uhmm... most do because most don't ever touch the postgresql.conf and >> those that do, don't touc

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Joshua D. Drake <[EMAIL PROTECTED]> wrote: I am not trying to pick on the issue but I do think it is important to recognize that literally only those in the know, are going to ever touch the postgresql.conf. I agree. -- Jonah H. Harris, Software Architect | phone: 732.331.1324 Ente

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Martijn van Oosterhout wrote: >> Generally, rolling back a transaction doesn't reduce the amount of disk >> used. Only VACUUM FULL actually shrinks relations. > Right, but what I mean was -- if we rollback because we hit quota we > could potentially

Re: [HACKERS] Proposal for Implenting read-only queries during wal replay (SoC 2007)

2007-02-28 Thread plabrh1
Thanks Josh, I'll look for the earlier one and try to add it there... -Paul -Original Message- From: Joshua D. Drake [mailto:[EMAIL PROTECTED] Sent: Wednesday, February 28, 2007 12:09 AM To: Paul Silveira Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Proposal for Implenting

Re: [HACKERS] Compilation errors

2007-02-28 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Gregory Stark wrote: >> flex 2.5.33 > Aha! Known to be broken, iirc. Use flex 2.5.4a No, the known breakages with flex were years ago; 2.5.33 has only been out a year. I think 2.5.31 might have been the one we saw big problems with (there's a note war

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people off and lose people if you say things like that rather

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Oleg Bartunov
On Wed, 28 Feb 2007, Tom Lane wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: Martijn van Oosterhout wrote: Generally, rolling back a transaction doesn't reduce the amount of disk used. Only VACUUM FULL actually shrinks relations. Right, but what I mean was -- if we rollback because we

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Tom Lane wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: Martijn van Oosterhout wrote: Generally, rolling back a transaction doesn't reduce the amount of disk used. Only VACUUM FULL actually shrinks relations. Right, but what I mean was -- if we rollback because we

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: "Jonah H. Harris" <[EMAIL PROTECTED]> writes: > Which is, of course, how everyone else does it. I happen to agree with your conclusion but this line of argument is exceptionally unconvincing. In fact in this crowd you'll tend to turn people o

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jonah H. Harris
On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: Am am not comfortable starting and having something fail later. Then do you have some other idea for protecting pages from being torn without storing an entire backup copy or performing a block-level consistency check? How other databases d

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: Gregory Stark wrote: flex 2.5.33 Aha! Known to be broken, iirc. Use flex 2.5.4a No, the known breakages with flex were years ago; 2.5.33 has only been out a year. I think 2.5.31 might have been the one we sa

Re: [HACKERS] Compilation errors

2007-02-28 Thread Andrew Dunstan
I wrote: I guess we could conditionally add prototypes for those functions to all the .l files if you really want to move to 2.5.33. Kinda yucky, though. Actually, we couldn't.The definition section from the .l file gets included after these functions. So we'd need to include something

Re: [HACKERS] Resumable vacuum proposal and design overview

2007-02-28 Thread Heikki Linnakangas
Gregory Stark wrote: "Simon Riggs" <[EMAIL PROTECTED]> writes: How much memory would it save during VACUUM on a 1 billion row table with 200 million dead rows? Would that reduce the number of cycles a normal non-interrupted VACUUM would perform? It would depend on how many dead tuples you hav

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Martijn van Oosterhout
On Wed, Feb 28, 2007 at 03:57:56PM -0500, Tom Lane wrote: > It's worse than that, because VACUUM FULL will actually bloat the > indexes on the way to being able to reduce the table size (since it has > to make new index entries for rows it moves). If the limit is strictly I was thinking that inde

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joshua D. Drake
> I don't know, but in my opinion, I don't see anything bad in requiring > dropping the data if the quota is full. That's what usually occurs in > the case of normal filesystem quota... If you don't have a space there, > you cannot edit files, copy them etc... > And that solution should be definit

Re: [HACKERS] Possible Bug: high CPU usage for stats collector in 8.2

2007-02-28 Thread Joshua D. Drake
Joshua D. Drake wrote: > Darcy Buskermolen wrote: >> I'm observing high CPU usage (95%) of a 2.6GHz opteron by the stats >> collector >> on an 8.2.3 box investigation has lead me to belive that the stats file is >> written a lot more often that once every 500ms the following shows this >> beh

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Jeff Davis
On Wed, 2007-02-28 at 21:13 +, Gregory Stark wrote: > Hm that's an interesting thought. We only really have to check pages that > would have received a full page write since the last checkpoint. So if we made Do we ever do a partial page write, or is what you're saying equivalent to "we only h

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Joachim Wieland
On Wed, Feb 28, 2007 at 12:56:13PM -0500, Tom Lane wrote: > It hasn't ever made it onto the TODO list, which means there's not a > consensus that we need it. Such a patch could improve the acceptance of PostgreSQL in shared hosting environments. Note that a database without quotas can be filled up

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joshua D. Drake wrote: The bad point is not that we would rollback the transaction. The bad point is what happens when you need to rollback a transaction and in your scenario it is quite plausible that a large rollback could occur, more than once, causing the requirement of

Re: [HACKERS] SOC & user quotas

2007-02-28 Thread Sergey E. Koposov
On Wed, 28 Feb 2007, Joachim Wieland wrote: On Wed, Feb 28, 2007 at 12:56:13PM -0500, Tom Lane wrote: It hasn't ever made it onto the TODO list, which means there's not a consensus that we need it. Such a patch could improve the acceptance of PostgreSQL in shared hosting environments. Note th

Re: [HACKERS] Compilation errors

2007-02-28 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > Actually, we couldn't.The definition section from the .l file gets > included after these functions. So we'd need to include something in > gram.y before including scan.c. Actually, since we don't use any of those functions, the following advice from

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Simon Riggs
On Tue, 2007-02-27 at 09:32 -0800, Josh Berkus wrote: > Simon, I think if you address me in a mail, it would be best not to explicitly *remove* my name from the address list. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com ---(end of broadcas

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Bruce Momjian
Jonah H. Harris wrote: > On 2/28/07, Bruce Momjian <[EMAIL PROTECTED]> wrote: > > Am am not comfortable starting and having something fail later. > > Then do you have some other idea for protecting pages from being torn > without storing an entire backup copy or performing a block-level > consiste

Re: [HACKERS] COMMIT NOWAIT Performance Option

2007-02-28 Thread Gregory Stark
"Jonah H. Harris" <[EMAIL PROTECTED]> writes: > On 2/28/07, Gregory Stark <[EMAIL PROTECTED]> wrote: >> "Jonah H. Harris" <[EMAIL PROTECTED]> writes: >> >> > Which is, of course, how everyone else does it. >> >> I happen to agree with your conclusion but this line of argument is >> exceptionally u

  1   2   >