Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Christopher Kings-Lynne
* AFAICS the only downside of not having a Relation available in smgr.c and md.c is that error messages could only refer to the RelFileNode numbers and not to the relation name. I'm not sure this is bad, since in my experience what you want to know about such errors is the actual disk filename, wh

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Marc G. Fournier
On Thu, 5 Feb 2004, Tom Lane wrote: > "Marc G. Fournier" <[EMAIL PROTECTED]> writes: > > k, but that would be a different scenario, no? As I mentioned in my > > original, a DROP TABLE would reset its timeout to -1, meaning to close it > > and drop it on the next checkpoint interval ... > > How wo

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Tom Lane
"Marc G. Fournier" <[EMAIL PROTECTED]> writes: > k, but that would be a different scenario, no? As I mentioned in my > original, a DROP TABLE would reset its timeout to -1, meaning to close it > and drop it on the next checkpoint interval ... How would it do that? These structs are local to part

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Marc G. Fournier
On Thu, 5 Feb 2004, Tom Lane wrote: > "Marc G. Fournier" <[EMAIL PROTECTED]> writes: > > Why? Setting it to the checkpoint interval itself should be sufficient, > > no? All you want to do is avoid closing any files that were used during > > that last checkpoint interval, since there is a good ch

Re: [HACKERS] Recursive queries?

2004-02-05 Thread Hans-Jürgen Schönig
evgen wrote: hello hackers, some time ago i played with PgSQL and have written simpliest working prototype of WITH clause for it. it don't do any checks and performs only simpliest selects, but it works. i can contribute it and develop it further to production state. regards, .evgen I suggest

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Tom Lane
"Marc G. Fournier" <[EMAIL PROTECTED]> writes: > Why? Setting it to the checkpoint interval itself should be sufficient, > no? All you want to do is avoid closing any files that were used during > that last checkpoint interval, since there is a good chance you'd have to > once more reopen them in

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Marc G. Fournier
On Thu, 5 Feb 2004, Tom Lane wrote: > "Marc G. Fournier" <[EMAIL PROTECTED]> writes: > > 'k, only comment is on this one ... would it not be a bit more efficient > > to add a flag to the "SMgrRelation *" structure that acts as a timer? > > Hm, we could try that, although I'm not sure it would help

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Alvaro Herrera
On Thu, Feb 05, 2004 at 04:55:44PM -0500, Tom Lane wrote: > Alvaro Herrera <[EMAIL PROTECTED]> writes: > > Another way to do it would be keeping the list of files to delete along > > with the deleting Xid, but that would also require keeping a list of > > which xacts aborted and which ones didn't.

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > On Thu, Feb 05, 2004 at 02:05:46PM -0500, Tom Lane wrote: >> * smgrclose closes the md.c-level file and drops the hashtable entry. >> Hashtable entries remain valid unless explicitly closed (thus, multiple >> smgropens for the same file are legal). > So

Re: [HACKERS] Preventing duplicate vacuums?

2004-02-05 Thread Josh Berkus
Tom, > Yes we do: there's a lock. Sorry, bad test. Forget I said anything. Personally, I would like to have the 2nd vacuum error out instead of blocking. However, I'll bet that a lot of people won't agree with me. -- -Josh Berkus Aglio Database Solutions San Francisco --

Re: [HACKERS] [PATCHES] log session end - again

2004-02-05 Thread Andrew Dunstan
Chester Kustarz wrote: On Thu, 5 Feb 2004, Bruce Momjian wrote: Wow, like the idea too --- pid plus time_t start time of backend. Actully, it would be good to have star time first so you can sort everything in order of start time. Why not just add a printf like pattern so the user can ou

Re: [HACKERS] Preventing duplicate vacuums?

2004-02-05 Thread Josh Berkus
Rod, > You have a 8 billion row table with some very high turn over tuples > (lots of updates to a few thousand rows). A partial or targeted vacuum > would be best, failing that you kick them off fairly frequently, > especially if IO isn't really an issue. Yes, but we don't have partial or target

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Alvaro Herrera
On Thu, Feb 05, 2004 at 02:05:46PM -0500, Tom Lane wrote: > * smgrclose closes the md.c-level file and drops the hashtable entry. > Hashtable entries remain valid unless explicitly closed (thus, multiple > smgropens for the same file are legal). So, will there be a refcount on each cache entry?

Re: [HACKERS] Preventing duplicate vacuums?

2004-02-05 Thread Rod Taylor
On Thu, 2004-02-05 at 15:37, Josh Berkus wrote: > Folks, > > Just occurred to me that we have no code to prevent a user from running two > simultaneos lazy vacuums on the same table.I can't think of any > circumstance why running two vacuums would be desirable behavior; how > difficult woul

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Tom Lane
"Marc G. Fournier" <[EMAIL PROTECTED]> writes: > 'k, only comment is on this one ... would it not be a bit more efficient > to add a flag to the "SMgrRelation *" structure that acts as a timer? Hm, we could try that, although I'm not sure it would help much. You'd have to set the timeout to be lo

Re: [HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Marc G. Fournier
On Thu, 5 Feb 2004, Tom Lane wrote: > * Because we don't smgrclose after a write, it is possible to have > "dangling" smgr entries that aren't useful any more, as well as open > file descriptors underneath them. This isn't too big a deal on Unix > but it will be fatal for the Windows port, since

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Chester Kustarz
I do not see the win32 list on http://www.postgresql.org/lists.html How would I find out about it and join? I probably did not subscribe to hackers when it started. On Thu, 5 Feb 2004, Bruce Momjian wrote: > I think the win32 email list has worked well. What is has allowed is > people who want to

Re: [HACKERS] Preventing duplicate vacuums?

2004-02-05 Thread Tom Lane
Josh Berkus <[EMAIL PROTECTED]> writes: > Just occurred to me that we have no code to prevent a user from running two > simultaneos lazy vacuums on the same table. Yes we do: there's a lock. regards, tom lane ---(end of broadcast)-

Re: [HACKERS] [PATCHES] log session end - again

2004-02-05 Thread Chester Kustarz
On Thu, 5 Feb 2004, Bruce Momjian wrote: > Wow, like the idea too --- pid plus time_t start time of backend. > Actully, it would be good to have star time first so you can sort > everything in order of start time. Why not just add a printf like pattern so the user can output whatever they like? %

Re: [HACKERS] Vacuum Delay feature

2004-02-05 Thread Jan Wieck
Attached is a corrected version that solves the query cancel problem by not napping any more and going full speed as soon as any signal is pending. If nobody objects, I'm going to commit this tomorrow. Jan Jan Wieck wrote: The attached patch applies to CVS tip as of 02/05/2004 and implements

[HACKERS] Preventing duplicate vacuums?

2004-02-05 Thread Josh Berkus
Folks, Just occurred to me that we have no code to prevent a user from running two simultaneos lazy vacuums on the same table.I can't think of any circumstance why running two vacuums would be desirable behavior; how difficult would it be to make this an exception? This becomes a more cruc

Re: [HACKERS] Recursive queries?

2004-02-05 Thread evgen
hello hackers, some time ago i played with PgSQL and have written simpliest working prototype of WITH clause for it. it don't do any checks and performs only simpliest selects, but it works. i can contribute it and develop it further to production state. regards, .evgen -

Re: [HACKERS] Why has postmaster shutdown gotten so slow?

2004-02-05 Thread Jan Wieck
Jan Wieck wrote: Tom Lane wrote: Shutdown of an idle postmaster used to take about two or three seconds (mostly due to the sync/sleep(2)/sync in md_sync). For the last couple of days it's taking more like a dozen seconds. I presume somebody broke something, but I'm unsure whether to pin the bla

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Austin Gonyou
On Thu, 2004-02-05 at 14:00, Nicolai Tufar wrote: > > -Original Message- > > From: Dave Page [mailto:[EMAIL PROTECTED] > > My SQL2K servers give me far more sleepless nights than PostgreSQL > ever > > did! > > You bet! I totally agree with you. > Technicians like you, me and most people on

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Nicolai Tufar
> -Original Message- > From: Dave Page [mailto:[EMAIL PROTECTED] > My SQL2K servers give me far more sleepless nights than PostgreSQL ever > did! You bet! I totally agree with you. Technicians like you, me and most people on this list Already know that PostgreSQL is stable and reliable. It

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Tom Lane
"Dave Page" <[EMAIL PROTECTED]> writes: > Ahh, that's not quite what I thought you meant. It sounded like you were > questioning the reliability of PostgreSQL, not it's ability to be > recovered to point of failure. I think the waters got muddied a bit by the suggestion elsewhere in the thread (no

[HACKERS] It's past time to redo the smgr API

2004-02-05 Thread Tom Lane
A long time ago Vadim proposed that we should revise smgr.c's API so that it does not depend on Relations (relcache entries); rather, only a RelFileNode value should be needed to access a file in smgr and lower levels. This would allow us to get rid of the concept of "blind writes". As of CVS tip

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Dave Page
> -Original Message- > From: Nicolai Tufar [mailto:[EMAIL PROTECTED] > Sent: 05 February 2004 17:35 > To: Dave Page; [EMAIL PROTECTED] > Subject: RE: [HACKERS] PITR Dead horse? > > > -Original Message- > > From: Dave Page [mailto:[EMAIL PROTECTED] > > Sent: Thursday, February

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Nicolai Tufar
> -Original Message- > From: Dave Page [mailto:[EMAIL PROTECTED] > Sent: Thursday, February 05, 2004 11:02 AM > To: [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: RE: [HACKERS] PITR Dead horse? > Of course, but I would argue that my claim that PostgreSQL is reliable > is backed up by the l

Re: [HACKERS] [pgsql-hackers-win32] Sync vs. fsync during checkpoint

2004-02-05 Thread Zeugswetter Andreas SB SD
> People keep saying that the bgwriter mustn't write pages synchronously > because it'd be bad for performance, but I think that analysis is > faulty. Performance of what --- the bgwriter? Nonsense, the *point* Imho that depends on the workload. For a normal OLTP workload this is certainly cor

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Austin Gonyou
Wow. What a wonderful response. Thanks all! On Thu, 2004-02-05 at 08:57, Bruce Momjian wrote: > Tatsuo Ishii wrote: > > > Has this been beaten to death now? Just curious if PITR was in Dev tree > > > yet. Been out of the loop. TIA. > > > > I and my co workers are very interested in implementing

Re: [HACKERS] dollar quoting

2004-02-05 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan <[EMAIL PROTECTED]> writes: IIRC, a consensus was reached on the actual format of the quote delimiters (either $$ or $identifier$), and Tom had a proof of concept patch to the parser to handle it, but work was needed on psql, plpgsql, pg_dump (and pg_restore?

Re: [HACKERS] dollar quoting

2004-02-05 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > IIRC, a consensus was reached on the actual format of the quote > delimiters (either $$ or $identifier$), and Tom had a proof of concept > patch to the parser to handle it, but work was needed on psql, plpgsql, > pg_dump (and pg_restore?) + docs. I t

Re: [HACKERS] [pgsql-hackers-win32] Sync vs. fsync during checkpoint

2004-02-05 Thread Tom Lane
Shridhar Daithankar <[EMAIL PROTECTED]> writes: > There are other benefits of writing pages earlier even though they might not > get synced immediately. Such as? > It would tell kernel that this is latest copy of updated buffer. Kernel VFS > should make that copy visible to every other backend

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Marc G. Fournier
[EMAIL PROTECTED] I set myself as owner, since I didn't figure it was something you really needed added to your plate? :) Just means you don't have to go through and do the Approvals for postings when they need it, I'll just do it as my normal stuff ... On Thu, 5 Feb 2004, Bruce Momjian wrote:

Re: [HACKERS] dollar quoting

2004-02-05 Thread Jon Jensen
On Thu, 5 Feb 2004, Andrew Dunstan wrote: > What has become of the "dollar quoting" mechanism that we had so much > discussion about back in August/September? > > IIRC, a consensus was reached on the actual format of the quote > delimiters (either $$ or $identifier$), and Tom had a proof of con

[HACKERS] prepared queries

2004-02-05 Thread Andrew Dunstan
The current release of DBD::Pg contains this in the CHANGES file: - $dbh->prepare() rewrites the SQL statement into an internal form, striping out comments and whitespace, and if PostgreSQL > 7.3 takes the stripped statement and passes that to Postgres' PREPAR

[HACKERS] dollar quoting

2004-02-05 Thread Andrew Dunstan
What has become of the "dollar quoting" mechanism that we had so much discussion about back in August/September? IIRC, a consensus was reached on the actual format of the quote delimiters (either $$ or $identifier$), and Tom had a proof of concept patch to the parser to handle it, but work was

[HACKERS] Vacuum Delay feature

2004-02-05 Thread Jan Wieck
The attached patch applies to CVS tip as of 02/05/2004 and implements the cost based vacuum delay feature. A detailed description with charts of different configuration settings can be found here: http://developer.postgresql.org/~wieck/vacuum_cost/ There is a problem left that seems to be

Re: [HACKERS] [pgsql-hackers-win32] Sync vs. fsync during checkpoint

2004-02-05 Thread Shridhar Daithankar
On Thursday 05 February 2004 20:24, Tom Lane wrote: > "Zeugswetter Andreas SB SD" <[EMAIL PROTECTED]> writes: > > So Imho the target should be to have not much IO open for the checkpoint, > > so the fsync is fast enough, even if serial. > > The best we can do is push out dirty pages with write() vi

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Bruce Momjian wrote: > Dave Page wrote: > > > > > > > -Original Message- > > > From: Nicolai Tufar [mailto:[EMAIL PROTECTED] > > > Sent: 05 February 2004 00:01 > > > To: [EMAIL PROTECTED] > > > Subject: Re: [HACKERS] PITR Dead horse? > > > > > > Totally agree. Robustness and rock-soli

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Marc G. Fournier wrote: > On Wed, 4 Feb 2004, Tatsuo Ishii wrote: > > > > I and some other developers are also interested in. > > > Do you think we can work together? > > > > Sure. Why not. I think it would be practical to decide who is the > > leader of this project, though. > > Is this somethin

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Tom Lane wrote: > "Marc G. Fournier" <[EMAIL PROTECTED]> writes: > > Is this something large enough, like the win32 stuff, that having a side > > list for discussions is worth setting up? > > In terms of the amount of code to be written, I expect it's larger than > the win32 porting effort. And i

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Nicolai Tufar wrote: > I would like to join this effort too. > I was afraid that people at RedHat are already > halfway though and were to release their work > shortly. But it does not seem to be the case. We are a long way away from completion: http://momjian.postgresql.org/main/writing

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Koichi Suzuki wrote: > Hi, This is Suzuki from NTT DATA Intellilink. > > I told Bruce Momjan that I and my colleagues are interested in > implementing PITR in BOF in NY LW2004. NTT's laboratory is very > interested in this issue and I'm planning to work with them. I hope we > could cooperate.

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Tatsuo Ishii wrote: > > Has this been beaten to death now? Just curious if PITR was in Dev tree > > yet. Been out of the loop. TIA. > > I and my co workers are very interested in implementing PITR. We will > tackle this for 7.5 if no one objects. I have put up a PITR project page: http:/

Re: [HACKERS] [pgsql-hackers-win32] Sync vs. fsync during checkpoint

2004-02-05 Thread Tom Lane
"Zeugswetter Andreas SB SD" <[EMAIL PROTECTED]> writes: > So Imho the target should be to have not much IO open for the checkpoint, > so the fsync is fast enough, even if serial. The best we can do is push out dirty pages with write() via the bgwriter and hope that the kernel will see fit to writ

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Bruce Momjian
Dave Page wrote: > > > > -Original Message- > > From: Nicolai Tufar [mailto:[EMAIL PROTECTED] > > Sent: 05 February 2004 00:01 > > To: [EMAIL PROTECTED] > > Subject: Re: [HACKERS] PITR Dead horse? > > > > Totally agree. Robustness and rock-solidness are the only > > things missing fo

Re: [HACKERS] [PATCHES] log session end - again

2004-02-05 Thread Andrew Dunstan
Bruce Momjian wrote: Tom Lane wrote: Larry's idea about combining PID and backend start time didn't sound too unreasonable to me. Wow, like the idea too --- pid plus time_t start time of backend. Actully, it would be good to have star time first so you can sort everything in order of st

Re: [HACKERS] [PATCHES] log session end - again

2004-02-05 Thread Bruce Momjian
Tom Lane wrote: > Andrew Dunstan <[EMAIL PROTECTED]> writes: > > Right. And if we have sessionids we would want them logged there, I > > think. And that would rule out anything based on xid or backend pid. > > Uh, what's wrong with backend pid? Since we fork before we start doing > anything wit

Re: [HACKERS] [pgsql-hackers-win32] Sync vs. fsync during checkpoint

2004-02-05 Thread Zeugswetter Andreas SB SD
> I don't think the bgwriter is going to be able to keep up with I/O bound > backends, but I do think it can scan and set those booleans fast enough > for the backends to then perform the writes. As long as the bgwriter does not do sync writes (which it does not, since that would need a whole lot

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Dave Page
> -Original Message- > From: Nicolai Tufar [mailto:[EMAIL PROTECTED] > Sent: 05 February 2004 08:15 > To: Dave Page > Subject: RE: [HACKERS] PITR Dead horse? > > > -Original Message- > > From: Dave Page [mailto:[EMAIL PROTECTED] Well I've > only been > > using PostgreSQL sin

Re: [HACKERS] PITR Dead horse?

2004-02-05 Thread Dave Page
> -Original Message- > From: Nicolai Tufar [mailto:[EMAIL PROTECTED] > Sent: 05 February 2004 00:01 > To: [EMAIL PROTECTED] > Subject: Re: [HACKERS] PITR Dead horse? > > Totally agree. Robustness and rock-solidness are the only > things missing for PostgreSQL to become the killer of