Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-07 Thread Simon Riggs
On Mon, 2006-02-06 at 21:07 -0700, Rick Gigger wrote: I was thinking the exact same thing. Except the and just fsync() dirty pages on commit part. Wouldn't that actually make the situation worse? I thought the whole point of WAL was that it was more efficient to fsync all of the

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-07 Thread Christopher Kings-Lynne
This would apply to only a single relation, so would be just as efficient a write to the database as to WAL. The proposed route is to sync to the database, but not to WAL, thus halving the required I/O. Yes, its designed for large data loads. A question - would setting fsync=off while

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-07 Thread Jim C. Nasby
On Wed, Feb 08, 2006 at 09:44:24AM +0800, Christopher Kings-Lynne wrote: This would apply to only a single relation, so would be just as efficient a write to the database as to WAL. The proposed route is to sync to the database, but not to WAL, thus halving the required I/O. Yes, its

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Simon Riggs
On Fri, 2006-02-03 at 22:29 -0500, Bruce Momjian wrote: Based on this, I think we should just implement the TRUNCATE/DROP option for the table, and avoid the idea of allowing non-logged operations on a table that has any data we want recovered after a crash. Well the only other option is

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Bruce Momjian
I have split up the TODO items as: * Allow WAL logging to be turned off for a table, but the table might be dropped or truncated during crash recovery [walcontrol] Allow tables to bypass WAL writes and just fsync() dirty pages on commit.

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Christopher Kings-Lynne
* Allow WAL logging to be turned off for a table, but the table might be dropped or truncated during crash recovery [walcontrol] Allow tables to bypass WAL writes and just fsync() dirty pages on commit. This should be implemented using ALTER TABLE,

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Doug McNaught
Bruce Momjian pgman@candle.pha.pa.us writes: TABLE PERSISTENCE [ DROP | TRUNCATE | DEFAULT ]. Tables using non-default logging should not use referential integrity with default-logging tables. I have to say this smells way too much like MySQL for me to feel

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Rick Gigger
I was thinking the exact same thing. Except the and just fsync() dirty pages on commit part. Wouldn't that actually make the situation worse? I thought the whole point of WAL was that it was more efficient to fsync all of the changes in one sequential write in one file rather than

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Bruce Momjian
Rick Gigger wrote: I was thinking the exact same thing. Except the and just fsync() dirty pages on commit part. Wouldn't that actually make the situation worse? I thought the whole point of WAL was that it was more efficient to fsync all of the changes in one sequential write in

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-06 Thread Rick Gigger
Rick Gigger wrote: I was thinking the exact same thing. Except the and just fsync() dirty pages on commit part. Wouldn't that actually make the situation worse? I thought the whole point of WAL was that it was more efficient to fsync all of the changes in one sequential write in one file

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-02-03 Thread Bruce Momjian
Based on this, I think we should just implement the TRUNCATE/DROP option for the table, and avoid the idea of allowing non-logged operations on a table that has any data we want recovered after a crash. --- Simon Riggs

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-05 Thread Bruce Momjian
Simon Riggs wrote: So, we need a name for EXCLUSIVE mode that suggests how it is different from TRUNCATE, and in this case, the difference is that EXCLUSIVE preserves the previous contents of the table on recovery, while TRUNCATE does not. Do you want to call the mode PRESERVE, or

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-05 Thread Bruce Momjian
Simon Riggs wrote: On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: Having COPY behave differently because it is in a transaction is fine as long as it is user-invisible, but once you require users to do that to get the speedup, it isn't user-invisible anymore. Since we're

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-05 Thread Simon Riggs
On Thu, 2006-01-05 at 12:27 -0500, Bruce Momjian wrote: Seems like a nice optimization. Negative thoughts: Toast tables have a toast index on them, yes? We have agreed that we cannot use the optimization if we have indexes on the main table. It follows that we cannot use the optimization if we

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Jim C. Nasby
On Thu, Dec 29, 2005 at 11:24:28AM -0500, Tom Lane wrote: Bruce Momjian pgman@candle.pha.pa.us writes: Andrew Dunstan wrote: I an horribly scared that this will be used as a performance boost for normal use. I would at least like to see some restrictions that make it harder to mis-use.

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Jim C. Nasby
On Fri, Dec 30, 2005 at 12:58:15PM -0500, Bruce Momjian wrote: Andrew Dunstan wrote: My concern is more about making plain that this is for special operations, not normal operations. Or maybe I have misunderstood the purpose. Rephrase that as full ownership rights must be

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Stephen Frost
* Jim C. Nasby ([EMAIL PROTECTED]) wrote: I dislike restricting to super-user, and to some extent even table owner. The reason is that if you have some automated batch process, you don't want that process running as a superuser. Also, it is often awkward to require that the user running that

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: Dumb question: if the ALTER is done inside a transaction, and then reverted at the end of the transaction, does that mean that no other transactions would have those permissions? I think the general use-case is that you only one the session doing the

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Tom Lane
Stephen Frost [EMAIL PROTECTED] writes: In general, I do prefer that permissions be seperably grantable. Being able to grant 'truncate' permissions would be really nice. Is the only reason such permission doesn't exist due to no one working on it, or is there other disagreement about it?

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Jim C. Nasby
On Tue, Jan 03, 2006 at 11:29:02AM -0500, Tom Lane wrote: Stephen Frost [EMAIL PROTECTED] writes: In general, I do prefer that permissions be seperably grantable. Being able to grant 'truncate' permissions would be really nice. Is the only reason such permission doesn't exist due to no

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Jim C. Nasby
On Tue, Jan 03, 2006 at 11:26:51AM -0500, Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: Dumb question: if the ALTER is done inside a transaction, and then reverted at the end of the transaction, does that mean that no other transactions would have those permissions? I think the

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Tom Lane
Jim C. Nasby [EMAIL PROTECTED] writes: On Tue, Jan 03, 2006 at 11:26:51AM -0500, Tom Lane wrote: Such an ALTER would certainly require exclusive lock on the table, so I'm not sure that I see much use-case for doing it like that. You'd want to do the ALTER and commit so as not to lock other

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Stephen Frost
* Tom Lane ([EMAIL PROTECTED]) wrote: Stephen Frost [EMAIL PROTECTED] writes: In general, I do prefer that permissions be seperably grantable. Being able to grant 'truncate' permissions would be really nice. Is the only reason such permission doesn't exist due to no one working on it, or

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Stephen Frost
* Jim C. Nasby ([EMAIL PROTECTED]) wrote: Yeah, I hadn't thought about that. I agree; if you trust some process enough to have MVCC-affecting rights then you should be able to trust it with full ownership rights. About that, I have to disagree. :) I've got a case where this isn't true, see my

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Bruce Momjian
Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: On Tue, Jan 03, 2006 at 11:26:51AM -0500, Tom Lane wrote: Such an ALTER would certainly require exclusive lock on the table, so I'm not sure that I see much use-case for doing it like that. You'd want to do the ALTER and commit so as

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Tom Lane
Stephen Frost [EMAIL PROTECTED] writes: The problem is that you might want to grant 'truncate' to people who *aren't* particularly trusted. For truncate, at least I have a real-world use-case for it. I don't find this use-case particularly convincing. If the users are allowed to delete all

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Stephen Frost
* Tom Lane ([EMAIL PROTECTED]) wrote: I don't find this use-case particularly convincing. If the users are allowed to delete all data in a given table, then that table must be dedicated to them anyway; so it's not that easy to see why you can't risk giving them ownership rights on it. The

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Jim C. Nasby
On Tue, Jan 03, 2006 at 12:08:05PM -0500, Bruce Momjian wrote: Tom Lane wrote: Jim C. Nasby [EMAIL PROTECTED] writes: On Tue, Jan 03, 2006 at 11:26:51AM -0500, Tom Lane wrote: Such an ALTER would certainly require exclusive lock on the table, so I'm not sure that I see much use-case

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Bruce Momjian
Jim C. Nasby wrote: I don't think it should (which implies that EXCLUSIVE is a bad name). Agreed, EXCLUSIVE was used to mean an _exclusive_ writer. The new words I proposed were PRESERVE or STABLE. This seems to seriously limit the usefulness, though. You'll only want to use

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Jim C. Nasby
On Tue, Jan 03, 2006 at 04:20:47PM -0500, Bruce Momjian wrote: Jim C. Nasby wrote: I don't think it should (which implies that EXCLUSIVE is a bad name). Agreed, EXCLUSIVE was used to mean an _exclusive_ writer. The new words I proposed were PRESERVE or STABLE. This seems to

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Bruce Momjian
Jim C. Nasby wrote: We would be creating a new lock type for this. Sorry if I've just missed this in the thread, but what would the new lock type do? My impression is that as it stands you can either do: BEGIN; ALTER TABLE EXCLUSIVE; ... ALTER TABLE SHARE; --fsync COMMIT; Which

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Simon Riggs
On Fri, 2005-12-30 at 16:14 -0500, Bruce Momjian wrote: Simon Riggs wrote: The implications of EXCLUSIVE are: - there will be a check on each and every I, U, D to check the state of the relation - *every* operation that attempts a write lock will attempt to acquire an EXCLUSIVE full

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Simon Riggs
On Tue, 2006-01-03 at 16:20 -0500, Bruce Momjian wrote: Jim C. Nasby wrote: Idealistically, if EXCLUSIVE/PRESERVE/STABLE does it's thing by only appending new pages, it would be nice if other backends could continue performing updates at the same time, assuming there's free space

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Simon Riggs
On Tue, 2006-01-03 at 17:38 -0500, Bruce Momjian wrote: Right, the DML will be single-threaded and fsync of all dirty pages will happen before commit of each transaction. heap_sync() would occur at end of statement, as it does with CTAS. We could delay until EOT but I'm not sure I see why; in

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Simon Riggs
On Sat, 2005-12-31 at 12:59 +0100, Michael Paesold wrote: Bruce Momjian wrote: The --single-transaction mode would apply even if the dump was created using an earlier version of pg_dump. pg_dump has *not* been altered at all. (And I would again add that the idea was not my own) I

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Simon Riggs
On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: Having COPY behave differently because it is in a transaction is fine as long as it is user-invisible, but once you require users to do that to get the speedup, it isn't user-invisible anymore. Since we're agreed on adding ALTER TABLE

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Josh Berkus
Bruce, Basically meaning your idea of update while EXCLUSIVE/PRESERVE/STABLE is happening is never going to be implemented because it is just too hard to do, and too prone to error. What I figured. Never hurts to ask though. :):) Actually, it does hurt because it generates

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2006-01-03 Thread Bruce Momjian
Josh Berkus wrote: Bruce, Basically meaning your idea of update while EXCLUSIVE/PRESERVE/STABLE is happening is never going to be implemented because it is just too hard to do, and too prone to error. What I figured. Never hurts to ask though. :):) Actually, it does

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-31 Thread Michael Paesold
Bruce Momjian wrote: The --single-transaction mode would apply even if the dump was created using an earlier version of pg_dump. pg_dump has *not* been altered at all. (And I would again add that the idea was not my own) I assume you mean this:

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-31 Thread August Zajonc
As a user and a list lurker I very much like Bruce's proposed ALTER TABLE syntax. COPY LOCK (and the variants I can imagine being required for all the other types of cases) don't seem as appealing. And ALTER TABLE seems to make it clear it is an object level change, feels like it fits the

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Simon Riggs
On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: My view would be that this thread has been complex because everybody has expressed a somewhat different requirement, which could be broken down as: 1. The need for a

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Andrew Dunstan
Simon Riggs said: Following Andrew's concerns, I'd also note that ALTER TABLE requires a much higher level of privilege to operate than does COPY. That sounds like it will make things more secure, but all it does is open up the administrative rights, since full ownership rights must be

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Tom Lane
Andrew Dunstan [EMAIL PROTECTED] writes: Simon Riggs said: Following Andrew's concerns, I'd also note that ALTER TABLE requires a much higher level of privilege to operate than does COPY. That sounds like it will make things more secure, but all it does is open up the administrative rights,

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Bruce Momjian
Simon Riggs wrote: On Thu, 2005-12-29 at 11:37 -0500, Bruce Momjian wrote: Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: My view would be that this thread has been complex because everybody has expressed a somewhat different requirement, which could be broken down as:

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Andrew Dunstan
Tom Lane wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Simon Riggs said: Following Andrew's concerns, I'd also note that ALTER TABLE requires a much higher level of privilege to operate than does COPY. That sounds like it will make things more secure, but all it does is open up

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Bruce Momjian
Andrew Dunstan wrote: My concern is more about making plain that this is for special operations, not normal operations. Or maybe I have misunderstood the purpose. Rephrase that as full ownership rights must be obtained to load data in a way that requires dropping any existing indexes

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Simon Riggs
On Fri, 2005-12-30 at 11:49 -0500, Bruce Momjian wrote: Yes, I know we agreed to the COPY LOCK, but new features now being requested, so we have to re-evaluate where we are going with COPY LOCK to get a more consistent solution. Thank you. Ah, but people wanted fast INSERT INTO ... SELECT,

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Bruce Momjian
Simon Riggs wrote: On Fri, 2005-12-30 at 11:49 -0500, Bruce Momjian wrote: Yes, I know we agreed to the COPY LOCK, but new features now being requested, so we have to re-evaluate where we are going with COPY LOCK to get a more consistent solution. Thank you. Good. I think we can be

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Greg Stark
As far as EXCLUSIVE or COPY LOCK goes, I think this would be useful functionality but perhaps there doesn't have to be any proprietary user interface to it at all. Why not just check if the conditions are already present to allow the optimization and if so go ahead. That is, if the current

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Bruce Momjian
Greg Stark wrote: As far as EXCLUSIVE or COPY LOCK goes, I think this would be useful functionality but perhaps there doesn't have to be any proprietary user interface to it at all. Why not just check if the conditions are already present to allow the optimization and if so go ahead. That

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Simon Riggs
On Fri, 2005-12-30 at 16:14 -0500, Bruce Momjian wrote: This was discussed on-list by 2 core team members, a committer and myself, but I see no requirements change here. You even accepted the invisible COPY optimization in your last post - why unpick that now? Please forgive my tone, but

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Bruce Momjian
Simon Riggs wrote: On Fri, 2005-12-30 at 16:14 -0500, Bruce Momjian wrote: This was discussed on-list by 2 core team members, a committer and myself, but I see no requirements change here. You even accepted the invisible COPY optimization in your last post - why unpick that now?

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Greg Stark
Bruce Momjian pgman@candle.pha.pa.us writes: BEGIN; LOCK TABLE foo; COPY foo from ... COMMIT; There could be a COPY LOCK option to obtain a lock, but it would be purely for user convenience so they don't have to bother with BEGIN and COMMIt. The only downside is a check to

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-30 Thread Bruce Momjian
Greg Stark wrote: Bruce Momjian pgman@candle.pha.pa.us writes: BEGIN; LOCK TABLE foo; COPY foo from ... COMMIT; There could be a COPY LOCK option to obtain a lock, but it would be purely for user convenience so they don't have to bother with BEGIN and COMMIt.

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Simon Riggs
On Wed, 2005-12-28 at 20:58 -0500, Bruce Momjian wrote: Having read through this thread, I would like to propose a syntax/behavior. I think we all now agree that the logging is more part of the table than the command itself. Right now we have a COPY LOCK patch, but people are going to want

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Rod Taylor
So, my thinking would be to separate things into two: a) Add a TODO item shared temp tables that caters for (1) and (4) ALTER TABLE name RELIABILITY {DELETE ROWS AT RECOVERY | FULL RECOVERY} (syntax TBD) DELETE ROWS AT RECOVERY would need to be careful or disallowed

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Simon Riggs
On Thu, 2005-12-29 at 09:35 -0500, Rod Taylor wrote: So, my thinking would be to separate things into two: a) Add a TODO item shared temp tables that caters for (1) and (4) ALTER TABLE name RELIABILITY {DELETE ROWS AT RECOVERY | FULL RECOVERY} (syntax TBD) DELETE

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Andrew Dunstan
Bruce Momjian said: DROP would drop the table on a restart after a non-clean shutdown. It would do _no_ logging on the table and allow concurrent access, plus index access. DELETE is the same as DROP, but it just truncates the table (perhaps TRUNCATE is a better word). EXCLUSIVE would

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Bruce Momjian
Andrew Dunstan wrote: Bruce Momjian said: DROP would drop the table on a restart after a non-clean shutdown. It would do _no_ logging on the table and allow concurrent access, plus index access. DELETE is the same as DROP, but it just truncates the table (perhaps TRUNCATE is a better

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: My view would be that this thread has been complex because everybody has expressed a somewhat different requirement, which could be broken down as: 1. The need for a multi-user-accessible yet temporary table 2. Loading data into a table immediately after

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Tom Lane
Bruce Momjian pgman@candle.pha.pa.us writes: Andrew Dunstan wrote: I an horribly scared that this will be used as a performance boost for normal use. I would at least like to see some restrictions that make it harder to mis-use. Perhaps restrict to superuser? Certainly restrict to table

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Bruce Momjian
Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: My view would be that this thread has been complex because everybody has expressed a somewhat different requirement, which could be broken down as: 1. The need for a multi-user-accessible yet temporary table 2. Loading data into a

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Greg Stark
Andrew Dunstan [EMAIL PROTECTED] writes: Bruce Momjian said: DROP would drop the table on a restart after a non-clean shutdown. It would do _no_ logging on the table and allow concurrent access, plus index access. DELETE is the same as DROP, but it just truncates the table (perhaps

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-29 Thread Bruce Momjian
Greg Stark wrote: Andrew Dunstan [EMAIL PROTECTED] writes: Bruce Momjian said: DROP would drop the table on a restart after a non-clean shutdown. It would do _no_ logging on the table and allow concurrent access, plus index access. DELETE is the same as DROP, but it just

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-28 Thread Martijn van Oosterhout
On Wed, Dec 28, 2005 at 12:47:31AM +0200, Hannu Krosing wrote: I've thought of one other possibility, which is kind of at the extreme end of system implementation. Given the suggestion about not losing a whole table on unclean shutdown, how about using a single table, split. snip How

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-28 Thread Bruce Momjian
Having read through this thread, I would like to propose a syntax/behavior. I think we all now agree that the logging is more part of the table than the command itself. Right now we have a COPY LOCK patch, but people are going to want to control logging for INSERT INTO ... SELECT, and UPDATE,

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-28 Thread Joshua D. Drake
now agree that the logging is more part of the table than the command itself. Right now we have a COPY LOCK patch, but people are going to want to control logging for INSERT INTO ... SELECT, and UPDATE, and all sorts of other things, so I think we are best adding an ALTER TABLE capability. I

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-28 Thread Bruce Momjian
Joshua D. Drake wrote: now agree that the logging is more part of the table than the command itself. Right now we have a COPY LOCK patch, but people are going to want to control logging for INSERT INTO ... SELECT, and UPDATE, and all sorts of other things, so I think we are best adding

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-27 Thread Jim C. Nasby
On Sat, Dec 24, 2005 at 02:01:17AM -0500, Greg Stark wrote: But I like the other user's suggestion too. If there's a standards blessed feature to have temporary tables that are visible in other sessions then perhaps that's what we've arrived at from another direction. Having a temporary table

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-27 Thread Martijn van Oosterhout
On Mon, Dec 26, 2005 at 11:00:51AM -0500, Bruce Momjian wrote: I think this brings up an interesting distinction, that having the NO LOGGING switch per command doesn't make sense because it modifying the table. It has to be a per-object switch, or something that operates only on empty

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-27 Thread Hannu Krosing
Ühel kenal päeval, T, 2005-12-27 kell 19:20, kirjutas Martijn van Oosterhout: On Mon, Dec 26, 2005 at 11:00:51AM -0500, Bruce Momjian wrote: I think this brings up an interesting distinction, that having the NO LOGGING switch per command doesn't make sense because it modifying the table.

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-27 Thread Hannu Krosing
Ühel kenal päeval, R, 2005-12-23 kell 23:06, kirjutas Tom Lane: Greg Stark [EMAIL PROTECTED] writes: It seems to me the only rational way to approach this is to have a per-table flag that sets that table to be non-logged. Essentially changing a table's behaviour to that of a temporary table

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-27 Thread Tom Lane
Hannu Krosing [EMAIL PROTECTED] writes: making one of WAL files (strategies) be /dev/null would almost get us non-logged writes, except for a little overhead in write() calls. fsync() on /dev/null should be instantaneous . No, you really do want to push it up to a higher level than that. The

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-26 Thread Simon Riggs
On Sat, 2005-12-24 at 10:32 -0500, Tom Lane wrote: An ALTER TABLE SET LOGGED/UNLOGGED switch might have some merit, but please don't muddy the waters by confusing this with temp-table status. I would not be against such a table-level switch, but the exact behaviour would need to be specified

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-26 Thread Martijn van Oosterhout
On Mon, Dec 26, 2005 at 12:03:27PM +, Simon Riggs wrote: I would not be against such a table-level switch, but the exact behaviour would need to be specified more closely before this became a TODO item, IMHO. Well, I think at a per table level is the only sensible level. If a table isn't

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-26 Thread Bruce Momjian
I think this brings up an interesting distinction, that having the NO LOGGING switch per command doesn't make sense because it modifying the table. It has to be a per-object switch, or something that operates only on empty tables. This is the exact same distinction we talked about for NO

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-24 Thread Martijn van Oosterhout
On Fri, Dec 23, 2005 at 11:19:01PM -0500, Rod Taylor wrote: In many cases you could use temporary tables, but sometimes you might want multiple processes or multiple transactions to be able to see the data. Could always implement GLOBAL TEMP tables that have the ability to use these kinds

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-24 Thread Simon Riggs
On Fri, 2005-12-23 at 22:41 -0500, Greg Stark wrote: Tom Lane [EMAIL PROTECTED] writes: Qingqing Zhou [EMAIL PROTECTED] writes: I revised the idea with MINIMAL XLOG (instead of WITHOUT XLOG) like the below. I think in this way, we can always gaurantee its correctness and can always

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-24 Thread Greg Stark
Martijn van Oosterhout kleptog@svana.org writes: CREATE GLOBAL TEMP TABLE blah INHERITS myhugetable; I don't think you can have your temp table inherit from a real table. That would make your real table have temp table semantics. Ie, records in it will spontaneously disappear on reboot. But

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-24 Thread Tom Lane
Greg Stark [EMAIL PROTECTED] writes: Martijn van Oosterhout kleptog@svana.org writes: CREATE GLOBAL TEMP TABLE blah INHERITS myhugetable; I don't think you can have your temp table inherit from a real table. Bear in mind also that this notion of a GLOBAL TEMP table has less than nothing to do

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Simon Riggs
On Thu, 2005-12-22 at 17:36 -0500, Stephen Frost wrote: * Simon Riggs ([EMAIL PROTECTED]) wrote: On Thu, 2005-12-22 at 21:18 +0100, Martijn van Oosterhout wrote: Considering WAL bypass is code for breaks PITR No it isn't. All of the WAL bypass logic does *not* operate when PITR is

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Qingqing Zhou
Torn pages (partial page write) are still a problem. I revised the idea with MINIMAL XLOG (instead of WITHOUT XLOG) like the below. I think in this way, we can always gaurantee its correctness and can always improve it. To Use It -- A BEGIN TRANSACTION MINIMAL XLOG/END block is a

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Tom Lane
Qingqing Zhou [EMAIL PROTECTED] writes: I revised the idea with MINIMAL XLOG (instead of WITHOUT XLOG) like the below. I think in this way, we can always gaurantee its correctness and can always improve it. I think the entire idea is a waste of time anyway. If we have the COPY case covered

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Qingqing Zhou
On Fri, 23 Dec 2005, Tom Lane wrote: Qingqing Zhou [EMAIL PROTECTED] writes: I revised the idea with MINIMAL XLOG (instead of WITHOUT XLOG) like the below. I think in this way, we can always gaurantee its correctness and can always improve it. I think the entire idea is a waste of time

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Greg Stark
Tom Lane [EMAIL PROTECTED] writes: Qingqing Zhou [EMAIL PROTECTED] writes: I revised the idea with MINIMAL XLOG (instead of WITHOUT XLOG) like the below. I think in this way, we can always gaurantee its correctness and can always improve it. I think the entire idea is a waste of time

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Qingqing Zhou
Greg Stark [EMAIL PROTECTED] wrote But I don't see turning on and off the WAL on a per-transaction basis to be useful. Every transaction in the system is affected by the WAL status of every other transaction working with the same tables. It doesn't serve any purpose to have one

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Tom Lane
Greg Stark [EMAIL PROTECTED] writes: It seems to me the only rational way to approach this is to have a per-table flag that sets that table to be non-logged. Essentially changing a table's behaviour to that of a temporary table except that other transactions can see it. But what's the point?

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Rod Taylor
In many cases you could use temporary tables, but sometimes you might want multiple processes or multiple transactions to be able to see the data. Could always implement GLOBAL TEMP tables that have the ability to use these kinds of shortcuts. -- ---(end of

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-23 Thread Greg Stark
Tom Lane [EMAIL PROTECTED] writes: Greg Stark [EMAIL PROTECTED] writes: It seems to me the only rational way to approach this is to have a per-table flag that sets that table to be non-logged. Essentially changing a table's behaviour to that of a temporary table except that other

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and DELETE?

2005-12-22 Thread Jim C. Nasby
On Thu, Dec 22, 2005 at 02:31:33PM +, Simon Riggs wrote: Having just optimized COPY to avoid writing WAL during the transaction in which a table was first created, it seems worth considering whether this should occur for INSERT, UPDATE and DELETE also. It is fairly common to do data

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and DELETE?

2005-12-22 Thread Jim C. Nasby
BTW, this should also probably be moved over to -hackers... -- Jim C. Nasby, Sr. Engineering Consultant [EMAIL PROTECTED] Pervasive Software http://pervasive.comwork: 512-231-6117 vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461 ---(end of

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and DELETE?

2005-12-22 Thread Martijn van Oosterhout
On Thu, Dec 22, 2005 at 12:37:51PM -0600, Jim C. Nasby wrote: I do think this needs to be something that is made either completely transparent or must be specifically enabled. As described, I believe this would break PITR, so users should have to specifically request that behavior (and they

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Simon Riggs
On Thu, 2005-12-22 at 21:18 +0100, Martijn van Oosterhout wrote: Considering WAL bypass is code for breaks PITR No it isn't. All of the WAL bypass logic does *not* operate when PITR is active. The WAL bypass logic is aimed at Data Warehouses, which typically never operate in PITR mode for

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Stephen Frost
* Simon Riggs ([EMAIL PROTECTED]) wrote: On Thu, 2005-12-22 at 21:18 +0100, Martijn van Oosterhout wrote: Considering WAL bypass is code for breaks PITR No it isn't. All of the WAL bypass logic does *not* operate when PITR is active. The WAL bypass logic is aimed at Data Warehouses, which

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Qingqing Zhou
Simon Riggs [EMAIL PROTECTED] wrote No it isn't. All of the WAL bypass logic does *not* operate when PITR is active. The WAL bypass logic is aimed at Data Warehouses, which typically never operate in PITR mode for performance reasons, however the choice is yours. To make things, is it

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Qingqing Zhou
Qingqing Zhou [EMAIL PROTECTED] wrote To make things, is it possible to add a GUC to let user disable *all* the xlogs? It may work like this: BEGIN TRANSACTION WITHOUT XLOG;/* forbidden vacuum, PITR etc */ BEGIN .../* no xlog during this peroid */ END; /* don't mark this

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Tom Lane
Qingqing Zhou [EMAIL PROTECTED] writes: BEGIN TRANSACTION WITHOUT XLOG;/* forbidden vacuum, PITR etc */ So during this peroid, if any transaction failed, the only consequence is add invisible garbage data. No, the likely consequence is irretrievable corruption of any table or index page

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Qingqing Zhou
Tom Lane [EMAIL PROTECTED] wrote Qingqing Zhou [EMAIL PROTECTED] writes: BEGIN TRANSACTION WITHOUT XLOG;/* forbidden vacuum, PITR etc */ So during this peroid, if any transaction failed, the only consequence is add invisible garbage data. No, the likely consequence is irretrievable

Re: [HACKERS] [Bizgres-general] WAL bypass for INSERT, UPDATE and

2005-12-22 Thread Tom Lane
Qingqing Zhou [EMAIL PROTECTED] writes: I guess I know (at least part) of what you mean. This is because we rely on replay all the xlog no matter it belongs to a committed transaction or not. Why? Because a failed transaction is not totally useless since later transaction may reply on some