Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-11 Thread Bruce Momjian
Simon Riggs wrote: On Sat, Aug 6, 2011 at 4:16 AM, Bruce Momjian br...@momjian.us wrote: Well, if the table is created in the same transaction (which is the only case under consideration), no other sessions can write to the table so you are just writing the entire table on commit, rather

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Heikki Linnakangas
On 06.08.2011 06:32, Gokulakannan Somasundaram wrote: However, for small operations it's a net loss - you avoid writing a WAL record, but you have to fsync() the heap instead. If you only modify a few rows, the extra fsync (or fsyncs if there are indexes too) is more expensive than writing the

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Simon Riggs
On Sat, Aug 6, 2011 at 4:16 AM, Bruce Momjian br...@momjian.us wrote: Well, if the table is created in the same transaction (which is the only case under consideration), no other sessions can write to the table so you are just writing the entire table on commit, rather than to the WAL. Below

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Heikki Linnakangas
On 06.08.2011 13:13, Simon Riggs wrote: I think we should remove the COPY optimisation because of this and definitely not extend INSERT SELECT to perform it automatically. It can be very helpful when loading a lot of data, so I'm not in favor of removing it altogether. Maybe WAL-log the first

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Jaime Casanova
On Sat, Aug 6, 2011 at 11:05 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 06.08.2011 13:13, Simon Riggs wrote: I think we should remove the COPY optimisation because of this and definitely not extend INSERT SELECT to perform it automatically. It can be very helpful

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Kevin Grittner
Heikki Linnakangas wrote: On 06.08.2011 13:13, Simon Riggs wrote: I think we should remove the COPY optimisation because of this and definitely not extend INSERT SELECT to perform it automatically. It can be very helpful when loading a lot of data, so I'm not in favor of removing it

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Tom Lane
Jaime Casanova ja...@2ndquadrant.com writes: On Sat, Aug 6, 2011 at 11:05 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It can be very helpful when loading a lot of data, so I'm not in favor of removing it altogether. Maybe WAL-log the first 1 rows or such normally,

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-06 Thread Jeff Davis
On Fri, 2011-08-05 at 23:16 -0400, Bruce Momjian wrote: Well, if the table is created in the same transaction (which is the only case under consideration), no other sessions can write to the table so you are just writing the entire table on commit, rather than to the WAL. The transaction can

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-05 Thread Simon Riggs
On Thu, Aug 4, 2011 at 11:07 PM, Bruce Momjian br...@momjian.us wrote: Simon Riggs wrote: On Thu, Aug 4, 2011 at 10:46 PM, Bruce Momjian br...@momjian.us wrote: Right. ?I brought up SELECT INTO because you could make the argument that INSERT ... SELECT is not a utility command like the

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-05 Thread Jeff Davis
On Thu, 2011-08-04 at 18:07 -0400, Bruce Momjian wrote: I am confused how generating WAL traffic that is larger than the heap file we are fsync'ing can possibly be slower. Are you just throwing out an idea to try to make me prove it? That's worded in a slightly confusing way, but here is the

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-05 Thread Bruce Momjian
Jeff Davis wrote: On Thu, 2011-08-04 at 18:07 -0400, Bruce Momjian wrote: I am confused how generating WAL traffic that is larger than the heap file we are fsync'ing can possibly be slower. Are you just throwing out an idea to try to make me prove it? That's worded in a slightly

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-05 Thread Gokulakannan Somasundaram
However, for small operations it's a net loss - you avoid writing a WAL record, but you have to fsync() the heap instead. If you only modify a few rows, the extra fsync (or fsyncs if there are indexes too) is more expensive than writing the WAL. We'd need a heuristic to decide whether to

[HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Bruce Momjian
We currently have reduced WAL logging for wal_level = minimum for these commands: CREATE TABLE AS CREATE INDEX CLUSTER COPY into tables that were created or truncated in the same transaction One thing we don't optimize is INSERT ... SELECT when the table

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: One thing we don't optimize is INSERT ... SELECT when the table is created or truncated in the same transaction. Seems we could. We optimize CREATE TABLE AS which is effectively SELECT ... INTO using a different syntax. Is this a TODO? Considering

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Andrew Dunstan
On 08/04/2011 04:55 PM, Tom Lane wrote: Bruce Momjianbr...@momjian.us writes: One thing we don't optimize is INSERT ... SELECT when the table is created or truncated in the same transaction. Seems we could. We optimize CREATE TABLE AS which is effectively SELECT ... INTO using a different

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Bruce Momjian
Andrew Dunstan wrote: On 08/04/2011 04:55 PM, Tom Lane wrote: Bruce Momjianbr...@momjian.us writes: One thing we don't optimize is INSERT ... SELECT when the table is created or truncated in the same transaction. Seems we could. We optimize CREATE TABLE AS which is effectively

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Simon Riggs
On Thu, Aug 4, 2011 at 10:46 PM, Bruce Momjian br...@momjian.us wrote: Right.  I brought up SELECT INTO because you could make the argument that INSERT ... SELECT is not a utility command like the other ones and therefore can't be done easily, but CREATE TABLE AS is internal SELECT INTO and

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Bruce Momjian
Simon Riggs wrote: On Thu, Aug 4, 2011 at 10:46 PM, Bruce Momjian br...@momjian.us wrote: Right. ?I brought up SELECT INTO because you could make the argument that INSERT ... SELECT is not a utility command like the other ones and therefore can't be done easily, but CREATE TABLE AS is

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Jeff Davis
On Thu, 2011-08-04 at 17:46 -0400, Bruce Momjian wrote: Right. I brought up SELECT INTO because you could make the argument that INSERT ... SELECT is not a utility command like the other ones and therefore can't be done easily, but CREATE TABLE AS is internal SELECT INTO and implemented in

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Bruce Momjian
Jeff Davis wrote: On Thu, 2011-08-04 at 17:46 -0400, Bruce Momjian wrote: Right. I brought up SELECT INTO because you could make the argument that INSERT ... SELECT is not a utility command like the other ones and therefore can't be done easily, but CREATE TABLE AS is internal SELECT

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Robert Haas
On Thu, Aug 4, 2011 at 8:55 PM, Bruce Momjian br...@momjian.us wrote: Jeff Davis wrote: On Thu, 2011-08-04 at 17:46 -0400, Bruce Momjian wrote: Right.  I brought up SELECT INTO because you could make the argument that INSERT ... SELECT is not a utility command like the other ones and

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Jeff Davis
On Thu, 2011-08-04 at 20:55 -0400, Bruce Momjian wrote: It would act like COPY, meaning the table would have to be truncated or created in the same transaction. Well, in that case it could work for any INSERT. No need for a SELECT to be involved. For that matter, why not make it work for DELETE

Re: [HACKERS] Reduce WAL logging of INSERT SELECT

2011-08-04 Thread Heikki Linnakangas
On 05.08.2011 04:23, Jeff Davis wrote: On Thu, 2011-08-04 at 20:55 -0400, Bruce Momjian wrote: It would act like COPY, meaning the table would have to be truncated or created in the same transaction. Well, in that case it could work for any INSERT. No need for a SELECT to be involved. For