Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Ilja Golshtein
>> Is it Ok to use COPY BINARY FROM STDIN instead of multiple INSERTs? >I don't think I would use BINARY, it seems likely to be susceptible >to changes in the underlying data type storage. From the docs: >"To determine the appropriate binary format for the actual tuple data >you should consult

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Casey Duncan
On Oct 18, 2006, at 5:20 AM, Ilja Golshtein wrote: When starting a database from scratch it is much faster to import the data and then create the indexes. The time to create index on a full table is less than the extra time from each index update from the inserts. The more indexes to update the

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Merlin Moncure
On 10/18/06, Ron Johnson <[EMAIL PROTECTED]> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/18/06 09:47, Merlin Moncure wrote: > On 10/18/06, Ron Johnson <[EMAIL PROTECTED]> wrote: >> > I tested binary quite a bit and only found it to be a win if moving >> > blobs in and out of the d

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Alvaro Herrera
Ron Johnson wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 10/18/06 09:47, Merlin Moncure wrote: > > On 10/18/06, Ron Johnson <[EMAIL PROTECTED]> wrote: > >> > I tested binary quite a bit and only found it to be a win if moving > >> > blobs in and out of the database. On 'normal'

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Ron Johnson
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/18/06 09:47, Merlin Moncure wrote: > On 10/18/06, Ron Johnson <[EMAIL PROTECTED]> wrote: >> > I tested binary quite a bit and only found it to be a win if moving >> > blobs in and out of the database. On 'normal' tables of mixed fields >> > type

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Merlin Moncure
On 10/18/06, Ron Johnson <[EMAIL PROTECTED]> wrote: > I tested binary quite a bit and only found it to be a win if moving > blobs in and out of the database. On 'normal' tables of mixed fields > types of small size, it can actually be slower. Binary is a bit > faster for native types and bytea,

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Ron Johnson
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/18/06 08:03, Merlin Moncure wrote: > On 10/18/06, Martijn van Oosterhout wrote: >> Binary may be slightly faster because the datum parsing can be >> partially skipped, but that's hardly much benefit over a text copy. > > I tested binary quite a

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Shane Ambler
Ilja Golshtein wrote: And my question remains. Is it Ok to use COPY BINARY FROM STDIN instead of multiple INSERTs? If it does what you want then it is OK to use it. -- Shane Ambler [EMAIL PROTECTED] Get Sheeky @ http://Sheeky.Biz ---(end of broadcast)---

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Merlin Moncure
On 10/18/06, Ilja Golshtein <[EMAIL PROTECTED]> wrote: I've tried to play with batches and with peculiar constructions like INSERT (SELECT .. UNION ALL SELECT ..) to improve performance, but not satisfied with the result I've got. postgresql 8.2 (beta) supports the 'multiple insert' syntax, so

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Merlin Moncure
On 10/18/06, Martijn van Oosterhout wrote: Binary may be slightly faster because the datum parsing can be partially skipped, but that's hardly much benefit over a text copy. I tested binary quite a bit and only found it to be a win if moving blobs in and out of the database. On 'normal' table

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Ilja Golshtein
>The reason why copy is faster is because it doesn't have to >parse/plan/execute all the queries. In exchange you can't use >expressions or joins to fill the table, only raw data. In other words, COPY has no hidden catches, and I should go with it and don't worry. Correct interpretation? ;) >Bi

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Martijn van Oosterhout
On Wed, Oct 18, 2006 at 04:20:41PM +0400, Ilja Golshtein wrote: > And my question remains. > Is it Ok to use COPY BINARY FROM STDIN instead of multiple INSERTs? The reason why copy is faster is because it doesn't have to parse/plan/execute all the queries. In exchange you can't use expressions or

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Ilja Golshtein
>When starting a database from scratch it is much faster to import the >data and then create the indexes. The time to create index on a full >table is less than the extra time from each index update from the >inserts. The more indexes to update the more time updating indexes takes. > >The proble

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Shane Ambler
Ilja Golshtein wrote: Sounds like your working with an existing database - if you are starting from scratch (inserting data into an empty database) then there are other things that can help too. I am working with existing database, though I am interested what "other things" you mean. Basica

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Ilja Golshtein
Hello! >Using COPY FROM STDIN is much faster than INSERT's (I am sure some out >there have test times to compare, I don't have any on hand) Yes, I know it is much faster. The question is about possible pay for this quickness. What if COPY, say, locks index until end of transaction (it is just

Re: [GENERAL] COPY FROM STDIN instead of INSERT

2006-10-18 Thread Shane Ambler
Ilja Golshtein wrote: Hello! One important use case in my libpq based application (PostgreSQL 8.1.4) is a sort of massive data loading. Currently it is implemented as a series of plain normal INSERTs (binary form of PQexecParams is used) and the problem here it is pretty slow. I've tried to p