Re: [HACKERS] COPY speedup

2009-08-18 Thread Pierre Frédéric Caillau d
AFAIK you need to run app with LD_LIBRARY_PATH=/usr/lib/debug, otherwise the debug packages won't be used. I had stupidly put the LD_LIBRARY_PATH on make rather than on postgres, ahem. OK, it works, thanks. I'm very carefully benchmarking everything everytime I make a modificatio

Re: [HACKERS] COPY speedup

2009-08-18 Thread Marko Kreen
On 8/18/09, Pierre Frédéric Caillaud wrote: > I'm doing some more exploration with oprofile... > > I've got the glibc-debug package installed (on kubuntu), but oprofile > doesn't seem to know about it. I wonder what part of glibc gets 60% of the > run time... do I have to set a magic option in t

Re: [HACKERS] COPY speedup

2009-08-17 Thread Pierre Frédéric Caillau d
I'm doing some more exploration with oprofile... I've got the glibc-debug package installed (on kubuntu), but oprofile doesn't seem to know about it. I wonder what part of glibc gets 60% of the run time... do I have to set a magic option in the postgres config ? samples %image nam

Re: [HACKERS] COPY speedup

2009-08-13 Thread Pierre Frédéric Caillau d
In the previous mails I made a mistake, writing "MTuples/s" instead of "MDatums/s", sorry about that. It is the number of rows x columns. The title was wrong, but the data was right. I've been doing some tests on COPY FROM ... BINARY. - inlines in various pg_get* etc - a faster buffer handli

Re: [HACKERS] COPY speedup

2009-08-13 Thread Pierre Frédéric Caillau d
But when I see a big red button, I just press it to see what happens. Ugly hacks are useful to know how fast the thing can go ; then the interesting part is to reimplement it cleanly, trying to reach the same performance... Right -- now that you've shown a 6x speedup increase, it is clear that

Re: [HACKERS] COPY speedup

2009-08-12 Thread Alvaro Herrera
Pierre Frédéric Caillaud escribió: > But when I see a big red button, I just press it to see what happens. > Ugly hacks are useful to know how fast the thing can go ; then the > interesting part is to reimplement it cleanly, trying to reach the > same performance... Right -- now that you've shown

Re: [HACKERS] COPY speedup

2009-08-12 Thread Pierre Frédéric Caillau d
>> We don't touch datatype APIs >> lightly, because it affects too much code. >> >>                        regards, tom lane > >        I definitely agree with that. Actually, let me clarify: When I modified the datatype API, I was feeling uneasy, like "I shouldn't really touch this". But

Re: [HACKERS] COPY speedup

2009-08-12 Thread Alvaro Herrera
Merlin Moncure escribió: > 2009/8/12 Pierre Frédéric Caillaud : > > > > > >> If you do as much damage to the I/O function API as the other patch > >> did, it will probably be rejected. > > > >        You mean, as the COPY patch in my previous message, or as another > > patch ? > >        (I just se

Re: [HACKERS] COPY speedup

2009-08-12 Thread Merlin Moncure
2009/8/12 Pierre Frédéric Caillaud : > > >> If you do as much damage to the I/O function API as the other patch >> did, it will probably be rejected. > >        You mean, as the COPY patch in my previous message, or as another > patch ? >        (I just search the archives and found one about CopyR

Re: [HACKERS] COPY speedup

2009-08-12 Thread Pierre Frédéric Caillau d
If you do as much damage to the I/O function API as the other patch did, it will probably be rejected. You mean, as the COPY patch in my previous message, or as another patch ? (I just search the archives and found one about CopyReadLine, but that's probably not what you are talki

Re: [HACKERS] COPY speedup

2009-08-12 Thread Tom Lane
=?utf-8?Q?Pierre_Fr=C3=A9d=C3=A9ric_Caillau?= =?utf-8?Q?d?= writes: > I've been examining the code path for COPY FROM too, and I think it is > possible to get the same kind of speedups on COPY FROM that the patch in > the previous message did for COPY TO, that is to say perhaps 2-3x faster > in B

Re: [HACKERS] COPY speedup

2009-08-12 Thread Pierre Frédéric Caillau d
Replying to myself... I've been examining the code path for COPY FROM too, and I think it is possible to get the same kind of speedups on COPY FROM that the patch in the previous message did for COPY TO, that is to say perhaps 2-3x faster in BINARY mode and 10-20% faster in TEXT mode (these figu