Re: [HACKERS] COPY speedup

2009-08-18 Thread Marko Kreen
On 8/18/09, Pierre Frédéric Caillaud li...@peufeu.com wrote: I'm doing some more exploration with oprofile... I've got the glibc-debug package installed (on kubuntu), but oprofile doesn't seem to know about it. I wonder what part of glibc gets 60% of the run time... do I have to set a magic

Re: [HACKERS] COPY speedup

2009-08-18 Thread Pierre Frédéric Caillau d
AFAIK you need to run app with LD_LIBRARY_PATH=/usr/lib/debug, otherwise the debug packages won't be used. I had stupidly put the LD_LIBRARY_PATH on make rather than on postgres, ahem. OK, it works, thanks. I'm very carefully benchmarking everything everytime I make a

Re: [HACKERS] COPY speedup

2009-08-17 Thread Pierre Frédéric Caillau d
I'm doing some more exploration with oprofile... I've got the glibc-debug package installed (on kubuntu), but oprofile doesn't seem to know about it. I wonder what part of glibc gets 60% of the run time... do I have to set a magic option in the postgres config ? samples %image

Re: [HACKERS] COPY speedup

2009-08-13 Thread Pierre Frédéric Caillau d
But when I see a big red button, I just press it to see what happens. Ugly hacks are useful to know how fast the thing can go ; then the interesting part is to reimplement it cleanly, trying to reach the same performance... Right -- now that you've shown a 6x speedup increase, it is clear

Re: [HACKERS] COPY speedup

2009-08-13 Thread Pierre Frédéric Caillau d
In the previous mails I made a mistake, writing MTuples/s instead of MDatums/s, sorry about that. It is the number of rows x columns. The title was wrong, but the data was right. I've been doing some tests on COPY FROM ... BINARY. - inlines in various pg_get* etc - a faster buffer handling

Re: [HACKERS] COPY speedup

2009-08-12 Thread Pierre Frédéric Caillau d
Replying to myself... I've been examining the code path for COPY FROM too, and I think it is possible to get the same kind of speedups on COPY FROM that the patch in the previous message did for COPY TO, that is to say perhaps 2-3x faster in BINARY mode and 10-20% faster in TEXT mode (these

Re: [HACKERS] COPY speedup

2009-08-12 Thread Tom Lane
=?utf-8?Q?Pierre_Fr=C3=A9d=C3=A9ric_Caillau?= =?utf-8?Q?d?= li...@peufeu.com writes: I've been examining the code path for COPY FROM too, and I think it is possible to get the same kind of speedups on COPY FROM that the patch in the previous message did for COPY TO, that is to say perhaps 2-3x

Re: [HACKERS] COPY speedup

2009-08-12 Thread Pierre Frédéric Caillau d
If you do as much damage to the I/O function API as the other patch did, it will probably be rejected. You mean, as the COPY patch in my previous message, or as another patch ? (I just search the archives and found one about CopyReadLine, but that's probably not what you are

Re: [HACKERS] COPY speedup

2009-08-12 Thread Alvaro Herrera
Merlin Moncure escribió: 2009/8/12 Pierre Frédéric Caillaud li...@peufeu.com: If you do as much damage to the I/O function API as the other patch did, it will probably be rejected.        You mean, as the COPY patch in my previous message, or as another patch ?        (I just

Re: [HACKERS] COPY speedup

2009-08-12 Thread Merlin Moncure
2009/8/12 Pierre Frédéric Caillaud li...@peufeu.com: If you do as much damage to the I/O function API as the other patch did, it will probably be rejected.        You mean, as the COPY patch in my previous message, or as another patch ?        (I just search the archives and found one

Re: [HACKERS] COPY speedup

2009-08-12 Thread Pierre Frédéric Caillau d
We don't touch datatype APIs lightly, because it affects too much code.                        regards, tom lane        I definitely agree with that. Actually, let me clarify: When I modified the datatype API, I was feeling uneasy, like I shouldn't really touch this. But when I see

Re: [HACKERS] COPY speedup

2009-08-12 Thread Alvaro Herrera
Pierre Frédéric Caillaud escribió: But when I see a big red button, I just press it to see what happens. Ugly hacks are useful to know how fast the thing can go ; then the interesting part is to reimplement it cleanly, trying to reach the same performance... Right -- now that you've shown a