Re: [PERFORM] copy vs. C function

2016-12-30 Thread rajmhn
Hi Jon This is exactly, what I was looking for. Need to read the data from delimited file with no header, and do few transformation as described below using Postgres C function and load it using pg_bulkload utility. Transformation below, can be handled with query after loading all the data

Re: [PERFORM] copy vs. C function

2011-12-14 Thread Kevin Martyn
? Please help me, Thank you, Danny -- *From:* Tom Lane t...@sss.pgh.pa.us *To:* Jon Nelson jnelson+pg...@jamponi.net *Cc:* pgsql-performance@postgresql.org *Sent:* Wednesday, December 14, 2011 8:18 AM *Subject:* Re: [PERFORM] copy vs. C function Jon Nelson

Re: [PERFORM] copy vs. C function

2011-12-14 Thread Jon Nelson
On Wed, Dec 14, 2011 at 12:18 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: The only thing I have left are these statements: get_call_result_type TupleDescGetAttInMetadata BuildTupleFromCStrings HeapTupleGetDatum and finally PG_RETURN_DATUM It turns

Re: [PERFORM] copy vs. C function

2011-12-14 Thread idc danny
to achieve my restriction? Please help me, Thank you, Danny From: Tom Lane t...@sss.pgh.pa.us To: Jon Nelson jnelson+pg...@jamponi.net Cc: pgsql-performance@postgresql.org Sent: Wednesday, December 14, 2011 8:18 AM Subject: Re: [PERFORM] copy vs. C function Jon

Re: [PERFORM] copy vs. C function

2011-12-14 Thread Tom Lane
Jon Nelson jnelson+pg...@jamponi.net writes: Regarding caching, I tried caching it across calls by making the TupleDesc static and only initializing it once. When I tried that, I got: ERROR: number of columns (6769856) exceeds limit (1664) I tried to find some documentation or examples

Re: [PERFORM] copy vs. C function

2011-12-14 Thread Jon Nelson
On Wed, Dec 14, 2011 at 9:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: Regarding caching, I tried caching it across calls by making the TupleDesc static and only initializing it once. When I tried that, I got: ERROR:  number of columns (6769856)

Re: [PERFORM] copy vs. C function

2011-12-14 Thread Jon Nelson
On Wed, Dec 14, 2011 at 9:40 AM, Jon Nelson jnelson+pg...@jamponi.net wrote: On Wed, Dec 14, 2011 at 9:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: Regarding caching, I tried caching it across calls by making the TupleDesc static and only initializing

Re: [PERFORM] copy vs. C function

2011-12-14 Thread Jon Nelson
On Wed, Dec 14, 2011 at 9:51 AM, Jon Nelson jnelson+pg...@jamponi.net wrote: On Wed, Dec 14, 2011 at 9:40 AM, Jon Nelson jnelson+pg...@jamponi.net wrote: On Wed, Dec 14, 2011 at 9:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: Regarding caching, I tried

Re: [PERFORM] copy vs. C function

2011-12-13 Thread Jon Nelson
On Mon, Dec 12, 2011 at 10:38 AM, Merlin Moncure mmonc...@gmail.com wrote: On Sat, Dec 10, 2011 at 7:27 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: I was experimenting with a few different methods of taking a line of text, parsing it, into a set of fields, and then getting that info into a

Re: [PERFORM] copy vs. C function

2011-12-13 Thread Tom Lane
Jon Nelson jnelson+pg...@jamponi.net writes: The only thing I have left are these statements: get_call_result_type TupleDescGetAttInMetadata BuildTupleFromCStrings HeapTupleGetDatum and finally PG_RETURN_DATUM It turns out that: get_call_result_type adds 43 seconds [total: 54],

Re: [PERFORM] copy vs. C function

2011-12-13 Thread idc danny
Nelson jnelson+pg...@jamponi.net Cc: pgsql-performance@postgresql.org Sent: Wednesday, December 14, 2011 8:18 AM Subject: Re: [PERFORM] copy vs. C function Jon Nelson jnelson+pg...@jamponi.net writes: The only thing I have left are these statements: get_call_result_type

Re: [PERFORM] copy vs. C function

2011-12-12 Thread Merlin Moncure
On Sat, Dec 10, 2011 at 7:27 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: I was experimenting with a few different methods of taking a line of text, parsing it, into a set of fields, and then getting that info into a table. The first method involved writing a C program to parse a file,

Re: [PERFORM] copy vs. C function

2011-12-10 Thread Craig Ringer
On 12/11/2011 09:27 AM, Jon Nelson wrote: The first method involved writing a C program to parse a file, parse the lines and output newly-formatted lines in a format that postgresql's COPY function can use. End-to-end, this takes 15 seconds for about 250MB (read 250MB, parse, output new data to

Re: [PERFORM] copy vs. C function

2011-12-10 Thread Sam Gendler
Start a transaction before the first insert and commit it after the last one and it will be much better, but I believe that the copy code path is optimized to perform better than any set of queries can, even in a single transaction Sent from my iPhone On Dec 10, 2011, at 5:27 PM, Jon Nelson

Re: [PERFORM] copy vs. C function

2011-12-10 Thread Jon Nelson
On Sat, Dec 10, 2011 at 8:32 PM, Craig Ringer ring...@ringerc.id.au wrote: On 12/11/2011 09:27 AM, Jon Nelson wrote: The first method involved writing a C program to parse a file, parse the lines and output newly-formatted lines in a format that postgresql's COPY function can use.