Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Sat, Dec 5, 2009 at 3:32 AM, Daniel Farina drfar...@acm.org wrote: On Mon, Nov 30, 2009 at 12:14 PM, Greg Smith g...@2ndquadrant.com wrote: Jeff Davis wrote: COPY target FROM FUNCTION foo() WITH RECORDS; In what format would the records be? As a not-quite aside, I think I have a better

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 5:57 PM, Robert Haas robertmh...@gmail.com wrote: I am very fuzzy on where we stand with this patch.  There's a link to the initial version on the commitfest application, but there has been so much discussion since then that I think there are probably some revisions to

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 9:23 PM, Daniel Farina drfar...@acm.org wrote: On Tue, Dec 29, 2009 at 5:57 PM, Robert Haas robertmh...@gmail.com wrote: I am very fuzzy on where we stand with this patch.  There's a link to the initial version on the commitfest application, but there has been so much

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 6:48 PM, Robert Haas robertmh...@gmail.com wrote: I think there's clear support for a version of COPY that returns rows like a SELECT statement, particularly for use with CTEs.  There seems to be support both for a mode that returns text[] or something like it and also

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Jeff Davis
On Tue, 2009-12-29 at 21:48 -0500, Robert Haas wrote: Anyway, my specific reaction to your suggestions in the email that I quoted is that it seems a bit baroque and that I'm not really sure what it's useful for in practice. I'm certainly not saying it ISN'T useful, because I can't believe

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 9:56 PM, Daniel Farina drfar...@acm.org wrote: On Tue, Dec 29, 2009 at 6:48 PM, Robert Haas robertmh...@gmail.com wrote: I think there's clear support for a version of COPY that returns rows like a SELECT statement, particularly for use with CTEs.  There seems to be

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Jeff Davis
On Tue, 2009-12-29 at 23:11 -0500, Robert Haas wrote: I fear that to make this really useful we would need to define some new SQL syntax, like: CREATE [OR REPLACE] COPY TARGET name (STARTUP function_name, STREAM function_name, SHUTDOWN function_name); DROP COPY TARGET name; GRANT USAGE ON

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 8:11 PM, Robert Haas robertmh...@gmail.com wrote: It might be possible to do this without introducing a notion of an explicit object, but there are a couple of problems with that.  First, it requires the user to specify a lot of details on every COPY invocation, which

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 11:44 PM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2009-12-29 at 23:11 -0500, Robert Haas wrote: I fear that to make this really useful we would need to define some new SQL syntax, like: CREATE [OR REPLACE] COPY TARGET name (STARTUP function_name, STREAM

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Tue, Dec 29, 2009 at 11:47 PM, Daniel Farina drfar...@acm.org wrote: On Tue, Dec 29, 2009 at 8:11 PM, Robert Haas robertmh...@gmail.com wrote: It might be possible to do this without introducing a notion of an explicit object, but there are a couple of problems with that.  First, it

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 9:11 PM, Robert Haas robertmh...@gmail.com wrote: I think we should try to put an interface layer in place in the first iteration.  I don't really see this as having much value without that.  If we implement this strictly as a series of COPY options, we're going to be

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Wed, Dec 30, 2009 at 12:26 AM, Daniel Farina drfar...@acm.org wrote: I think you may be right about this, so we could basically move most things from the COPY option list into a new DDL option list...I am more ambivalent, but it seems that would require a catalog, and so on, which is why I

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Daniel Farina
On Tue, Dec 29, 2009 at 9:46 PM, Robert Haas robertmh...@gmail.com wrote: Sure.  If you think you can make it work using bytea, that seems clearly better than using an internal type, all things being equal. I think both are perfectly viable and can be supported simultaneously, actually...I

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Robert Haas
On Wed, Dec 30, 2009 at 12:56 AM, Daniel Farina drfar...@acm.org wrote: On Tue, Dec 29, 2009 at 9:46 PM, Robert Haas robertmh...@gmail.com wrote: Sure.  If you think you can make it work using bytea, that seems clearly better than using an internal type, all things being equal. I think both

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-29 Thread Jeff Davis
On Wed, 2009-12-30 at 00:00 -0500, Robert Haas wrote: I generally agree with those ideas, although I think that trying to make the existing aggregate interface serve this purpose will probably turn out to be trying to make a square peg fit a round hole. I didn't think that they intended to

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-12-05 Thread Daniel Farina
On Mon, Nov 30, 2009 at 12:14 PM, Greg Smith g...@2ndquadrant.com wrote: Jeff Davis wrote: COPY target FROM FUNCTION foo() WITH RECORDS; In what format would the records be? As a not-quite aside, I think I have a better syntax suggestion. I have recently been digging around in the grammar

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-30 Thread Greg Smith
Jeff Davis wrote: COPY target FROM FUNCTION foo() WITH RECORDS; In what format would the records be? What was your intended internal format for this form to process? -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Jeff Davis
On Fri, 2009-11-27 at 20:28 -0500, Greg Smith wrote: In the context of the read case, I'm not as sure it's so black and white. While the current situation does map better to a function that produces a stream of bytes, that's not necessarily the optimal approach for all situations. It's

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Jeff Davis
On Thu, 2009-11-26 at 18:30 -0800, Daniel Farina wrote: Okay, so this thread sort of wandered into how we could refactor other elements of COPY. Do we have a good sense on what we should do to the current patch (or at least the idea represented by it) to get it into a committable state within

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Daniel Farina
On Sun, Nov 29, 2009 at 6:35 PM, Jeff Davis pg...@j-davis.com wrote: What if the network buffer is flushed in the middle of a line? Is that possible, or is there a guard against that somewhere? What do you mean? They both catenate onto one stream of bytes, it shouldn't matter where the flush

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-29 Thread Jeff Davis
On Sun, 2009-11-29 at 18:53 -0800, Daniel Farina wrote: On Sun, Nov 29, 2009 at 6:35 PM, Jeff Davis pg...@j-davis.com wrote: What if the network buffer is flushed in the middle of a line? Is that possible, or is there a guard against that somewhere? What do you mean? They both catenate

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-27 Thread Simon Riggs
On Tue, 2009-11-24 at 22:13 -0800, Jeff Davis wrote: My disagreement with the row-by-row approach is more semantics than performance. COPY translates records to bytes and vice-versa, and your original patch maintains those semantics. The bytes - records conversion is a costly one. Anything we

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-27 Thread Jeff Davis
On Fri, 2009-11-27 at 13:39 +, Simon Riggs wrote: On Tue, 2009-11-24 at 22:13 -0800, Jeff Davis wrote: My disagreement with the row-by-row approach is more semantics than performance. COPY translates records to bytes and vice-versa, and your original patch maintains those semantics.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-27 Thread Greg Smith
Jeff Davis wrote: All I mean is that the second argument to COPY should produce/consume bytes and not records. I'm not discussing the internal implementation at all, only semantics. In other words, STDIN is not a source of records, it's a source of bytes; and likewise for STDOUT. In the

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Andrew Dunstan
On Thu, November 26, 2009 2:22 am, Jeff Davis wrote: On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: I worry that we're getting further away from the original problem. Let's allow functions to get the bytes of data from a COPY,

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Andrew Dunstan
On Thu, November 26, 2009 2:22 am, Jeff Davis wrote: On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: I worry that we're getting further away from the original problem. Let's allow functions to get the bytes of data from a COPY,

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Pavel Stehule
2009/11/26 Jeff Davis pg...@j-davis.com: On Thu, 2009-11-26 at 05:01 +0100, Pavel Stehule wrote: It working like: 1. EXECUTE SELECT 0 FROM generate_series(1,...); 2. STORE RESULT TO TABLE zero; 3. EXECUTE SELECT 1/i FROM zero; 4. STORE RESULT TO TABLE tmp; Problem is in seq execution.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread David Fetter
On Thu, Nov 26, 2009 at 10:11:12AM -0500, Andrew Dunstan wrote: On Thu, November 26, 2009 2:22 am, Jeff Davis wrote: On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: I worry that we're getting further away from the original

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-26 Thread Daniel Farina
On Thu, Nov 26, 2009 at 6:13 PM, David Fetter da...@fetter.org wrote: It'd be nice to make this available to the next revision of DBI-Link and it'll be pretty handy for our SQL/MED whenever that happens. Okay, so this thread sort of wandered into how we could refactor other elements of COPY.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Pavel Stehule
2009/11/25 Jeff Davis pg...@j-davis.com: On Wed, 2009-11-25 at 07:36 +0100, Pavel Stehule wrote: Moving records from a function to a table can be done with:  INSERT INTO mytable SELECT * FROM myfunc(); And that already works fine. It works, but COPY FROM myfunc() should be significantly

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Wed, 2009-11-25 at 09:23 +0100, Pavel Stehule wrote: If SRFs use a tuplestore in that situation, it sounds like that should be fixed. Why do we need to provide alternate syntax involving COPY? It isn't problem of SRF function design. It allow both mode - row and tuplestor. select *

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Hannu Krosing
On Tue, 2009-11-24 at 21:13 -0800, Jeff Davis wrote: On Tue, 2009-11-24 at 23:44 -0500, Tom Lane wrote: If you do that, then there is no possibility of ever using this feature except with C-coded functions, which seems to me to remove most of whatever use-case there was. It fits the use

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Pavel Stehule
2009/11/25 Jeff Davis pg...@j-davis.com: On Wed, 2009-11-25 at 09:23 +0100, Pavel Stehule wrote: If SRFs use a tuplestore in that situation, it sounds like that should be fixed. Why do we need to provide alternate syntax involving COPY? It isn't problem of SRF function design. It allow both

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Daniel Farina
On Tue, Nov 24, 2009 at 10:39 PM, Pavel Stehule pavel.steh...@gmail.com wrote: yes - it is two features - and should be solved independently There are some common problems though. I was thinking about this with some mind towards my existing mental model of thinking of specifying some parameters

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Wed, 2009-11-25 at 09:23 +0100, Pavel Stehule wrote: If SRFs use a tuplestore in that situation, it sounds like that should be fixed. Why do we need to provide alternate syntax involving COPY? It isn't problem of SRF function design. It allow both mode

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Wed, 2009-11-25 at 11:32 +0100, Pavel Stehule wrote: 1. postgres=# select count(*) from generate_series(1,100); count ─ 100 (1 row) Time: 930,720 ms 2. postgres=# select count(*) from (select generate_series(1,100)) x; count ─ 100 (1 row)

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Pavel Stehule
2009/11/25 Jeff Davis pg...@j-davis.com: On Wed, 2009-11-25 at 11:32 +0100, Pavel Stehule wrote: 1. postgres=# select count(*) from generate_series(1,100);   count ─  100 (1 row) Time: 930,720 ms 2. postgres=# select count(*) from (select generate_series(1,100)) x;

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Andrew Dunstan
On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: I worry that we're getting further away from the original problem. Let's allow functions to get the bytes of data from a COPY, like the original proposal. I am not sure COPY is the best mechanism to move records around when INSERT ...

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Daniel Farina
On Wed, Nov 25, 2009 at 9:35 PM, Andrew Dunstan and...@dunslane.net wrote: On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: I worry that we're getting further away from the original problem. Let's allow functions to get the bytes of data from a COPY, like the original proposal. I am not

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Thu, 2009-11-26 at 05:01 +0100, Pavel Stehule wrote: It working like: 1. EXECUTE SELECT 0 FROM generate_series(1,...); 2. STORE RESULT TO TABLE zero; 3. EXECUTE SELECT 1/i FROM zero; 4. STORE RESULT TO TABLE tmp; Problem is in seq execution. Result is stored to destination after

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-25 Thread Jeff Davis
On Thu, 2009-11-26 at 00:35 -0500, Andrew Dunstan wrote: On Wed, November 25, 2009 3:56 pm, Jeff Davis wrote: I worry that we're getting further away from the original problem. Let's allow functions to get the bytes of data from a COPY, like the original proposal. I am not sure COPY is

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Mon, 2009-11-23 at 18:31 -0500, Greg Smith wrote: Robert Haas wrote: I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. This has been through some heavy

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Mon, 2009-11-23 at 16:25 -0800, Daniel Farina wrote: On Mon, Nov 23, 2009 at 4:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: pgsql-hackers had some preliminary discussions a couple months back on refactoring COPY to allow things like this --- see the thread starting here:

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 12:29 AM, Hannu Krosing ha...@krosing.net wrote: COPY stdin TO udf(); If stdin becomes (is?) a legitimate source of records, then this patch will Just Work. The patch is already quite useful in the COPY (SELECT ...) TO FUNCTION ... scenario. COPY udf() FROM stdin;

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 12:38 AM, Hannu Krosing ha...@2ndquadrant.com wrote: COPY func(rowtype) FROM stdin; I didn't consider rowtype...I did consider a type list, such as: COPY func(typea, typeb, typec) FROM ... Which would then operate just like a table, but be useless for the data-cleaning

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
Hello I thing, so this patch is maybe good idea. I am missing better function specification. Specification by name isn't enough - we can have a overloaded functions. This syntax doesn't allow to use explicit cast - from my personal view, the syntax is ugly - with type specification we don't need

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I thing, so this patch is maybe good idea. I am missing better function specification. Specification by name isn't enough - we can have a overloaded functions. This syntax doesn't allow to use explicit cast -

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 02:37 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I thing, so this patch is maybe good idea. I am missing better function specification. Specification by name isn't enough - we can have a

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing ha...@2ndquadrant.com wrote: Can't you use existing aggregate function design ? CREATE AGGREGATE name ( input_data_type [ , ... ] ) (    SFUNC = sfunc,    STYPE = state_data_type    [ , FINALFUNC = ffunc ]    [ , INITCOND = initial_condition ]

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina drfar...@gmail.com wrote: On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing ha...@2ndquadrant.com wrote: Can't you use existing aggregate function design ? CREATE AGGREGATE name ( input_data_type [ , ... ] ) (    SFUNC = sfunc,    STYPE =

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 02:56 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina drfar...@gmail.com wrote: On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing ha...@2ndquadrant.com wrote: Can't you use existing aggregate function design ? CREATE AGGREGATE name (

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 3:25 AM, Hannu Krosing ha...@2ndquadrant.com wrote: On Tue, 2009-11-24 at 02:56 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina drfar...@gmail.com wrote: On Tue, Nov 24, 2009 at 2:50 AM, Hannu Krosing ha...@2ndquadrant.com wrote: Can't

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/24 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I thing, so this patch is maybe good idea. I am missing better function specification. Specification by name isn't enough - we can have a overloaded functions.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2009/11/24 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I thing, so this patch is maybe good idea. I am missing better function

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 03:48 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 3:25 AM, Hannu Krosing ha...@2ndquadrant.com wrote: On Tue, 2009-11-24 at 02:56 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 2:51 AM, Daniel Farina drfar...@gmail.com wrote: On Tue, Nov 24, 2009 at 2:50

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/24 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule pavel.steh...@gmail.com wrote: 2009/11/24 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 2:10 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello I thing, so this patch is maybe good

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Hannu Krosing
On Tue, 2009-11-24 at 05:00 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule pavel.steh...@gmail.com wrote: then syntax should be: COPY table TO streamname(parameters) COPY table TO filestream('/tmp/foo.dta') ... COPY table TO

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Robert Haas
On Mon, Nov 23, 2009 at 8:46 PM, Greg Smith g...@2ndquadrant.com wrote: You know how people complain about how new contributors are treated here?  Throwing out comments like this, that come off as belittling to other people's work, doesn't help.  All I was suggesting was that Dan wasn't

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/24 Hannu Krosing ha...@2ndquadrant.com: On Tue, 2009-11-24 at 05:00 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 4:37 AM, Pavel Stehule pavel.steh...@gmail.com wrote: then syntax should be: COPY table TO streamname(parameters) COPY table TO filestream('/tmp/foo.dta')

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Robert Haas
On Mon, Nov 23, 2009 at 9:37 PM, Andrew Dunstan and...@dunslane.net wrote: Greg Smith wrote: I haven't heard anything from Andrew about ragged CVS import either.  I think that ultimately those features are useful, but just exceed what the existing code could be hacked to handle cleanly.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 00:54 -0800, Daniel Farina wrote: On Tue, Nov 24, 2009 at 12:29 AM, Hannu Krosing ha...@krosing.net wrote: COPY stdin TO udf(); If stdin becomes (is?) a legitimate source of records, then this patch will Just Work. STDIN is a source of bytes representing a set of

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 14:39 +0100, Pavel Stehule wrote: a) good designed C API like: initialise_functions(fcinfo) -- std fcinfo consument_process_tuple(fcinfo) -- gets standard row -- Datum dvalues[] + Row description producent_process_tuple(fcinfo) -- returns standard row --

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: Don't you still need the functions to accept an argument of type internal? Otherwise, we lose the ability to copy a buffer to the dblink connection, which was the original motivation. If you do that, then there is no possibility of ever using this feature

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Jeff Davis pg...@j-davis.com: On Tue, 2009-11-24 at 14:39 +0100, Pavel Stehule wrote: a) good designed C API  like:    initialise_functions(fcinfo) -- std fcinfo    consument_process_tuple(fcinfo) -- gets standard row -- Datum dvalues[] + Row description    

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule pavel.steh...@gmail.com wrote: It depends on design. I don't thing so internal is necessary. It is just wrong design. Depends on how lean you want to be when doing large COPY...right now the cost is restricted to having to call a function pointer

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 23:44 -0500, Tom Lane wrote: If you do that, then there is no possibility of ever using this feature except with C-coded functions, which seems to me to remove most of whatever use-case there was. It fits the use case involving dblink (or dblink-like modules). Maybe the

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 9:13 PM, Jeff Davis pg...@j-davis.com wrote: I still don't see any reason to force it to be record by record though. If the point is to push data from a table into a remote table, why should the copied data be translated out of binary format into a record, and then

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule pavel.steh...@gmail.com wrote: It depends on design. I don't thing so internal is necessary. It is just wrong design. Depends on how lean you want to be when doing large COPY...right now the cost is

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 9:35 PM, Pavel Stehule pavel.steh...@gmail.com wrote: 2009/11/25 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule pavel.steh...@gmail.com wrote: It depends on design. I don't thing so internal is necessary. It is just wrong design.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 9:35 PM, Pavel Stehule pavel.steh...@gmail.com wrote: 2009/11/25 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 8:45 PM, Pavel Stehule pavel.steh...@gmail.com wrote: It depends on design. I don't thing so

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Tue, 2009-11-24 at 21:42 -0800, Daniel Farina wrote: You are probably right. We could try coercing to bytea and back out to bytes, although it seems like a superfluous cost to force *everyone* to pay just to get the same bytes to a network buffer. Well, I suppose only performance will

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: I believe so using an internal minimalize necessary changes in COPY implementation. Using a funcapi needs more work inside COPY - you have to take some functionality from COPY to stream functions. Probably the most slow operations is

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Jeff Davis pg...@j-davis.com: On Tue, 2009-11-24 at 21:42 -0800, Daniel Farina wrote: You are probably right.  We could try coercing to bytea and back out to bytes, although it seems like a superfluous cost to force *everyone* to pay just to get the same bytes to a network buffer.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Daniel Farina
On Tue, Nov 24, 2009 at 10:23 PM, Jeff Davis pg...@j-davis.com wrote: On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: I believe so using an internal minimalize necessary changes in COPY implementation. Using a funcapi needs more work inside COPY -  you have to take some functionality

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Jeff Davis pg...@j-davis.com: On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: I believe so using an internal minimalize necessary changes in COPY implementation. Using a funcapi needs more work inside COPY -  you have to take some functionality from COPY to stream functions.

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 10:23 PM, Jeff Davis pg...@j-davis.com wrote: On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: I believe so using an internal minimalize necessary changes in COPY implementation. Using a funcapi needs more work inside

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Pavel Stehule
2009/11/25 Pavel Stehule pavel.steh...@gmail.com: 2009/11/25 Daniel Farina drfar...@gmail.com: On Tue, Nov 24, 2009 at 10:23 PM, Jeff Davis pg...@j-davis.com wrote: On Wed, 2009-11-25 at 06:35 +0100, Pavel Stehule wrote: I believe so using an internal minimalize necessary changes in COPY

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Wed, 2009-11-25 at 07:31 +0100, Pavel Stehule wrote: My disagreement with the row-by-row approach is more semantics than performance. COPY translates records to bytes and vice-versa, and your original patch maintains those semantics. uff, really COPY CSV ? CSV is like text or

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-24 Thread Jeff Davis
On Wed, 2009-11-25 at 07:36 +0100, Pavel Stehule wrote: Moving records from a function to a table can be done with: INSERT INTO mytable SELECT * FROM myfunc(); And that already works fine. It works, but COPY FROM myfunc() should be significantly faster. You can skip tuple store. If

[HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
Signed-off-by: Daniel Farina dfar...@truviso.com --- contrib/dblink/expected/dblink.out | 272 contrib/dblink/sql/dblink.sql | 112 +++ 2 files changed, 384 insertions(+), 0 deletions(-) diff --git a/contrib/dblink/expected/dblink.out

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Robert Haas
On Mon, Nov 23, 2009 at 4:34 PM, Daniel Farina dfar...@truviso.com wrote: Signed-off-by: Daniel Farina dfar...@truviso.com Thanks for the patch. You may want to take a look at this: http://wiki.postgresql.org/wiki/Submitting_a_Patch I'm fuzzy on what problem this is attempting to solve... as

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Greg Smith
Robert Haas wrote: I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. This has been through some heavy design discussions with a few PG hackers you know and some

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 2:16 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Nov 23, 2009 at 4:34 PM, Daniel Farina dfar...@truviso.com wrote: Signed-off-by: Daniel Farina dfar...@truviso.com Thanks for the patch.  You may want to take a look at this:

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Andrew Dunstan
Greg Smith wrote: Robert Haas wrote: I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. This has been through some heavy design discussions with a few PG

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 3:46 PM, Andrew Dunstan and...@dunslane.net wrote: I recently found myself trying to push data through dblink() and ended up writing code to make a call to the target to call a function which called back to the source to select the data and insert it. The speedup was

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 4:03 PM, Daniel Farina dfar...@truviso.com wrote: Yes.  Take a look at the tests introduced to core PostgeSQL (see patch 2), where instead of returning a text[] I return just a single text of the verbatim output of the copy.  You could imagine making that an SRF

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: Robert Haas wrote: I'm fuzzy on what problem this is attempting to solve... as mentioned in the above guidelines, it's usually good to start with some design discussions before writing/submitting code. This has been through some heavy design

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Daniel Farina
On Mon, Nov 23, 2009 at 4:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: pgsql-hackers had some preliminary discussions a couple months back on refactoring COPY to allow things like this --- see the thread starting here: http://archives.postgresql.org/pgsql-hackers/2009-09/msg00486.php While I

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Greg Smith
Tom Lane wrote: Those discussions don't have a lot of credibility if they didn't take place on the public mailing lists. You know how people complain about how new contributors are treated here? Throwing out comments like this, that come off as belittling to other people's work, doesn't

Re: [HACKERS] [PATCH 4/4] Add tests to dblink covering use of COPY TO FUNCTION

2009-11-23 Thread Andrew Dunstan
Greg Smith wrote: I haven't heard anything from Andrew about ragged CVS import either. I think that ultimately those features are useful, but just exceed what the existing code could be hacked to handle cleanly. The patch is attached for your edification/amusement. I have backpatched it