Re: [HACKERS] exposing COPY API

On Wed, Feb 9, 2011 at 12:45 PM, Andrew Dunstan wrote: > Itagaki-san published a patch for this about about 12 hours ago in the > file_fdw thread that looks pretty committable to me. OK, excellent. > This whole API thing is a breakout from file_fdw, because the original > file_fdw submission cop

Re: [HACKERS] exposing COPY API

On 02/09/2011 12:26 PM, Robert Haas wrote: On Wed, Feb 9, 2011 at 7:38 AM, Shigeru HANADA wrote: On Tue, 8 Feb 2011 08:49:36 -0500 Robert Haas wrote: On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA wrote: I'll submit revised file_fdw patch after removing IsForeignTable() catalog lookup a

Re: [HACKERS] exposing COPY API

On Wed, Feb 9, 2011 at 7:38 AM, Shigeru HANADA wrote: > On Tue, 8 Feb 2011 08:49:36 -0500 > Robert Haas wrote: >> On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA >> wrote: >> > I'll submit revised file_fdw patch after removing IsForeignTable() >> > catalog lookup along Heikki's proposal. >> >> So

Re: [HACKERS] exposing COPY API

On Tue, 8 Feb 2011 08:49:36 -0500 Robert Haas wrote: > On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA > wrote: > > I'll submit revised file_fdw patch after removing IsForeignTable() > > catalog lookup along Heikki's proposal. > > So I'm a bit confused. I don't see the actual copy API change pat

Re: [HACKERS] exposing COPY API

On Tue, Feb 8, 2011 at 4:42 AM, Shigeru HANADA wrote: > On Tue, 8 Feb 2011 17:49:09 +0900 > Itagaki Takahiro wrote: > >> On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan wrote: >> > Here is a patch against the latest revision of file_fdw to exercise this >> > API. It includes some regression tests,

Re: [HACKERS] exposing COPY API

On Tue, 8 Feb 2011 17:49:09 +0900 Itagaki Takahiro wrote: > On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan wrote: > > Here is a patch against the latest revision of file_fdw to exercise this > > API. It includes some regression tests, and I think apart from one or two > > small details plus a requ

Re: [HACKERS] exposing COPY API

On 02/08/2011 03:49 AM, Itagaki Takahiro wrote: On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan wrote: Here is a patch against the latest revision of file_fdw to exercise this API. It includes some regression tests, and I think apart from one or two small details plus a requirement for documenta

Re: [HACKERS] exposing COPY API

On Tue, Feb 8, 2011 at 09:38, Andrew Dunstan wrote: > Here is a patch against the latest revision of file_fdw to exercise this > API. It includes some regression tests, and I think apart from one or two > small details plus a requirement for documentation, is complete. The patch contains a few fi

Re: [HACKERS] exposing COPY API

On 02/07/2011 11:34 AM, Andrew Dunstan wrote: On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***field

Re: [HACKERS] exposing COPY API

On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It j

Re: [HACKERS] exposing COPY API

On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It j

Re: [HACKERS] exposing COPY API

Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It just returns separated fields in the next line. Fortunat

Re: [HACKERS] exposing COPY API

On 02/03/2011 10:43 PM, Itagaki Takahiro wrote: Also, a FDW allows the COPY to be used as a FROM target, giving it great flexibility. AFAICT this does not. BTW, which do you want to improve, FDW or COPY FROM? If FDW, the better API would be "raw" version of NextCopyFrom(). For example: b

Re: [HACKERS] exposing COPY API

On Fri, Feb 4, 2011 at 12:17, Andrew Dunstan wrote: >> http://pgbulkload.projects.postgresql.org/pg_bulkload.html#filter > AFAICT, this doesn't support ragged tables with too many columns, which is > my prime use case. If it supported variadic arguments in filter functions it > might come closer.

Re: [HACKERS] exposing COPY API

On 02/03/2011 09:43 PM, Itagaki Takahiro wrote: On Fri, Feb 4, 2011 at 11:32, Andrew Dunstan wrote: Umm, where? I can't find this in the documentation Here: http://pgbulkload.projects.postgresql.org/pg_bulkload.html#filter The ob

Re: [HACKERS] exposing COPY API

On Fri, Feb 4, 2011 at 11:32, Andrew Dunstan wrote: > Umm, where? I can't find this in the documentation > Here: http://pgbulkload.projects.postgresql.org/pg_bulkload.html#filter > The object, as I have explained previously, is to have

Re: [HACKERS] exposing COPY API

Of course, I want this so I could construct a text array from the read in data, but I could also imagine a foreign data wrapper wanting to mangle the data before handing it to postgres, say by filling in a field or hashing it. Could you explain the actual use-cases and examples? I think we ne

Re: [HACKERS] exposing COPY API

On Fri, Feb 4, 2011 at 09:48, Andrew Dunstan wrote: > I'd like to be able to add a callback function to construct the values for > the tuple. So it would become something like: >   typedef void (*copy_make_values) (CopyState cstate, NumFieldsRead int); You can do nothing interesting in the callba

[HACKERS] exposing COPY API

Revisiting this, it occurred to me that I could achieve what I need of we extend the proposed API a bit. Currently, it has: extern CopyState BeginCopyFrom(Relation rel, const char *filename, List *attnamelist, List *options); I'd like to be able to add a