Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-11 Thread Shigeru HANADA
Hi all, (2012/03/07 3:39), Tom Lane wrote: A bigger issue with postgresql_fdw_validator is that it supposes that the core backend is authoritative as to what options libpq supports, which is bad design on its face. It would be much more sensible for dblink to be asking libpq what options

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-09 Thread Thom Brown
2012/4/9 Shigeru HANADA shigeru.han...@gmail.com:    1) connect to the server at the beginning of the local query    2) execute EXPLAIN for foreign table foo    3) execute EXPLAIN for foreign table bar    4) execute actual query for foreign table foo    5) execute actual query for foreign

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-08 Thread Shigeru HANADA
(2012/04/08 5:19), Thom Brown wrote: 2012/4/7 Shigeru HANADAshigeru.han...@gmail.com: I've updated pgsql_fdw so that it can collect statistics from foreign data with new FDW API. I notice that if you restart the remote server, the connection is broken, but the client doesn't notice this

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-08 Thread Gerald Devotta
This is Gerald Devotta, Recruitment Specialist with Newt Global LLC, a global staffing solutions provider that has been serving the telecommunications and utility industries. I am contacting you to let you know that your resume came to my attention while I was conducting a job search for a

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-07 Thread Thom Brown
2012/4/7 Shigeru HANADA shigeru.han...@gmail.com: I've updated pgsql_fdw so that it can collect statistics from foreign data with new FDW API. I notice that if you restart the remote server, the connection is broken, but the client doesn't notice this until it goes to fire off another command.

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-06 Thread Shigeru HANADA
(2012/04/06 1:29), Tom Lane wrote: Albe Laurenzlaurenz.a...@wien.gv.at writes: Maybe the FDW API could be extended so that foreign data wrappers can provide a random sample to avoid a full table scan. The one thing that seems pretty clear from this discussion is that one size doesn't fit

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-06 Thread Kyotaro HORIGUCHI
Excuse me for cutting in, 2012/4/6 Shigeru HANADA shigeru.han...@gmail.com: To support foreign-table ANALYZE by adding a new hook, we would need a mechanism (or at least documented guide lines) to manage the chain of hook functions, because such hook might be used by multiple FDWs (or other

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-06 Thread Tom Lane
Shigeru HANADA shigeru.han...@gmail.com writes: (2012/04/06 1:29), Tom Lane wrote: The one thing that seems pretty clear from this discussion is that one size doesn't fit all. I think we really ought to provide a hook so that the FDW can determine whether ANALYZE applies to its foreign tables

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-06 Thread Shigeru Hanada
On Fri, Apr 6, 2012 at 11:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Another concern is the place where we hook the process of ANALYZE.  IOW, how much portion of ANALYZE should be overridable? Not much, IMO.  The FDW should be able to decide whether or not to analyze a particular table, and it

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-06 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: On Fri, Apr 6, 2012 at 11:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: In particular I do not like the specific way it's done in the v7 patch (I've not looked at v8 yet) because the interposed logic has a hard-wired assumption that foreign tables

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-05 Thread Etsuro Fujita
At 22:11 12/03/28 +0900, Shigeru HANADA wrote: ANALYZE support for foreign tables is proposed by Fujita-san in current CF, so I'd like to push it. I updated the patch to the latest HEAD. Please find attached a patch. Best regards, Etsuro Fujita postgresql-analyze-v7.patch Description:

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-05 Thread Etsuro Fujita
05, 2012 9:36 PM To: Shigeru HANADA; Albe Laurenz Cc: Tom Lane; Kevin Grittner; Robert Haas; PostgreSQL-development; Kohei KaiGai; Martijn van Oosterhout; Hitoshi Harada Subject: Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server At 22:11 12/03/28 +0900, Shigeru HANADA wrote: ANALYZE support

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-05 Thread Tom Lane
Albe Laurenz laurenz.a...@wien.gv.at writes: I think that the price of a remote table scan is something we should be willing to pay for good local statistics. And there is always the option not to analyze the foreign table if you are not willing to pay that price. Maybe the FDW API could be

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-04 Thread Albe Laurenz
Shigeru HANADA wrote: During a foreign scan, type input functions are used to convert the text representation of values. If a foreign table is misconfigured, you can get error messages from these functions, like: ERROR: invalid input syntax for type double precision: etwas or ERROR:

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-03 Thread Albe Laurenz
Shigeru HANADA wrote: Attached are latest version of pgsql_fdw patches. Note that pgsql_fdw_analyze.patch is only for test the effect of local statistics. Please apply patches in the order below: (1) pgsql_fdw_v18.patch (2) pgsql_fdw_pushdown_v11.patch (3) pgsql_fdw_analyze.patch (if you

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-30 Thread Albe Laurenz
I wrote: How about getting # of rows estimate by executing EXPLAIN for fully-fledged remote query (IOW, contains pushed-down WHERE clause), and estimate selectivity of local filter on the basis of the statistics which are generated by FDW via do_analyze_rel() and FDW-specific sampling

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-28 Thread Thom Brown
2012/3/28 Shigeru HANADA shigeru.han...@gmail.com: (2012/03/27 20:32), Thom Brown wrote: 2012/3/26 Shigeru HANADAshigeru.han...@gmail.com: * pgsql_fdw_v17.patch     - Adds pgsql_fdw as contrib module * pgsql_fdw_pushdown_v10.patch     - Adds WHERE push down capability to pgsql_fdw *

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-28 Thread Thom Brown
On 28 March 2012 08:13, Thom Brown t...@linux.com wrote: 2012/3/28 Shigeru HANADA shigeru.han...@gmail.com: (2012/03/27 20:32), Thom Brown wrote: 2012/3/26 Shigeru HANADAshigeru.han...@gmail.com: * pgsql_fdw_v17.patch     - Adds pgsql_fdw as contrib module * pgsql_fdw_pushdown_v10.patch    

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-28 Thread Thom Brown
On 28 March 2012 08:39, Thom Brown t...@linux.com wrote: On 28 March 2012 08:13, Thom Brown t...@linux.com wrote: 2012/3/28 Shigeru HANADA shigeru.han...@gmail.com: (2012/03/27 20:32), Thom Brown wrote: 2012/3/26 Shigeru HANADAshigeru.han...@gmail.com: * pgsql_fdw_v17.patch     - Adds

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-28 Thread Shigeru HANADA
(2012/03/28 16:18), Albe Laurenz wrote: I wrote: How about getting # of rows estimate by executing EXPLAIN for fully-fledged remote query (IOW, contains pushed-down WHERE clause), and estimate selectivity of local filter on the basis of the statistics which are generated by FDW via

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-28 Thread Shigeru HANADA
(2012/03/28 21:07), Albe Laurenz wrote: I found another limitation of this approach: pgsql_fdw_analyze() has to run as a user who can update pg_statistic, and this user needs a user mapping to a remote user who can read pg_statistic. This is not necessary for normal operation and needs to

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-27 Thread Albe Laurenz
Shigeru HANADA wrote: I've implemented pgsql_fdw's own deparser and enhanced some features since last post. Please apply attached patches in the order below: Changes from previous version = 1) Don't use remote EXPLAIN for cost/rows estimation, so now planner

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-27 Thread Shigeru HANADA
Thanks for the comments. (2012/03/27 18:36), Albe Laurenz wrote: 2) Defer planning stuffs as long as possible to clarify the role of each function. Currently GetRelSize just estimates result rows from local statistics, and GetPaths adds only one path which represents SeqScan on remote

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-27 Thread Thom Brown
2012/3/26 Shigeru HANADA shigeru.han...@gmail.com: (2012/03/15 23:06), Shigeru HANADA wrote: Although the patches are still WIP, especially in WHERE push-down part, but I'd like to post them so that I can get feedback of the design as soon as possible. I've implemented pgsql_fdw's own

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-27 Thread Albe Laurenz
Shigeru HANADA wrote: My gut feeling is that planning should be done by the server which will execute the query. Agreed, if selectivity of both local filtering and remote filtering were available, we can estimate result rows correctly and choose better plan. How about getting # of rows

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-27 Thread Shigeru HANADA
(2012/03/27 20:32), Thom Brown wrote: 2012/3/26 Shigeru HANADAshigeru.han...@gmail.com: * pgsql_fdw_v17.patch - Adds pgsql_fdw as contrib module * pgsql_fdw_pushdown_v10.patch - Adds WHERE push down capability to pgsql_fdw * pgsql_fdw_analyze_v1.patch - Adds pgsql_fdw_analyze

Corrected: Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-12 Thread Etsuro Fujita
(2012/03/12 13:04), Etsuro Fujita wrote: (2012/03/09 23:48), Tom Lane wrote: Etsuro Fujitafujita.ets...@lab.ntt.co.jp writes: 2. IMHO RelOptInfo.fdw_private seems confusing. How about renaming it to e.g., RelOptInfo.fdw_state? Why is that better? It seems just as open to confusion with

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-12 Thread Albe Laurenz
Tom Lane wrote: Shigeru Hanada shigeru.han...@gmail.com writes: Thanks for the review. Agreed to write own depraser for pgsql_fdw which handles nodes which can be pushed down. Every SQL-based FDW which constructs SQL statement for each local query would need such module inside. Yeah.

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-11 Thread Etsuro Fujita
(2012/03/09 23:48), Tom Lane wrote: Etsuro Fujitafujita.ets...@lab.ntt.co.jp writes: Thank you for your answer. 1. FilefdwPlanState.pages and FileFdwPlanState.ntuples seems redundant. Why not use RelOptInfo.pages and RelOptInfo.tuples? I intentionally avoided setting RelOptInfo.pages

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-10 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: I've not read whole of the patch yet, but I have basic questions. 1) IIUC, GetForeignRelSize should set baserel-rows to the number of rows the ForeignScan node returns to upper node, but not the number of rows FDW returns to core executor,

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-10 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: Thanks for the review. Agreed to write own depraser for pgsql_fdw which handles nodes which can be pushed down. Every SQL-based FDW which constructs SQL statement for each local query would need such module inside. Yeah. That's kind of

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-10 Thread Martijn van Oosterhout
On Sat, Mar 10, 2012 at 11:38:51AM -0500, Tom Lane wrote: Shigeru Hanada shigeru.han...@gmail.com writes: Thanks for the review. Agreed to write own depraser for pgsql_fdw which handles nodes which can be pushed down. Every SQL-based FDW which constructs SQL statement for each local query

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-09 Thread Etsuro Fujita
(2012/03/09 14:00), Tom Lane wrote: I wrote: There are a couple of other points that make me think we need to revisit the PlanForeignScan API definition some more, too. ... So we need to break down what PlanForeignScan currently does into three separate steps. The first idea that comes to

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-09 Thread Tom Lane
Etsuro Fujita fujita.ets...@lab.ntt.co.jp writes: (2012/03/09 14:00), Tom Lane wrote: Attached is a draft patch for that. 1. FilefdwPlanState.pages and FileFdwPlanState.ntuples seems redundant. Why not use RelOptInfo.pages and RelOptInfo.tuples? I intentionally avoided setting

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-09 Thread Shigeru Hanada
I've not read whole of the patch yet, but I have basic questions. 1) IIUC, GetForeignRelSize should set baserel-rows to the number of rows the ForeignScan node returns to upper node, but not the number of rows FDW returns to core executor, right? BTW, once Fujita-san's ANALYZE support patch is

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-09 Thread Shigeru Hanada
(2012/03/09 1:18), Tom Lane wrote: I've been looking at this patch a little bit over the past day or so. I'm pretty unhappy with deparse.c --- it seems like a real kluge, inefficient and full of corner-case bugs. After some thought I believe that you're ultimately going to have to abandon

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-09 Thread Shigeru Hanada
(2012/03/06 23:47), Albe Laurenz wrote: Shigeru Hanada wrote: Connection should be closed only when the trigger is a top level transaction and it's aborting, but isTopLevel flag was not checked. I fixed the bug and added regression tests for such cases. I wondered about that

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-08 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: [ pgsql_fdw_v15.patch ] I've been looking at this patch a little bit over the past day or so. I'm pretty unhappy with deparse.c --- it seems like a real kluge, inefficient and full of corner-case bugs. After some thought I believe that you're

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-08 Thread Tom Lane
I wrote: There are a couple of other points that make me think we need to revisit the PlanForeignScan API definition some more, too. ... So we need to break down what PlanForeignScan currently does into three separate steps. The first idea that comes to mind is to call them

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-07 Thread Peter Eisentraut
On tis, 2012-03-06 at 13:39 -0500, Tom Lane wrote: A bigger issue with postgresql_fdw_validator is that it supposes that the core backend is authoritative as to what options libpq supports, which is bad design on its face. It would be much more sensible for dblink to be asking libpq what

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-07 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: Agreed. Attached fdw_helper patch doesn't contain GetFdwOptionValue() any more, and pgsql_fdw patch accesses only necessary catalogs. I've committed the fdw_helper part of this, with some very minor improvements. regards,

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-06 Thread Albe Laurenz
Shigeru Hanada wrote: [pgsql_fdw_v12.patch] I know this is not the latest version, but I played around with it and tickled a bug. It seems to have a problem with rolled back subtransactions. test= \d+ remote Foreign table laurenz.remote Column | Type | Modifiers |

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-06 Thread Shigeru Hanada
(2012/03/06 6:19), Tom Lane wrote: I've committed the PlanForeignScan API change, with that change and some other minor editorialization. The pgsql_fdw patch now needs an update, so I set it back to Waiting On Author state. Thanks. I've revised pgsql_fdw to catch up to this change, but I'll

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-06 Thread Albe Laurenz
Shigeru Hanada wrote: Connection should be closed only when the trigger is a top level transaction and it's aborting, but isTopLevel flag was not checked. I fixed the bug and added regression tests for such cases. I wondered about that - is it really necessary to close the

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-06 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: On tor, 2012-03-01 at 20:56 +0900, Shigeru Hanada wrote: How about moving postgresql_fdw_validator into dblink, That's probably a good move. If this were C++, we might try to subclass this whole thing a bit, to avoid code duplication, but I don't see

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-06 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: Attached patches also contains changes to catch up to the redesign of PlanForeignScan. I started to look at this patch, which soon led me back to the prerequisite patch fdw_helper_v5.patch (that is the last version you posted of that one, right?).

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-05 Thread Shigeru Hanada
(2012/02/21 20:25), Etsuro Fujita wrote: Please find attached an updated version of the patch. This v2 patch can be applied on HEAD cleanly. Compile completed with only one expected warning of scan.c, and all regression tests for both core and contrib modules passed. This patch allows FDWs to

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-05 Thread Etsuro Fujita
(2012/03/05 18:21), Shigeru Hanada wrote: (2012/02/21 20:25), Etsuro Fujita wrote: Please find attached an updated version of the patch. This v2 patch can be applied on HEAD cleanly. Compile completed with only one expected warning of scan.c, and all regression tests for both core and

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-05 Thread Etsuro Fujita
(2012/03/05 21:00), Etsuro Fujita wrote: (2012/03/05 18:21), Shigeru Hanada wrote: (2012/02/21 20:25), Etsuro Fujita wrote: Please find attached an updated version of the patch. This v2 patch can be applied on HEAD cleanly. Compile completed with only one expected warning of scan.c, and all

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-05 Thread Shigeru Hanada
(2012/03/05 21:05), Etsuro Fujita wrote: (2012/03/05 21:00), Etsuro Fujita wrote: create_index_path builds multiple index paths for a plain relation. How about renaming the function to create_foreign_paths? I meant create_foreignscan_paths. I'm sorry about that. Perhaps you are confusing

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-05 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: So, just renaming create_foreignscan_path to plural form seems missing the point. I agree that that wouldn't be an improvement. What bothers me about the patch's version of this function is that it just creates a content-free Path node and leaves

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-05 Thread Tom Lane
I wrote: I'm inclined to think that if we provide this function in core at all, it should take a parameter list long enough to let it fill in the Path completely. That would imply that any future changes in Path structs would result in a change in the parameter list, which would break

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-01 Thread Shigeru Hanada
(2012/03/01 0:33), Tom Lane wrote: I don't think that creating such a dependency is acceptable. Even if we didn't mind the dependency, you said yourself that contrib/postgresql_fdw's validator will accept stuff that's not appropriate for dblink. Agreed. I think that these two contrib modules

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-01 Thread Peter Eisentraut
On tor, 2012-03-01 at 20:56 +0900, Shigeru Hanada wrote: How about moving postgresql_fdw_validator into dblink, That's probably a good move. If this were C++, we might try to subclass this whole thing a bit, to avoid code duplication, but I don't see an easy way to do that here. with

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-03-01 Thread Shigeru Hanada
2012/3/2 Peter Eisentraut pete...@gmx.net: with renaming to dblink_fdw_validator? Well, it's not the validator of the dblink_fdw, so maybe something like basic_postgresql_fdw_validator. -1 for same reason. It's not the validator of basic_postgresql_fdw. Using fdw in the name of validator

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-29 Thread Shigeru Hanada
(2012/02/29 4:07), Peter Eisentraut wrote: Let's at least be clear about the reasons here. The fact that postgresql_fdw_validator exists means (a) there is a possible naming conflict that has not been discussed yet, and/or (b) the name is already settled and we need to think of a way to make

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-29 Thread Pavel Stehule
2012/2/29 Shigeru Hanada shigeru.han...@gmail.com: (2012/02/29 4:07), Peter Eisentraut wrote: Let's at least be clear about the reasons here.  The fact that postgresql_fdw_validator exists means (a) there is a possible naming conflict that has not been discussed yet, and/or (b) the name is

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-29 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: How about removing postgresql_fdw_validator from backend binary, and changing dblink to use contrib/postgresql_fdw's validator? It breaks some backward compatibility and requires contrib/postgresql_fdw to be installed before using contrib/dblink

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-28 Thread Robert Haas
On Fri, Feb 24, 2012 at 5:31 PM, Peter Eisentraut pete...@gmx.net wrote: Could we name this postgresql_fdw instead?  We already have several ${productname}_fdw out there, and I don't want to get in the business of having to guess variant spellings. If you don't like variant spellings, having

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-28 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Feb 24, 2012 at 5:31 PM, Peter Eisentraut pete...@gmx.net wrote: Could we name this postgresql_fdw instead?  We already have several ${productname}_fdw out there, and I don't want to get in the business of having to guess variant spellings.

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-28 Thread Robert Haas
On Tue, Feb 28, 2012 at 11:02 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Feb 24, 2012 at 5:31 PM, Peter Eisentraut pete...@gmx.net wrote: Could we name this postgresql_fdw instead?  We already have several ${productname}_fdw out there, and I don't

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-28 Thread Peter Eisentraut
On tis, 2012-02-28 at 11:20 -0500, Robert Haas wrote: [ snicker ] But still, Peter has a point: pgsql is not a name for the product, it's at best an abbreviation. We aren't calling the other thing orcl_fdw or ora_fdw. I think either postgres_fdw or postgresql_fdw would be fine. I

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-28 Thread David E. Wheeler
On Feb 28, 2012, at 8:20 AM, Robert Haas wrote: I liked the shorter name, myself, but I'm not going to make a big deal about it. pg_ is used quite a bit. what about pg_fdw? David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-27 Thread Shigeru Hanada
(2012/02/25 7:31), Peter Eisentraut wrote: Could we name this postgresql_fdw instead? We already have several ${productname}_fdw out there, and I don't want to get in the business of having to guess variant spellings. I worry name conflict with existing postgresql_fdw_validator, which is

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-24 Thread Peter Eisentraut
Could we name this postgresql_fdw instead? We already have several ${productname}_fdw out there, and I don't want to get in the business of having to guess variant spellings. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-21 Thread Etsuro Fujita
(2012/02/15 20:50), Etsuro Fujita wrote: (2012/02/14 23:50), Tom Lane wrote: (2012/02/14 17:40), Etsuro Fujita wrote: As discussed at that thread, it would have to change the PlanForeignScan API to let the FDW generate multiple paths and dump them all to add_path instead of returning a

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-20 Thread Albe Laurenz
I wrote: Shigeru Hanada wrote: - Since a rescan is done by rewinding the cursor, is it necessary to have any other remote isolation level than READ COMMITED? There is only one query issued per transaction. If multiple foreign tables on a foreign server is used in a local query,

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-20 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: If your query involves foreign scans on two foreign tables on the same foreign server, these should always see the same snapshot, because that's how it works with two scans in one query on local tables. That makes sense. So I think it should

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-20 Thread Albe Laurenz
Kevin Grittner wrote: If your query involves foreign scans on two foreign tables on the same foreign server, these should always see the same snapshot, because that's how it works with two scans in one query on local tables. That makes sense. So I think it should be REPEATABLE READ

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-20 Thread Kevin Grittner
Albe Laurenz laurenz.a...@wien.gv.at wrote: I read the example carefully, and it seems to me that it is necessary for the read-only transaction (T3) to be SERIALIZABLE so that T1 is aborted and the state that T3 saw remains valid. Correct. If I understand right, I agree with your

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-20 Thread Shigeru Hanada
2012/02/21 0:58 Kevin Grittner kevin.gritt...@wicourts.gov: Albe Laurenz laurenz.a...@wien.gv.at wrote: I read the example carefully, and it seems to me that it is necessary for the read-only transaction (T3)v to be SERIALIZABLE so that T1 is aborted and the state that T3 saw remains

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-18 Thread Kohei KaiGai
2012年2月17日6:08 Shigeru Hanada shigeru.han...@gmail.com: (2012/02/17 2:02), Kohei KaiGai wrote: I found a strange behavior with v10. Is it available to reproduce? snip I tried to raise an error on remote side. postgres=# select * FROM ftbl WHERE 100 / (a - 3) 0; The connection to the

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-17 Thread Albe Laurenz
Shigeru Hanada wrote: - Since a rescan is done by rewinding the cursor, is it necessary to have any other remote isolation level than READ COMMITED? There is only one query issued per transaction. If multiple foreign tables on a foreign server is used in a local query, multiple

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-16 Thread Albe Laurenz
Shigeru Hanada wrote: Thanks for the review. Attached patches are revised version, though only fdw_helper_v5.patch is unchanged. Two questions: - Is it on purpose that you can specify all SSL client options except sslcompression? - Since a rescan is done by rewinding the cursor, is it

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-16 Thread Kohei KaiGai
I found a strange behavior with v10. Is it available to reproduce? In case of ftbl is declared as follows: postgres=# select * FROM ftbl; a | b ---+- 1 | aaa 2 | bbb 3 | ccc 4 | ddd 5 | eee (5 rows) I tried to raise an error on remote side. postgres=# select *

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-16 Thread Kohei KaiGai
2012年2月16日13:41 Shigeru Hanada shigeru.han...@gmail.com: Kaigai-san, Thanks for the review. Attached patches are revised version, though only fdw_helper_v5.patch is unchanged. (2012/02/16 0:09), Kohei KaiGai wrote: [memory context of tuple store] It calls tuplestore_begin_heap() under the

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-16 Thread Shigeru Hanada
(2012/02/17 0:15), Albe Laurenz wrote: Shigeru Hanada wrote: Thanks for the review. Attached patches are revised version, though only fdw_helper_v5.patch is unchanged. Two questions: - Is it on purpose that you can specify all SSL client options except sslcompression? No, just an

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-15 Thread Etsuro Fujita
(2012/02/14 23:50), Tom Lane wrote: Shigeru Hanadashigeru.han...@gmail.com writes: (2012/02/14 17:40), Etsuro Fujita wrote: As discussed at that thread, it would have to change the PlanForeignScan API to let the FDW generate multiple paths and dump them all to add_path instead of returning

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-15 Thread Kohei KaiGai
Harada-san, I checked the v9 patch, however, it still has some uncertain implementation. [memory context of tuple store] It calls tuplestore_begin_heap() under the memory context of festate-scan_cxt at pgsqlBeginForeignScan. On the other hand, tuplestore_gettupleslot() is called under the memory

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-14 Thread Etsuro Fujita
(2012/02/14 15:15), Shigeru Hanada wrote: (2012/02/13 20:50), Etsuro Fujita wrote: The patches have been applied, but role-related regression tests failed in my environment. I fixed it in a similar fashion of /src/test/regress/sql/foreign_data.sql. Please find attached a updated patch for

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-14 Thread Shigeru Hanada
(2012/02/14 17:40), Etsuro Fujita wrote: OK. But my question was about the PlanForeignScan API. Sorry for misunderstanding. :( As discussed at that thread, it would have to change the PlanForeignScan API to let the FDW generate

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-14 Thread Etsuro Fujita
(2012/02/14 19:42), Shigeru Hanada wrote: (2012/02/14 17:40), Etsuro Fujita wrote: As discussed at that thread, it would have to change the PlanForeignScan API to let the FDW generate multiple paths and dump them all to add_path instead

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-14 Thread Tom Lane
Shigeru Hanada shigeru.han...@gmail.com writes: (2012/02/14 17:40), Etsuro Fujita wrote: As discussed at that thread, it would have to change the PlanForeignScan API to let the FDW generate multiple paths and dump them all to add_path instead of returning a FdwPlan struct. Multiple valuable

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-13 Thread Etsuro Fujita
(2012/02/10 20:39), Shigeru Hanada wrote: (2012/02/08 20:51), Shigeru Hanada wrote: Attached revised patches. Changes from last version are below. snip I've found and fixed a bug which generates wrong remote query when any column of a foreign table has been dropped. Also regression test

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-13 Thread Shigeru Hanada
(2012/02/13 20:50), Etsuro Fujita wrote: The patches have been applied, but role-related regression tests failed in my environment. I fixed it in a similar fashion of /src/test/regress/sql/foreign_data.sql. Please find attached a updated patch for the regression tests. Good catch, thanks.

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-08 Thread Shigeru Hanada
(2012/02/02 18:24), Marko Kreen wrote: I think you want this instead: https://commitfest.postgresql.org/action/patch_view?id=769 With modified version of pgsql_fdw which uses row processor to retrieve result tuples, I found significant performance gain on simple read-only pgbench, though

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-06 Thread Shigeru Hanada
Thanks for the comments! (2012/02/06 5:08), Kohei KaiGai wrote: Yes. In my opinion, one significant benefit of pgsql_fdw is to execute qualifiers on the distributed nodes; that enables to utilize multiple CPU resources efficiently. Duplicate checks are reliable way to keep invisible tuples

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-05 Thread Kohei KaiGai
2012年2月1日12:15 Shigeru Hanada shigeru.han...@gmail.com: (2012/01/30 4:39), Kohei KaiGai wrote: I checked the fdw_helper_funcs_v3.patch, pgsql_fdw_v5.patch and pgsql_fdw_pushdown_v1.patch. My comments are below. Thanks for the review! [BUG] Even though pgsql_fdw tries to push-down

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-05 Thread Shigeru Hanada
2012/2/2 Marko Kreen mark...@gmail.com: I think you want this instead: Â https://commitfest.postgresql.org/action/patch_view?id=769 Somehow I've missed this cool feature. Thanks for the suggestion! -- Shigeru Hanada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-02 Thread Marko Kreen
On Tue, Jan 31, 2012 at 8:56 PM, Robert Haas robertmh...@gmail.com wrote: 2012/1/29 Kohei KaiGai kai...@kaigai.gr.jp:     Remote SQL: DECLARE pgsql_fdw_cursor_0 SCROLL CURSOR FOR SELECT a, b FROM public.t1 WHERE (a 2)  (3 rows) Shouldn't we be using protocol-level cursors rather than

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-02-01 Thread Shigeru Hanada
(2012/02/01 3:56), Robert Haas wrote: 2012/1/29 Kohei KaiGaikai...@kaigai.gr.jp: Remote SQL: DECLARE pgsql_fdw_cursor_0 SCROLL CURSOR FOR SELECT a, b FROM public.t1 WHERE (a 2) (3 rows) Shouldn't we be using protocol-level cursors rather than SQL-level cursors? Yes, we should, if

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-01-31 Thread Robert Haas
2012/1/29 Kohei KaiGai kai...@kaigai.gr.jp:     Remote SQL: DECLARE pgsql_fdw_cursor_0 SCROLL CURSOR FOR SELECT a, b FROM public.t1 WHERE (a 2)  (3 rows) Shouldn't we be using protocol-level cursors rather than SQL-level cursors? [Design comment] When BEGIN should be issued on the

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-01-29 Thread Kohei KaiGai
Hi Harada-san, I checked the fdw_helper_funcs_v3.patch, pgsql_fdw_v5.patch and pgsql_fdw_pushdown_v1.patch. My comments are below. [BUG] Even though pgsql_fdw tries to push-down qualifiers being executable on the remove side at the deparseSql(), it does not remove qualifiers being pushed down

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-19 Thread Greg Smith
On 12/14/2011 09:02 AM, Shigeru Hanada wrote: Here I'd like to propose three incremental patches: 1) fdw_helper_funcs_v3.patch...: This is not specific to pgsql_fdw, but probably useful for every FDWs which use FDW options... 2) pgsql_fdw_v5.patch: This patch provides simple pgsql_fdw

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-13 Thread Albe Laurenz
Tom Lane wrote: Shigeru Hanada shigeru.han...@gmail.com writes: (2011/12/12 22:59), Robert Haas wrote: ... I feel like we might need a system here that allows for more explicit user control about what to push down vs. not, rather than assuming we'll be able to figure it out behind the scenes.

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-13 Thread Heikki Linnakangas
On 13.12.2011 11:57, Albe Laurenz wrote: Tom Lane wrote: Shigeru Hanadashigeru.han...@gmail.com writes: (2011/12/12 22:59), Robert Haas wrote: ... I feel like we might need a system here that allows for more explicit user control about what to push down vs. not, rather than assuming we'll

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-13 Thread Shigeru Hanada
(2011/12/13 20:04), Heikki Linnakangas wrote: The SQL/MED spec handles this with the concept of routine mappings. There is syntax for defining which remote routines, meaning functions, correspond local functions: CREATE ROUTINE MAPPING routine mapping name FOR specific routine designator

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-13 Thread Shigeru Hanada
(2011/12/13 18:57), Albe Laurenz wrote: Maybe I'm missing something, but if pushdown worked as follows: - Push down only system functions and operators on system types. - Only push down what is guaranteed to work. Oh, I didn't care whether system data types. Indeed user defined types would

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-12 Thread Robert Haas
On Wed, Dec 7, 2011 at 2:34 AM, Shigeru Hanada shigeru.han...@gmail.com wrote: Sorry for delayed response. 2011/11/29 Albe Laurenz laurenz.a...@wien.gv.at: I think that this is not always safe even from PostgreSQL to PostgreSQL. If two databases have different collation, on strings will

Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2011-12-12 Thread Shigeru Hanada
(2011/12/12 22:59), Robert Haas wrote: It does seem like this might not be enough information for the FDW to make good decisions about pushdown. Even supposing the server on the other hand is also PostgreSQL, the collation names might not match (if, say, one is running Windows, and the other,

  1   2   >