Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-26 Thread Kouhei Kaigai
Hello, I checked the latest updatable security barrier view patch. Even though I couldn't find a major design problem in this revision, here are two minor comments below. I think, it needs to be reviewed by committer to stick direction to implement this feature. Of course, even I know Tom argued

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-01-27 Thread Kouhei Kaigai
Hackers, Is somebody available to volunteer to review the custom-scan patch? Even though Hanada-san acknowledged before, it seems to me this patch has potentially arguable implementations. Even if you have enough time to review whole of the code, it helps me if you can comment on the following

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-01-27 Thread Kouhei Kaigai
Hi Stephen, Thanks for your comments. * Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: Is somebody available to volunteer to review the custom-scan patch? I looked through it a bit and my first take away from it was that the patches to actually use the new hooks were also making more changes

Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-27 Thread Kouhei Kaigai
AFAICS the only area of objection is the handling of inherited relations, which occurs within the planner in the current patch. I can see that would be a cause for concern since the planner is pluggable and it would then be possible to bypass security checks. Obviously installing a new

Re: [HACKERS] inherit support for foreign tables

2014-01-27 Thread Kouhei Kaigai
I wonder what shall be the cases when foreign table is on a server which does not support *all* SQL features. Does a FDW need to have the possible inherit options mentioned in its documentation for this patch? The answer is no, in my understanding. The altering operation simply

Re: [HACKERS] WIP patch (v2) for updatable security barrier views

2014-01-28 Thread Kouhei Kaigai
Kouhei Kaigai kai...@ak.jp.nec.com writes: Let me ask an elemental question. What is the reason why inheritance expansion logic should be located on the planner stage, not on the tail of rewriter? I think it's mostly historical. You would however have to think of a way to preserve

Re: [HACKERS] Triggers on foreign tables

2014-01-29 Thread Kouhei Kaigai
Hello, I initially tried to keep track of them by allocating read pointers on the tuple store, but it turned out to be so expensive that I had to find another way (24bytes per stored tuple, which are not reclaimable until the end of the transaction). What do you think about this approach ?

Re: contrib/cache_scan (Re: [HACKERS] What's needed for cache-only table scan?)

2014-02-12 Thread Kouhei Kaigai
8. I am not able to find a protection mechanism in insert/delete and etc of a tuple in Ttree. As this is a shared memory it can cause problems. For design simplification, I put a giant lock per columnar-cache. So, routines in cscan.c acquires exclusive

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-24 Thread Kouhei Kaigai
/* Hook for plugins to add custom join path, in addition to default ones */ typedef void (*add_join_path_hook_type)(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel,

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-25 Thread Kouhei Kaigai
think it is a reasonable solution, however, I'm not 100% certain whether people have more graceful idea to implement it. If you have comments around above two topic, or others, please give your ideas. Thanks, 2014-01-28 9:14 GMT+09:00 Kouhei Kaigai kai

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-25 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: Instead of custom node, it might be better idea to improve FDW infrastructure to push join. For the starters, is it possible for the custom scan node hooks to create a ForeignScan node? In general, I think, it might be better

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-25 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: Yes, the part-1 patch provides a set of interface portion to interact between the backend code and extension code. Rest of part-2 and part-3 portions are contrib modules that implements its feature on top of custom-scan API. Just to come

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-26 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: This regular one means usual tables. Even though custom implementation may reference self-managed in-memory cache instead of raw heap, the table pointed in user's query shall be a usual table. In the past, Hanada-san had proposed

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-26 Thread Kouhei Kaigai
If you're looking to just use GPU acceleration for improving individual queries, I would think that Robert's work around backend workers would be a more appropriate way to go, with the ability to move a working set of data from shared buffers and on-disk representation of a relation

Re: contrib/cache_scan (Re: [HACKERS] What's needed for cache-only table scan?)

2014-02-26 Thread Kouhei Kaigai
Thanks for the information, I will apply other patches also and start testing. When try to run the pgbench test, by default the cache-scan plan is not chosen because of more cost. So I increased the cpu_index_tuple_cost to a maximum value or by turning off index_scan, so that the

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-26 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: IIUC, his approach was integration of join-pushdown within FDW APIs, however, it does not mean the idea of remote-join is rejected. For my part, trying to consider doing remote joins *without* going through FDWs is just nonsensical. What

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-02-26 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I (plan to) use custom-scan of course. Once a relation is referenced and optimizer decided GPU acceleration is cheaper, associated custom- scan node read the data from underlying relation (or in-memory cache if exists) then move to the shared

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-03-03 Thread Kouhei Kaigai
As I mentioned up-thread, I'd really like to see FDW join push-down, FDW aggregate push-down, parallel query execution, and parallel remote-FDW execution and I don't see this CustomScan approach as the right answer to any of those. In accordance with the above, what I'd like to see

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-03-05 Thread Kouhei Kaigai
Tom, thanks for your detailed comments. I apologize for not having paid much attention to this thread so far. It kept getting stuck on my to look at later queue. Anyway, I've taken a preliminary look at the v7 patch now. While the patch seems roughly along the lines of what we talked about

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-03-05 Thread Kouhei Kaigai
Kouhei Kaigai kai...@ak.jp.nec.com writes: * Please drop the whole register_custom_provider/get_custom_provider API. One thing I was worrying about is how copyObject() and nodeToString() support set of function pointer tables around custom-scan node, however, you suggested to change

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-03-06 Thread Kouhei Kaigai
I was thinking more like typedef struct CustomPathFuncs { const char *name; /* used for debugging purposes only */ NodeCopy_function node_copy; NodeTextOut_function node_textout; ... etc etc etc ... } CustomPathFuncs; typedef struct CustomPath {

Re: contrib/cache_scan (Re: [HACKERS] What's needed for cache-only table scan?)

2014-03-12 Thread Kouhei Kaigai
Thanks for your efforts! Head patched Diff Select - 500K772ms2659ms-200% Insert - 400K 3429ms 1948ms 43% (I am not sure how it improved in this case) delete - 200K

[HACKERS] pgstat wait timeout (RE: contrib/cache_scan)

2014-03-12 Thread Kouhei Kaigai
KaiGai Kohei kai...@ak.jp.nec.com -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai Sent: Wednesday, March 12, 2014 3:26 PM To: Haribabu Kommi; Kohei KaiGai Cc: Tom Lane; PgHacker; Robert Haas Subject: Re

Re: contrib/cache_scan (Re: [HACKERS] What's needed for cache-only table scan?)

2014-03-17 Thread Kouhei Kaigai
for cache-only table scan?) On Mon, Mar 17, 2014 at 11:45 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Hello, The attached patches are revised ones according to the latest custom-plan interface patch (v11). The cache-scan module was re-implemented on the newer interface, and also I

Re: [HACKERS] Triggers on foreign tables

2014-03-17 Thread Kouhei Kaigai
I hacked on this for awhile, but there remain two matters on which I'm uncertain about the right way forward. (1) To acquire the old tuple for UPDATE/DELETE operations, the patch closely parallels our handling for INSTEAD OF triggers on views. It adds a wholerow resjunk attribute, from

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-03-19 Thread Kouhei Kaigai
version. Thanks, -- NEC OSS Promotion Center / PG-Strom Project KaiGai Kohei kai...@ak.jp.nec.com -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai Sent: Monday, March 17, 2014 9:29 AM To: Kaigai Kouhei(海外

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-14 Thread Kouhei Kaigai
On 24 March 2014 10:25, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Brief summary of the current approach that has been revised from my original submission through the discussion on pgsql-hackers: The plannode was renamed to CustomPlan, instead of CustomScan, because it dropped all

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-14 Thread Kouhei Kaigai
Simon Riggs si...@2ndquadrant.com writes: [ assorted comments about custom-scan patch, but particularly ] * The prune hook makes me feel very uneasy. It seems weirdly specific implementation detail, made stranger by the otherwise lack of data maintenance API calls. Calling that for

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-15 Thread Kouhei Kaigai
Andres Freund and...@2ndquadrant.com writes: What I think this discussion shows that this patch isn't ready for 9.4. The first iteration of the patch came in 2013-11-06. Imo that's pretty damn late for a relatively complex patch. And obviously we don't have agreement on the course

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-15 Thread Kouhei Kaigai
On Tue, Apr 15, 2014 at 10:44 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Apr 14, 2014 at 4:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah. After a fast review of the custom-scan and cache-scan patches, it seems to me that my original fears

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Kouhei Kaigai
On 7 May 2014 02:05, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Prior to the development cycle towards v9.5, I'd like to reopen the discussion of custom-plan interface. Even though we had lots of discussion during the last three commit-fests, several issues are still under discussion. So

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Kouhei Kaigai
] Custom Plan API On 7 May 2014 08:17, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Let me clarify. This mechanism allows to add alternative scan/join paths including built-in ones, not only custom enhanced plan/exec node, isn't it? Probably, it is a variation of above proposition if we

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Kouhei Kaigai
Let me list up the things to be clarified / developed randomly. * Join replacement by FDW; We still don't have consensus about join replacement by FDW. Probably, it will be designed to remote-join implementation primarily, however, things to do is similar. We may need to revisit

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-07 Thread Kouhei Kaigai
* ForeignScan node that is not associated with a particular foreign-table. Once we try to apply ForeignScan node instead of Sort or Aggregate, existing FDW implementation needs to be improved. These nodes scan on a materialized relation (generated on the fly), however,

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
On Wed, May 7, 2014 at 4:01 AM, Simon Riggs si...@2ndquadrant.com wrote: Agreed. My proposal is that if the planner allows the lookaside to an FDW then we pass the query for full execution on the FDW. That means that the scan, aggregate and join could take place via the FDW. i.e. Custom

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
I also think that there are really two separate problems here: getting the executor to call a custom scan node when it shows up in the plan tree; and figuring out how to get it into the plan tree in the first place. I'm not sure we've properly separated those problems, and I'm not sure

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
On Thu, May 8, 2014 at 4:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: I thought that the executor side of his patch wasn't in bad shape. The real problems were in the planner, and indeed largely in the backend part of the planner where there's a lot of hard-wired logic for fixing up

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
On Thu, May 8, 2014 at 6:34 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Umm... I'm now missing the direction towards my goal. What approach is the best way to glue PostgreSQL and PGStrom? I haven't really paid any attention to PGStrom. Perhaps it's just that I missed it, but I would

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
So it seems reasonable to have a way to define/declare what is possible and what is not. But my take is that adding a new column to pg_operator for every CustomJoin node is probably out of the question, hence my suggestion to list the operators we know it can work with. It does seem

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I initially intended to allow extensions to add their custom-path based on their arbitrary decision, because the core backend cannot have expectation towards the behavior of custom-plan. However, of course, the custom-path that replaces built

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-08 Thread Kouhei Kaigai
* Kouhei Kaigai (kai...@ak.jp.nec.com) wrote: I didn't ask this before but it's been on my mind for a while- how will this work for custom data types, ala the 'geometry' type from PostGIS? There's user-provided code that we have to execute to check equality for those, but they're

Re: [HACKERS] [v9.5] Custom Plan API

2014-05-11 Thread Kouhei Kaigai
On 8 May 2014 22:55, Tom Lane t...@sss.pgh.pa.us wrote: We're past the prototyping stage and into productionising what we know works, AFAIK. If that point is not clear, then we need to discuss that first. OK, I'll bite: what here do we know works? Not a damn thing AFAICS; it's all

Re: [HACKERS] [v9.5] Custom Plan API

2014-07-03 Thread Kouhei Kaigai
10:09 GMT+09:00 Kouhei Kaigai kai...@ak.jp.nec.com: On 8 May 2014 22:55, Tom Lane t...@sss.pgh.pa.us wrote: We're past the prototyping stage and into productionising what we know works, AFAIK. If that point is not clear, then we need to discuss that first. OK, I'll bite: what

Re: [HACKERS] RLS Design

2014-07-04 Thread Kouhei Kaigai
Sorry for my late responding, now I'm catching up the discussion. * Robert Haas (robertmh...@gmail.com) wrote: On Tue, Jul 1, 2014 at 3:20 PM, Dean Rasheed dean.a.rash...@gmail.com wrote: If RLS quals are instead regarded as constraints on access, and multiple policies apply, then it

Re: [HACKERS] RLS Design

2014-07-04 Thread Kouhei Kaigai
Kaigai, On Thursday, July 3, 2014, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Sorry for my late responding, now I'm catching up the discussion. * Robert Haas (robertmh...@gmail.com javascript:; ) wrote: On Tue, Jul 1, 2014 at 3:20 PM, Dean Rasheed dean.a.rash

Re: [HACKERS] [v9.5] Custom Plan API

2014-07-17 Thread Kouhei Kaigai
Alvaro Herrera alvhe...@2ndquadrant.com writes: I haven't followed this at all, but I just skimmed over it and noticed the CustomPlanMarkPos thingy; apologies if this has been discussed before. It seems a bit odd to me; why isn't it sufficient to have a boolean flag in regular CustomPlan

Re: [HACKERS] [v9.5] Custom Plan API

2014-07-17 Thread Kouhei Kaigai
GMT+09:00 Kouhei Kaigai kai...@ak.jp.nec.com: Sorry, expected result of sanity-check test was not updated on renaming to pg_custom_plan_provider. The attached patch fixed up this point. I confirmed that all regression tests passed, so I marked the patch as Ready for committer. I

[HACKERS] what data type should be returned by sum(float4)

2014-09-07 Thread Kouhei Kaigai
Hello, http://www.postgresql.org/docs/devel/static/functions-aggregate.html The documentation says that return type of sum(expression) is... bigint for smallint or int arguments, numeric for bigint arguments, double precision for floating-point arguments, otherwise the same as the argument

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-11 Thread Kouhei Kaigai
On Thu, Sep 11, 2014 at 11:24 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Don't the changes to src/backend/optimizer/plan/createplan.c belong in patch #2? The borderline between #1 and #2 is little bit bogus. So, I moved most of portion into #1, however, invocation of InitCustomScan

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-15 Thread Kouhei Kaigai
On Thu, Sep 11, 2014 at 8:40 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Thu, Sep 11, 2014 at 11:24 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Don't the changes to src/backend/optimizer/plan/createplan.c belong in patch #2? The borderline between #1 and #2 is little bit

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-16 Thread Kouhei Kaigai
On Mon, Sep 15, 2014 at 8:38 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: The only reason why I put separate hooks here is, create_custom_scan() needs to know exact size of the CustomScan node (including private fields), however, it is helpful for extensions to kick its callback

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-17 Thread Kouhei Kaigai
Why does it need to know that? I don't see that it's doing anything that requires knowing the size of that node, and if it is, I think it shouldn't be. That should get delegated to the callback provided by the custom plan provider. Sorry, my explanation might be confusable. The

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-29 Thread Kouhei Kaigai
On Wed, Sep 17, 2014 at 7:40 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: At this moment, I revised the above portion of the patches. create_custom_plan() was modified to call PlanCustomPath callback next to the initialization of tlist and clauses. It's probably same as what you

Re: [HACKERS] [v9.5] Custom Plan API

2014-09-30 Thread Kouhei Kaigai
On Mon, Sep 29, 2014 at 9:04 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: Do we need to match the prototype of wrapper function with callback? Yes. OK, I fixed up the patch part-2, to fit declaration of GetSpecialCustomVar() with corresponding callback. Also, a noise in the part-3 patch,

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-03 Thread Kouhei Kaigai
On 10/03/2014 07:08 AM, Kouhei Kaigai wrote: Hello, I recently got a trouble on development of my extension that utilizes the shared buffer when it released each buffer page. This extension transfers contents of the shared buffers to GPU device using DMA feature, then kicks a device

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-03 Thread Kouhei Kaigai
On 2014-10-03 10:35:42 +0300, Heikki Linnakangas wrote: On 10/03/2014 07:08 AM, Kouhei Kaigai wrote: Hello, I recently got a trouble on development of my extension that utilizes the shared buffer when it released each buffer page. This extension transfers contents

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-04 Thread Kouhei Kaigai
On 03/10/2014 05:53, Kouhei Kaigai wrote: Yep, that's my pain. Even though usual query does not take many buffers pinned, my use case needs to fetch megabytes scale data at once because of performance reason; page-by-page synchronous scan makes GPU being idle. Doesn't your GPU have

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-04 Thread Kouhei Kaigai
Heikki Linnakangas hlinnakan...@vmware.com writes: On 10/03/2014 07:08 AM, Kouhei Kaigai wrote: What is the best way to solve the problem? How about creating a separate ResourceOwner for these buffer pins, and doing a wholesale ResourceOwnerRelease() on it when you're done? That's

Re: [HACKERS] How to make ResourceOwnerForgetBuffer() O(1), instead of O(N^2) scale

2014-10-15 Thread Kouhei Kaigai
On 10/03/2014 07:08 AM, Kouhei Kaigai wrote: Hello, I recently got a trouble on development of my extension that utilizes the shared buffer when it released each buffer page. This extension transfers contents of the shared buffers to GPU device using DMA feature, then kicks

Re: [HACKERS] [v9.5] Custom Plan API

2014-10-27 Thread Kouhei Kaigai
: Re: [HACKERS] [v9.5] Custom Plan API On 30 September 2014 07:27, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Mon, Sep 29, 2014 at 9:04 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: Do we need to match the prototype of wrapper function with callback? Yes

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-07 Thread Kouhei Kaigai
On Mon, Oct 27, 2014 at 2:35 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: FYI, patch v12 part 2 no longer applies cleanly. Thanks. I rebased the patch set according to the latest master branch. The attached v13 can be applied to the master. I've committed parts 1 and 2

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-10 Thread Kouhei Kaigai
On Sat, Nov 8, 2014 at 4:16 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Oct 27, 2014 at 2:35 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: FYI, patch v12 part 2 no longer applies cleanly. Thanks. I rebased the patch set according to the latest master branch. The attached

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-10 Thread Kouhei Kaigai
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Haribabu Kommi Sent: Tuesday, November 11, 2014 1:13 PM To: Amit Kapila Cc: Andres Freund; Robert Haas; Simon Riggs; Tom Lane; pgsql-hackers@postgresql.org

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-11 Thread Kouhei Kaigai
Given where we are with the infrastructure, there would be a number of unhandled problems, such as deadlock detection (needs group locking or similar), assessment of quals as to parallel-safety (needs proisparallel or similar), general waterproofing to make sure that pushing down a

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Kouhei Kaigai
On 12 November 2014 07:54, David Rowley dgrowle...@gmail.com wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit the mainline of BI applications and many

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-13 Thread Kouhei Kaigai
On Fri, Nov 14, 2014 at 2:12 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On 12 November 2014 07:54, David Rowley dgrowle...@gmail.com Take int4_avg_accum() for example it does: transdata-count++; transdata-sum += newval

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-14 Thread Kouhei Kaigai
On 14 November 2014 07:37, Jim Nasby jim.na...@bluetreble.com wrote: On 11/12/14, 1:54 AM, David Rowley wrote: On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs si...@2ndquadrant.com mailto:si...@2ndquadrant.com wrote: This plan type is widely used in reporting queries, so will hit

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-11-17 Thread Kouhei Kaigai
On 18 November 2014 05:19, Simon Riggs si...@2ndquadrant.com wrote: On 14 November 2014 11:02, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'd like to throw community folks a question. Did someone have a discussion to the challenge of aggregate push-down across

[HACKERS] T_CustomScan on ExplainTargetRel

2014-11-19 Thread Kouhei Kaigai
Hello, The attached obvious patch adds T_CustomScan on case-switch of ExplainTargetRel() that was oversight. It looked like working, but what it did was just printing referenced name, instead of table name. postgres=# explain select ctid, * from t0 hoge where ctid '(50,0)'::tid;

Re: [HACKERS] T_CustomScan on ExplainTargetRel

2014-11-19 Thread Kouhei Kaigai
...@postgresql.org] On Behalf Of Kouhei Kaigai Sent: Thursday, November 20, 2014 3:42 PM To: pgsql-hackers@postgresql.org Subject: [HACKERS] T_CustomScan on ExplainTargetRel Hello, The attached obvious patch adds T_CustomScan on case-switch of ExplainTargetRel() that was oversight. It looked

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-20 Thread Kouhei Kaigai
Robert Haas robertmh...@gmail.com writes: I've committed parts 1 and 2 of this, without the documentation, and with some additional cleanup. I am not sure that this feature is sufficiently non-experimental that it deserves to be documented, but if we're thinking of doing that then the

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-20 Thread Kouhei Kaigai
On Thu, Nov 20, 2014 at 7:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: I've done some preliminary cleanup on this patch, but I'm still pretty desperately unhappy about some aspects of it, in particular the way that it gets custom scan providers directly involved in setrefs.c,

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-21 Thread Kouhei Kaigai
Kouhei Kaigai kai...@ak.jp.nec.com writes: I've done some preliminary cleanup on this patch, but I'm still pretty desperately unhappy about some aspects of it, in particular the way that it gets custom scan providers directly involved in setrefs.c, finalize_primnode

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-21 Thread Kouhei Kaigai
Kouhei Kaigai kai...@ak.jp.nec.com writes: Let assume a custom-scan provider that runs on a materialized-view (or, something like a query cache in memory) instead of join. In this case, a reasonable design is to fetch a tuple from the materialized-view then put it on the ecxt_scantuple

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-24 Thread Kouhei Kaigai
Kouhei Kaigai kai...@ak.jp.nec.com writes: Let me explain the current idea of mine. CustomScan node will have a field that hold varnode mapping information that is constructed by custom-scan provider on create_customscan_plan, if they want. It is probably a list of varnode. If exists

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-25 Thread Kouhei Kaigai
On Mon, Nov 24, 2014 at 6:57 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Indeed, I don't think it is a good idea to start from this harder portion. Let's focus on just varno/varattno remapping to replace join relation by custom-scan, as an immediate target. We still need something like

Re: [HACKERS] [v9.5] Custom Plan API

2014-11-27 Thread Kouhei Kaigai
On 7 November 2014 at 22:46, Robert Haas robertmh...@gmail.com wrote: On Mon, Oct 27, 2014 at 2:35 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: FYI, patch v12 part 2 no longer applies cleanly. Thanks. I rebased the patch set according to the latest master branch. The attached v13 can

Re: [HACKERS] no test programs in contrib

2014-11-27 Thread Kouhei Kaigai
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Josh Berkus Sent: Friday, November 28, 2014 5:48 AM To: Alvaro Herrera; Pg Hackers Subject: Re: [HACKERS] no test programs in contrib On 11/24/2014 05:49 AM, Alvaro

Re: [HACKERS] [v9.5] Custom Plan API

2014-12-02 Thread Kouhei Kaigai
Eisentraut Subject: Re: [HACKERS] [v9.5] Custom Plan API On 27 November 2014 at 10:33, Kouhei Kaigai kai...@ak.jp.nec.com wrote: The reason why documentation portion was not yet committed is, sorry, it is due to quality of documentation from the standpoint of native English speaker. Now

Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2014-12-02 Thread Kouhei Kaigai
On Tue, Nov 25, 2014 at 3:44 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Today, I had a talk with Hanada-san to clarify which can be a common portion of them and how to implement it. Then, we concluded both of features can be shared most of the infrastructure. Let me put an introduction

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

2014-12-02 Thread Kouhei Kaigai
On Fri, Nov 14, 2014 at 02:51:32PM +1300, David Rowley wrote: Likely for most aggregates, like count, sum, max, min, bit_and and bit_or the merge function would be the same as the transition function, as the state type is just the same as the input type. It would only be aggregates like

Re: [HACKERS] [v9.5] Custom Plan API

2014-12-06 Thread Kouhei Kaigai
; Stephen Frost; Andres Freund; PgHacker; Jim Mlodgenski; Peter Eisentraut Subject: Re: [HACKERS] [v9.5] Custom Plan API On 27 November 2014 at 20:48, Simon Riggs si...@2ndquadrant.com wrote: On 27 November 2014 at 10:33, Kouhei Kaigai kai...@ak.jp.nec.com wrote: The reason why documentation

Re: [HACKERS] [v9.5] Custom Plan API

2014-12-09 Thread Kouhei Kaigai
, 2014 12:24 AM To: Kaigai Kouhei(海外 浩平) Cc: Robert Haas; Thom Brown; Kohei KaiGai; Tom Lane; Alvaro Herrera; Shigeru Hanada; Stephen Frost; Andres Freund; PgHacker; Jim Mlodgenski; Peter Eisentraut Subject: Re: [HACKERS] [v9.5] Custom Plan API On 7 December 2014 at 08:21, Kouhei Kaigai kai

ctidscan as an example of custom-scan (Re: [HACKERS] [v9.5] Custom Plan API)

2014-12-14 Thread Kouhei Kaigai
This attached patch adds a code example of custom-scan interface. This custom-scan provider (ctidscan) performs almost same as built-in SeqScan plan, but can produce same results with less page scan in case when qualifier of relation scan has inequality operators (, =, or =) on ctid system

Re: ctidscan as an example of custom-scan (Re: [HACKERS] [v9.5] Custom Plan API)

2014-12-15 Thread Kouhei Kaigai
On Mon, Dec 15, 2014 at 4:55 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'm not certain whether we should have this functionality in contrib from the perspective of workload that can help, but its major worth is for an example of custom-scan interface. worker_spi is now in src/test

Re: [HACKERS] Join push-down support for foreign tables

2014-12-15 Thread Kouhei Kaigai
Hanada-san, Thanks for proposing this great functionality people waited for. On Mon, Dec 15, 2014 at 3:40 AM, Shigeru Hanada shigeru.han...@gmail.com wrote: I'm working on $SUBJECT and would like to get comments about the design. Attached patch is for the design below. I'm glad you are

Re: [HACKERS] Join push-down support for foreign tables

2014-12-15 Thread Kouhei Kaigai
KaiGai Kohei kai...@ak.jp.nec.com -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai Sent: Tuesday, December 16, 2014 9:01 AM To: Robert Haas; Shigeru Hanada Cc: PostgreSQL-development Subject: Re

Re: [HACKERS] Combining Aggregates

2014-12-17 Thread Kouhei Kaigai
Simon, Its concept is good to me. I think, the new combined function should be responsible to take a state data type as argument and update state object of the aggregate function. In other words, combined function performs like transition function but can update state object according to the

Re: [HACKERS] Combining Aggregates

2014-12-17 Thread Kouhei Kaigai
Hi Atri, So are you proposing not calling transfuncs at all and just use combined functions? No. It is discretion of software component that distribute an aggregate into multiple partial-aggregates. That sounds counterintuitive to me. I am not able to see why you would want to avoid

Re: ctidscan as an example of custom-scan (Re: [HACKERS] [v9.5] Custom Plan API)

2014-12-24 Thread Kouhei Kaigai
On Mon, Dec 15, 2014 at 4:55 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I'm not certain whether we should have this functionality in contrib from the perspective of workload that can help, but its major worth is for an example of custom-scan interface. worker_spi is now in src

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-01-14 Thread Kouhei Kaigai
On Fri, Jan 9, 2015 at 10:51 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: When custom-scan node replaced a join-plan, it shall have at least two child plan-nodes. The callback handler of PlanCustomPath needs to be able to call create_plan_recurse() to transform the underlying path-nodes

Re: [HACKERS] Parallel Seq Scan

2015-01-21 Thread Kouhei Kaigai
On Wed, Jan 21, 2015 at 4:31 PM, Amit Langote amitlangot...@gmail.com wrote: On Wednesday, January 21, 2015, Amit Kapila amit.kapil...@gmail.com wrote: Does it happen only when parallel_seqscan_degree max_worker_processes? I have max_worker_processes set to the default of 8

Re: [HACKERS] Combining Aggregates

2015-02-18 Thread Kouhei Kaigai
Hi Rowley, Let me put some comments on this patch. This patch itself looks good as an infrastructure towards the big picture, however, we still don't reach the consensus how combined functions are used instead of usual translation functions. Aggregate function usually consumes one or more

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-02-13 Thread Kouhei Kaigai
4:38 PM To: Kaigai Kouhei(海外 浩平) Cc: Robert Haas; Tom Lane; pgsql-hackers@postgreSQL.org; Shigeru Hanada Subject: ##freemail## Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API) On Thu, Jan 15, 2015 at 8:02 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: On Fri

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-02-13 Thread Kouhei Kaigai
...@gmail.com wrote: On Fri, Feb 13, 2015 at 4:59 PM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Where are we on this? AFAIK, we have now a feature with no documentation and no example in-core to test those custom routine APIs, hence moved

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-02-15 Thread Kouhei Kaigai
Promotion Center / PG-Strom Project KaiGai Kohei kai...@ak.jp.nec.com -Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai Sent: Thursday, January 15, 2015 8:03 AM To: Robert Haas Cc: Tom Lane; pgsql-hackers

Re: [HACKERS] Commit fest 2015-12 enters money time

2015-02-16 Thread Kouhei Kaigai
I couldn't find operation to add new entry on the current CF. Go to the bottom of this CF: https://commitfest.postgresql.org/4/ Then click on New patch. I couldn't find the New patch link on both of IE and Chrome, even though I logged in. Did you do something special? Thanks, -- NEC OSS

Re: [HACKERS] Commit fest 2015-12 enters money time

2015-02-16 Thread Kouhei Kaigai
On Tue, Feb 17, 2015 at 8:50 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: I couldn't find operation to add new entry on the current CF. Go to the bottom of this CF: https://commitfest.postgresql.org/4/ Then click on New patch. I couldn't find the New patch link on both of IE

Re: [HACKERS] Join push-down support for foreign tables

2015-02-16 Thread Kouhei Kaigai
my dev branch into unexpected state. I'll check it soon. 2015-02-16 13:13 GMT+09:00 Kouhei Kaigai kai...@ak.jp.nec.com: Hanada-san, Your patch mixtures enhancement of custom-/foreign-scan interface and enhancement of contrib/postgres_fdw... Probably, it is a careless mis- operation

Re: Custom/Foreign-Join-APIs (Re: [HACKERS] [v9.5] Custom Plan API)

2015-01-09 Thread Kouhei Kaigai
On Tue, Jan 6, 2015 at 9:17 AM, Kouhei Kaigai kai...@ak.jp.nec.com wrote: The attached patch is newer revision of custom-/foreign-join interface. It seems that the basic purpose of this patch is to allow a foreign scan or custom scan to have scanrelid == 0, reflecting the case where we

  1   2   3   4   5   >