Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

> Hello, > > On Wed, March 1, 2017 7:21 pm, Kouhei Kaigai wrote: > >> I've looked at the patch, and as I'm not that familiar with the > >> pg-sourcecode, customs and so on, this isn't a review, more like food > >> for thought and all should be taken with a

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

> Hello all, > > as this is my first mail to pgsql-hackers, please be gentle :) > Welcome to pgsql-hackers, > I've looked at the patch, and as I'm not that familiar with the pg-sourcecode, > customs and so on, this isn't a review, more like food for thought and all > should be taken with a grain

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

he next CF". > > Its status has not been changed since the last update. (Code was revised > according to the last comment by Jeevan, but CF-Nov was time up at that > time.) > > How do I handle the patch? > > 2016-12-05 16:49 GMT+09:00 Kouhei Kaigai <kai...@ak.jp.nec.com>:

Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside ExecEndGather)

...@gmail.com>; Amit Kapila > <amit.kapil...@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org> > Subject: Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside > ExecEndGather) > > On Fri, Feb 17, 2017 at 12:46 PM, Kouhei Kaigai <kai...@a

Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside ExecEndGather)

<pgsql-hackers@postgresql.org> > Subject: Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside > ExecEndGather) > > On Mon, Feb 6, 2017 at 1:42 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > I also had thought an idea to have extra space to Instrumentati

Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside ExecEndGather)

...@gmail.com>; Robert Haas > <robertmh...@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org> > Subject: Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside > ExecEndGather) > > On Mon, Feb 6, 2017 at 1:00 AM, Claudio Freire <klaussfre...@gmail.com>

Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside ExecEndGather)

...@gmail.com>; Robert Haas > <robertmh...@gmail.com>; pgsql-hackers <pgsql-hackers@postgresql.org> > Subject: Re: ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside > ExecEndGather) > > On Mon, Oct 31, 2016 at 11:33 AM, Kouhei Kaigai <kai...@ak.jp.nec.co

Re: [HACKERS] T_Float morph to T_Integer after nodeRead

> Kouhei Kaigai <kai...@ak.jp.nec.com> writes: > > Simplified description of what I did is: > > fval = makeFloat(psprintf("%.0f", plan_nrows)); > > custom_scan->custom_private = list_make1(fval); > > So don't do that. The lexer would never produ

[HACKERS] T_Float morph to T_Integer after nodeRead

I noticed a strange behavior when T_Float value is serialized, then deserialized on the worker process for cpu parallel execution. Simplified description of what I did is: fval = makeFloat(psprintf("%.0f", plan_nrows)); custom_scan->custom_private = list_make1(fval); This string expression

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas > Sent: Thursday, January 05, 2017 5:29 AM > To: Kohei KaiGai > Cc: Kaigai Kouhei(海外 浩平) ; Jeevan Chalke >

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

Hello, Sorry for my late response. The attached patch reflects your comments. > Here are few comments on latest patch: > > > 1. > make/make check is fine, however I am getting regression failure in > postgres_fdw contrib module (attached regression.diff). > Please investigate and fix. > It was

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

> > Cc: pgsql-hackers@postgresql.org; Jeevan Chalke > <jeevan.cha...@enterprisedb.com>; Etsuro Fujita > <fujita.ets...@lab.ntt.co.jp>; Andres Freund <and...@anarazel.de> > Subject: ##freemail## Re: PassDownLimitBound for ForeignScan/CustomScan > [take-2] > > On M

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

> On Tue, Nov 8, 2016 at 6:54 AM, Jeevan Chalke > wrote: > > 1. ps_numTuples is declared as long, however offset and count members in > > LimitState struct and bound member in SortState struct is int64. However > > long on 32 bit machine may be 32 bits and thus I

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

ita; Andres Freund > Subject: Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2] > > On Mon, Oct 31, 2016 at 10:20 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > As an example, I enhanced postgres_fdw to understand the ps_numTuples > > if it is se

Re: [HACKERS] Proposal: scan key push down to heap [WIP]

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Dilip Kumar > Sent: Saturday, October 29, 2016 3:48 PM > To: Andres Freund > Cc: Tom Lane; Alvaro Herrera; pgsql-hackers > Subject: Re: [HACKERS] Proposal: scan key

ParallelFinish-hook of FDW/CSP (Re: [HACKERS] Steps inside ExecEndGather)

, 2016 11:22 AM > To: Kaigai Kouhei(海外 浩平) > Cc: Robert Haas; pgsql-hackers > Subject: ##freemail## Re: [HACKERS] Steps inside ExecEndGather > > On Mon, Oct 17, 2016 at 6:22 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Hello, > > > > I'm now trying to carr

[HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

ber 16, 2016 12:39 AM > To: Kaigai Kouhei(海外 浩平) > Cc: Jeevan Chalke; pgsql-hackers@postgresql.org; Etsuro Fujita; Andres Freund > Subject: ##freemail## Re: [HACKERS] PassDownLimitBound for > ForeignScan/CustomScan > > On Tue, Sep 13, 2016 at 9:07 PM, Kouhei Kaigai <kai..

Re: [HACKERS] Steps inside ExecEndGather

> On Mon, Oct 17, 2016 at 6:22 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Hello, > > > > I'm now trying to carry extra performance statistics on CustomScan > > (like DMA transfer rate, execution time of GPU kernels, etc...) > > from parallel work

Re: [HACKERS] Steps inside ExecEndGather

> I'm now trying to carry extra performance statistics on CustomScan > (like DMA transfer rate, execution time of GPU kernels, etc...) > from parallel workers to the leader process using the DSM segment > attached by the parallel-context. > We can require an arbitrary length of DSM using

[HACKERS] Steps inside ExecEndGather

Hello, I'm now trying to carry extra performance statistics on CustomScan (like DMA transfer rate, execution time of GPU kernels, etc...) from parallel workers to the leader process using the DSM segment attached by the parallel-context. We can require an arbitrary length of DSM using

Re: [HACKERS] palloc() too large on pg_buffercache with large shared_buffers

> On Wed, Sep 14, 2016 at 12:13 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > It looks to me pg_buffercache tries to allocate more than 1GB using > > palloc(), when shared_buffers is more than 256GB. > > > > # show shared_buffers ; > > shared_buffers

[HACKERS] palloc() too large on pg_buffercache with large shared_buffers

Hello, It looks to me pg_buffercache tries to allocate more than 1GB using palloc(), when shared_buffers is more than 256GB. # show shared_buffers ; shared_buffers 280GB (1 row) # SELECT buffers, d.datname, coalesce(c.relname, '???') FROM (SELECT count(*) buffers,

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan

> On Tue, Sep 13, 2016 at 3:48 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Sorry for my late. > > > > The attached patch fixed the wording problems on SGML part. > > I agree that we should have some way for foreign data wrappers and > custom scans a

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan

Sorry for my late. The attached patch fixed the wording problems on SGML part. Best regards, -- NEC Business Creation Division / PG-Strom Project KaiGai Kohei > -Original Message- > From: Jeevan Chalke [mailto:jeevan.cha...@enterprisedb.com] > Sent: Tuesday,

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai > Sent: Monday, September 05, 2016 12:58 PM > To: Jeevan Chalke > Cc: pgsql-hackers@postgresql.org; Etsuro Fujita >

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan

> On Mon, Aug 29, 2016 at 7:25 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > > Hello, > > The attached patch adds an optional callback to support special > optimization > if ForeignScan/CustomScan are located under the Limit node in plan-

[HACKERS] PassDownLimitBound for ForeignScan/CustomScan

Hello, The attached patch adds an optional callback to support special optimization if ForeignScan/CustomScan are located under the Limit node in plan-tree. Our sort node wisely switches the behavior when we can preliminary know exact number of rows to be produced, because all the Sort node has

[HACKERS] comment fix for CUSTOMPATH_* flags

Hello, I noticed the source code comment around CustomPath structure says "see above" for definition of CUSTOMPATH_* flags. It was originally right, but it was moved to nodes/extensible.h on the further development. So, no comments are above. The attached patch corrects the comment for the right

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> On 2016/08/02 22:02, Kouhei Kaigai wrote: > > I wrote: > >> I removed the Relations line. Here is an updated version of the patch. > >> > >> * As I said upthread, I left the upper-relation handling for another > >> patch. Currently, the patch prints

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> -Original Message- > From: Etsuro Fujita [mailto:fujita.ets...@lab.ntt.co.jp] > Sent: Tuesday, August 02, 2016 9:36 PM > To: Kaigai Kouhei(海外 浩平); pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown > plans > > On 2016/08/01 20:15,

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

pushdown > plans > > On 2016/08/02 13:32, Kouhei Kaigai wrote: > > I wrote: > >> My concern here is EXPLAIN for foreign joins. I think it's another > >> problem how we handle Foreign Scan plan nodes representing > >> post-scan/join operations in EXPLAIN, s

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> On 2016/08/01 22:25, Kouhei Kaigai wrote: > > I wrote: > >>> a broader > >>> word like "Processing" seems to work well because we allow various > >>> kinds of operations to the remote side, in addition to scans/joins, > >&

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> -Original Message- > From: Etsuro Fujita [mailto:fujita.ets...@lab.ntt.co.jp] > Sent: Monday, August 01, 2016 8:26 PM > To: Ashutosh Bapat > Cc: Kaigai Kouhei(海外 浩平); pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown > plans > > On

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> I wrote: > >> That may be so, but my point is that the target relations involved in > >> the foreign join (ie, ft1 and ft2) should be printed somewhere in the > >> EXPLAIN output by core, as in EXPLAIN for a simple foreign table scan. > > > Why? According to your rule, Hash Join should take "on

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> On 2016/07/28 10:01, Kouhei Kaigai wrote: > > What I'm saying is here: > > > EXPLAIN (COSTS false, VERBOSE) > > SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY > > t1.c3, t1.c1 OFFSET 100 LIMIT 10; > &

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> On 2016/07/27 13:51, Kouhei Kaigai wrote: > > This output is, at least, not incorrect. > > This ForeignScan actually scan a relation that consists of two joined > > tables on the remote side. So, not incorrect, but may not convenient for > > better understanding b

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

> I noticed that currently the core doesn't show any information on the > target relations involved in a foreign/custom join in EXPLAIN, by > itself. Here is an example: > > -- join two tables > EXPLAIN (COSTS false, VERBOSE) > SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER

Re: [HACKERS] make clean didn't clean up files generated from *.(y|l)

> Kouhei Kaigai <kai...@ak.jp.nec.com> writes: > > I tried to build the latest master branch just after the switch from > > REL9_5_STABLE and "make clean", however, repl_gram.c was not cleaned > > up correctly. So, my problem is that repl_gram.l was the

[HACKERS] make clean didn't clean up files generated from *.(y|l)

Hello, I got the build error below. It concerns RESERVE_WAL is not defined, however, it should not be a problem to be oversight for a long time. I tried to build the latest master branch just after the switch from REL9_5_STABLE and "make clean", however, repl_gram.c was not cleaned up correctly.

Re: [HACKERS] Does people favor to have matrix data type?

o have matrix data type? > > On 5/30/16 9:05 PM, Kouhei Kaigai wrote: > > Due to performance reason, location of each element must be deterministic > > without walking on the data structure. This approach guarantees we can > > reach individual element with 2 steps.

Re: [HACKERS] Does people favor to have matrix data type?

vor to have matrix data type? > > On 31/05/16 12:01, Kouhei Kaigai wrote: > >> On 05/29/2016 04:55 PM, Kouhei Kaigai wrote: > >>> For the closer integration, it may be valuable if PL/R and PL/CUDA can > >>> exchange > >>> the data structu

Re: [HACKERS] Does people favor to have matrix data type?

> On 05/29/2016 04:55 PM, Kouhei Kaigai wrote: > > For the closer integration, it may be valuable if PL/R and PL/CUDA can > > exchange > > the data structure with no serialization/de-serialization when PL/R code > > tries > > to call SQL functions. > &

Re: [HACKERS] Does people favor to have matrix data type?

> On 05/28/2016 03:33 PM, Kouhei Kaigai wrote: > >> -Original Message- > >> From: Joe Conway [mailto:m...@joeconway.com] > >> Sent: Sunday, May 29, 2016 1:40 AM > >> To: Kaigai Kouhei(海外 浩平); Jim Nasby; Ants Aasma; Simon Riggs > >> C

Re: [HACKERS] Does people favor to have matrix data type?

> > On 05/28/2016 07:12 AM, Kouhei Kaigai wrote: > > Sparse matrix! It is a disadvantaged area for the current array format. > > > > I have two ideas. HPC folks often split a large matrix into multiple > > grid. A grid is typically up to 1024x1024 matrix, for example

Re: [HACKERS] Does people favor to have matrix data type?

> On 5/25/16 7:46 AM, Kouhei Kaigai wrote: > > My only concern is that domain type is not allowed to define type cast. > > If we could add type cast on domain, we can define type transformation from > > other array type to matrix. > > I've actually wished for that in t

Re: [HACKERS] Does people favor to have matrix data type?

> On Wed, May 25, 2016 at 10:38 AM, Simon Riggs <si...@2ndquadrant.com> wrote: > > On 25 May 2016 at 03:52, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > >> > >> In a few days, I'm working for a data type that represents matrix in > >> mathematica

Re: [HACKERS] Does people favor to have matrix data type?

> -Original Message- > From: Simon Riggs [mailto:si...@2ndquadrant.com] > Sent: Wednesday, May 25, 2016 4:39 PM > To: Kaigai Kouhei(海外 浩平) > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Does people favor to have matrix data type? > > On 25 May 2016

[HACKERS] Does people favor to have matrix data type?

In a few days, I'm working for a data type that represents matrix in mathematical area. Does people favor to have this data type in the core, not only my extension? Like oidvector or int2vector, it is one of array type with a few restrictions: - 2 dimensional only - never contains NULL -

Re: [HACKERS] asynchronous and vectorized execution

HACKERS] asynchronous and vectorized execution > > On 10 May 2016 at 13:38, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > My concern about ExecProcNode is, it is constructed with a large switch > > ... case statement. It involves tons of comparison operation at run-time. >

Re: [HACKERS] asynchronous and vectorized execution

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas > Sent: Tuesday, May 10, 2016 2:34 AM > To: pgsql-hackers@postgresql.org > Subject: [HACKERS] asynchronous and vectorized execution > > Hi, > > I realize

[HACKERS] EPQ recheck across HashJoin, what does it actuall check?

Folks, Could you correct me if I misunderstand the execution path. We may have the following example. Query try to get row level locking on the table 't1' that is under the 'Hash' node. postgres=# EXPLAIN postgres-# SELECT * FROM t0 NATURAL JOIN t1 WHERE ax between 10 and 30 FOR UPDATE OF t1;

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

Re: [HACKERS] Reworks of CustomScan serialization/deserialization > > On Mon, Mar 28, 2016 at 9:36 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > I don't have a strong reason to keep these stuff in separate files. > > Both stuffs covers similar features and amount of code a

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Robert Haas > Sent: Friday, March 25, 2016 12:27 AM > To: Petr Jelinek > Cc: Kaigai Kouhei(海外 浩平); pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] Reworks of

Re: [HACKERS] WIP: Upper planner pathification

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai > Sent: Saturday, March 19, 2016 8:57 AM > To: Tom Lane > Cc: Robert Haas; Petr Jelinek; David Rowley; pgsql-hackers@postgresql

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

> >> On 15/03/16 05:03, Kouhei Kaigai wrote: > >>> Petr, > >>> > >>> The attached patch is the revised one that follows the new extensible- > >>> node routine. > >>> > >>> It is almost same the previous version e

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

Re: [HACKERS] Reworks of CustomScan serialization/deserialization > > On 15/03/16 05:03, Kouhei Kaigai wrote: > > Petr, > > > > The attached patch is the revised one that follows the new extensible- > > node routine. > > > > It is almost same the previous versio

Re: [HACKERS] WIP: Upper planner pathification

> Robert Haas <robertmh...@gmail.com> writes: > > On Wed, Mar 16, 2016 at 2:47 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > >> Robert Haas <robertmh...@gmail.com> writes: > >>> On Mon, Mar 14, 2016 at 9:21 PM, Kouhei Kaigai <kai...@ak.jp.nec.com&

Re: [HACKERS] WIP: Upper planner pathification

> > Robert Haas <robertmh...@gmail.com> writes: > > > On Mon, Mar 14, 2016 at 9:21 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> > > > wrote: > > >> So, even though we don't need to define multiple hook declarations, > > >> I thin

Re: [HACKERS] WIP: Upper planner pathification

> Robert Haas <robertmh...@gmail.com> writes: > > On Mon, Mar 14, 2016 at 9:21 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > >> So, even though we don't need to define multiple hook declarations, > >> I think the hook invocation is needed just

Re: [HACKERS] WIP: Upper planner pathification

.org > Subject: Re: [HACKERS] WIP: Upper planner pathification > > Kouhei Kaigai <kai...@ak.jp.nec.com> writes: > > On Wed, Mar 16, 2016 at 2:47 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > >> I do not, however, like the proposal to expose wflists and so forth. >

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

ackers@postgresql.org > Subject: ##freemail## Re: [HACKERS] Reworks of CustomScan > serialization/deserialization > > On Sun, Mar 13, 2016 at 9:53 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > OK, I split the previous small patch into two tiny patches. > > The

Re: [HACKERS] WIP: Upper planner pathification

ion > > Petr Jelinek <p...@2ndquadrant.com> writes: > > On 14/03/16 02:43, Kouhei Kaigai wrote: > >> Even though I couldn't check the new planner implementation entirely, > >> it seems to be the points below are good candidate to inject CustomPath > >>

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

> On Sun, Mar 13, 2016 at 9:53 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > OK, I split the previous small patch into two tiny patches. > > The one is bugfix around max length of the extnodename. > > The other replaces Assert() by ereport() according to t

Re: [HACKERS] Use %u to print user mapping's umid and userid

> -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Etsuro Fujita > Sent: Monday, March 14, 2016 4:59 PM > To: Ashutosh Bapat; Tom Lane > Cc: pgsql-hackers > Subject: Re: [HACKERS] Use %u to print user mapping's umid

Re: [HACKERS] WIP: Upper planner pathification

gt; > On 14/03/16 02:43, Kouhei Kaigai wrote: > > > > CustomPath node is originally expected to generate various kind of plan > > node, not only scan/join, and its interface is designed to support them. > > For example, we can expect a CustomPath that generates &quo

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

> On 14/03/16 02:53, Kouhei Kaigai wrote: > >> -Original Message- > >> From: pgsql-hackers-ow...@postgresql.org > >> [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Petr Jelinek > >> Sent: Friday, March 11, 2016 12:27 AM > >> To: Ka

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

orks of CustomScan serialization/deserialization > > On 10/03/16 08:08, Kouhei Kaigai wrote: > >> > >>>> Also in RegisterCustomScanMethods > >>>> +Assert(strlen(methods->CustomName) <= CUSTOM_NAME_MAX_LEN); > >>>> > >>>>

Re: [HACKERS] WIP: Upper planner pathification

Hello, I'm now checking the new planner implementation to find out the way to integrate CustomPath to the upper planner also. CustomPath node is originally expected to generate various kind of plan node, not only scan/join, and its interface is designed to support them. For example, we can expect

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

orks of CustomScan serialization/deserialization > > On 10/03/16 02:18, Kouhei Kaigai wrote: > > > >> I am not sure I like the fact that we have this EXTNODENAME_MAX_LEN and > >> now the CUSTOM_NAME_MAX_LEN with the same length and also they are both > >> same le

Re: [HACKERS] Reworks of CustomScan serialization/deserialization

> On 29/02/16 13:07, Kouhei Kaigai wrote: > > > > I'd like to adjust a few of custom-scan interface prior to v9.6 freeze. > > > > The major point is serialization/deserialization mechanism. > > Now, extension has to give LibraryName and SymbolName to rep

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

Subject: Re: [HACKERS] Way to check whether a particular block is on the > shared_buffer? > > On Thu, Mar 3, 2016 at 8:54 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > I found one other, but tiny, problem to implement SSD-to-GPU direct > > data transfer feature

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

ql.org; Amit Langote > > Subject: Re: [HACKERS] Way to check whether a particular block is on the > > shared_buffer? > > > > On Thu, Feb 11, 2016 at 9:05 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > > Hmm. In my experience, it is often not a productive

[HACKERS] Reworks of CustomScan serialization/deserialization

Hello, I'd like to adjust a few of custom-scan interface prior to v9.6 freeze. The major point is serialization/deserialization mechanism. Now, extension has to give LibraryName and SymbolName to reproduce same CustomScanMethods on the background worker process side. Indeed, it is sufficient

[HACKERS] A trivial fix on extensiblenode

Hello, RegisterExtensibleNodeMethods() initializes its hash table with keysize=NAMEDATALEN, instead of EXTNODENAME_MAX_LEN. The attached patch fixes it. Thanks, -- NEC Business Creation Division / PG-Strom Project KaiGai Kohei

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

gote > Subject: Re: [HACKERS] Way to check whether a particular block is on the > shared_buffer? > > On Thu, Feb 11, 2016 at 9:05 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Hmm. In my experience, it is often not a productive discussion whether > > a feature is nic

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

bject: Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan > support > on readfuncs.c) > > On Wed, Feb 10, 2016 at 1:25 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > It is pretty good! > > > > The attached patch (primary one) implements the above id

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

> On Tue, Feb 9, 2016 at 6:35 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Unfortunately, it was not sufficient. > > > > Due to the assumption, the buffer page to be suspended does not exist > > when a backend process issues a series P2P DMA command. (If

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

a particular block is > on the shared_buffer? > > On Sun, Feb 7, 2016 at 9:49 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > On the other hands, it also became clear we have to guarantee OS buffer > > or storage block must not be updated partially during the P2P DMA. > &

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

t; CustomScan support on readfuncs.c) > > On Sun, Feb 7, 2016 at 7:28 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > The new callbacks of T_ExtensibleNode will replace the necessity to > > form and deform process of private values, like as: > > https://githu

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

> On Wed, Feb 3, 2016 at 11:57 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > At this moment, I tried to write up description at nodes/nodes.h. > > The amount of description is about 100lines. It is on a borderline > > whether we split off this chunk into anoth

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

a particular block is > on the shared_buffer? > > On Thu, Feb 4, 2016 at 11:34 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > I can agree that smgr hooks shall be primarily designed to make storage > > systems pluggable, even if we can use this hooks for suspend & resume of > &g

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

a particular block is on the > shared_buffer? > > On 2/4/16 12:30 AM, Kouhei Kaigai wrote: > >> 2. A feature to suspend i/o write-out towards a particular blocks > >> >that are registered by other concurrent backend, unless it is not > >> >unregistered (usually, at the

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

> On 2/4/16 12:30 AM, Kouhei Kaigai wrote: > >> 2. A feature to suspend i/o write-out towards a particular blocks > >> >that are registered by other concurrent backend, unless it is not > >> >unregistered (usually, at the end of P2P DMA). > >> &

Re: [HACKERS] CustomScan under the Gather node?

Jan 28, 2016 at 8:14 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > >> total ForeignScandiff > >> 0 workers: 17584.319 ms 17555.904 ms 28.415 ms > >> 1 workers: 18464.476 ms 18110.968 ms 353.508 ms > >> 2 workers: 19

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

> On Wed, Jan 27, 2016 at 9:36 PM, Robert Haas <robertmh...@gmail.com> wrote: > > On Mon, Jan 25, 2016 at 8:06 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > >> Sorry for my late response. I've been unavailable to have enough > >> ti

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

> > KaiGai-san, > > > > On 2016/02/01 10:38, Kouhei Kaigai wrote: > > > As an aside, background of my motivation is the slide below: > > > http://www.slideshare.net/kaigai/sqlgpussd-english > > > (LT slides in JPUG conference last Dec) > > &

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

bject: Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan > support > on readfuncs.c) > > On Wed, Feb 3, 2016 at 8:00 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > >> Well, looking at this a bit more, it seems like the documentation > >> you've wr

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

> > > On 1/31/16 7:38 PM, Kouhei Kaigai wrote: > > > > To answer your direct question, I'm no expert, but I haven't seen any > > > functions that do exactly what you want. You'd have to pull relevant > > > bits from ReadBuffer_*. Or maybe a better method wou

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

> KaiGai-san, > > On 2016/02/01 10:38, Kouhei Kaigai wrote: > > As an aside, background of my motivation is the slide below: > > http://www.slideshare.net/kaigai/sqlgpussd-english > > (LT slides in JPUG conference last Dec) > > > > I'm under investigat

Re: [HACKERS] Way to check whether a particular block is on the shared_buffer?

> On 1/31/16 7:38 PM, Kouhei Kaigai wrote: > > I'm under investigation of SSD-to-GPU direct feature on top of > > the custom-scan interface. It intends to load a bunch of data > > blocks on NVMe-SSD to GPU RAM using P2P DMA, prior to the data > > loading onto CPU/

[HACKERS] Way to check whether a particular block is on the shared_buffer?

Hello, Do we have a reliable way to check whether a particular heap block is already on the shared buffer, but not modify? Right now, ReadBuffer and ReadBufferExtended are entrypoint of the buffer manager for extensions. However, it tries to acquire an available buffer pool instead of the victim

Re: [HACKERS] CustomScan under the Gather node?

> To: 'Robert Haas' > Cc: pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] CustomScan under the Gather node? > > > On Tue, Jan 26, 2016 at 1:30 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > > What enhancement will be necessary to implement similar feature of

Re: [HACKERS] CustomScan under the Gather node?

> On Thu, Jan 28, 2016 at 10:50 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > >> If I would make a proof-of-concept patch with interface itself, it > >> seems to me file_fdw may be a good candidate for this enhancement. > >> It is not a field for postg

Re: [HACKERS] CustomScan under the Gather node?

ms307.134 ms Thanks, -- NEC Business Creation Division / PG-Strom Project KaiGai Kohei <kai...@ak.jp.nec.com> > -Original Message- > From: pgsql-hackers-ow...@postgresql.org > [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Kouhei Kaigai > Sent: Friday, Ja

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

> On Mon, Jan 25, 2016 at 8:06 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Sorry for my late response. I've been unavailable to have enough > > time to touch code for the last 1.5 month. > > > > The attached patch is a revised one to handle private data of

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

> On Thu, Jan 28, 2016 at 10:18 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > Do you think we shall allow to register same extensible node name for > > different node types? Like, "GpuJoin" for any of CustomPath, CustomScan > > and CustomScanState. Or,

Re: [HACKERS] CustomScan under the Gather node?

> On Tue, Jan 26, 2016 at 1:30 AM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > What enhancement will be necessary to implement similar feature of > > partial seq-scan using custom-scan interface? > > > > It seems to me callbacks on the three points below are n

Re: [HACKERS] CustomScan under the Gather node?

Jan 26, 2016 at 12:00 PM, Kouhei Kaigai <kai...@ak.jp.nec.com> wrote: > > > > Hello, > > > > What enhancement will be necessary to implement similar feature of > > partial seq-scan using custom-scan interface? > > > > It seems to me ca

[HACKERS] CustomScan under the Gather node?

Hello, What enhancement will be necessary to implement similar feature of partial seq-scan using custom-scan interface? It seems to me callbacks on the three points below are needed. * ExecParallelEstimate * ExecParallelInitializeDSM * ExecParallelInitializeWorker Anything else? Does

Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan support on readfuncs.c)

e- > From: Kaigai Kouhei(海外 浩平) > Sent: Wednesday, December 02, 2015 5:52 PM > To: 'Robert Haas' > Cc: Andres Freund; Amit Kapila; pgsql-hackers > Subject: Re: CustomScan in a larger structure (RE: [HACKERS] CustomScan > support > on readfuncs.c) > > > On Thu, No

  1   2   3   4   5   >