Re: [HACKERS] Declarative partitioning - another take

2016-09-27 Thread Ashutosh Bapat
On Tue, Sep 27, 2016 at 2:46 PM, Amit Langote wrote: > On 2016/09/27 15:44, Ashutosh Bapat wrote: >>> By the way, I fixed one thinko in your patch as follows: >>> >>> -result->oids[i] = oids[mapping[i]]; >>> +result->oids[mapping[

Re: [HACKERS] Declarative partitioning - another take

2016-09-27 Thread Ashutosh Bapat
tch is > certainly to be blamed here; if I revert the patch, the problem goes away. > > Based on 2 above, I attempted to add logic for AppendPath in > reparameterize_path() as in the attached. It fixes the reported problem > and does not break any regression tests. If it's

Re: [HACKERS] Declarative partitioning - another take

2016-09-26 Thread Ashutosh Bapat
as corresponding OIDs. Partition-wise join simply joins the RelOptInfos at the same positions from both the parent RelOptInfos. I can not spot an error in this logic too. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing l

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-26 Thread Ashutosh Bapat
On Tue, Sep 27, 2016 at 8:48 AM, Etsuro Fujita wrote: > On 2016/09/26 20:20, Ashutosh Bapat wrote: >> >> On Mon, Sep 26, 2016 at 4:06 PM, Etsuro Fujita >> wrote: >>> >>> On 2016/09/26 18:06, Ashutosh Bapat wrote: >>>> >>>

Re: [HACKERS] Aggregate Push Down - Performing aggregation on foreign server

2016-09-26 Thread Ashutosh Bapat
pt the FIXME over there as > I think it should have done by the core itself. > > 30, 31, 32, 33. All done. > > Let me know your views. > > Thanks > -- > Jeevan B Chalke > Principal Software Engineer, Product Development > EnterpriseDB Corporation > The Enterprise Post

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-09-26 Thread Ashutosh Bapat
On Mon, Sep 26, 2016 at 5:25 PM, Masahiko Sawada wrote: > On Mon, Sep 26, 2016 at 7:28 PM, Ashutosh Bapat > wrote: >> My original patch added code to manage the files for 2 phase >> transactions opened by the local server on the remote servers. This >> code was mostly i

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-26 Thread Ashutosh Bapat
On Mon, Sep 26, 2016 at 4:06 PM, Etsuro Fujita wrote: > On 2016/09/26 18:06, Ashutosh Bapat wrote: >> >> On Mon, Sep 26, 2016 at 1:05 PM, Etsuro Fujita >> wrote: > > >>> ISTM that the use of the same RTI for subqueries in multi-levels in a >>> remote

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-09-26 Thread Ashutosh Bapat
optimized. One of the things I wanted to do is see, if those optimizations are applicable here as well. Have you considered that? [1]. https://www.postgresql.org/message-id/74355FCF-AADC-4E51-850B-47AF59E0B215%40postgrespro.ru On Fri, Aug 26, 2016 at 11:43 AM, Ashutosh Bapat wrote: > > &g

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-26 Thread Ashutosh Bapat
On Mon, Sep 26, 2016 at 1:05 PM, Etsuro Fujita wrote: > On 2016/09/15 15:29, Ashutosh Bapat wrote: >> >> On Wed, Sep 14, 2016 at 8:52 PM, Robert Haas >> wrote: > > >>> I'm not sure why it wouldn't work >>> to just use the lowest RTI in

Re: [HACKERS] Aggregate Push Down - Performing aggregation on foreign server

2016-09-26 Thread Ashutosh Bapat
1 where c1 = 13) q(a) right join ft2 on (q.a = ft2.c1) where ft2.c1 between 10 and 15 group by q.a order by 1 nulls last; 26. Instead of the testcase below a test which has window aggregates over a pushed down aggregate makes sense in the context of aggregate pushdown. +-- WindowAgg +explain (verb

Re: [HACKERS] Declarative partitioning - another take

2016-09-22 Thread Ashutosh Bapat
On Thu, Sep 22, 2016 at 1:02 PM, Ashutosh Bapat wrote: > For list partitions, the ListInfo stores the index maps for values > i.e. the index of the partition to which the value belongs. Those > indexes are same as the indexes in partition OIDs array and come from > the catalogs. In

Re: [HACKERS] Declarative partitioning - another take

2016-09-22 Thread Ashutosh Bapat
scheme match to the OIDs array to find matching RelOptInfos for partition-wise join. On Thu, Sep 22, 2016 at 11:12 AM, Ashutosh Bapat wrote: > Hi Amit, > Following sequence of DDLs gets an error > -- > -- multi-leveled partitions > -- > CREATE TABLE prt1_l (a int, b int, c var

Re: [HACKERS] Declarative partitioning - another take

2016-09-21 Thread Ashutosh Bapat
) PARTITION BY LIST (j); >> CREATE TABLE test_array_p1 PARTITION OF test_array FOR VALUES IN ('{1}'); >> CREATE TABLE test_array_p2 PARTITION OF test_array FOR VALUES IN ('{2,2}'); >> >> INSERT INTO test_array (i,j[1],k[1]) VALUES (1,1,1); >> INS

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2016-09-20 Thread Ashutosh Bapat
Hi Rajkumar, On Fri, Sep 16, 2016 at 6:00 PM, Rajkumar Raghuwanshi wrote: > > On Fri, Sep 9, 2016 at 3:17 PM, Ashutosh Bapat > wrote: >> >> Hi All, >> >> PFA the patch to support partition-wise joins for partitioned tables. The >> patch >> is base

[HACKERS] Re: [HACKERS] Error running custom plugin: “output plugins have to declare the _PG_output_plugin_init symbol”

2016-09-20 Thread Ashutosh Bapat
at https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Windows. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Calculation of param_source_rels in add_paths_to_joinrel

2016-09-19 Thread Ashutosh Bapat
path(). Also, the way this code has been written, the declaration of variable sjinfo masks the earlier declaration with the same name. I am not sure if that's intentional, but may be we should use another variable name for the inner sjinfo. I have not included that change in the patch. -- Best

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-18 Thread Ashutosh Bapat
Thanks a lot. On Fri, Sep 16, 2016 at 7:07 PM, Tom Lane wrote: > Ashutosh Bapat writes: >>> I'd suggest that this is parallel to nodeToString() and therefore >>> (a) should be placed beside it, > >> Done. Added it after nodeToString(). > > Pushed, th

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2016-09-18 Thread Ashutosh Bapat
On Fri, Sep 16, 2016 at 6:00 PM, Rajkumar Raghuwanshi wrote: > > On Fri, Sep 9, 2016 at 3:17 PM, Ashutosh Bapat > wrote: >> >> Hi All, >> >> PFA the patch to support partition-wise joins for partitioned tables. The >> patch >> is based on the declarat

Re: [HACKERS] Improvements in psql hooks for variables

2016-09-18 Thread Ashutosh Bapat
ove changes, > I think we should hear it first. > > [1] > https://www.postgresql.org/message-id/f2cb5838-0ee9-4fe3-acc0-df77aeb7d4c7%40mm > [2] > https://www.postgresql.org/message-id/4695.1473961140%40sss.pgh.pa.us > > > Best regards, > -- > Daniel Vérité > Pos

Re: [HACKERS] Why postgres take RowExclusiveLock on all partition

2016-09-16 Thread Ashutosh Bapat
On Fri, Sep 16, 2016 at 4:31 PM, Sachin Kotwal wrote: > Hi Hackers, > > > I checked if there is update transaction on master table involved in > partition. > Postgresql takes RowExclusiveLock on all partition tables. > > constraint exclusion is set to on. I checked this under the debugger and f

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-16 Thread Ashutosh Bapat
ids) $4 = 0x2105998 "(b 1 3 4)" (gdb) p bmsToString(joinrel->lateral_relids) $5 = 0x2105db0 "(b)" (gdb) p joinrel->lateral_relids $6 = (Relids) 0x0 -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company diff --git a/src/backend/nodes/outfu

Re: [HACKERS] Parallel sec scan in plpgsql

2016-09-15 Thread Ashutosh Bapat
ql. So, I guess, answer to your question is yes. If you are expecting something else, more context will help. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] Declarative partitioning - another take

2016-09-15 Thread Ashutosh Bapat
vation 3 : Getting cache lookup failed, when selecting list >> partition table containing array. >> >> CREATE TABLE test_array ( i int,j int[],k text[]) PARTITION BY LIST (j); >> CREATE TABLE test_array_p1 PARTITION OF test_array FOR VALUES IN ('{1}'); >> CREAT

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-14 Thread Ashutosh Bapat
On Wed, Sep 14, 2016 at 8:52 PM, Robert Haas wrote: > On Tue, Sep 13, 2016 at 11:38 PM, Ashutosh Bapat > wrote: >> On Tue, Sep 13, 2016 at 10:28 PM, Robert Haas wrote: >>> On Tue, Sep 6, 2016 at 9:07 AM, Ashutosh Bapat >>> wrote: >>>> That's not

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
On Wed, Sep 14, 2016 at 8:45 PM, Alvaro Herrera wrote: > Tom Lane wrote: >> Ashutosh Bapat writes: >> > While working on partition-wise join, I had to examine Relids objects >> > many times. Printing the Bitmapset::words[] in binary format and then >> > in

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
here are other places where we manipulate Bitmapsets. And not every Relids object computed is contained in a Node. So, pprint() doesn't help much. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
tion like the one I have in the patch makes things easy for all the debuggers and may be developers not familiar with python. > > The complete API reference is available here > http://lldb.llvm.org/python_reference/index.html > > Looks like an interesting SoC project to write useful

[HACKERS] Re: [HACKERS] Error running custom plugin: “output plugins have to declare the _PG_output_plugin_init symbol”

2016-09-14 Thread Ashutosh Bapat
_decoding/test_decoding.c > > I'm not sure if the deployment process is ok (just copying the dll) or if > there is some other step to take. Can anyone shed some light? > It's hard to tell what's wrong exactly, without seeing the changes you have made. But, it looks like wh

[HACKERS] Printing bitmap objects in the debugger

2016-09-14 Thread Ashutosh Bapat
and compile it for debugging. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 29b7712..b4cae11 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-13 Thread Ashutosh Bapat
On Tue, Sep 13, 2016 at 10:28 PM, Robert Haas wrote: > On Tue, Sep 6, 2016 at 9:07 AM, Ashutosh Bapat > wrote: >> That's not true with the alias information. As long as we detect which >> relations need subqueries, their RTIs are enough to create unique aliases >>

Re: [HACKERS] Nested loop join condition does not get pushed down to foreign scan

2016-09-13 Thread Ashutosh Bapat
ths for that foreign relation. If using a parameterized path reduces cost of the join, it will use a nested loop join with inner relation parameterized by the outer relation, pushing join conditions down into the foreign scan. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-12 Thread Ashutosh Bapat
a clue as to what error context is >> about. >> > > I don't think so. Consider an example of the conversion error message, > which is from the regression test: > > SELECT ft1.c1, ft2.c2, ft1 FROM ft1, ft2 WHERE ft1.c1 = ft2.c1 AND > ft1.c1 = 1; > ERROR: invalid input syntax for integer: "foo" > CONTEXT: whole-row reference to foreign table "ft1" > > As shown in the example, the error message is displayed under a remote > query for execution. So, ISTM it's reasonable to print something like > "expression at position %d in select list" in the context if an expression > in a PHV. > I missed it. Sorry. Looks ok. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Aggregate Push Down - Performing aggregation on foreign server

2016-09-08 Thread Ashutosh Bapat
of just assigning the pointer. 30. By the time postgresGetForeignUpperPaths() gets called, the core has already added its own paths, so it doesn't make much sense to set rows and width grouped_rel in create_foreign_grouping_paths(). 31. fpinfo->server and user fields are being set twice, on

Re: [HACKERS] Push down more UPDATEs/DELETEs in postgres_fdw

2016-09-06 Thread Ashutosh Bapat
ERE clause can be applied on it. But it's not needed if we are pushing down the query. If we eliminate the targetlist of the query, we could construct a remote query without having subquery in it, making it more readable. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-09-06 Thread Ashutosh Bapat
On Fri, Sep 2, 2016 at 3:55 PM, Etsuro Fujita wrote: > Hi Ashutosh, > > On 2016/08/22 15:49, Ashutosh Bapat wrote: > >> 1. deparsePlaceHolderVar looks odd - each of the deparse* function is >> named as deparse + > into>. PlaceHolderVar is not a parser node, so no s

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Ashutosh Bapat
This patch uses RangeBound structure. There's also a structure defined with the same name in rangetypes.h with some slight differences. Should we rename the one in partition.c as PartRangeBound or something like that to avoid the confusion? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corpor

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Ashutosh Bapat
ve of parent's, selecting a parent's NOT NULL > column might return nulls from the child table that no longer has the > constraint. > > I recently came across a related proposal whereby dropping *inherited* NOT > NULL from child tables will be prevented. Problems in letting it

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
On Fri, Sep 2, 2016 at 12:23 PM, Amit Langote wrote: > On 2016/09/02 15:22, Ashutosh Bapat wrote: > >> > >> > >>> 2. A combination of constraints on the partitions should be applicable > to > >>> the parent. We aren't doing that. > >&

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
0 or a < 600)). It will probably get excluded, if constraint exclusion is smart enough to understand ORing. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
nge partition, an error will be thrown. > > 0008-Update-DDL-Partitioning-chapter.patch > > This patch updates the partitioning section in the DDL chapter to reflect > the new methods made available for creating and managing partitioned table > and its partitions. Especially considering th

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
-wise join. > Then as we build successively higher levels of > joinrels, new entries will be made for those joinrels for which we added > pairwise join paths, with relids matching the corresponding joinrels. > Does that make sense? > > I don't think we will make any ne

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-08-25 Thread Ashutosh Bapat
On Fri, Aug 26, 2016 at 11:37 AM, Masahiko Sawada wrote: > On Fri, Aug 26, 2016 at 3:03 PM, Ashutosh Bapat > wrote: > > > > > > On Fri, Aug 26, 2016 at 11:22 AM, Masahiko Sawada > > > wrote: > >> > >> On Fri, Aug 26, 2016 at

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2016-08-25 Thread Ashutosh Bapat
ginal patch, patch of pg_fdw_xact_resolver and > documentation are lacked. > I am not able to understand the last statement. Do you mean to say that your patches do not have pg_fdw_xact_resolver() and documentation that my patches had? OR you mean to say that my patches did not have (lacked) pg_fdw_xact_resolver() and documenation OR some combination of those? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning - another take

2016-08-25 Thread Ashutosh Bapat
On Thu, Aug 25, 2016 at 12:22 PM, Amit Langote < langote_amit...@lab.ntt.co.jp> wrote: > On 2016/08/22 13:51, Ashutosh Bapat wrote: > > The parent-child relationship of multi-level partitioned tables is not > > retained when creating the AppendRelInfo nodes. We create RelOp

Re: [HACKERS] Push down more full joins in postgres_fdw

2016-08-21 Thread Ashutosh Bapat
3, c4, c5, c6, > c7, c8), "C 1" FROM "S 1"."T 1") ss2(c1, c2) ON (TRUE)) WHERE ((ss1.c3 = > ss2.\ > c2)) ORDER BY ss1.c4 ASC NULLS LAST, ss1.c3 ASC NULLS LAST > (6 rows) > > I'll add this to the next CF. Comments are welcome! > > Best rega

Re: [HACKERS] Declarative partitioning - another take

2016-08-21 Thread Ashutosh Bapat
e available for creating and managing partitioned table > and its partitions. Especially considering that it is no longer necessary > to define CHECK constraints and triggers/rules manually for constraint > exclusion and tuple routing, respectively. > > TODO (in short term): > *

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Ashutosh Bapat
On Wed, Aug 17, 2016 at 11:51 AM, Amit Langote < langote_amit...@lab.ntt.co.jp> wrote: > On 2016/08/17 14:33, Ashutosh Bapat wrote: > >> +relid_is_partition(Oid relid) > >> +{ > >> + return SearchSysCacheExists1(PARTRELID, > ObjectIdGetDatum(relid));

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Ashutosh Bapat
another question. Can we have rd_partdesc non null and rd_partkey null or vice-versa. If not, should we club those into a single structure like Partition (similar to Relation)? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning - another take

2016-08-15 Thread Ashutosh Bapat
ar extracted out of a date column e.g. order_date. They will find it convenient to use an expression (extract(month from date)) as a partition key, instead of storing month or year as a separate column. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-08-10 Thread Ashutosh Bapat
rt (0) end ((1/5) ... Shouldn't we allow constant expressions here? If this has been already discussed, please forgive me and point out the relevant mail chain. On Tue, Aug 9, 2016 at 12:48 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > What strikes me od

Re: [HACKERS] Declarative partitioning

2016-08-09 Thread Ashutosh Bapat
tuple routing. If the > > algorithm I proposed above turns out to be too slow for matching > > partitions, then we could keep both this representation and the > > previous one. We are not limited to just one. But I don't think > > that's likely to be the case. > > I agree. Let's see how the option 2 turns out. > > > Also, note that all of this presupposes we're doing range > > partitioning, or perhaps list partitioning with a btree opclass. For > > partitioning based on a hash opclass, you'd organize the data based on > > the hash values rather than range comparisons. > > Yes, the current patch does not implement hash partitioning, although I > have to think about how to support the hash case when designing the > internal data structures. > > > By the way, I am planning to start a new thread with the latest set of > patches which I will post in a day or two. I have tried to implement all > the bug fixes and improvements that have been suggested on this thread so > far. Thanks to all those who reviewed and gave their comments. Please > check this page to get a link to the new thread: > https://commitfest.postgresql.org/10/611/ > > Thanks, > Amit > > > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-08-07 Thread Ashutosh Bapat
ge suggested above, it should be easy to check partition-wise join compatibilty for simplest case. I agree that for generic case it will be difficult. E.g. Table 1#: p3 {'a', 'e'}, p4{'a#', 'l'}, p2 {'b', 'f'}, p1 {'c', 'd'} being (INNER) joined with Table 2: p2 {'a', 'e'}, p1 {'b', 'f'}, p3 {'c', 'd'} assuming 'a'< 'a#' < 'b'; in this case, we should be able to match p3-p2, p2-p1, p1-p3 for partition-wise join, even though canonical representations of both partitions differ, because of an extra partition in between. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Ashutosh Bapat
;)} which are essentially same, are represented in different ways. It makes matching partitions for partition-wise join a bit tedius. We have to make sure that the first array matches for both the joining relations and then make sure that all the values belonging to the same partition for one table also belong to the same partition in the other table. Some more complex logic for matching subsets of lists for partition-wise join. At least for straight forward partitioned table matching it helps to have both these array look same independent of the user specification. From that point of view, the partition be ordered by their lowest or highest list values and the second array is the index in the ordered set. For both the specifications above, the list will look like [ 'a', 'b', 'd', 'e', f', 'h', 'i', 'l', 'm' ] [1, 1, 2, 3, 3, 2, 3, 1, 2] -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Ashutosh Bapat
On Fri, Aug 5, 2016 at 5:21 PM, Robert Haas wrote: > On Fri, Aug 5, 2016 at 6:53 AM, Ashutosh Bapat > wrote: > > The lists for list partitioned tables are stored as they are specified by > > the user. While searching for a partition to route tuple to, we compare > it >

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Ashutosh Bapat
responding partitions. We might be able to eliminate search in a given partition if its lowest value is higher than the given value or its higher value is lower than the given value. On Thu, Jul 21, 2016 at 10:10 AM, Amit Langote < langote_amit...@lab.ntt.co.jp> wrote: > On 2016/07/19 22:53, A

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

2016-08-01 Thread Ashutosh Bapat
pthread, we can see those relations from that, not the Relations line. The join type is missing in that description. > Also we can see the join tree structure from the deparsed query in the > Remote SQL line. The remote SQL has the names of the table on the foreign server. It does not help to identify the local names. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

2016-07-28 Thread Ashutosh Bapat
", such as > aggregation, window functions, distinct, order by, row locking, table > modification, or combinations of them. > > "Scan" is a better word than "Processing". From plan's perspective it's ultimately a Scan (on the data produced by the foreign server) and not processing. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Oddity in EXPLAIN for foreign/custom join pushdown plans

2016-07-26 Thread Ashutosh Bapat
ike Foreign Grouping, aggregation on ... Foreign Join on ... But then the question is a foreign scan node can be pushing down many operations together e.g. join, aggregation, sort OR join aggregation and windowing OR join and insert. How would we clearly convey this? May be we say Foreign Scan operations: join on ..., aggregation, ... That wouldn't be so great and might be clumsy for many operations. Any better idea? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2016-07-19 Thread Ashutosh Bapat
Sorry forgot to mention: this patch applies on top of the v7 patches posted by Amit Langote on 27th June ( https://www.postgresql.org/message-id/81371428-bb4b-1e33-5ad6-8c5c51b52cb7%40lab.ntt.co.jp ). On Tue, Jul 19, 2016 at 7:41 PM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com>

Re: [HACKERS] Declarative partitioning

2016-07-19 Thread Ashutosh Bapat
I am seeing following warning with this set of patches. gram.y:4734:24: warning: assignment from incompatible pointer type [enabled by default] On Tue, Jul 5, 2016 at 10:18 AM, Amit Langote wrote: > On 2016/07/04 21:31, Ashutosh Bapat wrote: > > Hi Amit, > > I observed that the

Re: [HACKERS] Oddity in handling of cached plans for FDW queries

2016-07-14 Thread Ashutosh Bapat
> > Join between views on foreign tables or between foreign tables and views containing foreign tables won't be rare. This feature is yet to be released, so we don't know if PostgreSQL users would find it useful. But I do see Oracle users joining views on dblink tables. I would guess same would be the case in PostgreSQL. But I would like to hear from other PostgreSQL FDW users. In such cases, being able to push down a join between foreign tables across view boundaries will be useful. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Oddity in handling of cached plans for FDW queries

2016-07-14 Thread Ashutosh Bapat
On Thu, Jul 14, 2016 at 10:02 PM, Tom Lane wrote: > Ashutosh Bapat writes: > > Exactly, for a rare scenario, should we be penalizing large number of > plans > > or just continue to use a previously prepared plan when an optimal plan > has > > become available beca

Re: [HACKERS] Oddity in handling of cached plans for FDW queries

2016-07-14 Thread Ashutosh Bapat
On Thu, Jul 14, 2016 at 5:10 PM, Etsuro Fujita wrote: > On 2016/07/13 18:00, Ashutosh Bapat wrote: > >> To fix the first, I'd like to propose (1) replacing the existing >> has_foreign_join flag in the CachedPlan data structure with a new >> flag, say uses

Re: [HACKERS] Oddity in handling of cached plans for FDW queries

2016-07-13 Thread Ashutosh Bapat
. If we do that, we might not set hasForeignJoin flag in create_foreignscan_plan() when the user mapping for pushed down join is invalid. That will keep FDWs which do not use user mappings out of plan cache invalidation. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-07-04 Thread Ashutosh Bapat
are valid, but then onwards they are restamped with the varno of the first partition. Please add testcases to your patch to catch such types of issues. On Mon, Jun 27, 2016 at 3:56 PM, Amit Langote wrote: > > Hi Ashutosh, > > On 2016/06/24 23:08, Ashutosh Bapat wrote: > > Hi

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-07-01 Thread Ashutosh Bapat
On Fri, Jul 1, 2016 at 7:45 PM, Robert Haas wrote: > On Tue, Jun 28, 2016 at 8:20 AM, Ashutosh Bapat > wrote: > >> > postgres_fdw resets the search path to pg_catalog while opening > >> > connection > >> > to the server. The reason behind this is

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-28 Thread Ashutosh Bapat
On Tue, Jun 28, 2016 at 12:52 PM, Etsuro Fujita wrote: > On 2016/06/28 15:23, Ashutosh Bapat wrote: > >> The wording "column "whole-row reference ..." doesn't look good. >> Whole-row reference is not a column. The error context itself should be >>

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-28 Thread Ashutosh Bapat
ill be tempted to create more instances of schema-qualification. I think we should revert the schema qualification. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-27 Thread Ashutosh Bapat
On Tue, Jun 28, 2016 at 11:43 AM, Etsuro Fujita wrote: > On 2016/06/28 13:53, Ashutosh Bapat wrote: > >> Ideally, we should point out the specific column that faced the >> conversion problem and report it, instead of saying the whole row >> reference conversion caused t

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-27 Thread Ashutosh Bapat
On Tue, Jun 28, 2016 at 9:00 AM, Etsuro Fujita wrote: > On 2016/06/27 18:56, Ashutosh Bapat wrote: > >> On Mon, Jun 27, 2016 at 3:06 PM, Etsuro Fujita >> mailto:fujita.ets...@lab.ntt.co.jp>> wrote: >> > > I found another bug in error handling of whole-ro

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-27 Thread Ashutosh Bapat
case reproducing the bug here? It would be good to include that test in the regression. There is a always a possibility that a user would create a table (which can be used as target for the foreign table) with column named 'wholerow', in which case s/he will get confused with this error messag

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-26 Thread Ashutosh Bapat
alified; otherwise, qualify. */ static char * deparse_type_name(Oid type_oid, int32 typemod) { if (is_builtin(type_oid)) return format_type_with_typemod(type_oid, typemod); else return format_type_with_typemod_qualified(type_oid, typemod); } -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-06-24 Thread Ashutosh Bapat
t1_l SELECT i, to_char(i, 'FM00'), i FROM generate_series(1, 600, 2) i; CREATE TABLE upt1_l AS SELECT * FROM pt1_l; The last statement gives error "ERROR: cache lookup failed for function 0". Let me know if this problem is reproducible. On Thu, Jun 9, 2016 at 7:20 AM, Amit Lan

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-24 Thread Ashutosh Bapat
On Fri, Jun 24, 2016 at 1:59 PM, Amit Langote wrote: > On 2016/06/24 15:44, Ashutosh Bapat wrote: > >> > >> I think the proposed idea of applying record::text explicit coercion to > a > >> whole-row reference in the IS NOT NULL condition in the CASE WHEN > >

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-23 Thread Ashutosh Bapat
a record is null and not null in the way we want (as described upthread). I didn't find any quickly. Do you have any suggestions? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-22 Thread Ashutosh Bapat
On Wed, Jun 22, 2016 at 3:57 PM, Etsuro Fujita wrote: > On 2016/06/22 18:16, Ashutosh Bapat wrote: > >> On Wed, Jun 22, 2016 at 2:26 PM, Etsuro Fujita >> mailto:fujita.ets...@lab.ntt.co.jp>> wrote: >> > > I think we could address this in another

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-22 Thread Ashutosh Bapat
also address this by replacing the whole-row reference in the IS NOT NULL > condition in that conversion with the system column reference. > > That would not work again as the system column reference would make sense locally but may not be available at the foreign server e.g. foreign table targeting a view a tableoid is requested. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-22 Thread Ashutosh Bapat
ids (which isn't a big deal I guess). That's something more than necessary for fixing the bug, which is the focus in beta stage right now. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-21 Thread Ashutosh Bapat
D". IMO I think that that would > be much simpler than Ashutosh's approach. > > A foreign table can have a view, a regular table, another foreign table or a materialised view a its target. A view does not support any of the system columns, so none of them are available. -- Best

Re: [HACKERS] Postgres_fdw join pushdown - wrong results with whole-row reference

2016-06-21 Thread Ashutosh Bapat
something of the pre-8.3 2261 * behavior that many types had implicit (yipes!) casts to text. 2262 */ PFA the patch with the cast to text. This is probably uglier than expected, but I don't know any better test to find nullness of a record, the way we want here. The patch als

Re: [HACKERS] Declarative partitioning

2016-06-21 Thread Ashutosh Bapat
artitioned table by range on expression chr(a) it works. On Thu, Jun 9, 2016 at 7:20 AM, Amit Langote wrote: > On 2016/06/08 22:22, Ashutosh Bapat wrote: > > On Mon, May 23, 2016 at 3:35 PM, Amit Langote wrote > >> > >> [...] > >> > >> I made a

[HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2016-06-15 Thread Ashutosh Bapat
://users.cs.duke.edu/~shivnath/papers/sigmod295-herodotou.pdf [3]. https://users.cs.duke.edu/~shivnath/tmp/paqo_draft.pdf -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company partitioned_join.out Description: Binary data \set num_samples 100 drop table t1 cascade; drop table t2

Re: [HACKERS] [sqlsmith] Failed assertion in postgres_fdw/deparse.c:1116

2016-06-13 Thread Ashutosh Bapat
rpose for which I added the Assert, but in a better manner. May be the error message can read "non-Var nodes/targets/expressions not expected in target list". I am not sure what do we call individual (whole) members of target list. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-06-08 Thread Ashutosh Bapat
ion.c:747 > > [...] > > I made a mistake in the last version of the patch which caused a relcache > field to be pfree'd unexpectedly. Attached updated patches. > > Thanks, > Amit > > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) >

Re: [HACKERS] [sqlsmith] Failed assertion in postgres_fdw/deparse.c:1116

2016-06-08 Thread Ashutosh Bapat
On Wed, Jun 8, 2016 at 12:25 PM, Amit Langote wrote: > On 2016/06/08 14:13, Ashutosh Bapat wrote: > > On Tue, Jun 7, 2016 at 6:19 PM, Amit Langote wrote: > >> On Tue, Jun 7, 2016 at 7:47 PM, Ashutosh Bapat wrote: > >>> Looks good to me. If we add a column from the

Re: [HACKERS] [sqlsmith] Failed assertion in postgres_fdw/deparse.c:1116

2016-06-07 Thread Ashutosh Bapat
at 7:47 PM, Ashutosh Bapat wrote: > > On Tue, Jun 7, 2016 at 4:07 PM, Amit Langote wrote: > >> On 2016/06/07 19:13, Ashutosh Bapat wrote: > >> > So, your patch looks to be the correct approach (even after we support > >> > deparsing subqueries). Can you please

Re: [HACKERS] [sqlsmith] Failed assertion in postgres_fdw/deparse.c:1116

2016-06-07 Thread Ashutosh Bapat
On Tue, Jun 7, 2016 at 4:07 PM, Amit Langote wrote: > On 2016/06/07 19:13, Ashutosh Bapat wrote: > > I thought, columns of inner relation will be set to null during > projection > > from ForeignScan for joins. But I was wrong. If we want to push-down > joins > >

Re: [HACKERS] [sqlsmith] Failed assertion in postgres_fdw/deparse.c:1116

2016-06-07 Thread Ashutosh Bapat
ush-down joins in this case, we have two solutions 1. Build queries with subqueries at the time of deparsing. Thus a base relation or join has to include placeholders while being deparsed as a subquery. This means that the deparser should deparse expression represented by the placeholder. This may not

Re: [HACKERS] [sqlsmith] Failed assertion in postgres_fdw/deparse.c:1116

2016-06-07 Thread Ashutosh Bapat
ing down the join. Although, I haven't tried the exact query given in the report. Please let me know if the patch fixes issue with that query as well. The query generated by sqlsmith doesn't seem to return any result since it assigns 31 to subq_0.c0 and the WHERE clause checks 92 =subq_0.c0.

Re: [HACKERS] Postgres_fdw join pushdown - getting server crash in left outer join of three table

2016-05-18 Thread Ashutosh Bapat
one Ashutosh sent. Oops. > > Reverting Michael's patch and applying Ashutosh's doesn't work any > more due to conflicts in the regression tests. And in re-rereviewing > Ashutosh's patch I came to feel like the comments in this area needed > a lot more work. > Tha

Re: [HACKERS] Use %u to print user mapping's umid and userid

2016-05-11 Thread Ashutosh Bapat
On Wed, May 11, 2016 at 1:34 PM, Etsuro Fujita wrote: > On 2016/05/11 16:49, Ashutosh Bapat wrote: > >> The patch is calculating user mapping when it's readily available >> through RelOptInfo::fdw_private. That incurs a catalog lookup >> unnecessarily. Instead, ca

Re: [HACKERS] Use %u to print user mapping's umid and userid

2016-05-11 Thread Ashutosh Bapat
adily available through RelOptInfo::fdw_private. That incurs a catalog lookup unnecessarily. Instead, can we add new function makeOid, oidVal on the lines of makeInteger and intVal to store and retrieve an OID resp. and also corresponding print function? It might be helpful in future. -- Best Wishes,

Re: [HACKERS] Declarative partitioning

2016-05-09 Thread Ashutosh Bapat
gt; > > > You're right, silly mistake. :-( > > > > Will fix > > Attached updated version fixes this. I'll take time to send the next > version but I'd very much appreciate it if you keep reporting anything > that doesn't look/work right like you did so far. > > Thanks, > Amit > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] UNION ALL - Var attno

2016-04-29 Thread Ashutosh Bapat
r, >> but assuming it's a Var? The fact that 141 is the pg_proc OID of int4mul >> lends considerable weight to this suspicion ... >> >> regards, tom lane >> > > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] postgres_fdw : Not able to update foreign table referring to a local table's view when use_remote_estimate = true

2016-04-24 Thread Ashutosh Bapat
or something like "DMLs are not supported on foreign tables referring to views/non-tables on foreign server" is not supported. While creating the foreign table a user can specify whether the object being referred is updatable (writable?) or not, Import foreign schema can set the status by looking at pg_class type entry. The efforts required may not be worth the usage given that this case is highly unlikely. May be we should just update the documents saying that a user may encounter such an error if s/he attempts to update/delete such a foreign table. -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company

Re: [HACKERS] Declarative partitioning

2016-04-18 Thread Ashutosh Bapat
On Mon, Apr 18, 2016 at 1:23 PM, Amit Langote wrote: > On 2016/04/18 15:38, Ashutosh Bapat wrote: > >> There was no KeyTypeCollInfo in early days of the patch and then I found > >> myself doing a lot of: > >> > >> partexprs_item = list_head(key->pa

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Ashutosh Bapat
> On 2016/04/15 18:46, Ashutosh Bapat wrote: > > > > 3. PartitionKeyData contains KeyTypeCollInfo, whose contents can be > > obtained by calling functions exprType, exprTypemod on partexprs. Why do > we > > need to store that information as a separate member? >

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Ashutosh Bapat
OREIGN TABLE ... PARTITION OF)? That seems a better way. Otherwise users might wonder whether we keep the partitions of a foreign table on the foreign server which won't be true. But then we allow foreign tables to have local tables as children in inheritance, so somebody from that back

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Ashutosh Bapat
On Fri, Apr 15, 2016 at 10:30 PM, Robert Haas wrote: > On Fri, Apr 15, 2016 at 5:46 AM, Ashutosh Bapat > wrote: > > Retaining the partition hierarchy would help to push-down join across > > partition hierarchy effectively. > > -1. You don't get to insert cruf

Re: [HACKERS] Odd system-column handling in postgres_fdw join pushdown patch

2016-04-15 Thread Ashutosh Bapat
On Fri, Apr 15, 2016 at 10:17 PM, Robert Haas wrote: > On Fri, Apr 15, 2016 at 12:16 PM, Ashutosh Bapat > wrote: > > The testcases had tableoid::regclass which outputs the foreign table's > local > > name, which won't change across runs. Isn't that so? >

<    2   3   4   5   6   7   8   9   10   >