Re: STATISTICS retained in CREATE TABLE ... LIKE (INCLUDING ALL)?

2018-03-05 Thread David Rowley
Right now, there's nothing to do here, so we just copy the list. + */ +static void +transformExtendedStatistics(CreateStmtContext *cxt) +{ + cxt->alist = list_concat(cxt->alist, cxt->extstats); -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Runtime Partition Pruning

2018-04-25 Thread David Rowley
On 25 April 2018 at 05:10, Alvaro Herrera <alvhe...@alvh.no-ip.org> wrote: > Pushed. Thanks! Thanks! -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Runtime Partition Pruning

2018-04-25 Thread David Rowley
st(a); create table lp_1 partition of lp for values in(1); create table lp_2 partition of lp for values in(2); create function lp_value(p_a int) returns int as $$ select value from lp where a = p_a; $$ language sql; insert into lp values(1,10),(2,20); select sum(lp_value(x)) from generate_Series(1,2)

Re: Generating partitioning tuple conversion maps faster

2018-06-28 Thread David Rowley
On 29 June 2018 at 11:10, David Rowley wrote: > On further inspection, the slowdown is coming from the planner in > make_inh_translation_list(). The INSERT path does not hit that since > the planner's job is pretty simple for simple INSERTs. I've attached a patch that uses SearchSysCac

Re: Regression on PostgreSQL 10 ORDER/GROUP BY expression not found in targetlist

2018-06-29 Thread David Rowley
ere edge_id = 1 > ORDER BY 1; Can you try in REL_10_STABLE? There was a fix committed [1] for something very similar just a week ago. [1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=a4c95b0b80c70677c09c0d5c82a6fba875160288 -- David Rowley http://www.2ndQuadran

Re: Generating partitioning tuple conversion maps faster

2018-06-28 Thread David Rowley
ttnum does not have the same name. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Performance of the partitioning in the large scale

2018-09-27 Thread David Rowley
wise than a generic plan with run-time partition pruning. Unless you're running with plan_cache_mode = 'force_generic_plan' then the overhead of the partitioned cases likely includes planning too. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Some incorrect comments and out-dated README from run-time pruning

2018-09-27 Thread David Rowley
for one for each corresponding plan node. This is no longer completely true. Some Append / MergeAppend subnodes may be skipped when performing run-time pruning at executor startup. I thought it might be best to mention that in the README. Patch attached. -- David Rowley http://www

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-09-27 Thread David Rowley
On Thu, 20 Sep 2018 at 11:31, Andres Freund wrote: > > On 2018-09-19 12:06:47 +0900, Michael Paquier wrote: > > On Wed, Sep 19, 2018 at 01:14:10PM +1200, David Rowley wrote: > > > Wouldn't it be better to modify copy.c to just perform the heap_sync > > > on

Re: Small performance tweak to run-time partition pruning

2018-10-07 Thread David Rowley
On 7 September 2018 at 19:29, David Rowley wrote: > While reviewing some other patches to improve partitioning performance > I noticed that one of the loops in ExecFindInitialMatchingSubPlans() > could be coded a bit more efficiently. The current code loops over > all the origi

Re: executor relation handling

2018-10-07 Thread David Rowley
he partitioned table, all the way to the top of the hierarchy. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: executor relation handling

2018-10-07 Thread David Rowley
ne in PG12, then the RTE_DUMMYs would not exist, as we'd only have RTEs in the range table for partitions that survived plan-time pruning. It also leaves a problem to solve in the unneeded locks being taken on partitions for PREPAREd queries using run-time pruning. [1] https://commitfest.postg

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-05 Thread David Rowley
On 17 September 2018 at 21:15, David Rowley wrote: > v9 patch attached. Fixes conflict with 6b78231d. v10 patch attached. Fixes conflict with cc2905e9. I'm not so sure we need to zero the partition_tuple_slots[] array at all since we always set a value there is there's a corresponding

Re: Calculate total_table_pages after set_base_rel_sizes()

2018-10-07 Thread David Rowley
On 6 October 2018 at 18:20, Edmund Horner wrote: > David Rowley said: >> I am considering this a bug fix, but I'm proposing this for PG12 only >> as I don't think destabilising plans in the back branches is a good >> idea. I'll add this to the September commitfest. >

Small run-time pruning doc fix

2018-10-08 Thread David Rowley
mention: Both of these behaviors are likely to be changed in a future release of PostgreSQL. The attached patch updates this to mention that ModifyTable is unsupported which I think makes the above fragment make sense again. -- David Rowley http://www

Re: Soon-to-be-broken regression test case

2018-10-11 Thread David Rowley
If people are writing that then probably they're doing so to trick the planner, perhaps to hide some stats that get outdated easily. I'd imagine we'd upset more people than we'd please. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Calculate total_table_pages after set_base_rel_sizes()

2018-10-12 Thread David Rowley
On 12 October 2018 at 23:35, Amit Langote wrote: > On 2018/10/11 13:45, Amit Langote wrote: >> On 2018/10/07 17:43, David Rowley wrote: >>> Amit Langote has since posted a patch to delay the RangeTblEntry >>> creation until after pruning. His patch happens to also mov

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-22 Thread David Rowley
; need quite a number of changes but given the weight of this change maybe it > could be considered? It's very unlikely to happen, especially so with the 0002 patch, which I've so far just attached as a demonstration of where the performance could end up. -- David Rowley ht

Re: Small performance tweak to run-time partition pruning

2018-10-22 Thread David Rowley
other bottlenecks are removed. More partitions may show a larger increase, but more partitions also means that a larger range table array gets built during ExecInitRangeTable(), which is also slow. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Tr

Re: Multi-insert into a partitioned table with before insert row trigger causes server crash on latest HEAD

2018-10-19 Thread David Rowley
d from this problem. I'm not sure that doubling up the `insertMethod == CIM_MULTI_CONDITIONAL` test is the cleanest fix. Personally, I liked the way it was in the v6 edition of the patch, but I'm used to getting outvoted. [1] https://www.postgresql.org/message-id/CAKJS1f9f8yuj04X_rffNu2JPbvhy+YP_aVH6iw

A small tweak to some comments for PartitionKeyData

2018-10-24 Thread David Rowley
to mention how they should be used in the comments. Patch attached. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services reword_comments_in_partitionkeydata_struct.patch Description: Binary data

Re: A small tweak to some comments for PartitionKeyData

2018-10-24 Thread David Rowley
uot; on the end, but I didn't think it was worth having the extra line caused by the 80 line length limit, so I deleted it. Seems "element" is mentioned 73 times in [1], but "member" does not get a mention. [1] https://en.wikipedia.org/wiki/Array_data_structure --

Re: A small tweak to some comments for PartitionKeyData

2018-10-24 Thread David Rowley
On 25 October 2018 at 17:05, Amit Langote wrote: > On 2018/10/25 12:54, David Rowley wrote: >> On 25 October 2018 at 16:46, Amit Langote >> wrote: >>> +* key, one >>> for each zero-valued partatt

Re: Log timestamps at higher resolution

2018-10-24 Thread David Rowley
ater. I don't think we have any variables that zero padding would be that useful for, but maybe someone might want to zero pad their pids or xids... ? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Ordered Partitioned Table Scans

2018-10-28 Thread David Rowley
me yet to give it a throughout post write review, but on first look it seems okay. The known limitations are: * Disables the optimisation even if the DEFAULT partition is pruned. * Disables the optimisation if LIST partitioned tables have any partitions allowing > 1 value. * Fails to optimise UNION ALLs w

Re: Ordered Partitioned Table Scans

2018-10-29 Thread David Rowley
On 29 October 2018 at 13:44, David Rowley wrote: > v2 of the patch is attached. I've not had time yet to give it a > throughout post write review, but on first look it seems okay. Added to the November 'fest. https://commitfest.postgresql.org/20/1850/ -- David Rowley

Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread David Rowley
er 'fest. Likely I'll decided that based on any feedback received before then. The "superkey" name is borrowed from OOP. Think "superclass". If it turns out not to be a dud, we can call it something else. [1] https://commitfest.postgresql.org/20/1850/ -- David Rowley

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-31 Thread David Rowley
Agreed. That's a good idea. Thanks. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: PostgreSQL Limits and lack of documentation about them.

2018-10-31 Thread David Rowley
;> I'm not so sure about detailing limits of GUCs since the limits of >> those are mentioned in pg_settings. > > Maybe we could just have a link to that section in the docs. That's likely a good idea. I was just unable to find anything better than the link to the pg_settings view

Re: Ordered Partitioned Table Scans

2018-10-31 Thread David Rowley
t into a separate function. It does. It's just done via the call to cost_append(). > * Likewise, create_append_plan() / create_merge_append_plan() are going to be > more similar then before, so some refactoring could also make sense. > > Although it's not too much code, I

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-31 Thread David Rowley
ions for everything that's possible to implement. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-10-31 Thread David Rowley
ged onto the end. > Other than that the patch looks fine and works as expected. > > The new status of this patch is: Waiting on Author Thanks for looking at this. I've attached an updated patch. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services fix_incorrect_setting_of_hi_options_for_partitioned_tables_v3.patch Description: Binary data

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-31 Thread David Rowley
but how would we know that was better. There might be an index on extract('year', dt). -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-31 Thread David Rowley
On 1 November 2018 at 13:35, Amit Langote wrote: > On 2018/11/01 8:58, David Rowley wrote: >> On 1 November 2018 at 06:45, Robert Haas wrote: >>> I think a better way to shorten the name would be to truncate the >>> PartitionTupRouting() prefix in some way, e.g. droppi

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-31 Thread David Rowley
On 1 November 2018 at 06:45, Robert Haas wrote: > On Wed, Aug 22, 2018 at 8:30 AM David Rowley > wrote: >> On 22 August 2018 at 19:08, Amit Langote >> wrote: >> > +#define PartitionTupRoutingGetToParentMap(p, i) \ >> > +#define PartitionTupRoutingGetToCh

Re: PostgreSQL Limits and lack of documentation about them.

2018-10-30 Thread David Rowley
On 26 October 2018 at 11:40, Haribabu Kommi wrote: > On Fri, Oct 26, 2018 at 9:30 AM David Rowley > wrote: >> >> For a long time, we documented our table size, max columns, max column >> width limits, etc. in https://www.postgresql.org/about/ , but that >> info

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread David Rowley
On 31 October 2018 at 08:52, Tom Lane wrote: > David Rowley writes: >> I've started working on something I've ended up calling "Super >> PathKeys". The idea here is to increase the likelihood of a Path with >> PathKeys being used for a purpose that requires

Re: Ordered Partitioned Table Scans

2018-10-30 Thread David Rowley
On 31 October 2018 at 12:24, Julien Rouhaud wrote: > On Mon, Oct 29, 2018 at 1:44 AM David Rowley > wrote: >> >> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> > I just had a look at your patch. I see that you implemented only a >> > subset of the p

Re: Super PathKeys (Allowing sort order through precision loss functions)

2018-10-30 Thread David Rowley
are actually there for that sort of thing and I've seen WHERE indexcol+0 = used many times to disable the use of indexes, so making pathkeys see through those might be more annoying than useful... But it's a thought... -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v2-0001-Allow-Pathkeys-to-derive-their-order-from-a-paren.patch Description: Binary data

Re: Ordered Partitioned Table Scans

2018-10-31 Thread David Rowley
On 31 October 2018 at 13:05, David Rowley wrote: >>> On 28 October 2018 at 03:49, Julien Rouhaud wrote: >> I've registered as a reviewer. I still didn't have a deep look at >> the patch yet, but thanks a lot for working on it! > > Thanks for signing up to revi

PostgreSQL Limits and lack of documentation about them.

2018-10-25 Thread David Rowley
of 32 TB table size limit. I don't have a patch, but I propose we include this information in the docs, perhaps on a new page in the preface part of the documents. Does anyone else have any thoughts about this? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-11-03 Thread David Rowley
On 2 November 2018 at 15:43, Steve Singer wrote: > I am happy with the changes. > I think the patch is ready for a committer. Many thanks for your review. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-04 Thread David Rowley
On 1 November 2018 at 22:39, Amit Langote wrote: > On 2018/11/01 10:30, David Rowley wrote: >> It's great to know the patch is now so perfect that we've only the >> macro naming left to debate ;-) > > I looked over v12 again and noticed a couple minor issues. > > + *

Re: Small run-time pruning doc fix

2018-11-03 Thread David Rowley
ng run-time pruning for ModifyTable will require the planner to use the new partition pruning code. There's a patch for that in [1], so maybe it'll all be a little less confusing in PG12. [1] https://commitfest.postgresql.org/20/1778/ -- David Rowley http://www.2ndQuadrant.com/

Ordered Partitioned Table Scans

2018-10-25 Thread David Rowley
? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v1-0001-Allow-Append-to-be-used-in-place-of-MergeAppend-f.patch Description: Binary data

Re: Ordered Partitioned Table Scans

2018-10-25 Thread David Rowley
n't aware, or ... forgot. Looks like back then was tricky times to be doing this. Hopefully, the dust has settled a little bit now. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Multi-insert into a partitioned table with before insert row trigger causes server crash on latest HEAD

2018-10-20 Thread David Rowley
stgresql.org/message-id/CAKJS1f9f8yuj04X_rffNu2JPbvhy+YP_aVH6iwCTJ1OL=yw...@mail.gmail.com Oops. I was completely wrong about all of what I said above. Seems my analysis was rushed and very incorrect. Thanks for writing the patch Ashutosh and thanks Peter for committing the modified version of it. --

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-11-07 Thread David Rowley
9-6151-728901897b79%40redhat.com [2] https://www.postgresql.org/message-id/CAKJS1f_1RJyFquuCKRFHTdcXqoPX-PYqAd7nz%3DGVBwvGh4a6xA%40mail.gmail.com -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-11-07 Thread David Rowley
tition table could receive the tuple regardless of what might have been pruned. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Should new partitions inherit their tablespace from their parent?

2018-11-07 Thread David Rowley
-id/flat/CAKJS1f9PXYcT%2Bj%3DoyL-Lquz%3DScNwpRtmD7u9svLASUygBdbN8w%40mail.gmail.com#089ce41fe9a33c340f7433e5f0018912 -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Calculate total_table_pages after set_base_rel_sizes()

2018-11-07 Thread David Rowley
On 8 November 2018 at 06:17, Tom Lane wrote: > Pushed with cosmetic adjustments. Thanks for adjusting and pushing. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: PostgreSQL Limits and lack of documentation about them.

2018-11-07 Thread David Rowley
e first. Hopefully I I've addressed the other things mentioned too. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services Title: Appendix B. Database Limitations Appendix B. Database LimitationsPrev UpPart VIII. AppendixesH

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-08 Thread David Rowley
pt just > one line: > > + * Initially we must only setup 1 PartitionDispatch object; the one for > > setup -> set up Changed too. I've attached v15 and a delta from v14 to ease re-review. I also ran pgindent on this version. That's not part of the delta but is in the main patch. -- David

Re: Removing unneeded self joins

2018-11-07 Thread David Rowley
joins we process to control the quadratic behavior. */ + if (n > join_collapse_limit) + break; 6. In remove_self_joins_one_level() I think you should collect the removed relations in a Relids rather than a list. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Performance improvements of INSERTs to a partitioned table

2018-11-10 Thread David Rowley
can document that. Perhaps it's good enough. A guess there's at least a workaround of doing LOCK TABLE in the CREATE INDEX / TRUNCATE session. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-11-08 Thread David Rowley
and lack of toast table on pg_class. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-08 Thread David Rowley
On 9 November 2018 at 00:28, David Rowley wrote: > I've attached v15 and a delta from v14 to ease re-review. I just revived the 0002 patch, which is still just for testing at this stage. There was mention over on [1] about removing the find_all_inheritors() call. Also some benchmarks from

Re: speeding up planning with partitions

2018-11-10 Thread David Rowley
lace shows a number of problems. Likely I should only trigger the warning when bms_membership(em->em_relids) == BMS_SINGLETON, but it never-the-less appears to highlight various possible issues. Applying the same on master only appears to show the cases where em->em_relids isn't a singleton

Re: Performance improvements of INSERTs to a partitioned table

2018-11-08 Thread David Rowley
Pcv1J_C-mG%3DCs2UwhsD6cqwg%40mail.gmail.com [2] https://www.postgresql.org/message-id/CA%2BTgmoZGJsy-nRFnzurhZQJtHdDh5fzJKvbvhS0byN6_46pB9Q%40mail.gmail.com [3] https://commitfest.postgresql.org/20/1778/ -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Cache relation sizes?

2018-11-08 Thread David Rowley
we get [1] as we'll no longer ask for the size of any pruned partitions. Queries that don't prune any partitions are less likely to notice the extra overhead of the lseek(SEEK_END) since they've probably got more work to do elsewhere. [1] https://commitfest.postgresql.org/20/1778/ -- David Rowl

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-12 Thread David Rowley
tuple to the partition. [1] https://www.postgresql.org/message-id/25C1C6B2E7BE044889E4FE8643A58BA963B5796B@G01JPEXMBKW03 -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v16-0001-Speed-up-INSERT-and-UPDATE-on-partition

Re: PostgreSQL Limits and lack of documentation about them.

2018-11-13 Thread David Rowley
On 13 November 2018 at 19:46, John Naylor wrote: > On 11/8/18, David Rowley wrote: >> On 8 November 2018 at 22:46, Peter Eisentraut >> wrote: >>> Could you adjust this to use fewer capital letters, unless they start >>> sentences or similar? >> >>

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-07 Thread David Rowley
On 5 November 2018 at 20:17, Amit Langote wrote: > On 2018/11/04 19:07, David Rowley wrote: >> Perhaps a better design would be to instead of having random special >> partitioned-table-only fields in ResultRelInfo, just have an extra >> struct there that contains the extra

Re: Tid scan improvements

2018-11-05 Thread David Rowley
); + if (tuple) + ExecStoreBufferHeapTuple(tuple, /* tuple to store */ + slot, /* slot to store in */ + scandesc->rs_cbuf); /* buffer associated + * with this tuple */ + else + ExecClearTuple(slot); + + return slot; 0003: Saw nothing wrong: 0004: Not yet reviewed. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-13 Thread David Rowley
dPartition(). There's now no need to check if the map ! NULL as if the slot is there then the map must exist too. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services v17-0001-Speed-up-INSERT-and-UPDATE-on-partitione

Re: [HACKERS] Decimal64 and Decimal128

2018-11-12 Thread David Rowley
On 13 November 2018 at 10:39, Thomas Munro wrote: > ... and it has just been voted into the next revision of the C language: > > https://gustedt.wordpress.com/2018/11/12/c2x/ Nice. Maybe we can get DECFLOAT into core around PostgreSQL 32 or so :-) -- David Rowley h

Re: Small run-time pruning doc fix

2018-11-12 Thread David Rowley
On 13 November 2018 at 02:46, Peter Eisentraut wrote: > Committed your change. Thanks. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-11-14 Thread David Rowley
t think either is wrong, but I guess something must be misleading, so could perhaps be improved. We're simply allocating enough space for PARTITION_ROUTING_INITSIZE but we're only initialising 1 item. That leaves space for PARTITION_ROUTING_INITSIZE - 1 more items before we'd need to reallocate

Re: Ordered Partitioned Table Scans

2018-11-04 Thread David Rowley
ds, we can't rely on a failure to create the pathkey meaning that a more strict sort order is not required. I'm a bit unsure on how safe it would be to pass "create_it" as true to make_pathkey_from_sortinfo(). We might be building partition path keys for some sub-partitioned table. In this case the eclass should likely have a its member added with em_is_child = true. The existing code always sets em_is_child to false. It's not that clear to me that setting up a new eclass with a single em_is_child = true member is correct. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Small performance tweak to run-time partition pruning

2018-10-08 Thread David Rowley
o be possible to form the loop so that it calls bms_next_member() then store the result and loop until we reach that number. That would only save the bms_is_member call per loop, but I'd much rather do the array idea as it should also speed up plenty of other things. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-10-09 Thread David Rowley
On 9 October 2018 at 15:49, Amit Langote wrote: > On 2018/10/05 21:55, David Rowley wrote: >> I'm not so sure we need to zero the partition_tuple_slots[] array at >> all since we always set a value there is there's a corresponding map >> stored. I considered pulling that o

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-09-30 Thread David Rowley
On 28 September 2018 at 19:22, David Rowley wrote: > I've edited that in the attached patch. Also reworded a comment that > Amit mentioned and made a small change to the COPY FREEZE docs to > mention no support for partitioned tables. Added to the November 'fes

Re: Some incorrect comments and out-dated README from run-time pruning

2018-09-30 Thread David Rowley
On 28 September 2018 at 09:20, David Rowley wrote: > I've noticed that the comments above the PartitionedRelPruneInfo > struct incorrectly document how subplan_map and subpart_map are > indexed. This seems to have snuck in on 4e232364033. > > Also, while reading the executor

Re: Tid scan improvements

2018-10-02 Thread David Rowley
.00 rows=1 width=10) (actual time=0.025..0.065 rows=200 loops=1) TID Cond: (ctid <= '(0,200)'::tid) Planning Time: 0.081 ms Execution Time: 0.088 ms (4 rows) You can likely add on "(offset / avg_tuples_per_page) / rel->pages" to the selectivity and get a fairly accurate estimate... at least when there are no dead tuples in the heap -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-09-27 Thread David Rowley
performed a FREEZE for some partitions but not others. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: heap_sync seems rather oblivious to partitioned tables (wal_level=minimal)

2018-09-28 Thread David Rowley
On 28 September 2018 at 15:12, Michael Paquier wrote: > On Fri, Sep 28, 2018 at 02:46:30PM +1200, David Rowley wrote: >> I don't agree that we can skip explaining why one of the optimisations >> can't be applied just because we've explained why a similar >> optimisat

Re: executor relation handling

2018-09-28 Thread David Rowley
ly should be changed to avoid confusion. Apart from that, I see nothing wrong with the patches, so I think we should get someone else to look. I'm marking it as ready for committer. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Removing LEFT JOINs in more cases

2018-09-28 Thread David Rowley
ext 'fest. Cheers -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: executor relation handling

2018-09-28 Thread David Rowley
On 28 September 2018 at 20:28, Amit Langote wrote: > On 2018/09/28 17:21, David Rowley wrote: >> I think we maybe should switch the word "assert" for "verifies". The >> Assert is just checking we didn't get a NoLock and I don't think >> you're using "

Re: Small performance tweak to run-time partition pruning

2018-10-10 Thread David Rowley
On 11 October 2018 at 16:00, Imai, Yoshikazu wrote: > On Thu, Sept 6, 2018 at 7:30 PM, David Rowley wrote: >> I've also included an additional test to ensure the other_subplans >> gets updated correctly. The other tests for this seem to only perform >> run-time pruning du

Re: Subplan result caching

2018-10-01 Thread David Rowley
On 19 July 2018 at 06:27, Robert Haas wrote: > On Mon, Jul 9, 2018 at 5:08 AM, David Rowley >> "LazyMaterialize" seems like a good option for a name. It seems better >> than "LazyHash" since you may not want to restrict it to a hash table >> based cache

Re: [HACKERS] Secondary index access optimizations

2018-10-04 Thread David Rowley
On 4 October 2018 at 22:11, Konstantin Knizhnik wrote: > On 04.10.2018 06:19, David Rowley wrote: >> Please, can you also add a test which tests this code which has a >> partition with columns in a different order than it's parent. Having >> an INT and a TEXT column is best a

Re: [HACKERS] Secondary index access optimizations

2018-10-03 Thread David Rowley
st causes a crash if you temporarily put the code back to using the untranslated qual. Thanks for working on this. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services skip_implied_child_quals_allpaths.diff Description: Binary data

Re: [HACKERS] Secondary index access optimizations

2018-10-07 Thread David Rowley
ng about suggesting set_append_rel_size() as a good place to remove these quals. It should perhaps be done later, or maybe we can add some sort of marker to the qual to say it does not need to be enforced during execution. Probably the former would be best as we don't want to show these in EXPLAIN. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Some incorrect comments and out-dated README from run-time pruning

2018-10-09 Thread David Rowley
I don't see any PartitionDesc referenced or > mentioned at or near that struct. Perhaps it should have mentioned: the PartitionDesc belonging to the partitioned table referenced by 'rtindex'. I've attached another version. -- David Rowley http://www.2ndQuadrant.com/ Pos

Re: executor relation handling

2018-10-01 Thread David Rowley
he same lock level as the old code. Of course, I might be wrong. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: typcache.c typos

2018-09-02 Thread David Rowley
to change the existing wording. > > +1. "To speed up lookup" would be OK but I don't think "to speed > lookup" is wrong, and "to speedup lookup" does sound wrong. Not sure where changing it to "to speedup lookup" was proposed. In any case, it s

Re: Make executor's Range Table an array instead of a List

2018-09-04 Thread David Rowley
On 4 September 2018 at 19:31, Amit Langote wrote: > On 2018/08/24 7:22, David Rowley wrote: >> On 24 August 2018 at 02:26, Amit Langote wrote: >>> One of the patches I sent last week does the same thing, among a >>> couple of other things with regard to handling re

Re: Calculate total_table_pages after set_base_rel_sizes()

2018-09-25 Thread David Rowley
t; // NEW CODE > > /* Generate access paths. */ > set_base_rel_pathlists(root); Thanks for looking at this. I've changed that in the attached updated patch. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Error while executing initdb...

2018-12-31 Thread David Rowley
he first line of the output, so probably what's going on is you're printing out your additional line even when postgres is called with -V or --version... Would it not be easier just not to do that? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: slight tweaks to documentation about runtime pruning

2018-12-31 Thread David Rowley
On Tue, 18 Dec 2018 at 03:49, Alvaro Herrera wrote: > Pushed, thanks. I just noticed that this is still open on the CF app. Marking as committed... -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: rewrite ExecPartitionCheckEmitError

2018-12-31 Thread David Rowley
On Sat, 29 Dec 2018 at 06:52, Alvaro Herrera wrote: > Point taken; pushed with that change. Marking this as committed in the CF app. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Using Btree to Provide Sorting on Suffix Keys with LIMIT

2018-12-29 Thread David Rowley
the current status, other than not yet committed. [1] https://www.postgresql.org/message-id/flat/7f70bd5a-5d16-e05c-f0b4-2fdfc8873489%40BlueTreble.com [2] https://www.postgresql.org/message-id/flat/CAPpHfds1waRZ=NOmueYq0sx1ZSCnt+5QJvizT8ndT2=etze...@mail.gmail.com -- David Rowley http

Re: Performance issue in foreign-key-aware join estimation

2018-12-29 Thread David Rowley
ing shows that there's still a big regression to what the performance is like without the foreign keys. (Average of EXPLAIN over 60 seconds) foreign key qual matching code commented out: 2486.204 ms Master: 13909.551 ms v1 patch: 7310.719 ms -- David Rowley http://www.2ndQuadrant.com/ P

Re: START/END line number for COPY FROM

2019-01-05 Thread David Rowley
ike a bit of a waste of effort, and effort that wouldn't be required if the COPY TO had been done in multiple pieces. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: pg_dump multi VALUES INSERT

2019-01-02 Thread David Rowley
On Thu, 3 Jan 2019 at 01:50, Surafel Temesgen wrote: > On Mon, Dec 31, 2018 at 12:38 PM David Rowley > wrote: >> Just looking at the v5 patch, it seems not to handle 0 column tables >> correctly. > The attach patch contain a fix for it + /* if it is zero-column

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2019-01-06 Thread David Rowley
ScanKeyword struct. What I'm most interested in is how long it took to generate the hash function in hash2.c? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: "SELECT ... FROM DUAL" is not quite as silly as it appears

2019-01-05 Thread David Rowley
06.681296 (excluding connections establishing) Patched: tps = 9845.451081 (excluding connections establishing) tps = 9936.377521 (excluding connections establishing) tps = 9915.724816 (excluding connections establishing) (0.21% drop) -- David Rowley http://www.2ndQuadrant.com/ P

Re: Ordered Partitioned Table Scans

2019-01-05 Thread David Rowley
On Thu, 20 Dec 2018 at 18:20, Julien Rouhaud wrote: > > On Wed, Dec 19, 2018 at 11:08 PM David Rowley > wrote: > > create table listp (a int) partition by list (a); > > create table listp12 partition of listp for values in(1,2); > > create table listp03 partition

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-01-16 Thread David Rowley
e people must be pronouncing it, somehow!? surf... serf... ? easier, there's a SRF returning detailed information about the MCV lists. * Is it better to put a working SQL in here? SELECT * FROM pg_mcv_list_items(stxmcv); maybe like: SELECT s.* FROM pg_statistic_ext, LATERAL pg_mcv_list_items(stxmcv) s;

<    1   2   3   4   5   6   7   8   9   10   >