Re: BUG #15572: Misleading message reported by "Drop function operation" on DB with functions having same name

2019-03-20 Thread David Rowley
that case too. > The new status of this patch is: Ready for Committer Great! -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services drop_func_if_not_exists_fix_v9.patch Description: Binary data

Re: Should we add GUCs to allow partition pruning to be disabled?

2019-03-20 Thread David Rowley
On Thu, 14 Mar 2019 at 02:10, Robert Haas wrote: > > On Tue, Mar 12, 2019 at 7:28 PM David Rowley > wrote: > > I think I've done that in the attached patch. > > Cool, thanks. Just so I don't forget about this, I've added it to the July 'fest.

Re: Removing unneeded self joins

2019-03-20 Thread David Rowley
als. Doing that could cause unique joins not to work when the join search is done via GEQO. I also think this way would give you the subquery GROUP BY / DISTINCT self join removal for just about free. However, there might be more cleanup to be done in that case... I've not thought about that t

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-21 Thread David Rowley
/* type of MCV list (BASIC) */ I see: #define STATS_MCV_TYPE_BASIC 1 /* basic MCV list type */ but it's not really clear to me what else could exist. Maybe the "type" comment can explain there's only one type for now, but more might exist in the future? 29. Looking at the

Re: BUG #15572: Misleading message reported by "Drop function operation" on DB with functions having same name

2019-03-21 Thread David Rowley
ue > HINT: Specify the argument list to select the procedure unambiguously. > > Arguably, because I said "drop procedure", there's no ambiguity here; > but we don't account for objtype while doing the lookup. Yeah. I went with reporting the objtype that was spec

Re: Performance issue in foreign-key-aware join estimation

2019-03-22 Thread David Rowley
for v12, then I'll happily look over your patch, otherwise, I'll look sometime before July. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: speeding up planning with partitions

2019-03-22 Thread David Rowley
pruning at all before doing any of the work that's necessary > for run-time to work. I had a quick look at the make_partitionedrel_pruneinfo() code earlier before this patch appeared and I agree that something like this could be done. I've not gone over the patch in detail though.

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
, for example, so setting them afterwards does not seem like an improvement. They also allow us to cost the cheaper startup cost properly, however, you did seem to argue that you have no confidence in cheap startup plans, which I'm still confused by. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Removing unneeded self joins

2019-03-22 Thread David Rowley
we are looking for agreement on the detection part, could you also > review the removal part? I'm sure it has its own share of problems. I'd rather focus on the detection method before reviewing the removal code. If there's some blocker in the detection code then the removal code is not useful. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
and your reasoning here. I could understand it if Append needed the sorts but MergeAppend did not, but they both need the sorts if there's not a path that already provides the required ordering. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
On Sat, 23 Mar 2019 at 04:56, Tom Lane wrote: > > David Rowley writes: > > Append has the additional > > saving of not having to determine to perform a sort on the top row > > from each subpath. > > Uh, what? sorted-Append and MergeAppend would need pre-sort

Re: Ordered Partitioned Table Scans

2019-03-22 Thread David Rowley
ee it's not pretty to lie about the startup cost, but it does kinda seem silly to fall back on a more expensive MergeAppend when we know fine well Append is cheaper. Probably the danger would be that someone pulls it out thinking its a bug. So we'd need to clearly comment why we're d

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2019-03-23 Thread David Rowley
On Sun, 24 Mar 2019 at 12:41, Tomas Vondra wrote: > > On 3/21/19 4:05 PM, David Rowley wrote: > > 11. In get_mincount_for_mcv_list() it's probably better to have the > > numerical literals of 0.0 instead of just 0. > > Why? Isn't it what we do for float and doub

Re: Ordered Partitioned Table Scans

2019-03-24 Thread David Rowley
anges to equivalence classes a few years ago in [1] and I failed to get that idea floating. In ways, this is similar as it requires having equivalence classes that are not used in all cases. I think to get something working a week before code cutoff is a step too far for this, but certainl

Re: Ordered Partitioned Table Scans

2019-03-24 Thread David Rowley
On Sat, 23 Mar 2019 at 19:42, Tom Lane wrote: > > David Rowley writes: > > On Sat, 23 Mar 2019 at 05:40, Tom Lane wrote: > >> BTW, another thing we could possibly do to answer this objection is to > >> give the ordered-Append node an artificially pessimistic star

Assert failure when validating foreign keys

2019-03-24 Thread David Rowley
] https://www.postgresql.org/message-id/CAK%3D1%3DWrnNmBbe5D9sm3t0a6dnAq3cdbF1vXY816j1wsMqzC8bw%40mail.gmail.com -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Removing unneeded self joins

2019-03-24 Thread David Rowley
On Sat, 23 Mar 2019 at 04:13, David Rowley wrote: > > On Sat, 23 Mar 2019 at 03:39, Alexander Kuzmenkov > > This is a bug indeed. Unique index search is not exhaustive, so if many > > indexes match the join quals, we might not find the same index for both > > sides. I th

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-03-25 Thread David Rowley
nce but we'll also have to obtain and release a lock for each partition for each execution of the prepared statement. LockReleaseAll() is going to be slow in that case because it actually has to release a lot of locks. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Removing [Merge]Append nodes which contain a single subpath

2019-03-25 Thread David Rowley
On Tue, 26 Mar 2019 at 09:00, Tom Lane wrote: > > David Rowley writes: > > [ drop-useless-merge-appends-15.patch ] > > Pushed with some minor adjustments. Thank for all your work on this, and thanks for the final push. FWIW, you should probably have credited yourself as the m

Re: Ordered Partitioned Table Scans

2019-03-25 Thread David Rowley
On Tue, 26 Mar 2019 at 09:02, Julien Rouhaud wrote: > FTR this patch doesn't apply since single child [Merge]Append > suppression (8edd0e7946) has been pushed. Thanks for letting me know. I've attached v14 based on current master. -- David Rowley http://www

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-03-26 Thread David Rowley
Without the patch, each transaction's LockReleaseAll() has to > scan the bloated large hash table. Oh. I think I see what you're saying. Really the table in #2 would have to be some completely different table that's not partitioned. I think in that case it should make a diff

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-03-26 Thread David Rowley
On Tue, 26 Mar 2019 at 21:55, David Rowley wrote: > > On Tue, 26 Mar 2019 at 21:23, Tsunakawa, Takayuki > wrote: > > Thank you David for explaining. Although I may not understand the effect > > of "speeding up planning with partitions" patch, this patch ta

Should the docs have a warning about pg_stat_reset()?

2019-03-26 Thread David Rowley
ge-id/CAKJS1f_YbXC2qTMPyCbmsPiKvZYwpuQNQMohiRXLj1r=8_r...@mail.gmail.com -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Ordered Partitioned Table Scans

2019-03-26 Thread David Rowley
mentioned in ddl.sgml (or > somewhere else)? I'm not too sure about this. We don't generally detail out planner optimisations in the docs. However, maybe it's worth users knowing about it as it may control their design choices of partition hierarchies. I'd just not

Re: speeding up planning with partitions

2019-03-26 Thread David Rowley
; expand the target inheritance without having created any RelOptInfos. Perhaps the way to make this work, at least in the long term is to do in the planner what we did in the executor in d73f4c74dd34. I'm not quite sure how exactly you'd know what size to make the array, but if it&#x

Re: speeding up planning with partitions

2019-03-26 Thread David Rowley
On Wed, 27 Mar 2019 at 18:39, Amit Langote wrote: > > On 2019/03/27 14:26, David Rowley wrote: > > Perhaps the way to make this work, at least in the long term is to do > > in the planner what we did in the executor in d73f4c74dd34. > > Maybe you meant 9ddef36278a9? Pr

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
a good solution to that. The problem with moving it to partbounds.c is that nothing there knows about RelOptInfo currently. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
s in RelOptInfo. Something like that would allow us to make this better. Julien and I have talked about other possible cases to optimise if we have that. e.g if the default partition is pruned then we can optimise a RANGE partitioned table with a default. So there's definitely more to be done on this. I think there's a general consensus that what we're doing in the patch already is enough to be useful. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Berserk Autovacuum (let's save next Mandrill)

2019-03-27 Thread David Rowley
ot;Search and Url tables". I can imagine "search" never needs any UPDATEs, so quite possibly those were append-only, in which case the anti-wraparound vacuum would have had quite a lot of work on its hands since possibly every page needed frozen. A table receiving regular au

Re: Should the docs have a warning about pg_stat_reset()?

2019-03-27 Thread David Rowley
ttention. Anti-wraparound vacuums on the current database may get neglected too. I'm not saying let's not do it, I'm just saying we need to think of what bad things could happen as a result of such a change. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Thu, 28 Mar 2019 at 14:34, Amit Langote wrote: > > On 2019/03/28 7:29, David Rowley wrote: > > On Wed, 27 Mar 2019 at 19:48, Amit Langote > > It does need to know how many partitions the partitioned table has, > > which it gets from partrel->nparts, so yeah, RelOpt

Re: Ordered Partitioned Table Scans

2019-03-27 Thread David Rowley
On Thu, 28 Mar 2019 at 15:40, David Rowley wrote: > Thanks for the review. I've attached a patch that mostly just moved > the code around. Here's one that fixes up the compiler warning from the last one. Thanks CF bot... -- David Rowley http://www

Re: COPY FROM WHEN condition

2019-03-28 Thread David Rowley
ach partition, we'll end up with quite a lot of slots. So, in short. I think the patch either is going to cause possible memory problems during COPY FROM, or it leaves code that is pretty much redundant. ... Or I've just misunderstood the patch :) Does this make sense? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Berserk Autovacuum (let's save next Mandrill)

2019-03-28 Thread David Rowley
On Thu, 28 Mar 2019 at 22:04, Darafei "Komяpa" Praliaskouski wrote: > > On Thu, Mar 28, 2019 at 2:36 AM David Rowley > wrote: >> I thought recently that it would be good to have some sort of >> pro-active auto-vacuum mode that made use of idle workers. > >

Re: Ordered Partitioned Table Scans

2019-03-28 Thread David Rowley
On Fri, 29 Mar 2019 at 00:00, Amit Langote wrote: > > On 2019/03/28 8:04, David Rowley wrote: > > If it's *always* scanned last then it's fine for ORDER BY partkey > > NULLS LAST. If they have ORDER BY partkey NULLS FIRST then we won't > > match on the

Re: COPY FROM WHEN condition

2019-03-31 Thread David Rowley
On Fri, 29 Mar 2019 at 01:15, Andres Freund wrote: > On 2019-03-28 20:48:47 +1300, David Rowley wrote: > > I had a look at this and performance has improved again, thanks. > > However, I'm not sure if the patch is exactly what we need, let me > > explain. > > I&#x

Re: speeding up planning with partitions

2019-03-31 Thread David Rowley
y best put in their own thread. As far as I know, nothing is broken with the code today. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: COPY FROM WHEN condition

2019-04-01 Thread David Rowley
nd-slotify-COPY.patch # copy listp from program $$perl ~/bench_same.pl$$ delimiter '|'; Time: 19234.960 ms (00:19.235) master + attached # copy listp from program $$perl ~/bench_same.pl$$ delimiter '|'; Time: 9064.802 ms (00:09.065) Of course, it is possible that some

Re: Change ereport level for QueuePartitionConstraintValidation

2019-07-22 Thread David Rowley
g, the votes so far appear to be: NOTICE: Robert, Amit DEBUG1: Tom, Alvaro (I'm entirely basing this on the fact that they mentioned possible ways to test with DEBUG1) I'll be happy with DEBUG1 if we can get tests to test it. -- David Rowley http://www.2ndQuad

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-22 Thread David Rowley
Otherwise, I plan to push it about this time tomorrow. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services shrink_bloated_locallocktable_v7.patch Description: Binary data

Re: Runtime pruning problem

2019-07-23 Thread David Rowley
On Sat, 25 May 2019 at 18:55, David Rowley wrote: > and an updated patch, rebased after the pgindent run. > > Hopefully, this will make the CF bot happy again. and rebased again due to a conflict with some List changes that touched ruleutils.c. I also made another couple of passes

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-23 Thread David Rowley
Thanks for having a look at this. On Wed, 24 Jul 2019 at 04:13, Tom Lane wrote: > > David Rowley writes: > > I'm pretty happy with v7 now. If anyone has any objections to it, > > please speak up very soon. Otherwise, I plan to push it about this > > time tomorrow.

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-23 Thread David Rowley
On Wed, 24 Jul 2019 at 15:05, David Rowley wrote: > To be able to reduce the threshold down again we'd need to make a > hash_get_num_entries(LockMethodLocalHash) call before performing the > guts of LockReleaseAll(). We could then weight that onto some running > average counter

Re: Change atoi to strtol in same place

2019-07-23 Thread David Rowley
that takes a lower and upper bound and sets some output param with the value and returns a bool to indicate if it's within range or not? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-07-24 Thread David Rowley
On Wed, 24 Jul 2019 at 16:16, David Rowley wrote: > > On Wed, 24 Jul 2019 at 15:05, David Rowley > wrote: > > To be able to reduce the threshold down again we'd need to make a > > hash_get_num_entries(LockMethodLocalHash) call before performing the > > guts of

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-07-24 Thread David Rowley
le before I can get a chance to look at it again. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-07-24 Thread David Rowley
On Thu, 25 Jul 2019 at 11:33, Andres Freund wrote: > > On 2019-07-25 10:36:26 +1200, David Rowley wrote: > > 2) Planner trying to give nodeAgg.c a sorted path to work with on > > DISTINCT / ORDER BY aggs > > That'll have to be a best effort thing though, i.e. the

Re: Runtime pruning problem

2019-07-30 Thread David Rowley
On Wed, 31 Jul 2019 at 10:27, Tom Lane wrote: > > David Rowley writes: > > The part I wouldn't mind another set of eyes on is the ruleutils.c > > changes. > > Um, sorry for not getting to this sooner. > > What I had in mind was to revert 1cc29fe7c'

Re: Runtime pruning problem

2019-07-30 Thread David Rowley
Append should say "part", not "part_p1". > In that way they'd look the same regardless of which partitions > have been pruned or not. That seems perfectly reasonable for Append / MergeAppend that are for scanning partitioned tables. What do you propose we d

Re: Tid scan improvements

2019-08-01 Thread David Rowley
uce the number of patches getting kicked out to the next CF all the time. Maybe some icons that appear if it's been waiting on author for more than 2 months, or if it went through an entire CF as waiting on author. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: POC: converting Lists into arrays

2019-08-08 Thread David Rowley
okay with the patch once that one improvement is done. I think if we want to think about freeing the 2nd input List then we can do that in another commit. Removing the redundant list_copy() calls seems quite separate from that. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: POC: converting Lists into arrays

2019-08-08 Thread David Rowley
tty good to remove code that could be a future bottleneck if we ever manage to do something else with the locking of all partitions during UPDATE/DELETE. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: POC: converting Lists into arrays

2019-08-09 Thread David Rowley
On Sat, 10 Aug 2019 at 09:03, Tom Lane wrote: > > David Rowley writes: > > On Fri, 9 Aug 2019 at 09:55, Tom Lane wrote: > >> I still have hopes for getting rid of es_range_table_array though, > >> and will look at that tomorrow or so. > > > Yes, pl

Re: Custom table AMs need to include heapam.h because of BulkInsertState

2019-08-13 Thread David Rowley
ded up with ExecGetRoutedToRelations. I'm open to better ideas. I also chose to leave the change of function signatures done in f7c830f1a in place. I don't think the additional now unused parameter is that out of place. Also, the function is inlined, so removing it wouldn't help pe

Re: Speed up transaction completion faster after many relations are accessed in a transaction

2019-08-14 Thread David Rowley
which might be costly in a bloated table. For now, I'm out of ideas. If anyone else feels like suggesting something of picking this up, feel free. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: Converting NOT IN to anti-joins during planning

2019-09-02 Thread David Rowley
unlikely I'll get to this in September. I'll kick it out to the next 'fest now. Antonin, Thank you for the review. I will respond to it when I get time again. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Runtime Partition Pruning

2018-01-01 Thread David Rowley
patching master with Amit's v17 faster partition pruning patch [1]. [1] https://www.postgresql.org/message-id/58c3e20a-a964-4fdb-4e7d-bd833e9be...@lab.ntt.co.jp -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services runtime_prune_drowley_v5.patch Description: Binary data

Re: [HACKERS] UPDATE of partition key

2018-01-01 Thread David Rowley
e memory for the structure, sets it up then returns a pointer to the new struct. That's not very unusual. It seems unusual for a function to return void and modify a single parameter pointer to get the value to the caller rather than just to return that value. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: TODO list (was Re: Contributing with code)

2018-01-02 Thread David Rowley
ing things because they need PostgreSQL to that, then they've not used the TODO list for that so is not a concern of this thread. I think the warning that's on the TODO list [1] is a great idea. Perhaps it should be bigger or more verbose. [1] https://wiki.postgresql.org/wiki/Todo --

Re: [HACKERS] UPDATE of partition key

2018-01-03 Thread David Rowley
d, target :-( 17. You still have the following comment in ModifyTableState but you've moved all those fields out to PartitionTupleRouting: /* Tuple-routing support info */ 18. Should the following not be just called partKeyUpdate (without the 'd')? bool partKeyUpdated; /* some part key in hierarchy updated */ This occurs in the planner were the part key is certainly being updated. 19. In pathnode.h you've named a parameter partColsUpdated, but the function in the .c file calls it partKeyUpdated. I'll try to look at the tests tomorrow and also do some testing. So far I've only read the code and the docs. Overall, the patch appears to look quite good. Good to see the various cleanups going in like the new PartitionTupleRouting struct. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] UPDATE of partition key

2018-01-03 Thread David Rowley
On 4 January 2018 at 02:52, David Rowley wrote: > I'll try to look at the tests tomorrow and also do some testing. So > far I've only read the code and the docs. There are a few more things I noticed on another pass I made today: 20. "carried" -> "carried out t

Re: [HACKERS] Runtime Partition Pruning

2018-01-04 Thread David Rowley
d) { printf("%.*f", 0, nfiltered / nloops); return 0; } but on your machine it must be outputting "0"? I'm not sure if I should just try to have even row numbers in the tests, or if we should be fixing EXPLAIN so it is consistent. -- David Rowley

Re: [HACKERS] Runtime Partition Pruning

2018-01-04 Thread David Rowley
was failing due to platform dependent behaviour in printf. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services runtime_prune_drowley_v5.patch Description: Binary data

Re: [HACKERS] path toward faster partition pruning

2018-01-04 Thread David Rowley
that already. [1] https://commitfest.postgresql.org/16/1353/ -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] path toward faster partition pruning

2018-01-04 Thread David Rowley
interest this patch should have in that problem. (The problem gets more complex again when doing run-time pruning, but that's not a topic for this thread) -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-04 Thread David Rowley
as parents of that index, then they'll be dropped. I'll go off and look at the code now... -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Removing useless DISTINCT clauses

2018-01-06 Thread David Rowley
's been a while since I looked at this. I remember thinking something similar at the time but I must have not changed it. I'll look again and post an updated patch. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Removing useless DISTINCT clauses

2018-01-06 Thread David Rowley
On 6 January 2018 at 23:08, David Rowley wrote: >> I think remove_functionally_dependant_groupclauses should have a more >> generic name, like remove_functionally_dependant_clauses. > > It's been a while since I looked at this. I remember thinking > something similar a

Re: [HACKERS] Removing [Merge]Append nodes which contain a single subpath

2018-01-06 Thread David Rowley
On 28 December 2017 at 18:31, David Rowley wrote: > I've attached a rebased patch. > > The previous patch was conflicting with parallel Hash Join (180428404) And another one. Thanks to the patch tester [1], I realised that I didn't make check-world and there was a postgre

Re: [HACKERS] Runtime Partition Pruning

2018-01-07 Thread David Rowley
On 7 January 2018 at 00:03, David Rowley wrote: > I've fixed this in the attached, but I did so by calling > adjust_appendrel_attrs() from the nodeAppend.c, which did, of course, > mean that the AppendRelInfo needed to be given to the executor. I was > also a bit unsure what ex

Re: [HACKERS] UPDATE of partition key

2018-01-08 Thread David Rowley
On 4 January 2018 at 02:52, David Rowley wrote: > I'll try to look at the tests tomorrow and also do some testing. I've made a pass over the tests. Again, sometimes I'm probably a bit pedantic. The reason for that is that the tests are not that easy to follow. Moving creat

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-08 Thread David Rowley
#x27;t do any harm, but is the following still needed? I originally thought this would have been for pg_index changes. What's changed? -#define CATALOG_VERSION_NO 201712251 +#define CATALOG_VERSION_NO 201712291 I admit to not quite being as thorough in my review with the pg_dump code. The tests seem fine, but I'd like to see a test for #14 once that's fixed. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Removing useless DISTINCT clauses

2018-01-09 Thread David Rowley
t; > I do not remember for sure at the moment, but it may be that this > correspondence is only important for the case of DISTINCT ON, in which > case we could dodge the problem by not applying the optimization unless > it's plain DISTINCT. That doesn't help us with point 1

Re: [HACKERS] Removing LEFT JOINs in more cases

2018-01-09 Thread David Rowley
On 10 January 2018 at 08:44, Tom Lane wrote: > I'll set the patch back to Waiting on Author. Many thanks for looking at this. I'll try to resolve the things you've mentioned this coming weekend. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] path toward faster partition pruning

2018-01-09 Thread David Rowley
abc_a2 for values in (2); select * from ab_c where a between 1 and 2 and b <= '2'; -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Runtime Partition Pruning

2018-01-09 Thread David Rowley
On 9 January 2018 at 22:22, Beena Emerson wrote: > ERROR: operator 1057 is not a member of opfamily 1976 Thanks for finding these. I'm looking into the above, and the other ones you've mentioned now. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Deve

Re: [HACKERS] Runtime Partition Pruning

2018-01-09 Thread David Rowley
pr. I'll report to Amit on the thread for that patch. I'll also investigate the duplication in RelOptInfo->partition_rels and report that in another thread. Can you confirm that case 1 and 2 are working with the attached? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services runtime_prune_drowley_v8.patch Description: Binary data

Re: [HACKERS] path toward faster partition pruning

2018-01-09 Thread David Rowley
OpExpr clauses, but does not do the same for leftop for the ScalarArrayOpExpr test. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] path toward faster partition pruning

2018-01-09 Thread David Rowley
d_int(partitioned_rels, lfirst_int(lc)); } Although it seems pretty fragile. It would probably be better to find a nicer way of handling all this. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] UPDATE of partition key

2018-01-10 Thread David Rowley
et is actually in a different paragraph. The reason it's > there is to ensure we have reset it regardless of the earlier cleanup. hmm, I was reviewing the .out file, which does not have the empty lines. Still seems a bit surplus. > Attached v35 patch. Thanks. Thanks. I'll try to

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-10 Thread David Rowley
t this case and post an updated version later. I guess CIC will need to check if the index has a parent index when setting indisvalid = true, and do likewise to the parent index if all other siblings are valid. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-11 Thread David Rowley
Thanks for addressing that list. Just one thing to reply on before I look at the updated version: On 11 January 2018 at 22:52, Amit Langote wrote: > On 2018/01/10 10:55, David Rowley wrote: >> One more thing I discovered while troubleshooting a bug Beena reported >> in the run

Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-11 Thread David Rowley
the run-time prune patch, here's case that break in v18. create table xy (a int, b text) partition by range (a,b); create table xy1 partition of xy for values from (0,'a') to (10, 'b'); select * from xy where a = 1 and b in('x','y'); ERROR: o

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-11 Thread David Rowley
On 12 January 2018 at 15:27, Amit Langote wrote: > On 2018/01/11 19:23, David Rowley wrote: >> ERROR: operator 531 is not a member of opfamily 1976 > > You'll be able to see that the error no longer appears with the attached > updated set of patches, but I'm now se

Re: [HACKERS] path toward faster partition pruning

2018-01-12 Thread David Rowley
On 10 January 2018 at 17:18, David Rowley wrote: > Basically, the changes to add_paths_to_append_rel() are causing > duplication in partition_rels. > > A test case is: > > create table part (a int, b int) partition by list(a); > create table part1 partition of part for valu

Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-12 Thread David Rowley
On 12 January 2018 at 22:51, Amit Langote wrote: > On 2018/01/12 18:09, David Rowley wrote: >> On 10 January 2018 at 17:18, David Rowley >> wrote: >>> Basically, the changes to add_paths_to_append_rel() are causing >>> duplication in partition_rels. > &g

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-15 Thread David Rowley
(a INT NOT NULL, b INT NOT NULL); CREATE TABLE p1 (a INT NOT NULL, b INT NOT NULL) INHERITS (p); CREATE INDEX ON p (a); > I'll set this as Ready for Committer now. Will set back to waiting on author until the above things are addressed. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] UPDATE of partition key

2018-01-15 Thread David Rowley
et too, however, that's likely much cheaper than the palloc() anyway. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: ALTER TABLE ADD COLUMN fast default

2018-01-15 Thread David Rowley
On 2 January 2018 at 05:01, Andrew Dunstan wrote: > New version of the patch that fills in the remaining piece in > equalTupleDescs. This no longer applies to current master. Can send an updated patch? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] Proposal: Local indexes for partitioned table

2018-01-16 Thread David Rowley
a partitioned table. > > That's a leftover from previous versions too. YAGNI principle says I > should remove it rather than activate it, I think, since the optimizer > is not going to use the data for anything. Agreed. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-16 Thread David Rowley
On 16 January 2018 at 21:08, Amit Langote wrote: > On 2018/01/12 12:30, David Rowley wrote: >> 8. The code in get_partitions_from_ne_clauses() does perform quite a >> few nested loops. I think a more simple way to would be to track the >> offsets you've seen in a Bitm

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-16 Thread David Rowley
tion_bounding_keys() to have to match each clause to a partition key again, and classify_partition_bounding_keys is even coded to ignore clauses that don't' match any key, so it makes me wonder what is match_clauses_to_partkey actually for? I'm going to stop reviewing there as if you remove match_cl

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-17 Thread David Rowley
On 17 January 2018 at 17:05, David Rowley wrote: > 6. Which brings me to; why do we need match_clauses_to_partkey at all? > classify_partition_bounding_keys seems to do all the work > match_clauses_to_partkey does, plus more. Item #3 above is caused by > an inconsistency between the

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-17 Thread David Rowley
d be in charge of building keyclauses_all. That way the remaining work during the executor would never need to match clauses to a partition key as they'd be in lists dedicated to each key. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-17 Thread David Rowley
On 18 January 2018 at 00:13, David Rowley wrote: > On 17 January 2018 at 23:48, Amit Langote wrote: >> I'm concerned that after your patch to remove >> match_clauses_to_partkey(), we'd be doing more work than necessary in >> some cases. For example, consider the

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-18 Thread David Rowley
have to follow very soon... Thanks for merging that in. I'll have a try at making this work tomorrow, although I'm not sure yet how much time I'll have to dedicate to it as I have a few other things to do too. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-19 Thread David Rowley
On 19 January 2018 at 16:00, Kyotaro HORIGUCHI wrote: > And I'd like to ask David to check out his mail environment so > that SPF record is available for his message. Will investigate -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Suppor

Re: [Sender Address Forgery]Re: pg_(total_)relation_size and partitioned tables

2018-01-20 Thread David Rowley
this outmost leaf should be listed. hmm, that's thoroughly confusing. Just in case anyone else is stuck on that, I just need to mention that a leaf is the does not have branches, in nature or computer science. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Developm

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-22 Thread David Rowley
Hi Amit , On 19 January 2018 at 04:03, David Rowley wrote: > On 18 January 2018 at 23:56, Amit Langote > wrote: >> So, I've been assuming that the planner changes in the run-time pruning >> patch have to do with selecting clauses (restriction clauses not >> containi

Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [Sender Address Forgery]Re: [HACKERS] path toward faster partition pruning

2018-01-23 Thread David Rowley
On 23 January 2018 at 23:22, Amit Langote wrote: > On 2018/01/23 15:44, David Rowley wrote: >> Attached is what I had in mind about how to do this. > > Thanks for the delta patch. I will start looking at it tomorrow. Thanks. I've been looking more at this and I've made

<    1   2   3   4   5   6   7   8   9   10   >