Re: [HACKERS] Combining Aggregates

2016-03-19 Thread David Rowley
On 18 March 2016 at 13:39, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > On Thu, Mar 17, 2016 at 10:59 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com> wrote: >> Hi, >> >> On 03/17/2016 12:53 PM, David Rowley wrote: >>> >> ... >>>

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
> +} > + > +static Node * > +fix_combine_agg_expr_mutator(Node *node, > fix_upper_expr_context *context) > > Don't we want to handle the case of context->subplan_itlist->has_non_vars as > it is handled in fix_upper_expr_mutator()? If not then, I think adding the > r

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread David Rowley
On 17 March 2016 at 16:30, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > On Wed, Mar 16, 2016 at 10:08 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> On 16 March 2016 at 23:54, Haribabu Kommi <kommi.harib...@gmail.com> wrote: >>> On Wed

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
On 19 March 2016 at 06:15, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Mar 18, 2016 at 9:16 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> I've attached an updated patch. > > This looks substantially better than earlier versions, although I

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
at change now. > 6. > + /* XXX this causes some redundant cost calculation ... */ > + input_target = set_pathtarget_cost_width(root, > input_target); > + return input_target; > > Can't we use return set_pathtarget_cost_width() directly rather than > fetching it in inp

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
On 17 March 2016 at 18:05, David Rowley <david.row...@2ndquadrant.com> wrote: > Updated patch attached. Please disregard 0001-Allow-aggregation-to-happen-in-parallel_2016-03-17.patch. This contained a badly thought through last minute change to how the Gather path is generated and is b

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
On 17 March 2016 at 00:57, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Mar 16, 2016 at 6:49 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> On 16 March 2016 at 15:04, Robert Haas <robertmh...@gmail.com> wrote: >>> I don't think I'd be obj

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
we do find and put them into partial mode, * this adjusts the Aggref's return type so that the partially calculated * aggregate value can make its way up the execution tree up to the Finalize * Aggregate node. */ I will post an updated patch once I've addressed Amit's points. -- David Rowley

Re: [HACKERS] Parallel Aggregate

2016-03-19 Thread David Rowley
other purposes. Parallel Aggregate is just one of many possible use cases for this, so it makes little sense to give it a name according to a single use case. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent vi

Re: [HACKERS] Choosing parallel_degree

2016-03-19 Thread David Rowley
setting up all the > parallel stuff takes time. It would be really nice if it were possible to drop the setting really low, so that combined with a low parallel_setup_cost we could enable parallel query on small tables in the regression test suite. -- David Rowley http://w

Re: [HACKERS] Parallel Aggregate

2016-03-18 Thread David Rowley
On 18 March 2016 at 01:22, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Thu, Mar 17, 2016 at 10:35 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> >> On 17 March 2016 at 01:19, Amit Kapila <amit.kapil...@gmail.com> wrote: >> > Few

Re: [HACKERS] Parallel Aggregate

2016-03-18 Thread David Rowley
On 19 March 2016 at 05:46, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Mar 16, 2016 at 5:05 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >>> Cool! Why not initialize aggpartialtype always? >> >> Because the follow-on p

Re: [HACKERS] Parallel Aggregate

2016-03-18 Thread David Rowley
generator. This is because the combine Aggref->args still point to "num", instead of sum(num). > Back at make_partialgroup_input_target, the comment says "so that they > can be found later in Combine Aggregate nodes during setrefs". I think > it's better to be

Re: [HACKERS] Combining Aggregates

2016-03-16 Thread David Rowley
On 16 March 2016 at 23:54, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > On Wed, Mar 16, 2016 at 8:34 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> Yes me too, so I spent several hours yesterday writing all of the >> combine functions and

Re: [HACKERS] Parallel Aggregate

2016-03-16 Thread David Rowley
the newly created Var instead of doing that in fix_combine_agg_expr_mutator(), as the last version did. Thanks for the suggestion. New patch attached. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001-Allow-aggr

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 14:08, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Mar 15, 2016 at 8:55 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> On 16 March 2016 at 13:42, Robert Haas <robertmh...@gmail.com> wrote: >>> On Tue, Mar 15, 2016

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 13:42, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Mar 15, 2016 at 8:04 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> On 16 March 2016 at 12:58, Robert Haas <robertmh...@gmail.com> wrote: >>> ...and why would one

Re: [HACKERS] Choosing parallel_degree

2016-03-15 Thread David Rowley
llow the parallel_setup_cost to make parallel plans look less favourable for smaller relations. I assume that this is so that we don't burden the planner with the overhead of generating parallel paths for smaller relations? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQ

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 12:58, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Mar 15, 2016 at 6:55 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> On 16 March 2016 at 11:00, Robert Haas <robertmh...@gmail.com> wrote: >>> I don't see wh

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
ggref->aggref, aggref)) break; } if equals() compared the aggpartial then this code would fail to find the Aggref in the subnode due to the aggpartial field being true on one and false on the other Aggref. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Dev

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 16 March 2016 at 09:23, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Mar 14, 2016 at 7:56 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> A comment does explain this, but perhaps it's not good enough, so I've >> rewritten it to become

Re: [HACKERS] Combining Aggregates

2016-03-15 Thread David Rowley
or regular users to deduce that as it's related to > the type of the internal state and not to the input types. An example of > that is the SUM(bigint) example mentioned above. I agree. I will look for a suitable place. -- David Rowley http://www.2ndQuadrant.com/ P

Re: [HACKERS] Parallel Aggregate

2016-03-15 Thread David Rowley
On 15 March 2016 at 13:48, David Rowley <david.row...@2ndquadrant.com> wrote: > An updated patch will follow soon. I've attached an updated patch which addresses some of Robert's and Tomas' concerns. I've not done anything about the exprCollation() yet, as I'm still unsure of what it sho

Re: [HACKERS] Choosing parallel_degree

2016-03-14 Thread David Rowley
getting 2 workers for only 4 pages. I've not tested in Postgres, but if you do this and: SET parallel_setup_cost = 0; then I'd imagine it should generate a parallel plan. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Service

[HACKERS] Choosing parallel_degree

2016-03-14 Thread David Rowley
pages there will be 14 workers (rel size 12455648 MB, 12163 GB) [1] http://www.postgresql.org/message-id/CANkGpBtUvzpdvF2=_iq64ujmvrpycs6d4i9-wepbusq1sq+...@mail.gmail.com -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Serv

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread David Rowley
t let's start a new thread for that. I don't want this one getting bloated with debates on that. It's not code I'm planning on going anywhere near for this patch. I'll start a thread... -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Traini

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread David Rowley
onest. I just put it there because the patch never persisted the value of a PartialAggType in any struct field anywhere and checks it later in some other file. In all places where we use PartialAggType we're also calling aggregates_allow_partial(), which does require clauses.h. So that's why

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread David Rowley
aggstate->finalizeAggs == true) > > We usually just say if (a) not if (a == true) when it's a boolean. > Similarly !a rather than a == false. hmm, thanks. It appears that I've not been all that consistent in that area. I didn't know that was convention. I see

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
ating on one child table at a time currently? There is nothing in the planner yet, or any patch that I know of to push the Partial Aggregate node to below an Append node. That will most likely come in 9.7. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
about the internals of that function, which is likely just never going to happen. (3) is just going to make the outlook of a hash plan look a little brighter, although you'll likely need a work_mem of over 1GB to make the plan change. -- David Rowley http://www.2ndQuadrant.

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
after 9.6. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
run EXPLAIN ANALYZE on this with the 6 workers, does the actual number of Gather rows come out at 105? I'd just like to get an idea of my cost estimate for the Gather are going to be accurate for real world data sets. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Parallel Aggregate

2016-03-13 Thread David Rowley
On 12 March 2016 at 16:31, David Rowley <david.row...@2ndquadrant.com> wrote: > I've attached an updated patch which is based on commit 7087166, > things are really changing fast in the grouping path area at the > moment, but hopefully the dust is starting to settle now. The attac

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-13 Thread David Rowley
which supports the distinctness, and then ensure none of the other rels which make up the join rel can cause tuple duplication of that rel. But this just causes missed optimisation opportunities. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Su

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-12 Thread David Rowley
ess() to do something a bit smarter than just return false if there's no indexes. That might not buy us much though, but at least relations tend to have very little unique indexes, even when they have lots of indexes. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development,

Re: [HACKERS] Parallel Aggregate

2016-03-11 Thread David Rowley
On 11 March 2016 at 03:39, David Rowley <david.row...@2ndquadrant.com> wrote: > A couple of things which I'm not 100% happy with. > > 1. make_partialgroup_input_target() doing lookups to the syscache. > Perhaps this job can be offloaded to a new function in a more suitable &

Re: [HACKERS] Parallel Aggregate

2016-03-10 Thread David Rowley
On 8 March 2016 at 11:15, David Rowley <david.row...@2ndquadrant.com> wrote: > The setrefs.c parts of the patch remain completely broken. I've not > had time to look at this again yet, sorry. Ok, so again, apologies for previously sending such a broken patch. I've since managed to f

Re: [HACKERS] Parallel Aggregate

2016-03-10 Thread David Rowley
On 9 March 2016 at 04:06, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Mar 7, 2016 at 5:15 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> 3. Nothing in create_grouping_paths() looks at the force_parallel_mode >> GUC. I had a quick look at th

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-08 Thread David Rowley
YAML seems most compact. [1] http://www.postgresql.org/message-id/8907.1440383...@sss.pgh.pa.us -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-08 Thread David Rowley
On 23 January 2016 at 05:36, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > Hi, > > On 12/17/2015 02:17 PM, David Rowley wrote: >> >> On 17 December 2015 at 19:11, Simon Riggs <si...@2ndquadrant.com >> <mailto:si...@2ndquadrant.com>> wrote:

Re: [HACKERS] Parallel Aggregate

2016-03-08 Thread David Rowley
On 9 March 2016 at 04:06, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Mar 7, 2016 at 5:15 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> My concerns are: >> 1. Since there's no cheapest_partial_path in RelOptInfo the code is >> curr

[HACKERS] Typo in logicaldecoding docs

2016-03-07 Thread David Rowley
The attached fixes a small error in the logicaldecoding docs. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services logicaldecoding_docs_typo_fix.patch Description: Binary data -- Sent via pgsql-hackers mailing list (p

[HACKERS] Fix misspelling of "parallel"

2016-03-07 Thread David Rowley
The attached fixes a small spelling error in a comment. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services parallel_spelling_fix.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Parallel Aggregate

2016-03-07 Thread David Rowley
On 5 March 2016 at 07:25, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Mar 3, 2016 at 11:00 PM, David Rowley >> 3. The code never attempts to mix and match Grouping Agg and Hash Agg >> plans. e.g it could be an idea to perform Partial Hash Aggregate -> >> Ga

Re: [HACKERS] WIP: Upper planner pathification

2016-03-07 Thread David Rowley
On 8 March 2016 at 10:01, Tom Lane <t...@sss.pgh.pa.us> wrote: > I've pushed it now; we'll soon see if the buildfarm finds any problems. Fantastic! I'm looking forward to all the future optimisation opportunities that this opens up. Thanks for making this happen. -- Dav

Re: [HACKERS] Parallel Aggregate

2016-03-06 Thread David Rowley
stages, so I really don't see why we need to give the impression that there is in EXPLAIN. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-03-05 Thread David Rowley
0.1405522...@sss.pgh.pa.us -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel Aggregate

2016-03-03 Thread David Rowley
atch a little earlier, but wanted to do so since this is quite a hot area in the code at the moment and I wanted to post for transparency. To apply the patch please apply [1] first. [1] http://www.postgresql.org/message-id/3795.1456689...@sss.pgh.pa.us -- David Rowley http:

Re: [HACKERS] WIP: Upper planner pathification

2016-03-03 Thread David Rowley
a bit special as it's a "leaf node" in the path tree. Propagating these would mean I could remove that parameter again. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailin

Re: [HACKERS] WIP: Upper planner pathification

2016-03-03 Thread David Rowley
On 3 March 2016 at 22:57, David Rowley <david.row...@2ndquadrant.com> wrote: > On 3 March 2016 at 18:04, Tom Lane <t...@sss.pgh.pa.us> wrote: >> If you come across any points where it seems like it could be >> done better or more easily, that would be tre

Re: [HACKERS] WIP: Upper planner pathification

2016-03-03 Thread David Rowley
On 3 March 2016 at 18:04, Tom Lane <t...@sss.pgh.pa.us> wrote: > David Rowley <david.row...@2ndquadrant.com> writes: >> I agree that it would be good to get this in as soon as possible. I'm >> currently very close to being done with writing Parallel Aggregate on >>

Re: [HACKERS] WIP: Upper planner pathification

2016-03-02 Thread David Rowley
the upper planner changes in first, but soon! not at the end of March 'fest, as doing so would most likely kill parallel aggregate for 9.6, and I kinda think that would be silly as (I think) it's pretty much the biggest missing piece of the parallel query set. -- David Rowley h

Re: [HACKERS] Random note of encouragement

2016-02-24 Thread David Rowley
t to use int128 internally instead of NUMERIC, it didn't make any changes to any NUMERIC aggregate functions. It would be interesting to see the explain analyze buffers for both. Perhaps 9.5 just read more buffers from disk than 9.6 did. -- David Rowley http://www.2ndQuadrant.com/ Postg

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-02-18 Thread David Rowley
On 23/01/2016 12:42 am, "David Rowley" <david.row...@2ndquadrant.com> wrote: > > On 23 January 2016 at 05:36, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > > Otherwise I think the patch is ready for committer - I think this is a > > valua

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-02-14 Thread David Rowley
On 14/02/2016 5:11 pm, "Tom Lane" <t...@sss.pgh.pa.us> wrote: > > David Rowley <david.row...@2ndquadrant.com> writes: > > On 12/02/2016 12:01 am, "Tom Lane" <t...@sss.pgh.pa.us> wrote: > > I can't quite understand what you're seeing here. >

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-02-14 Thread David Rowley
On 12/02/2016 12:01 am, "Tom Lane" <t...@sss.pgh.pa.us> wrote: > > David Rowley <david.row...@2ndquadrant.com> writes: > > [ prune_group_by_clause_ab4f491_2016-01-23.patch ] > > [ check_functional_grouping_refactor.patch ] > > I've committed th

Re: [HACKERS] WIP: Make timestamptz_out less slow.

2016-02-08 Thread David Rowley
On 7/02/2016 4:14 am, "Tom Lane" <t...@sss.pgh.pa.us> wrote: > > David Rowley <david.row...@2ndquadrant.com> writes: > [ timestamp_out_speedup_2015-11-05.patch ] > > Pushed with a bunch of cosmetic tweaks. Great. Thanks for pushing this. David

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-31 Thread David Rowley
there are any crashes. I'm really not that sure of how else to increase the confidence levels on this. Do you have ideas? -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-24 Thread David Rowley
On 24 January 2016 at 08:20, Tom Lane <t...@sss.pgh.pa.us> wrote: > David Rowley <david.row...@2ndquadrant.com> writes: >> On 23 January 2016 at 12:44, Tom Lane <t...@sss.pgh.pa.us> wrote: >>> * What you did to join.sql/join.out seems a bit bizarre. The exis

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-24 Thread David Rowley
On 25 January 2016 at 10:17, Tom Lane <t...@sss.pgh.pa.us> wrote: > David Rowley <david.row...@2ndquadrant.com> writes: >> I've looked into why the join is not removed; since the redundant >> GROUP BY columns are removed during planning, and since the outer >> quer

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-23 Thread David Rowley
m being removed therefore allowing the join removal conditions to be met. On testing again with the old test query I see this no longer happens, so I've removed the change, although the expected output still differs due to the group by item being removed. > I'm going to set this back to &q

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-23 Thread David Rowley
ng about bms_num_members() is that it's going to loop over each word in the bitmap no matter what, whereas a subset check can abort early in some cases. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hac

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-01-22 Thread David Rowley
changes as a refactor patch first. If committer wants that, I can separate these out, but I'll hold off for a response before doing that. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services unique_joins_ba5b9cb_2016-01-23.patch D

Re: [HACKERS] Combining Aggregates

2016-01-22 Thread David Rowley
On 23 January 2016 at 09:17, Jeff Janes <jeff.ja...@gmail.com> wrote: > On Wed, Jan 20, 2016 at 11:06 AM, Robert Haas <robertmh...@gmail.com> wrote: >> On Wed, Jan 20, 2016 at 7:38 AM, David Rowley >> <david.row...@2ndquadrant.com> wrote: >>> Agreed. So

Re: [HACKERS] Parallel Aggregate

2016-01-22 Thread David Rowley
how they propagate through calculations is the subject of an entire branch of mathematics and computer science and will not be discussed here, except for the following points:" [1] [1] http://www.postgresql.org/docs/devel/static/datatype-numeric.html -- David Rowley http://www.2

Re: [HACKERS] Combining Aggregates

2016-01-22 Thread David Rowley
On 23 January 2016 at 09:44, David Rowley <david.row...@2ndquadrant.com> wrote: > On 23 January 2016 at 09:17, Jeff Janes <jeff.ja...@gmail.com> wrote: >> On Wed, Jan 20, 2016 at 11:06 AM, Robert Haas <robertmh...@gmail.com> wrote: >>> On Wed, Jan 20, 2016 at

Re: [HACKERS] Parallel Aggregate

2016-01-21 Thread David Rowley
ocess then you could be quite content with PAT_INTERNAL_ONLY. You'll just need to pull out the logic that checks for serial and deserial functions, since that's not in yet, and just have it return PAT_INTERNAL_ONLY if INTERNAL aggregates are found which have combine functions set. -- Davi

Re: [HACKERS] Combining Aggregates

2016-01-21 Thread David Rowley
On 22 January 2016 at 06:56, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Jan 20, 2016 at 8:32 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: >> The other two usages which I have thought of are; >> >> 1) Aggregating before UNION ALL

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-21 Thread David Rowley
On 20 January 2016 at 06:08, Anastasia Lubennikova <a.lubennik...@postgrespro.ru> wrote: > > > > 18.01.2016 01:02, David Rowley пишет: > > On 14 January 2016 at 08:24, David Rowley <david.row...@2ndquadrant.com> > wrote: >> >> I will try to rev

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 21 January 2016 at 08:06, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Jan 20, 2016 at 7:38 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > Agreed. So I've attached a version of the patch which does not have any > of > > the serialise/

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 21 January 2016 at 04:59, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Jan 20, 2016 at 7:53 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > On 21 January 2016 at 01:44, Robert Haas <robertmh...@gmail.com> wrote: > >> > >&g

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
an here. I've still got a bit of work to do (in the not too distant future) on the serial/deserial part, so would be better to keep this thread for discussion on that. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 21 January 2016 at 01:44, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Jan 20, 2016 at 7:38 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: > >> To my mind, priority #1 ought to be putting this fine new > >> functionality to some use. Expan

Re: [HACKERS] Combining Aggregates

2016-01-20 Thread David Rowley
On 20 January 2016 at 10:54, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Jan 19, 2016 at 4:50 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: > >> Oh, one more point: is there any reason why all of this needs to be a > >> single (gia

Re: [HACKERS] Combining Aggregates

2016-01-19 Thread David Rowley
On 20 January 2016 at 05:58, Robert Haas <robertmh...@gmail.com> wrote: > > On Mon, Dec 21, 2015 at 4:02 AM, David Rowley > > <david.row...@2ndquadrant.com> wrote: > >> Now, there has been talk of this previously, on various threads, but I > don't > >>

Re: [HACKERS] Combining Aggregates

2016-01-19 Thread David Rowley
On 20 January 2016 at 05:56, Robert Haas <robertmh...@gmail.com> wrote: > > On Mon, Dec 21, 2015 at 4:02 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > Now, there has been talk of this previously, on various threads, but I > don't > > believe

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
On 18 January 2016 at 16:44, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Jan 17, 2016 at 9:26 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > hmm, so wouldn't that mean that the transition function would need to > (for > > each input tuple):

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
r into a GroupAggregate plan, which is faster, but due to the the hash agg executor code not giving any regard to work_mem. If I set work_mem to 140MB (which is more realistic for this VM), it does cause the same swapping problems to occur. Probably setting aggtransspace for this aggregate to 1024 would help the costing problem, but it would also cause hashagg to be a less chosen option during planning. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
On 19 January 2016 at 18:04, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > Hi, > > On 01/19/2016 05:00 AM, David Rowley wrote: > >> On 19 January 2016 at 06:03, Pavel Stehule <pavel.steh...@gmail.com >> <mailto:pavel.steh...@gmail.com>> wrote

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
On 19 January 2016 at 02:44, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > On Mon, Jan 18, 2016 at 10:32 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > I just thought like direct mapping of the structure with text pointer. > something like > the be

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread David Rowley
l datatype functions to make sure that expanded objects > >> live where it wants them to. > > > > That's how I did it in my prototype, but the problem with that is that > > spinning up a memory context for every group sucks when there are many > > groups with only a

[HACKERS] Compiler warning in pg_am changes

2016-01-17 Thread David Rowley
Hi, I've attached a small patch to fix new compiler warning which is new as of 65c5fcd3 -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services amcostestimate_cast.patch Description: Binary data -- Sent via pgsql-hac

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-17 Thread David Rowley
On 14 January 2016 at 08:24, David Rowley <david.row...@2ndquadrant.com> wrote: > I will try to review the omit_opclass_4.0.patch soon. > Hi, as promised, here's my review of the omit_opclass_4.0.patch patch. The following comment needs to be updated: * indexkeys[], ind

Re: [HACKERS] Combining Aggregates

2016-01-17 Thread David Rowley
On 18 January 2016 at 14:36, Haribabu Kommi <kommi.harib...@gmail.com> wrote: > On Sat, Jan 16, 2016 at 12:00 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > On 16 January 2016 at 03:03, Robert Haas <robertmh...@gmail.com> wrote: > >> >

Re: [HACKERS] Combining Aggregates

2016-01-15 Thread David Rowley
On 16 January 2016 at 03:03, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Dec 29, 2015 at 7:39 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: > >> No, the idea I had in mind was to allow it to continue to exist in the > >> expanded format unt

Re: [HACKERS] Spelling corrections

2016-01-14 Thread David Rowley
On 15 January 2016 at 17:19, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Jan 10, 2016 at 5:17 AM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > On 10 January 2016 at 19:34, Peter Geoghegan <p...@heroku.com> wrote: > >> > >&g

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-14 Thread David Rowley
st == NULL || list_length(varlist) < 2) > + continue; > > To be perfectly correct, the comment should say "at least two Vars". > > Changed per discussion from you and Geoff > Except the small remaining typos, this patch looks very fine to me.

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-13 Thread David Rowley
On 14 January 2016 at 02:58, Anastasia Lubennikova < a.lubennik...@postgrespro.ru> wrote: > 13.01.2016 04:27, David Rowley: > >> I've also done some testing: >> >> create table ab (a int, b int); >> insert into ab select a,b from generate_Series(1,10) a

Re: [HACKERS] Removing Functionally Dependent GROUP BY Columns

2016-01-13 Thread David Rowley
if we find other Vars which make up the table's primary key. I didn't make these changes as I thought it was a less likely scenario. It wouldn't be too much extra code to add however. I've went and added an XXX comment to explain that there might be future optimisation opportunities in the future

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-12 Thread David Rowley
On 13 January 2016 at 05:59, Anastasia Lubennikova < a.lubennik...@postgrespro.ru> wrote: > 08.01.2016 00:12, David Rowley: > > On 7 January 2016 at 06:36, Jeff Janes <jeff.ja...@gmail.com> wrote: > >> On Tue, Jan 5, 2016 at 11:55 PM, David Rowley >>

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-12 Thread David Rowley
o review, I think things would make more sense of the omit_opclass_4.0.patch was included together with this. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] POC, WIP: OR-clause support for indexes

2016-01-10 Thread David Rowley
to always skip the first element of the keyData array. If that's really the intention, then wouldn't it be better to just make the initial condition of the for() look i = 1 ? + stack[stackPos] = i; + } + + Assert(stackPos == 0); + pfree(stack); +} I'd like to review more, but it feels like a job that's more difficult than it needs to be due to lack of comments. Would it be possible to update the patch to try and explain things a little better? Many thanks David -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Spelling corrections

2016-01-10 Thread David Rowley
On 10 January 2016 at 19:34, Peter Geoghegan <p...@heroku.com> wrote: > Attached patch fixes a couple of misspellings. > Snap! http://www.postgresql.org/message-id/CAKJS1f-AGgQaurTwhY=wkJjspgDcmDUE8Yx03XTXCDz=kae...@mail.gmail.com -- David Rowley http://www.2nd

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread David Rowley
on the more efficient hashing of the hash value that you're doing in the current patch, and hash the complete value in the scan node, then hash them again if they make it into the hash join node. That does not sound like it would be a win if hashing longer varlana values. -- David Rowley http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-07 Thread David Rowley
On 7 January 2016 at 06:36, Jeff Janes <jeff.ja...@gmail.com> wrote: > On Tue, Jan 5, 2016 at 11:55 PM, David Rowley > <david.row...@2ndquadrant.com> wrote: > > create table ab (a int,b int); > > insert into ab select x,y from generate_series(1,20) x(x), &g

Re: [HACKERS] Optimizer questions

2016-01-06 Thread David Rowley
e that it does seem like an interesting route for optimisation. It seems worthwhile to investigate how we might go about improving this so that the evaluation of the target list happens after LIMIT, at least for the columns which are not required before LIMIT. Konstantin, are you thinking of lo

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-05 Thread David Rowley
On 4 January 2016 at 21:49, David Rowley <david.row...@2ndquadrant.com> wrote: > I've not tested the patch yet. I will send another email soon with the > results of that. > Hi, As promised I've done some testing on this, and I've found something which is not quite right: create

Re: [HACKERS]WIP: Covering + unique indexes.

2016-01-04 Thread David Rowley
*indexIncludingParams; /* additional columns to index: a list of IndexElem */ This should wrap at 80 chars. struct RestrictInfo has some examples of how this is normally done. /* + * RelationGetNumberOfAttributes + * Returns the number of attributes in a relation. + */ +#define IndexR

Re: [HACKERS] More thorough planning for OLAP queries (was: [PATCH] Equivalence Class Filters)

2015-12-30 Thread David Rowley
On 30 December 2015 at 21:12, Benedikt Grundmann <bgrundm...@janestreet.com> wrote: > On Wed, Dec 30, 2015 at 7:16 AM, David Rowley < > david.row...@2ndquadrant.com> wrote: > >> >> A number of ideas were suggested on the other thread about how we might >&g

[HACKERS] Re: More thorough planning for OLAP queries (was: [PATCH] Equivalence Class Filters)

2015-12-30 Thread David Rowley
On 31 December 2015 at 01:24, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: > On 12/30/2015 08:16 AM, David Rowley wrote: > >> >> I do strongly believe that we need to come up with something to >> solve this problem. I already summarised my thoughts on t

<    1   2   3   4   5   6   7   8   9   10   >