Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, On 2016-12-09 15:21:36 -0800, Mark Dilger wrote: > Andres, > > Your patch, below, appears to have been applied to master in commit > 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5. It makes mention of a > function, tuplehash_start_iterate, in a macro, but the function is not > defined or

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Andres, Your patch, below, appears to have been applied to master in commit 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5. It makes mention of a function, tuplehash_start_iterate, in a macro, but the function is not defined or declared, and its signature and intention is not clear. Is there any

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 10/11/2016 05:56 PM, Andres Freund wrote: On 2016-10-11 04:29:31 +0200, Tomas Vondra wrote: On 10/11/2016 04:07 AM, Andres Freund wrote: On 2016-10-10 17:46:22 -0700, Andres Freund wrote: TPC-DS (tpcds.ods) -- In this case, I'd say the results are less convincing. There

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 2016-10-11 04:29:31 +0200, Tomas Vondra wrote: > On 10/11/2016 04:07 AM, Andres Freund wrote: > > On 2016-10-10 17:46:22 -0700, Andres Freund wrote: > > > > TPC-DS (tpcds.ods) > > > > -- > > > > > > > > In this case, I'd say the results are less convincing. There are quite > >

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 10/11/2016 04:07 AM, Andres Freund wrote: On 2016-10-10 17:46:22 -0700, Andres Freund wrote: TPC-DS (tpcds.ods) -- In this case, I'd say the results are less convincing. There are quite a few queries that got slower by ~10%, which is well above - for example queries 22 and

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 2016-10-10 17:46:22 -0700, Andres Freund wrote: > > TPC-DS (tpcds.ods) > > -- > > > > In this case, I'd say the results are less convincing. There are quite a few > > queries that got slower by ~10%, which is well above - for example queries > > 22 and 67. There are of course

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, On 2016-10-11 02:38:26 +0200, Tomas Vondra wrote: > Yes, I've done a bunch of TPC-H and TPC-DS tests on the patch, but it took > quite a bit of time. These tests were done on a fairly small machine (the > usual i5-2500k with 8GB of RAM), with only 1GB data sets for both > benchmarks, to keep

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, Attached is an updated version of the patchset. The main changes are - address most of Tomas' feedback - address regression test output changes by adding explicit ORDER BYs, in a separate commit. - fix issues around hash tables sized up to PG_UINT32_MAX - fix a bug in iteration with

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, On 2016-10-03 13:26:09 +0200, Arthur Silva wrote: > On Sat, Oct 1, 2016 at 2:44 AM, Andres Freund wrote: > A couple of comments. > * 80% occupancy is a bit conservative for RH hashing, it works well up to > 90% if you use the early stops due to distance. So that TODO is

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On Sat, Oct 1, 2016 at 2:44 AM, Andres Freund wrote: > Hi, > > On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > > In the attached patch I've attached simplehash.h, which can be > > customized by a bunch of macros, before being inlined. There's also a > > patch using this

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 2016-10-02 02:59:04 +0200, Tomas Vondra wrote: > On 10/02/2016 02:17 AM, Andres Freund wrote: > > Hi, > > ... > > > > > > > I think a crucial part of the benchmarking will be identifying and > > > > > measuring corner cases, e.g. a lot of rows with the same key, etc. > > > > > Although that

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 10/02/2016 02:17 AM, Andres Freund wrote: Hi, > ... I think a crucial part of the benchmarking will be identifying and measuring corner cases, e.g. a lot of rows with the same key, etc. Although that probably is not a major issue for the two places switched to the new implementation (e.g.

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, On 2016-10-02 01:37:35 +0200, Tomas Vondra wrote: > On 10/01/2016 09:59 PM, Andres Freund wrote: > >>What about running a bigger benchmark - say, TPC-DS - and evaluating > >>the impact? > > > >Worthwhile, although obviously the impact will be a lot smaller, > >since they're not entirely

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 10/01/2016 09:59 PM, Andres Freund wrote: Hi, On 2016-10-01 20:19:21 +0200, Tomas Vondra wrote: On 10/01/2016 02:44 AM, Andres Freund wrote: Hi, On 2016-07-26 17:43:33 -0700, Andres Freund wrote: In the attached patch I've attached simplehash.h, which can be customized by a bunch of

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 2016-10-01 20:04:18 +0100, Greg Stark wrote: > On Sat, Oct 1, 2016 at 1:44 AM, Andres Freund wrote: > > > > Unfortunately I'm running out battery right now, so I don't want to > > re-run the benchmarks posted upthread (loading takes a while). But the > > last time I ran

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, On 2016-10-01 20:19:21 +0200, Tomas Vondra wrote: > On 10/01/2016 02:44 AM, Andres Freund wrote: > > Hi, > > > > On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > > > In the attached patch I've attached simplehash.h, which can be > > > customized by a bunch of macros, before being

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On Sat, Oct 1, 2016 at 1:44 AM, Andres Freund wrote: > > Unfortunately I'm running out battery right now, so I don't want to > re-run the benchmarks posted upthread (loading takes a while). But the > last time I ran them all the results after the patches were better than >

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 10/01/2016 02:44 AM, Andres Freund wrote: Hi, On 2016-07-26 17:43:33 -0700, Andres Freund wrote: In the attached patch I've attached simplehash.h, which can be customized by a bunch of macros, before being inlined. There's also a patch using this for tidbitmap.c and nodeAgg/nodeSubplan/...

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > In the attached patch I've attached simplehash.h, which can be > customized by a bunch of macros, before being inlined. There's also a > patch using this for tidbitmap.c and nodeAgg/nodeSubplan/... via > execGrouping.c. Attached is a

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On 2016-07-27 10:04:52 -0400, Robert Haas wrote: > On Tue, Jul 26, 2016 at 8:43 PM, Andres Freund wrote: > > As previously mentioned, dynahash isn't particularly fast. In many cases > > that doesn't particularly matter. But e.g. hash aggregation and bitmap > > index scans are

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

On Tue, Jul 26, 2016 at 8:43 PM, Andres Freund wrote: > As previously mentioned, dynahash isn't particularly fast. In many cases > that doesn't particularly matter. But e.g. hash aggregation and bitmap > index scans are quite sensitive to hash performance. > > The biggest

[HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

Hi, As previously mentioned, dynahash isn't particularly fast. In many cases that doesn't particularly matter. But e.g. hash aggregation and bitmap index scans are quite sensitive to hash performance. The biggest problems with dynahash (also discussed at [1]) are a) two level structure doubling