Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-12-09 Thread Andres Freund
Hi, On 2016-12-09 15:21:36 -0800, Mark Dilger wrote: > Andres, > > Your patch, below, appears to have been applied to master in commit > 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5. It makes mention of a > function, tuplehash_start_iterate, in a macro, but the function is not > defined or

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-12-09 Thread Mark Dilger
Andres, Your patch, below, appears to have been applied to master in commit 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5. It makes mention of a function, tuplehash_start_iterate, in a macro, but the function is not defined or declared, and its signature and intention is not clear. Is there any

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-11 Thread Tomas Vondra
On 10/11/2016 05:56 PM, Andres Freund wrote: On 2016-10-11 04:29:31 +0200, Tomas Vondra wrote: On 10/11/2016 04:07 AM, Andres Freund wrote: On 2016-10-10 17:46:22 -0700, Andres Freund wrote: TPC-DS (tpcds.ods) -- In this case, I'd say the results are less convincing. There

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-11 Thread Andres Freund
On 2016-10-11 04:29:31 +0200, Tomas Vondra wrote: > On 10/11/2016 04:07 AM, Andres Freund wrote: > > On 2016-10-10 17:46:22 -0700, Andres Freund wrote: > > > > TPC-DS (tpcds.ods) > > > > -- > > > > > > > > In this case, I'd say the results are less convincing. There are quite > >

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-10 Thread Tomas Vondra
On 10/11/2016 04:07 AM, Andres Freund wrote: On 2016-10-10 17:46:22 -0700, Andres Freund wrote: TPC-DS (tpcds.ods) -- In this case, I'd say the results are less convincing. There are quite a few queries that got slower by ~10%, which is well above - for example queries 22 and

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-10 Thread Andres Freund
On 2016-10-10 17:46:22 -0700, Andres Freund wrote: > > TPC-DS (tpcds.ods) > > -- > > > > In this case, I'd say the results are less convincing. There are quite a few > > queries that got slower by ~10%, which is well above - for example queries > > 22 and 67. There are of course

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-10 Thread Andres Freund
Hi, On 2016-10-11 02:38:26 +0200, Tomas Vondra wrote: > Yes, I've done a bunch of TPC-H and TPC-DS tests on the patch, but it took > quite a bit of time. These tests were done on a fairly small machine (the > usual i5-2500k with 8GB of RAM), with only 1GB data sets for both > benchmarks, to keep

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-09 Thread Andres Freund
Hi, Attached is an updated version of the patchset. The main changes are - address most of Tomas' feedback - address regression test output changes by adding explicit ORDER BYs, in a separate commit. - fix issues around hash tables sized up to PG_UINT32_MAX - fix a bug in iteration with

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-03 Thread Andres Freund
Hi, On 2016-10-03 13:26:09 +0200, Arthur Silva wrote: > On Sat, Oct 1, 2016 at 2:44 AM, Andres Freund wrote: > A couple of comments. > * 80% occupancy is a bit conservative for RH hashing, it works well up to > 90% if you use the early stops due to distance. So that TODO is

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-03 Thread Arthur Silva
On Sat, Oct 1, 2016 at 2:44 AM, Andres Freund wrote: > Hi, > > On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > > In the attached patch I've attached simplehash.h, which can be > > customized by a bunch of macros, before being inlined. There's also a > > patch using this

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-02 Thread Andres Freund
On 2016-10-02 02:59:04 +0200, Tomas Vondra wrote: > On 10/02/2016 02:17 AM, Andres Freund wrote: > > Hi, > > ... > > > > > > > I think a crucial part of the benchmarking will be identifying and > > > > > measuring corner cases, e.g. a lot of rows with the same key, etc. > > > > > Although that

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Tomas Vondra
On 10/02/2016 02:17 AM, Andres Freund wrote: Hi, > ... I think a crucial part of the benchmarking will be identifying and measuring corner cases, e.g. a lot of rows with the same key, etc. Although that probably is not a major issue for the two places switched to the new implementation (e.g.

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Andres Freund
Hi, On 2016-10-02 01:37:35 +0200, Tomas Vondra wrote: > On 10/01/2016 09:59 PM, Andres Freund wrote: > >>What about running a bigger benchmark - say, TPC-DS - and evaluating > >>the impact? > > > >Worthwhile, although obviously the impact will be a lot smaller, > >since they're not entirely

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Tomas Vondra
On 10/01/2016 09:59 PM, Andres Freund wrote: Hi, On 2016-10-01 20:19:21 +0200, Tomas Vondra wrote: On 10/01/2016 02:44 AM, Andres Freund wrote: Hi, On 2016-07-26 17:43:33 -0700, Andres Freund wrote: In the attached patch I've attached simplehash.h, which can be customized by a bunch of

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Andres Freund
On 2016-10-01 20:04:18 +0100, Greg Stark wrote: > On Sat, Oct 1, 2016 at 1:44 AM, Andres Freund wrote: > > > > Unfortunately I'm running out battery right now, so I don't want to > > re-run the benchmarks posted upthread (loading takes a while). But the > > last time I ran

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Andres Freund
Hi, On 2016-10-01 20:19:21 +0200, Tomas Vondra wrote: > On 10/01/2016 02:44 AM, Andres Freund wrote: > > Hi, > > > > On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > > > In the attached patch I've attached simplehash.h, which can be > > > customized by a bunch of macros, before being

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Greg Stark
On Sat, Oct 1, 2016 at 1:44 AM, Andres Freund wrote: > > Unfortunately I'm running out battery right now, so I don't want to > re-run the benchmarks posted upthread (loading takes a while). But the > last time I ran them all the results after the patches were better than >

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-10-01 Thread Tomas Vondra
On 10/01/2016 02:44 AM, Andres Freund wrote: Hi, On 2016-07-26 17:43:33 -0700, Andres Freund wrote: In the attached patch I've attached simplehash.h, which can be customized by a bunch of macros, before being inlined. There's also a patch using this for tidbitmap.c and nodeAgg/nodeSubplan/...

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-09-30 Thread Andres Freund
Hi, On 2016-07-26 17:43:33 -0700, Andres Freund wrote: > In the attached patch I've attached simplehash.h, which can be > customized by a bunch of macros, before being inlined. There's also a > patch using this for tidbitmap.c and nodeAgg/nodeSubplan/... via > execGrouping.c. Attached is a

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-07-27 Thread Andres Freund
On 2016-07-27 10:04:52 -0400, Robert Haas wrote: > On Tue, Jul 26, 2016 at 8:43 PM, Andres Freund wrote: > > As previously mentioned, dynahash isn't particularly fast. In many cases > > that doesn't particularly matter. But e.g. hash aggregation and bitmap > > index scans are

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-07-27 Thread Robert Haas
On Tue, Jul 26, 2016 at 8:43 PM, Andres Freund wrote: > As previously mentioned, dynahash isn't particularly fast. In many cases > that doesn't particularly matter. But e.g. hash aggregation and bitmap > index scans are quite sensitive to hash performance. > > The biggest