Re: [HACKERS] tbm_lossify causes "unbalanced" hashtables / bitmaps

2016-09-23 Thread Andres Freund
On 2016-09-23 17:40:13 -0400, Tom Lane wrote: > My idea of an appropriate fix would be to resume the scan at the same > point where the last scan stopped, and work circularly around the table > when necessary. I've played with that idea, and it does help a great deal. Not that surprisingly, it's b

Re: [HACKERS] tbm_lossify causes "unbalanced" hashtables / bitmaps

2016-09-23 Thread Tom Lane
Andres Freund writes: > Because iteration (both in my implementation and dynahash) is > essentially in bucket order, the front of the hashtable will be mostly > empty, whereas later parts of the hashtable will be full (i.e. a lot of > collisions). The reason for that is that we'll find all parts o

Re: [HACKERS] tbm_lossify causes "unbalanced" hashtables / bitmaps

2016-09-23 Thread Andres Freund
On 2016-09-23 13:58:43 -0700, Andres Freund wrote: > static void > tbm_lossify(TIDBitmap *tbm) > { > ... > pagetable_start_iterate_random(tbm->pagetable, &i); Uh, obviously that's from one of my attempts to address the problem. Greetings, Andres Freund -- Sent via pgsql-hackers mailing

[HACKERS] tbm_lossify causes "unbalanced" hashtables / bitmaps

2016-09-23 Thread Andres Freund
Hi, Playing around with my hashtable implementation [1] and using it for bitmapscans/tidbitmap.c I noticed something curious. When using small work_mem (4MB in my case) for large-ish tables (~5GB), the performance tanks. That's primarily caused by some issues in the hashtable code (since fixed), b