Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Mon, May 22, 2017 at 6:39 AM, Thomas Munro wrote: > On Thu, Apr 27, 2017 at 11:03 AM, Thomas Munro > wrote: >> On Thu, Apr 27, 2017 at 5:13 AM, Oleg Golovanov wrote: >>> Can you actualize your patch set? The error

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Apr 27, 2017 at 11:03 AM, Thomas Munro wrote: > On Thu, Apr 27, 2017 at 5:13 AM, Oleg Golovanov wrote: >> Can you actualize your patch set? The error got from >> 0010-hj-parallel-v12.patch. > > I really should get around to setting up a

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Apr 27, 2017 at 5:13 AM, Oleg Golovanov wrote: > Can you actualize your patch set? The error got from > 0010-hj-parallel-v12.patch. I really should get around to setting up a cron job to tell me about that. Here's a rebased version. The things currently on my list for

[HACKERS] Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi. Thanks for rebased patch set v12. Currently I try to use this patch on my new test site and get following: Hmm...  The next patch looks like a unified diff to me... The text leading up to this was: -- |diff --git a/src/include/access/parallel.h

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Apr 13, 2017 at 10:04 PM, Oleg Golovanov wrote: > bash-4.2$ grep Hunk *.log | grep FAILED > 0005-hj-leader-gate-v11.patch.log:Hunk #1 FAILED at 14. > 0010-hj-parallel-v11.patch.log:Hunk #2 FAILED at 2850. > 0010-hj-parallel-v11.patch.log:Hunk #1 FAILED at 21. >

[HACKERS] Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi. I got errors of patching on CentOS 7: bash-4.2$ grep Hunk *.log | grep FAILED 0005-hj-leader-gate-v11.patch.log:Hunk #1 FAILED at 14. 0010-hj-parallel-v11.patch.log:Hunk #2 FAILED at 2850. 0010-hj-parallel-v11.patch.log:Hunk #1 FAILED at 21. 0010-hj-parallel-v11.patch.log:Hunk #3 FAILED at

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Tue, Apr 4, 2017 at 9:11 AM, Andres Freund wrote: > Hi, > > On 2017-03-31 17:53:12 +1300, Thomas Munro wrote: >> Thanks very much to Rafia for testing, and to Andres for his copious >> review feedback. Here's a new version. Changes: > > I unfortunately think it's too late

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, On 2017-03-31 17:53:12 +1300, Thomas Munro wrote: > Thanks very much to Rafia for testing, and to Andres for his copious > review feedback. Here's a new version. Changes: I unfortunately think it's too late to get this into v10. There's still heavy development going on, several pieces

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi Thomas, On 2017-03-31 17:53:12 +1300, Thomas Munro wrote: > Thanks very much to Rafia for testing, and to Andres for his copious > review feedback. Here's a new version. Changes: I've not looked at that aspect, but one thing I think would be good is to first add patch that increases

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi hackers, Thanks very much to Rafia for testing, and to Andres for his copious review feedback. Here's a new version. Changes: 1. Keep all the backing files that are part of a BufFileSet in subdirectories, as suggested by Andres. Now, instead of that unpopular logic for scanning ranges of

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Tue, Mar 28, 2017 at 11:11 AM, Rafia Sabih wrote: > On Mon, Mar 27, 2017 at 12:20 PM, Thomas Munro > wrote: >> >> On Sun, Mar 26, 2017 at 3:56 PM, Thomas Munro >> wrote: >> > But... what you said

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, On 2017-03-27 22:33:03 -0700, Andres Freund wrote: > On 2017-03-23 20:35:09 +1300, Thomas Munro wrote: > > Here is a new patch series responding to feedback from Peter and Andres: > > Here's a review of 0007 & 0010 together - they're going to have to be > applied together anyway... > ... >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi Thomas, On 3/28/17 1:41 AM, Rafia Sabih wrote: On Mon, Mar 27, 2017 at 12:20 PM, Thomas Munro I thought this last point about Windows might be fatal to my design, but it seems that Windows since at least version 2000 has support for Unixoid unlinkability via the special flag

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On 2017-03-23 20:35:09 +1300, Thomas Munro wrote: > Here is a new patch series responding to feedback from Peter and Andres: Here's a review of 0007 & 0010 together - they're going to have to be applied together anyway... diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sun, Mar 26, 2017 at 3:56 PM, Thomas Munro wrote: > But... what you said above must be a problem for Windows. I believe > it doesn't allow files to be unlinked if they are open, and I see that > DSM segments are cleaned up in resowner's phase == >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sun, Mar 26, 2017 at 6:50 PM, Thomas Munro wrote: > Like you, I also tend to suspect that people would be more likely to > use RAID type technologies to stripe things like this for both > bandwidth and space reasons these days. Tablespaces seem to make more >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Mon, Mar 27, 2017 at 12:12 PM, Peter Geoghegan wrote: > On Sun, Mar 26, 2017 at 3:41 PM, Thomas Munro > wrote: >>> 1. Segments are what buffile.c already calls the individual >>> capped-at-1GB files that it manages. They are an implementation >>>

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sun, Mar 26, 2017 at 3:41 PM, Thomas Munro wrote: >> 1. Segments are what buffile.c already calls the individual >> capped-at-1GB files that it manages. They are an implementation >> detail that is not part of buffile.c's user interface. There seems to >> be

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Mon, Mar 27, 2017 at 11:03 AM, Thomas Munro >> Is there a risk that this ends up running afoul of filename length >> limits on some platforms? > > Hmm. I didn't think so. Do we have a project guideline on maximum > path lengths based on some kind of survey?

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Mon, Mar 27, 2017 at 9:41 AM, Andres Freund wrote: > Hi, > > > SharedBufFile allows temporary files to be created by one backend and > then exported for read-only access by other backends, with clean-up > managed by reference counting associated with a DSM segment. This

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On 2017-03-23 20:35:09 +1300, Thomas Munro wrote: > Here is a new patch series responding to feedback from Peter and Andres: + +/* Per-participant shared state. */ +typedef struct SharedTuplestoreParticipant +{ + LWLock lock; Hm. No padding (ala LWLockMinimallyPadded / LWLockPadded) - but

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, SharedBufFile allows temporary files to be created by one backend and then exported for read-only access by other backends, with clean-up managed by reference counting associated with a DSM segment. This includes changes to fd.c and buffile.c to support new kinds of temporary file. diff

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Mar 25, 2017 at 7:56 PM, Thomas Munro wrote: > On Sun, Mar 26, 2017 at 1:53 PM, Peter Geoghegan wrote: >> ISTM that your patch now shares a quality with parallel tuplesort: You >> may now hold files open after an unlink() of the original

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sun, Mar 26, 2017 at 1:53 PM, Peter Geoghegan wrote: > ISTM that your patch now shares a quality with parallel tuplesort: You > may now hold files open after an unlink() of the original link/path > that they were opened using. As Robert pointed out when discussing > parallel

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Mar 23, 2017 at 12:35 AM, Thomas Munro wrote: > Thanks. I really appreciate your patience with the resource > management stuff I had failed to think through. It's a surprisingly difficult problem, that almost requires prototyping just to explain. No need

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, Here is a new patch series responding to feedback from Peter and Andres: 1. Support pgstat_report_tempfile and log_temp_files, which I had overlooked as Peter pointed out. 2. Use a patch format that is acceptable to git am, per complaint off-list from Andres. (Not actually made with git

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Mar 22, 2017 at 3:17 AM, Thomas Munro wrote: >> If I follow the new code correctly, then it doesn't matter that you've >> unlink()'d to take care of the more obvious resource management chore. >> You can still have a reference leak like this, if I'm not

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, Here is a new version addressing feedback from Peter and Andres. Please see below. On Wed, Mar 22, 2017 at 3:18 PM, Peter Geoghegan wrote: > On Tue, Mar 21, 2017 at 5:07 AM, Thomas Munro > wrote: >>> buffile.c should stop pretending to care

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Tue, Mar 21, 2017 at 7:18 PM, Peter Geoghegan wrote: >> As shown in 0008-hj-shared-buf-file-v8.patch. Thoughts? > > A less serious issue I've also noticed is that you add palloc() calls, > implicitly using the current memory context, within buffile.c. > BufFileOpenTagged() has

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Tue, Mar 21, 2017 at 5:07 AM, Thomas Munro wrote: >> buffile.c should stop pretending to care about anything other than >> temp files, IMV. 100% of all clients that want temporary files go >> through buffile.c. 100% of all clients that want non-temp files (files

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, Here is a new version of the patch series addressing complaints from Rafia, Peter, Andres and Robert -- see below. First, two changes not already covered in this thread: 1. Today Robert asked me a question off-list that I hadn't previously considered: since I am sharing tuples between

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Tue, Mar 14, 2017 at 8:03 AM, Thomas Munro wrote: > On Mon, Mar 13, 2017 at 8:40 PM, Rafia Sabih > wrote: >> In an attempt to test v7 of this patch on TPC-H 20 scale factor I found a >> few regressions, >> Q21: 52 secs on HEAD and

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Mon, Mar 13, 2017 at 8:40 PM, Rafia Sabih wrote: > In an attempt to test v7 of this patch on TPC-H 20 scale factor I found a > few regressions, > Q21: 52 secs on HEAD and 400 secs with this patch Thanks Rafia. Robert just pointed out off-list that there is a

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Mar 9, 2017 at 5:28 PM, Thomas Munro wrote: > On Wed, Mar 8, 2017 at 12:58 PM, Andres Freund wrote: > > 0002: Check hash join work_mem usage at the point of chunk allocation. > > > > Modify the existing hash join code to detect work_mem

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Mar 9, 2017 at 3:58 AM, Thomas Munro wrote: > In the meantime, here is a new patch series addressing the other > things you raised. Please see my remarks on 0007-hj-shared-buf-file-v7.patch over on the "on_dsm_detach() callback and parallel tuplesort

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Mar 8, 2017 at 12:58 PM, Andres Freund wrote: > 0002: Check hash join work_mem usage at the point of chunk allocation. > > Modify the existing hash join code to detect work_mem exhaustion at > the point where chunks are allocated, instead of checking after every >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Mar 8, 2017 at 1:15 PM, Tom Lane wrote: > Andres Freund writes: >> +++ b/src/include/storage/barrier.h >> +#include "postgres.h" > >> Huh, that normally shouldn't be in a header. I see you introduced that >> in a bunch of other places too - that

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Andres Freund writes: > +++ b/src/include/storage/barrier.h > +#include "postgres.h" > Huh, that normally shouldn't be in a header. I see you introduced that > in a bunch of other places too - that really doesn't look right to me. That is absolutely not project style and is

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, 0001: Do hash join work_mem accounting in chunks. Don't think there's much left to say. 0002: Check hash join work_mem usage at the point of chunk allocation. Modify the existing hash join code to detect work_mem exhaustion at the point where chunks are allocated, instead of checking after

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, On 2017-03-07 02:57:30 +1300, Thomas Munro wrote: > I'm not sure why nodeHashjoin.c is doing raw batchfile read/write > operations anyway; why not use tuplestore.c for that (as > tuplestore.c's comments incorrectly say is the case)? Another reason presumably is that using tuplestores would

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Mar 1, 2017 at 10:40 PM, Thomas Munro wrote: > I'm testing a new version which incorporates feedback from Andres and > Ashutosh, and is refactored to use a new SharedBufFileSet component to > handle batch files, replacing the straw-man implementation from

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 16, 2017 at 9:08 PM, Thomas Munro wrote: > On Thu, Feb 16, 2017 at 3:36 PM, Andres Freund wrote: >> That's it for now... > > Thanks! Plenty for me to go away and think about. I will post a new > version soon. I'm testing a new

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Feb 15, 2017 at 9:36 PM, Andres Freund wrote: > 0002-hj-add-dtrace-probes-v5.patch > > Hm. I'm personally very unenthusiastic about addming more of these, and > would rather rip all of them out. I tend to believe that static > problems simply aren't a good approach

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 16, 2017 at 3:36 PM, Andres Freund wrote: > Hi, Thanks for the review! > FWIW, I'd appreciate if you'd added a short commit message to the > individual patches - I find it helpful to have a littlebit more context > while looking at them than just the titles.

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, Just to summarize what you could read between the lines in the previous mail: From a higher level POV the design here makes sense to me, I do however think there's a good chunk of code-level improvements needed. Regards, Andres -- Sent via pgsql-hackers mailing list

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi, On 2017-02-13 23:57:00 +1300, Thomas Munro wrote: > Here's a new version to fix the problems reported by Rafia above. The > patch descriptions are as before but it starts from 0002 because 0001 > was committed as 7c5d8c16 (thanks, Andres). FWIW, I'd appreciate if you'd added a short commit

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Out of archeological curiosity, I was digging around in the hash join code and RCS history from Postgres 4.2[1], and I was astounded to discover that it had a parallel executor for Sequent SMP systems and was capable of parallel hash joins as of 1991. At first glance, it seems to follow

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 9, 2017 at 2:03 AM, Ashutosh Bapat wrote: >> >> 0004-hj-refactor-batch-increases-v4.patch: >> >> Modify the existing hash join code to detect work_mem exhaustion at >> the point where chunks are allocated, instead of checking after every >> tuple

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 2, 2017 at 4:57 PM, Rafia Sabih wrote: > On Thu, Feb 2, 2017 at 1:19 AM, Thomas Munro > wrote: >> On Thu, Feb 2, 2017 at 3:34 AM, Rafia Sabih >> wrote: >>> [ regressions ] >> >> Thanks Rafia.

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

> > 0004-hj-refactor-batch-increases-v4.patch: > > Modify the existing hash join code to detect work_mem exhaustion at > the point where chunks are allocated, instead of checking after every > tuple insertion. This matches the logic used for estimating, and more > importantly allows for some

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 2, 2017 at 4:57 PM, Rafia Sabih wrote: > Apart from the previously reported regression, there appear one more > issue in this set of patches. At times, running a query using parallel > hash it hangs up and all the workers including the master shows the >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 2, 2017 at 1:19 AM, Thomas Munro wrote: > On Thu, Feb 2, 2017 at 3:34 AM, Rafia Sabih > wrote: >> 9 | 62928.88 | 59077.909 > > Thanks Rafia. At first glance this plan is using the Parallel Shared > Hash in one

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Feb 2, 2017 at 3:34 AM, Rafia Sabih wrote: > 9 | 62928.88 | 59077.909 Thanks Rafia. At first glance this plan is using the Parallel Shared Hash in one place where it should pay off, that is loading the orders table, but the numbers are terrible.

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

> >> However, ExecHashIncreaseNumBatches() may change the >> number of buckets; the patch does not seem to account for spaceUsed changes >> because of that. > > That's what this hunk is intended to do: > > @@ -795,6 +808,12 @@ ExecHashIncreaseNumBuckets(HashJoinTable hashtable) >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Feb 1, 2017 at 2:10 AM, Ashutosh Bapat wrote: >> >> 0003-hj-refactor-memory-accounting-v4.patch: >> [...] >> > I looked at this patch. I agree that it accounts the memory usage more > accurately. Here are few comments. Thanks for the review! > spaceUsed

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

> > 0003-hj-refactor-memory-accounting-v4.patch: > > Modify the existing hash join code to work in terms of chunks when > estimating and later tracking memory usage. This is probably more > accurate than the current tuple-based approach, because it tries to > take into account the space used by

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Jan 28, 2017 at 10:03 AM, Thomas Munro wrote: > I have broken this up into a patch series, harmonised the private vs > shared hash table code paths better and fixed many things including > the problems with rescans and regression tests mentioned upthread. >

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Jan 7, 2017 at 9:01 AM, Thomas Munro wrote: > Stepping back a bit, I am aware of the following approaches to hash > join parallelism: > > 1. Run the inner plan and build a private hash table in each > participant [...]. > > 2. Run a partition-wise hash

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

Hi Thomas, On Fri, Jan 27, 2017 at 5:03 PM, Thomas Munro wrote: > I have broken this up into a patch series, harmonised the private vs > shared hash table code paths better and fixed many things including > the problems with rescans and regression tests mentioned

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Fri, Jan 13, 2017 at 2:36 PM, Peter Geoghegan wrote: > [...] > Yeah. That's basically what the BufFile unification process can > provide you with (or will, once I get around to implementing the > refcount thing, which shouldn't be too hard). As already noted, I'll > also want

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Jan 11, 2017 at 7:37 PM, Thomas Munro wrote: > Hmm. Yes, that is an entirely bogus use of isInterXact. I am > thinking about how to fix that with refcounts. Cool. As I said, the way I'd introduce refcounts would not be very different from what I've

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Jan 12, 2017 at 9:07 AM, Thomas Munro wrote: > On Wed, Jan 11, 2017 at 2:56 PM, Peter Geoghegan wrote: > > On Fri, Jan 6, 2017 at 12:01 PM, Thomas Munro > > wrote: > >> Here is a new WIP patch. I have

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Jan 11, 2017 at 2:56 PM, Peter Geoghegan wrote: > On Fri, Jan 6, 2017 at 12:01 PM, Thomas Munro > wrote: >> Here is a new WIP patch. I have plenty of things to tidy up (see note >> at end), but the main ideas are now pretty clear and I'd

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Jan 11, 2017 at 12:05 PM, Robert Haas wrote: > On Wed, Jan 11, 2017 at 2:20 PM, Peter Geoghegan wrote: >> You'd probably still want to throw an error when workers ended up not >> deleting BufFile segments they owned, though, at least for parallel

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Jan 11, 2017 at 11:20 AM, Peter Geoghegan wrote: >> If multiple processes are using the same file via the BufFile >> interface, I think that it is absolutely necessary that there should >> be a provision to track the "attach count" of the BufFile. Each >> process that

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Jan 11, 2017 at 2:20 PM, Peter Geoghegan wrote: > You'd probably still want to throw an error when workers ended up not > deleting BufFile segments they owned, though, at least for parallel > tuplesort. Don't see why. > This idea is something that's much more limited

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Wed, Jan 11, 2017 at 10:57 AM, Robert Haas wrote: > On Tue, Jan 10, 2017 at 8:56 PM, Peter Geoghegan wrote: >> Instead of all this, I suggest copying some of my changes to fd.c, so >> that resource ownership within fd.c differentiates between a vfd that

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Tue, Jan 10, 2017 at 8:56 PM, Peter Geoghegan wrote: > Instead of all this, I suggest copying some of my changes to fd.c, so > that resource ownership within fd.c differentiates between a vfd that > is owned by the backend in the conventional sense, including having a > need

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Fri, Jan 6, 2017 at 12:01 PM, Thomas Munro wrote: > Here is a new WIP patch. I have plenty of things to tidy up (see note > at end), but the main ideas are now pretty clear and I'd appreciate > some feedback. I have some review feedback for your V3. I've chosen

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Jan 7, 2017 at 9:01 AM, Thomas Munro wrote: > On Tue, Jan 3, 2017 at 10:53 PM, Thomas Munro > wrote: >> I will post a new rebased version soon with that and >> some other nearby problems fixed. > > Here is a new WIP patch. To

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Jan 7, 2017 at 9:01 AM, Thomas Munro wrote: > On Tue, Jan 3, 2017 at 10:53 PM, Thomas Munro > wrote: >> I will post a new rebased version soon with that and >> some other nearby problems fixed. > > Here is a new WIP patch. I

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Mon, Jan 2, 2017 at 3:17 PM, Peter Geoghegan wrote: > I noticed a bug in your latest revision: > >> + /* >> +* In HJ_NEED_NEW_OUTER, we already selected the current inner batch for >> +* reading from. If there is a shared hash table, we may have already >> +*

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Dec 31, 2016 at 2:52 AM, Thomas Munro wrote: > Unfortunately it's been a bit trickier than I anticipated to get the > interprocess batch file sharing and hash table shrinking working > correctly and I don't yet have a new patch in good enough shape to > post

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Sat, Dec 3, 2016 at 1:38 AM, Haribabu Kommi wrote: > Moved to next CF with "waiting on author" status. Unfortunately it's been a bit trickier than I anticipated to get the interprocess batch file sharing and hash table shrinking working correctly and I don't yet have

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

On Thu, Nov 3, 2016 at 4:19 PM, Thomas Munro wrote: > Obviously I'm actively working on developing and stabilising all this. > Some of the things I'm working on are: work_mem accounting, batch > increases, rescans and figuring out if the resource management for >

[HACKERS] WIP: [[Parallel] Shared] Hash

Hi hackers, In PostgreSQL 9.6, hash joins can be parallelised under certain conditions, but a copy of the hash table is built in every participating backend. That means that memory and CPU time are wasted. In many cases, that's OK: if the hash table contents are small and cheap to compute, then