On Fri, 19 May 2023 17:23:56 +0200
Tomas Vondra wrote:
> On 5/19/23 00:27, Melanie Plageman wrote:
> > v10 LGTM.
>
> Thanks!
>
> I've pushed 0002 and 0003, after some general bikeshedding and minor
> rewording (a bit audacious, admittedly).
Thank you Melanie et Tomas for your help, review et
Tomas Vondra writes:
> I didn't push 0001, I don't think generally do separate pgindent patches
> like this (I only run pgindent on large patches to ensure it doesn't
> cause massive breakage, not separately like this, but YMMV).
It's especially pointless when the main pgindent run for v16 is
On 5/19/23 00:27, Melanie Plageman wrote:
> On Wed, May 17, 2023 at 6:35 PM Jehan-Guillaume de Rorthais
> wrote:
>>
>> On Wed, 17 May 2023 13:46:35 -0400
>> Melanie Plageman wrote:
>>
>>> On Wed, May 17, 2023 at 07:10:08PM +0200, Jehan-Guillaume de Rorthais wrote:
On Tue, 16 May 2023
On Wed, May 17, 2023 at 6:35 PM Jehan-Guillaume de Rorthais
wrote:
>
> On Wed, 17 May 2023 13:46:35 -0400
> Melanie Plageman wrote:
>
> > On Wed, May 17, 2023 at 07:10:08PM +0200, Jehan-Guillaume de Rorthais wrote:
> > > On Tue, 16 May 2023 16:00:52 -0400
> > > Melanie Plageman wrote:
> > > >
On Wed, 17 May 2023 13:46:35 -0400
Melanie Plageman wrote:
> On Wed, May 17, 2023 at 07:10:08PM +0200, Jehan-Guillaume de Rorthais wrote:
> > On Tue, 16 May 2023 16:00:52 -0400
> > Melanie Plageman wrote:
> > > ...
> > > There are some existing indentation issues in these files, but you can
>
On Wed, May 17, 2023 at 07:10:08PM +0200, Jehan-Guillaume de Rorthais wrote:
> On Tue, 16 May 2023 16:00:52 -0400
> Melanie Plageman wrote:
> > > From 309ad354b7a9e4dfa01b2985bd883829f5e0eba0 Mon Sep 17 00:00:00 2001
> > > From: Jehan-Guillaume de Rorthais
> > > Date: Tue, 16 May 2023 15:42:14
On Tue, 16 May 2023 16:00:52 -0400
Melanie Plageman wrote:
> On Tue, May 16, 2023 at 04:00:51PM +0200, Jehan-Guillaume de Rorthais wrote:
>
> > From e5ecd466172b7bae2f1be294c1a5e70ce2b43ed8 Mon Sep 17 00:00:00 2001
> > From: Melanie Plageman
> > Date: Thu, 30 Apr 2020 07:16:28 -0700
> >
On Sun, May 14, 2023 at 12:10:00AM +0200, Tomas Vondra wrote:
> On 5/12/23 23:36, Melanie Plageman wrote:
> > Thanks for continuing to work on this.
> >
> > Are you planning to modify what is displayed for memory usage in
> > EXPLAIN ANALYZE?
> >
>
> We could do that, but we can do that
On Tue, May 16, 2023 at 04:00:51PM +0200, Jehan-Guillaume de Rorthais wrote:
> From e5ecd466172b7bae2f1be294c1a5e70ce2b43ed8 Mon Sep 17 00:00:00 2001
> From: Melanie Plageman
> Date: Thu, 30 Apr 2020 07:16:28 -0700
> Subject: [PATCH v8 1/3] Describe hash join implementation
>
> This is just a
Hi,
On Tue, 16 May 2023 12:01:51 +0200 Tomas Vondra
wrote:
> On 5/16/23 00:15, Jehan-Guillaume de Rorthais wrote:
> > On Sat, 13 May 2023 23:47:53 +0200
> > Tomas Vondra wrote:
> ...
> >> I'm not really sure about calling this "hybrid hash-join". What does it
> >> even mean? The new comments
On 5/16/23 00:15, Jehan-Guillaume de Rorthais wrote:
> Hi,
>
> Thanks for your review!
>
> On Sat, 13 May 2023 23:47:53 +0200
> Tomas Vondra wrote:
>
>> Thanks for the patches. A couple mostly minor comments, to complement
>> Melanie's review:
>>
>> 0001
>>
>> I'm not really sure about calling
Hi,
Thanks for your review!
On Sat, 13 May 2023 23:47:53 +0200
Tomas Vondra wrote:
> Thanks for the patches. A couple mostly minor comments, to complement
> Melanie's review:
>
> 0001
>
> I'm not really sure about calling this "hybrid hash-join". What does it
> even mean? The new comments
On Sun, 14 May 2023 00:10:00 +0200
Tomas Vondra wrote:
> On 5/12/23 23:36, Melanie Plageman wrote:
> > Thanks for continuing to work on this.
> >
> > Are you planning to modify what is displayed for memory usage in
> > EXPLAIN ANALYZE?
Yes, I already start to work on this. Tracking spilling
On 5/12/23 23:36, Melanie Plageman wrote:
> Thanks for continuing to work on this.
>
> Are you planning to modify what is displayed for memory usage in
> EXPLAIN ANALYZE?
>
We could do that, but we can do that separately - it's a separate and
independent improvement, I think.
Also, do you have
Hi,
Thanks for the patches. A couple mostly minor comments, to complement
Melanie's review:
0001
I'm not really sure about calling this "hybrid hash-join". What does it
even mean? The new comments simply describe the existing batching, and
how we increment number of batches, etc.
When someone
Thanks for continuing to work on this.
Are you planning to modify what is displayed for memory usage in
EXPLAIN ANALYZE?
Also, since that won't help a user who OOMs, I wondered if the spillCxt
is helpful on its own or if we need some kind of logging message for
users to discover that this is
Thank you for your review!
On Mon, 8 May 2023 11:56:48 -0400
Melanie Plageman wrote:
> ...
> > 4. accessor->read_buffer is currently allocated in accessor->context as
> > well.
> >
> >This buffer holds tuple read from the fileset. This is still a buffer,
> > but not related to any file
Thanks for continuing to work on this!
On Thu, May 04, 2023 at 07:30:06PM +0200, Jehan-Guillaume de Rorthais wrote:
> On Fri, 21 Apr 2023 16:44:48 -0400 Melanie Plageman
> wrote:
...
> > I think the biggest change that is needed is to implement this memory
> > context usage for parallel hash
Hi,
On Fri, 21 Apr 2023 16:44:48 -0400
Melanie Plageman wrote:
> On Fri, Apr 7, 2023 at 8:01 PM Jehan-Guillaume de Rorthais
> wrote:
> >
> > On Fri, 31 Mar 2023 14:06:11 +0200
> > Jehan-Guillaume de Rorthais wrote:
> >
> > > > [...]
> > > > >> Hmmm, not sure is WARNING is a good approach,
On Fri, Apr 7, 2023 at 8:01 PM Jehan-Guillaume de Rorthais
wrote:
>
> On Fri, 31 Mar 2023 14:06:11 +0200
> Jehan-Guillaume de Rorthais wrote:
>
> > > [...]
> > > >> Hmmm, not sure is WARNING is a good approach, but I don't have a better
> > > >> idea at the moment.
> > > >
> > > > I stepped it
On 21.04.2023 1:51 AM, Melanie Plageman wrote:
On Thu, Apr 20, 2023 at 12:42 PM Konstantin Knizhnik wrote:
On 11.04.2023 8:14 PM, Jehan-Guillaume de Rorthais wrote:
On Sat, 8 Apr 2023 02:01:19 +0200
Jehan-Guillaume de Rorthais wrote:
On Fri, 31 Mar 2023 14:06:11 +0200
Jehan-Guillaume de
On Thu, Apr 20, 2023 at 12:42 PM Konstantin Knizhnik wrote:
> On 11.04.2023 8:14 PM, Jehan-Guillaume de Rorthais wrote:
> > On Sat, 8 Apr 2023 02:01:19 +0200
> > Jehan-Guillaume de Rorthais wrote:
> >
> >> On Fri, 31 Mar 2023 14:06:11 +0200
> >> Jehan-Guillaume de Rorthais wrote:
> >>
> >>
On 11.04.2023 8:14 PM, Jehan-Guillaume de Rorthais wrote:
On Sat, 8 Apr 2023 02:01:19 +0200
Jehan-Guillaume de Rorthais wrote:
On Fri, 31 Mar 2023 14:06:11 +0200
Jehan-Guillaume de Rorthais wrote:
[...]
After rebasing Tomas' memory balancing patch, I did some memory measures
to answer
On Sat, 8 Apr 2023 02:01:19 +0200
Jehan-Guillaume de Rorthais wrote:
> On Fri, 31 Mar 2023 14:06:11 +0200
> Jehan-Guillaume de Rorthais wrote:
>
> [...]
>
> After rebasing Tomas' memory balancing patch, I did some memory measures
> to answer some of my questions. Please, find in attachment
On Tue, 28 Mar 2023 17:25:49 +0200
Tomas Vondra wrote:
...
> * Note that BufFile structs are allocated with palloc(), and therefore
> * will go away automatically at query/transaction end. Since the
> underlying
> * virtual Files are made with OpenTemporaryFile, all resources for
> * the
On 3/28/23 15:17, Jehan-Guillaume de Rorthais wrote:
> On Tue, 28 Mar 2023 00:43:34 +0200
> Tomas Vondra wrote:
>
>> On 3/27/23 23:13, Jehan-Guillaume de Rorthais wrote:
>>> Please, find in attachment a patch to allocate bufFiles in a dedicated
>>> context. I picked up your patch, backpatch'd
Hi,
Sorry for the late answer, I was reviewing the first patch and it took me some
time to study and dig around.
On Thu, 23 Mar 2023 08:07:04 -0400
Melanie Plageman wrote:
> On Fri, Mar 10, 2023 at 1:51 PM Jehan-Guillaume de Rorthais
> wrote:
> > > So I guess the best thing would be to go
On Tue, 28 Mar 2023 00:43:34 +0200
Tomas Vondra wrote:
> On 3/27/23 23:13, Jehan-Guillaume de Rorthais wrote:
> > Please, find in attachment a patch to allocate bufFiles in a dedicated
> > context. I picked up your patch, backpatch'd it, went through it and did
> > some minor changes to it. I
On 3/27/23 23:13, Jehan-Guillaume de Rorthais wrote:
> Hi,
>
> On Mon, 20 Mar 2023 15:12:34 +0100
> Jehan-Guillaume de Rorthais wrote:
>
>> On Mon, 20 Mar 2023 09:32:17 +0100
>> Tomas Vondra wrote:
>>
> * Patch 1 could be rebased/applied/backpatched
Would it help if I rebase
Hi,
On Mon, 20 Mar 2023 15:12:34 +0100
Jehan-Guillaume de Rorthais wrote:
> On Mon, 20 Mar 2023 09:32:17 +0100
> Tomas Vondra wrote:
>
> > >> * Patch 1 could be rebased/applied/backpatched
> > >
> > > Would it help if I rebase Patch 1 ("move BufFile stuff into separate
> > > context")?
On Thu, Mar 23, 2023 at 2:49 PM Tomas Vondra
wrote:
> > On Mon, Mar 20, 2023 at 10:12 AM Jehan-Guillaume de Rorthais
> > wrote:
> >> BNJL and/or other considerations are for 17 or even after. In the meantime,
> >> Melanie, who authored BNLJ, +1 the balancing patch as it can coexists with
> >>
On 3/23/23 13:07, Melanie Plageman wrote:
> On Fri, Mar 10, 2023 at 1:51 PM Jehan-Guillaume de Rorthais
> wrote:
>>> So I guess the best thing would be to go through these threads, see what
>>> the status is, restart the discussion and propose what to do. If you do
>>> that, I'm happy to
On Fri, Mar 10, 2023 at 1:51 PM Jehan-Guillaume de Rorthais
wrote:
> > So I guess the best thing would be to go through these threads, see what
> > the status is, restart the discussion and propose what to do. If you do
> > that, I'm happy to rebase the patches, and maybe see if I could improve
>
On Mon, 20 Mar 2023 09:32:17 +0100
Tomas Vondra wrote:
> >> * Patch 1 could be rebased/applied/backpatched
> >
> > Would it help if I rebase Patch 1 ("move BufFile stuff into separate
> > context")?
>
> Yeah, I think this is something we'd want to do. It doesn't change the
> behavior, but
On 3/19/23 20:31, Justin Pryzby wrote:
> On Fri, Mar 17, 2023 at 05:41:11PM +0100, Tomas Vondra wrote:
* Patch 2 is worth considering to backpatch
>>
>> I'm not quite sure what exactly are the numbered patches, as some of the
>> threads had a number of different patch ideas, and I'm not sure
On Fri, Mar 17, 2023 at 05:41:11PM +0100, Tomas Vondra wrote:
> >> * Patch 2 is worth considering to backpatch
>
> I'm not quite sure what exactly are the numbered patches, as some of the
> threads had a number of different patch ideas, and I'm not sure which
> one was/is the most promising one.
On 3/17/23 09:18, Jehan-Guillaume de Rorthais wrote:
> Hi there,
>
> On Fri, 10 Mar 2023 19:51:14 +0100
> Jehan-Guillaume de Rorthais wrote:
>
>>> So I guess the best thing would be to go through these threads, see what
>>> the status is, restart the discussion and propose what to do. If you
Hi there,
On Fri, 10 Mar 2023 19:51:14 +0100
Jehan-Guillaume de Rorthais wrote:
> > So I guess the best thing would be to go through these threads, see what
> > the status is, restart the discussion and propose what to do. If you do
> > that, I'm happy to rebase the patches, and maybe see if I
Hi,
> So I guess the best thing would be to go through these threads, see what
> the status is, restart the discussion and propose what to do. If you do
> that, I'm happy to rebase the patches, and maybe see if I could improve
> them in some way.
OK! It took me some time, but I did it. I'll try
On 3/2/23 23:57, Jehan-Guillaume de Rorthais wrote:
> On Thu, 2 Mar 2023 19:53:14 +0100
> Tomas Vondra wrote:
>> On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote:
> ...
>
>>> There was some thoughts about how to make a better usage of the memory. As
>>> memory is exploding way beyond
On Thu, 2 Mar 2023 19:53:14 +0100
Tomas Vondra wrote:
> On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote:
...
> > There was some thoughts about how to make a better usage of the memory. As
> > memory is exploding way beyond work_mem, at least, avoid to waste it with
> > too many buffers of
On 3/2/23 19:15, Jehan-Guillaume de Rorthais wrote:
> Hi!
>
> On Thu, 2 Mar 2023 13:44:52 +0100
> Tomas Vondra wrote:
>> Well, yeah and no.
>>
>> In principle we could/should have allocated the BufFiles in a different
>> context (possibly hashCxt). But in practice it probably won't make any
On Thu, 2 Mar 2023 19:15:30 +0100
Jehan-Guillaume de Rorthais wrote:
[...]
> For what it worth, these two patches seems really interesting to me. Do you
> need any help to revive it?
To avoid confusion, the two patches I meant were:
* 0001-move-BufFile-stuff-into-separate-context.patch
*
Hi!
On Thu, 2 Mar 2023 13:44:52 +0100
Tomas Vondra wrote:
> Well, yeah and no.
>
> In principle we could/should have allocated the BufFiles in a different
> context (possibly hashCxt). But in practice it probably won't make any
> difference, because the query will probably run all the hashjoins
On 3/2/23 13:08, Jehan-Guillaume de Rorthais wrote:
> ...
> [...]
>> But I have another idea - put a breakpoint on makeBufFile() which is the
>> bit that allocates the temp files including the 8kB buffer, and print in
>> what context we allocate that. I have a hunch we may be allocating it in
>>
On Thu, 2 Mar 2023 01:30:27 +0100
Tomas Vondra wrote:
> On 3/2/23 00:18, Jehan-Guillaume de Rorthais wrote:
> >>> ExecHashIncreaseNumBatches was really chatty, having hundreds of thousands
> >>> of calls, always short-cut'ed to 1048576, I guess because of the
> >>> conditional block «/* safety
On 3/2/23 00:18, Jehan-Guillaume de Rorthais wrote:
> Hi,
>
> On Wed, 1 Mar 2023 20:29:11 +0100
> Tomas Vondra wrote:
>> On 3/1/23 18:48, Jehan-Guillaume de Rorthais wrote:
>>> On Tue, 28 Feb 2023 20:51:02 +0100
>>> Tomas Vondra wrote:
On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote:
Hi,
On Wed, 1 Mar 2023 20:29:11 +0100
Tomas Vondra wrote:
> On 3/1/23 18:48, Jehan-Guillaume de Rorthais wrote:
> > On Tue, 28 Feb 2023 20:51:02 +0100
> > Tomas Vondra wrote:
> >> On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote:
> >>> * HashBatchContext goes up to 1441MB after 240s then
On Wed, 1 Mar 2023 20:34:08 +0100
Tomas Vondra wrote:
> On 3/1/23 19:09, Jehan-Guillaume de Rorthais wrote:
> > On Wed, 1 Mar 2023 18:48:40 +0100
> > Jehan-Guillaume de Rorthais wrote:
> > ...
> >> You'll find some intermediate stats I already collected in attachment:
> >>
> >> * break 1, 2
On 3/1/23 19:09, Jehan-Guillaume de Rorthais wrote:
> On Wed, 1 Mar 2023 18:48:40 +0100
> Jehan-Guillaume de Rorthais wrote:
> ...
>> You'll find some intermediate stats I already collected in attachment:
>>
>> * break 1, 2 and 3 are from AllocSetAlloc, break 4 is from AllocSetFree.
>> * most
On 3/1/23 18:48, Jehan-Guillaume de Rorthais wrote:
> Hi Tomas,
>
> On Tue, 28 Feb 2023 20:51:02 +0100
> Tomas Vondra wrote:
>> On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote:
>>> * HashBatchContext goes up to 1441MB after 240s then stay flat until the end
>>> (400s as the last record)
On Wed, 1 Mar 2023 18:48:40 +0100
Jehan-Guillaume de Rorthais wrote:
...
> You'll find some intermediate stats I already collected in attachment:
>
> * break 1, 2 and 3 are from AllocSetAlloc, break 4 is from AllocSetFree.
> * most of the non-free'd chunk are allocated since the very beginning,
Hi Tomas,
On Tue, 28 Feb 2023 20:51:02 +0100
Tomas Vondra wrote:
> On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote:
> > * HashBatchContext goes up to 1441MB after 240s then stay flat until the end
> > (400s as the last record)
>
> That's interesting. We're using HashBatchContext for
On 3/1/23 10:46, Jehan-Guillaume de Rorthais wrote:
> Hi Justin,
>
> On Tue, 28 Feb 2023 12:25:08 -0600
> Justin Pryzby wrote:
>
>> On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote:
>>> Hello all,
>>>
>>> A customer is facing out of memory query which looks similar
Hi Justin,
On Tue, 28 Feb 2023 12:25:08 -0600
Justin Pryzby wrote:
> On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote:
> > Hello all,
> >
> > A customer is facing out of memory query which looks similar to this
> > situation:
> >
> >
> >
On 2/28/23 19:06, Jehan-Guillaume de Rorthais wrote:
> Hello all,
>
> A customer is facing out of memory query which looks similar to this
> situation:
>
>
> https://www.postgresql.org/message-id/flat/12064.1555298699%40sss.pgh.pa.us#eb519865575bbc549007878a5fb7219b
>
> This PostgreSQL
On 2/28/23 19:25, Justin Pryzby wrote:
> On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote:
>> Hello all,
>>
>> A customer is facing out of memory query which looks similar to this
>> situation:
>>
>>
>>
On Tue, Feb 28, 2023 at 07:06:43PM +0100, Jehan-Guillaume de Rorthais wrote:
> Hello all,
>
> A customer is facing out of memory query which looks similar to this
> situation:
>
>
> https://www.postgresql.org/message-id/flat/12064.1555298699%40sss.pgh.pa.us#eb519865575bbc549007878a5fb7219b
>
58 matches
Mail list logo
