Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-03-16 Thread Claudio Freire
On Fri, Feb 9, 2018 at 1:05 PM, Claudio Freire wrote: > Turns out that it was a tad oversized. 300k tuples seems enough. > > Attached is a new patch version that: > > - Uses an unlogged table to make the large mwm test faster > - Uses a wait_barrier helper that waits for concurrent transactions >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-19 Thread Claudio Freire
On Fri, Aug 18, 2017 at 8:39 AM, Claudio Freire wrote: > On Fri, Apr 7, 2017 at 10:51 PM, Claudio Freire > wrote: >> Indeed they do, and that's what motivated this patch. But I'd need >> TB-sized tables to set up something like that. I don't have the >> hardware or time available to do that (vac

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-09 Thread Claudio Freire
On Fri, Feb 9, 2018 at 10:32 AM, Alvaro Herrera wrote: > Claudio Freire wrote: >> On Thu, Feb 8, 2018 at 8:39 PM, Alvaro Herrera >> wrote: > >> During the process of developing the patch, I got seriously broken >> code that passed the tests nonetheless. The test as it was was very >> ineffective

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-09 Thread Alvaro Herrera
Claudio Freire wrote: > On Thu, Feb 8, 2018 at 8:39 PM, Alvaro Herrera > wrote: > During the process of developing the patch, I got seriously broken > code that passed the tests nonetheless. The test as it was was very > ineffective at actually detecting issues. > > This new test may be slow, b

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-09 Thread Claudio Freire
On Thu, Feb 8, 2018 at 8:39 PM, Alvaro Herrera wrote: > Claudio Freire wrote: > >> I don't like looping, though, seems overly cumbersome. What's worse? >> maintaining that fragile weird loop that might break (by causing >> unexpected output), or a slight slowdown of the test suite? >> >> I don't k

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-08 Thread Alvaro Herrera
Claudio Freire wrote: > I don't like looping, though, seems overly cumbersome. What's worse? > maintaining that fragile weird loop that might break (by causing > unexpected output), or a slight slowdown of the test suite? > > I don't know how long it might take on slow machines, but in my > machin

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Claudio Freire
On Thu, Feb 8, 2018 at 2:44 AM, Kyotaro HORIGUCHI wrote: > Hello, I looked this a bit closer. > > In upthread[1] Robert mentioned the exponentially increasing size > of additional segments. > >>> Hmm, I had imagined making all of the segments the same size rather >>> than having the size grow expo

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Kyotaro HORIGUCHI
Hello, I looked this a bit closer. In upthread[1] Robert mentioned the exponentially increasing size of additional segments. >> Hmm, I had imagined making all of the segments the same size rather >> than having the size grow exponentially. The whole point of this is >> to save memory, and even i

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Claudio Freire
On Thu, Feb 8, 2018 at 12:13 AM, Claudio Freire wrote: > On Wed, Feb 7, 2018 at 11:29 PM, Alvaro Herrera > wrote: >> Claudio Freire wrote: >>> On Wed, Feb 7, 2018 at 8:52 PM, Alvaro Herrera >>> wrote: >>> >> Waiting as you say would be akin to what the patch does by putting >>> >> vacuum on it

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Claudio Freire
On Wed, Feb 7, 2018 at 11:29 PM, Alvaro Herrera wrote: > Claudio Freire wrote: >> On Wed, Feb 7, 2018 at 8:52 PM, Alvaro Herrera >> wrote: >> >> Waiting as you say would be akin to what the patch does by putting >> >> vacuum on its own parallel group. >> > >> > I don't think it's the same. We d

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Alvaro Herrera
Claudio Freire wrote: > On Wed, Feb 7, 2018 at 8:52 PM, Alvaro Herrera > wrote: > >> Waiting as you say would be akin to what the patch does by putting > >> vacuum on its own parallel group. > > > > I don't think it's the same. We don't need to wait until all the > > concurrent tests are done --

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Claudio Freire
On Wed, Feb 7, 2018 at 8:52 PM, Alvaro Herrera wrote: >> Waiting as you say would be akin to what the patch does by putting >> vacuum on its own parallel group. > > I don't think it's the same. We don't need to wait until all the > concurrent tests are done -- we only need to wait until the trans

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Alvaro Herrera
Claudio Freire wrote: > On Wed, Feb 7, 2018 at 7:57 PM, Alvaro Herrera > wrote: > > Claudio Freire wrote: > > Hmm, this solution is not very friendly to the goal of reducing test > > runtime, particularly since the new test creates a nontrivial-sized > > table. Maybe we can find a better altern

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Claudio Freire
On Wed, Feb 7, 2018 at 7:57 PM, Alvaro Herrera wrote: > Claudio Freire wrote: > >> - vacuum test on its own parallel group > > Hmm, this solution is not very friendly to the goal of reducing test > runtime, particularly since the new test creates a nontrivial-sized > table. Maybe we can find a be

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Alvaro Herrera
Claudio Freire wrote: > - vacuum test on its own parallel group Hmm, this solution is not very friendly to the goal of reducing test runtime, particularly since the new test creates a nontrivial-sized table. Maybe we can find a better alternative. Can we use some wait logic instead? Maybe some

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-07 Thread Claudio Freire
On Wed, Feb 7, 2018 at 12:50 AM, Kyotaro HORIGUCHI wrote: > Hello, > > At Tue, 6 Feb 2018 10:41:22 -0300, Claudio Freire > wrote in >> On Tue, Feb 6, 2018 at 10:18 AM, Claudio Freire >> wrote: >> > On Tue, Feb 6, 2018 at 4:35 AM, Kyotaro HORIGUCHI >> > wrote: >> >>> It's starting to look lik

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-06 Thread Kyotaro HORIGUCHI
Hello, At Tue, 6 Feb 2018 10:41:22 -0300, Claudio Freire wrote in > On Tue, Feb 6, 2018 at 10:18 AM, Claudio Freire > wrote: > > On Tue, Feb 6, 2018 at 4:35 AM, Kyotaro HORIGUCHI > > wrote: > >>> It's starting to look like a timing effect indeed. > >> > >> It seems to be truncation skip, may

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-06 Thread Claudio Freire
On Tue, Feb 6, 2018 at 10:18 AM, Claudio Freire wrote: > On Tue, Feb 6, 2018 at 4:35 AM, Kyotaro HORIGUCHI > wrote: >>> It's starting to look like a timing effect indeed. >> >> It seems to be truncation skip, maybe caused by concurrent >> autovacuum. > > Good point, I'll also disable autovacuum o

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-06 Thread Claudio Freire
On Tue, Feb 6, 2018 at 4:35 AM, Kyotaro HORIGUCHI wrote: >> It's starting to look like a timing effect indeed. > > It seems to be truncation skip, maybe caused by concurrent > autovacuum. Good point, I'll also disable autovacuum on vactst. > See lazy_truncate_heap() for details. Updates of > pg_

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-05 Thread Kyotaro HORIGUCHI
Hello, At Fri, 2 Feb 2018 19:52:02 -0300, Claudio Freire wrote in > On Thu, Jan 25, 2018 at 6:21 PM, Thomas Munro > wrote: > > On Fri, Jan 26, 2018 at 9:38 AM, Claudio Freire > > wrote: > >> I had the tests running in a loop all day long, and I cannot reproduce > >> that variance. > >> > >>

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-02-02 Thread Claudio Freire
On Thu, Jan 25, 2018 at 6:21 PM, Thomas Munro wrote: > On Fri, Jan 26, 2018 at 9:38 AM, Claudio Freire > wrote: >> I had the tests running in a loop all day long, and I cannot reproduce >> that variance. >> >> Can you share your steps to reproduce it, including configure flags? > > Here are two

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-25 Thread Thomas Munro
On Fri, Jan 26, 2018 at 9:38 AM, Claudio Freire wrote: > I had the tests running in a loop all day long, and I cannot reproduce > that variance. > > Can you share your steps to reproduce it, including configure flags? Here are two build logs where it failed: https://travis-ci.org/postgresql-cfbo

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-25 Thread Claudio Freire
On Thu, Jan 25, 2018 at 10:56 AM, Claudio Freire wrote: > On Thu, Jan 25, 2018 at 4:11 AM, Thomas Munro > wrote: >> On Thu, Jan 18, 2018 at 9:17 AM, Claudio Freire >> wrote: >>> Huh. That was simpler than I thought. >>> >>> Attached rebased versions. >> >> Hi Claudio, >> >> FYI the regression t

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-25 Thread Alvaro Herrera
Claudio Freire wrote: > On Thu, Jan 25, 2018 at 4:11 AM, Thomas Munro > wrote: > > *** 128,134 > > SELECT pg_relation_size('vactst', 'main'); > >pg_relation_size > > -- > > ! 0 > > (1 row) > > > > SELECT count(*) FROM vactst; > > --- 128,134 >

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-25 Thread Claudio Freire
On Thu, Jan 25, 2018 at 4:11 AM, Thomas Munro wrote: > On Thu, Jan 18, 2018 at 9:17 AM, Claudio Freire > wrote: >> Huh. That was simpler than I thought. >> >> Attached rebased versions. > > Hi Claudio, > > FYI the regression test seems to have some run-to-run variation. > Though it usually succe

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-24 Thread Thomas Munro
On Thu, Jan 18, 2018 at 9:17 AM, Claudio Freire wrote: > Huh. That was simpler than I thought. > > Attached rebased versions. Hi Claudio, FYI the regression test seems to have some run-to-run variation. Though it usually succeeds, recently I have seen a couple of failures like this: = C

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-17 Thread Claudio Freire
On Wed, Jan 17, 2018 at 5:02 PM, Claudio Freire wrote: > > > On Sat, Jan 6, 2018 at 7:35 PM, Stephen Frost wrote: > >> Greetings, >> >> * Michael Paquier (michael.paqu...@gmail.com) wrote: >> > On Mon, Dec 4, 2017 at 2:38 PM, Claudio Freire >> wrote: >> > > They did apply at the time, but I thi

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-17 Thread Claudio Freire
On Sat, Jan 6, 2018 at 7:35 PM, Stephen Frost wrote: > Greetings, > > * Michael Paquier (michael.paqu...@gmail.com) wrote: > > On Mon, Dec 4, 2017 at 2:38 PM, Claudio Freire > wrote: > > > They did apply at the time, but I think major work on vacuum was > > > pushed since then, and also I was tr

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2018-01-06 Thread Stephen Frost
Greetings, * Michael Paquier (michael.paqu...@gmail.com) wrote: > On Mon, Dec 4, 2017 at 2:38 PM, Claudio Freire wrote: > > They did apply at the time, but I think major work on vacuum was > > pushed since then, and also I was traveling so out of reach. > > > > It may take some time to rebase the

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2017-12-03 Thread Michael Paquier
On Mon, Dec 4, 2017 at 2:38 PM, Claudio Freire wrote: > They did apply at the time, but I think major work on vacuum was > pushed since then, and also I was traveling so out of reach. > > It may take some time to rebase them again. Should I move to needs > review myself after that? Sure, if you c

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2017-12-03 Thread Claudio Freire
On Tue, Nov 28, 2017 at 10:37 PM, Michael Paquier wrote: > On Mon, Oct 2, 2017 at 11:02 PM, Claudio Freire > wrote: >> Rebased version of the patches attached > > The status of the patch is misleading: > https://commitfest.postgresql.org/15/844/. This was marked as waiting > on author but a new

Re: [HACKERS] Vacuum: allow usage of more than 1GB of work mem

2017-11-28 Thread Michael Paquier
On Mon, Oct 2, 2017 at 11:02 PM, Claudio Freire wrote: > Rebased version of the patches attached The status of the patch is misleading: https://commitfest.postgresql.org/15/844/. This was marked as waiting on author but a new version has been published. Let's be careful. The last patches I am aw