Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-15 Thread Tom Lane
Peter Eisentraut writes: > make check-world -O -j6 PROVE_FLAGS=-j6 > 2 min 34 seconds > Nice! One thing I've noticed with parallel check-world is that it's possible for a sub-job to fail and for the resulting complaint from make to scroll offscreen before

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-15 Thread Peter Eisentraut
make check-world -O -j6 PROVE_FLAGS=-j6 2 min 34 seconds Nice! -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-14 Thread Andres Freund
On 2017-03-13 00:35:06 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-03-11 22:14:07 -0500, Tom Lane wrote: > >> This looks generally sane to me, although I'm not very happy about folding > >> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check ---

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-13 Thread Andrew Dunstan
On 03/13/2017 07:21 PM, Andrew Dunstan wrote: > > On 03/13/2017 12:35 AM, Tom Lane wrote: >> Andres Freund writes: >>> On 2017-03-11 22:14:07 -0500, Tom Lane wrote: This looks generally sane to me, although I'm not very happy about folding the "$(MKDIR_P)

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-13 Thread Andrew Dunstan
On 03/13/2017 12:35 AM, Tom Lane wrote: > Andres Freund writes: >> On 2017-03-11 22:14:07 -0500, Tom Lane wrote: >>> This looks generally sane to me, although I'm not very happy about folding >>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that >>>

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-12 Thread Tom Lane
Andres Freund writes: > On 2017-03-11 22:14:07 -0500, Tom Lane wrote: >> This looks generally sane to me, although I'm not very happy about folding >> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that >> seems weird and unlike the way it's done for the

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-12 Thread Andres Freund
0;115;0c On 2017-03-11 22:14:07 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-03-11 11:48:31 -0800, Andres Freund wrote: > >> I think that'd be a good plan. We probably should also keep --outputdir > >> seperate (which test_decoding/Makefile does, but > >>

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Tom Lane
Andres Freund writes: > On 2017-03-11 11:48:31 -0800, Andres Freund wrote: >> I think that'd be a good plan. We probably should also keep --outputdir >> seperate (which test_decoding/Makefile does, but >> pg_isolation_regress_check doesn't)? > Here's a patch doing that

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Andres Freund
On 2017-03-11 11:48:31 -0800, Andres Freund wrote: > On 2017-03-11 12:05:23 -0500, Tom Lane wrote: > > I wrote: > > > I believe the core problem is that contrib/test_decoding's regresscheck > > > and isolationcheck targets each want to use ./tmp_check as their > > > --temp-instance. make has no

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Andres Freund
On 2017-03-11 14:17:42 -0800, Andres Freund wrote: > On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote: > > FWIW, +1 on improving matters here. > > > > Andres Freund wrote: > > > > > The best I can come up so far is a toplevel target that creates the temp > > > install, starts a cluster and then

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Andres Freund
On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote: > FWIW, +1 on improving matters here. > > Andres Freund wrote: > > > The best I can come up so far is a toplevel target that creates the temp > > install, starts a cluster and then runs the 'installcheck-or-check' > > target on all the

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Jim Nasby
On 3/11/17 2:06 PM, Tom Lane wrote: Jim Nasby writes: It's actually a lot harder to mess up providing a git repo link than manually submitting patches to the mailing list. Yeah, we've heard that proposal before. We're still not doing it though. Insisting on patches

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Tom Lane
Jim Nasby writes: > It's actually a lot harder to mess up providing a git repo link than > manually submitting patches to the mailing list. Yeah, we've heard that proposal before. We're still not doing it though. Insisting on patches being actually submitted to the

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Andres Freund
On 2017-03-11 12:05:23 -0500, Tom Lane wrote: > I wrote: > > I believe the core problem is that contrib/test_decoding's regresscheck > > and isolationcheck targets each want to use ./tmp_check as their > > --temp-instance. make has no reason to believe it shouldn't run those > > two sub-jobs in

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Jim Nasby
On 3/10/17 6:06 PM, Peter Eisentraut wrote: On 3/10/17 19:00, Jim Nasby wrote: Maybe instead of having the commitfest app try and divine patches from the list it should be able to send patches to the list from a specified git repo/branch. Anyone that provides that info would have tests run

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-11 Thread Tom Lane
I wrote: > I believe the core problem is that contrib/test_decoding's regresscheck > and isolationcheck targets each want to use ./tmp_check as their > --temp-instance. make has no reason to believe it shouldn't run those > two sub-jobs in parallel, but when it does, we get two postmasters trying

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Tom Lane
Peter Eisentraut writes: make check-world -j2 seems to run fine for me. > I have been pounding it a bit, and every so often the test_decoding > tests fail in mysterious ways, but otherwise it seems to work fine. I'm > curious what you are seeing. For me,

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Peter Eisentraut
On 3/10/17 19:00, Jim Nasby wrote: > Maybe instead of having the commitfest app try and divine patches from > the list it should be able to send patches to the list from a specified > git repo/branch. Anyone that provides that info would have tests run > automagically, patches sent, etc. Anyone

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Peter Eisentraut
On 3/10/17 14:43, Andres Freund wrote: > I do get regular issues, although the happen to not end up in visible > failures. All the tests output their regression.diffs into the same > place - which means there'll every now be a failure to remove the file, > and if there were an actual failure it'd

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Jim Nasby
On 3/10/17 5:57 PM, Peter Eisentraut wrote: On 3/10/17 14:53, Jim Nasby wrote: The biggest win we'd get from something like Travis would be if the commitfest monitored for new patch files coming in for monitored threads and it created a new branch, applied the patches, and if they applied

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Peter Eisentraut
On 3/10/17 14:53, Jim Nasby wrote: > The biggest win we'd get from something like Travis would be if the > commitfest monitored for new patch files coming in for monitored threads > and it created a new branch, applied the patches, and if they applied > without error commit the branch and push

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Jim Nasby
On 3/10/17 2:18 PM, Magnus Hagander wrote: But if you can put together something that picks up the individual patches out of the mail threads in the CF app and keeps branch-tips in a git repo up-to-date with those, including feeding the results back into the app, then go for it :) Seems like

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Magnus Hagander
On Fri, Mar 10, 2017 at 3:11 PM, Jim Nasby wrote: > On 3/10/17 2:05 PM, Magnus Hagander wrote: > >> >> Travis specifically would not help us with this, due to the dependency >> on gifhub, but something that knows how to run "patch ... && configure >> && make && make check"

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Jim Nasby
On 3/10/17 2:05 PM, Magnus Hagander wrote: Travis specifically would not help us with this, due to the dependency on gifhub, but something that knows how to run "patch ... && configure && make && make check" in a container would. Who's updating https://github.com/postgres/postgres/ right now?

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Magnus Hagander
On Fri, Mar 10, 2017 at 2:53 PM, Jim Nasby wrote: > On 3/10/17 1:09 PM, Peter Eisentraut wrote: > >> On 3/10/17 03:27, Jim Nasby wrote: >> >>> Perhaps https://travis-ci.org/ or something similar could be used for >>> this. That avoids any issues about random code. >>> >>

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Jim Nasby
On 3/10/17 1:09 PM, Peter Eisentraut wrote: On 3/10/17 03:27, Jim Nasby wrote: Perhaps https://travis-ci.org/ or something similar could be used for this. That avoids any issues about random code. That doesn't achieve any platform coverage, which is the main point here. I don't think

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Andres Freund
On 2017-03-10 01:59:53 -0500, Peter Eisentraut wrote: > On 3/8/17 16:49, Andres Freund wrote: > >> make check-world -j2 seems to run fine for me. > > > > Hm, I at least used to get a lot of spurious failures with this. I > > e.g. don't think the free port selection is race free. > > I was also

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Peter Eisentraut
On 3/10/17 03:27, Jim Nasby wrote: > Perhaps https://travis-ci.org/ or something similar could be used for > this. That avoids any issues about random code. That doesn't achieve any platform coverage, which is the main point here. -- Peter Eisentraut http://www.2ndQuadrant.com/

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Peter Eisentraut
On 3/10/17 13:02, Magnus Hagander wrote: > for their largest feature development. Could be a good idea to for > example ave an example yml file somewhere on the wiki that people could > put into their branches. https://github.com/petere/postgresql/blob/travis/.travis.yml -- Peter Eisentraut

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Magnus Hagander
On Fri, Mar 10, 2017 at 11:29 AM, Dagfinn Ilmari Mannsåker < ilm...@ilmari.org> wrote: > Magnus Hagander writes: > > > AFAIK travis-ci would require us to use github as our hoster for all > those > > things, and embrace that workflow, they don't support anything else. > > >

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Dagfinn Ilmari Mannsåker
Magnus Hagander writes: > AFAIK travis-ci would require us to use github as our hoster for all those > things, and embrace that workflow, they don't support anything else. > > There might be others that do, just not travis. It merely requires the repository to exist on

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Magnus Hagander
On Fri, Mar 10, 2017 at 3:27 AM, Jim Nasby wrote: > On 3/7/17 9:52 PM, Magnus Hagander wrote: > >> There have also on and off been discussions about building arbitrary >> patches as they are sent to the mailinglists. Doing that without any >> committer (or other trusted

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-10 Thread Jim Nasby
On 3/7/17 9:52 PM, Magnus Hagander wrote: There have also on and off been discussions about building arbitrary patches as they are sent to the mailinglists. Doing that without any committer (or other trusted party) as a filter is a completely different challenge of course, given that it

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-09 Thread Peter Eisentraut
On 3/8/17 16:49, Andres Freund wrote: >> make check-world -j2 seems to run fine for me. > > Hm, I at least used to get a lot of spurious failures with this. I > e.g. don't think the free port selection is race free. I was also not sure about that, but as Michael has pointed out, that doesn't

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-08 Thread Michael Paquier
On Thu, Mar 9, 2017 at 6:49 AM, Andres Freund wrote: > On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote: >> On 3/6/17 19:53, Andres Freund wrote: >> > I'm just not quite sure what the best way is to make it easier to run >> > tests in parallel within the tree. >> >> make

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-08 Thread Andres Freund
On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote: > On 3/6/17 19:53, Andres Freund wrote: > > I'm just not quite sure what the best way is to make it easier to run > > tests in parallel within the tree. > > make check-world -j2 seems to run fine for me. Hm, I at least used to get a lot of

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-08 Thread Peter Eisentraut
On 3/6/17 19:53, Andres Freund wrote: > I'm just not quite sure what the best way is to make it easier to run > tests in parallel within the tree. make check-world -j2 seems to run fine for me. With higher -j I appear to be running out of memory or disks space, so I haven't checked that any

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-08 Thread Robert Haas
On Tue, Mar 7, 2017 at 11:23 PM, Andres Freund wrote: > Personally that's not addressing my main concern, which is that the > latency of getting done with some patch/topic takes a long while. If I > have to wait for the buildfarm to check some preliminary patch, I still > have

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-07 Thread Andres Freund
On 2017-03-07 20:58:35 +0800, Simon Riggs wrote: > On 7 March 2017 at 20:36, Alvaro Herrera wrote: > > > FWIW, +1 on improving matters here. > > +1 also. > > I don't see what's wrong with relying on buildfarm though; testing is > exactly what its there for. > > If we

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-07 Thread Magnus Hagander
On Tue, Mar 7, 2017 at 7:24 AM, Andrew Dunstan < andrew.duns...@2ndquadrant.com> wrote: > > > On 03/07/2017 07:58 AM, Simon Riggs wrote: > > On 7 March 2017 at 20:36, Alvaro Herrera > wrote: > > > >> FWIW, +1 on improving matters here. > > +1 also. > > > > I don't see

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-07 Thread Andrew Dunstan
On 03/07/2017 07:58 AM, Simon Riggs wrote: > On 7 March 2017 at 20:36, Alvaro Herrera wrote: > >> FWIW, +1 on improving matters here. > +1 also. > > I don't see what's wrong with relying on buildfarm though; testing is > exactly what its there for. > > If we had a

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-07 Thread Dagfinn Ilmari Mannsåker
Simon Riggs writes: > On 7 March 2017 at 20:36, Alvaro Herrera wrote: > >> FWIW, +1 on improving matters here. > > +1 also. > > I don't see what's wrong with relying on buildfarm though; testing is > exactly what its there for. > > If we had a

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-07 Thread Simon Riggs
On 7 March 2017 at 20:36, Alvaro Herrera wrote: > FWIW, +1 on improving matters here. +1 also. I don't see what's wrong with relying on buildfarm though; testing is exactly what its there for. If we had a two-stage process, where committers can issue "trial commits"

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-07 Thread Alvaro Herrera
FWIW, +1 on improving matters here. Andres Freund wrote: > The best I can come up so far is a toplevel target that creates the temp > install, starts a cluster and then runs the 'installcheck-or-check' > target on all the subdirectories via recursion. Individual makefiles can > either use the

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-06 Thread Andres Freund
On 2017-03-06 19:45:27 -0500, Tom Lane wrote: > Stephen Frost writes: > > * Andres Freund (and...@anarazel.de) wrote: > >> I'm not quite sure what the best way to attack this is, but I think we > >> need to do something. > > > I tend to agree with this, though I haven't got

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-06 Thread Tom Lane
Stephen Frost writes: > * Andres Freund (and...@anarazel.de) wrote: >> I'm not quite sure what the best way to attack this is, but I think we >> need to do something. > I tend to agree with this, though I haven't got any great answers, > unfortunately. I don't want to reduce

Re: [HACKERS] Need a builtin way to run all tests faster manner

2017-03-06 Thread Stephen Frost
Andres, * Andres Freund (and...@anarazel.de) wrote: > There's also some tests simply taking way too long, e.g. the pg_dump > tests just do largely redundant tests for 30s. About half of the time in the pg_dump case, at least, appears to be in 010_dump_connstr.pl. I've not looked into it, but

[HACKERS] Need a builtin way to run all tests faster manner

2017-03-06 Thread Andres Freund
Hi, Right now check-world, a sensible thing to run before commits that aren't very narrow, takes a long time. To the point it seems to impact development velocity enough that it's more sensible to just skip it, and rely on the buildfarm. That's not a great solution. A large amount of the time