Re: [HACKERS] make check-world output

2017-04-03 Thread Noah Misch 
On Mon, Apr 03, 2017 at 03:03:44PM +0900, Michael Paquier wrote:
> On Sat, Apr 1, 2017 at 4:28 PM, Noah Misch  wrote:
> > The pg_upgrade test suite originated in an age when "make check-world" was
> > forbidden to depend on Perl; the choice was a shell script or a C program.  
> > We
> > do maintain vcregress.pl:upgradecheck(), a Windows-specific Perl port of the
> > suite.  Maintaining both versions has been tedious.  I'd welcome a
> > high-quality rewrite using src/test/perl facilities.
> 
> I need to check in my archives, but I recall having a WIP patch doing that:
> - remove the shell scripts in src/bin/pg_upgrade.
> - remove upgradecheck in vcregress.pl.
> - add a TAP test in src/bin/pg_upgrade to run the upgrade tests.
> So you are interested by that?

No, not really.  A WIP patch is the easy part.  The bulk of the work is giving
it all the coverage, debuggability, and robustness of the current harness.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-04-03 Thread Michael Paquier 
On Sat, Apr 1, 2017 at 4:28 PM, Noah Misch  wrote:
> The pg_upgrade test suite originated in an age when "make check-world" was
> forbidden to depend on Perl; the choice was a shell script or a C program.  We
> do maintain vcregress.pl:upgradecheck(), a Windows-specific Perl port of the
> suite.  Maintaining both versions has been tedious.  I'd welcome a
> high-quality rewrite using src/test/perl facilities.

I need to check in my archives, but I recall having a WIP patch doing that:
- remove the shell scripts in src/bin/pg_upgrade.
- remove upgradecheck in vcregress.pl.
- add a TAP test in src/bin/pg_upgrade to run the upgrade tests.
So you are interested by that?
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-04-01 Thread Noah Misch 
On Sat, Mar 11, 2017 at 11:14:36AM -0500, Tom Lane wrote:
> Alvaro Herrera  writes:
> > Tom Lane wrote:
> >> What about just reverting 2f227656076a?
> 
> > That works for me too, if we think we no longer need that level of
> > detail.
> 
> A general issue with this sort of messaging is that when things are
> working more or less normally, you'd just as soon not see it ... but
> when you need to debug problems with the test scaffolding itself,
> you want verbosity.

Yep, and it's important for buildfarm logs to capture the verbose output.

> An entirely different big-picture point is "what the heck are we
> doing using a shell script here at all?  It is useless for testing
> on Windows".  Somebody should look into rewriting it in Perl,
> perhaps using the facilities of PostgresNode.pm.

The pg_upgrade test suite originated in an age when "make check-world" was
forbidden to depend on Perl; the choice was a shell script or a C program.  We
do maintain vcregress.pl:upgradecheck(), a Windows-specific Perl port of the
suite.  Maintaining both versions has been tedious.  I'd welcome a
high-quality rewrite using src/test/perl facilities.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-28 Thread Peter Eisentraut 
On 3/10/17 19:15, Jeff Janes wrote:
> Should --enable-tap-tests be mentioned in "32.1.3. Additional Test
> Suites"?   Or at least cross-referenced from "32.4. TAP Tests"?  

Done.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-28 Thread Peter Eisentraut 
On 3/13/17 05:35, Dagfinn Ilmari Mannsåker wrote:
> Another thing I noticed is that there's a bunch of 'diag' calls in the
> tests scripts (particularly ssl/t/001_ssltests.pl and
> recovery/t/001_stream_rep.pl) that should probably be 'note's instead,
> so they don't pollute STDERR in non-verbose mode.  'diag' should only be
> used to output extra diagnostics in the case of test failures, 'note' is
> for test progress/status updates.

This has been fixed.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-13 Thread Jeff Janes 
On Fri, Mar 10, 2017 at 6:19 PM, Tom Lane  wrote:

> Alvaro Herrera  writes:
> > Jeff Janes wrote:
> >> There was some recent discussion about making "make check-world" faster.
> >> I'm all for that, but how about making it quieter?  On both machines
> I've
> >> run it on (CentOS6.8 and Ubuntu 16.04.2), it dumps some gibberish to
> >> stderr, example attached.
>
> > I think you're complaining about the test code added by commit
> > fcd15f13581f.  Together with behavior introduced by 2f227656076a, it is
> > certainly annoying.  I would vote for redirecting that output to a log
> > file which can be ignored/removed unless there is a failure.
>
> What about just reverting 2f227656076a?
>

That takes care of most of it, but leaves:

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
NOTICE:  database "regression" does not exist, skipping

Cheers,

Jeff

Re: [HACKERS] make check-world output

2017-03-13 Thread Dagfinn Ilmari Mannsåker 
Tom Lane  writes:

> For the basic build process, we've largely solved that through the
> use of "make -s".  But we don't really have a comparable "be quiet"
> option for test runs, especially not the TAP tests.  Maybe we need
> to think a bit more globally about what it is we're trying ton
> accomplish.

Removing --verbose from PROVE_FLAGS cuts the check-world output from
about 9.4k lines to 3.8k, i.e. ~60%.  Doing 'make -s' cuts it by another
1.3k lines.

Another thing I noticed is that there's a bunch of 'diag' calls in the
tests scripts (particularly ssl/t/001_ssltests.pl and
recovery/t/001_stream_rep.pl) that should probably be 'note's instead,
so they don't pollute STDERR in non-verbose mode.  'diag' should only be
used to output extra diagnostics in the case of test failures, 'note' is
for test progress/status updates.

- ilmari

-- 
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least.   - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article.  - Calle Dybedahl


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-11 Thread Tom Lane 
Alvaro Herrera  writes:
> Tom Lane wrote:
>> What about just reverting 2f227656076a?

> That works for me too, if we think we no longer need that level of
> detail.

A general issue with this sort of messaging is that when things are
working more or less normally, you'd just as soon not see it ... but
when you need to debug problems with the test scaffolding itself,
you want verbosity.

For the basic build process, we've largely solved that through the
use of "make -s".  But we don't really have a comparable "be quiet"
option for test runs, especially not the TAP tests.  Maybe we need
to think a bit more globally about what it is we're trying to
accomplish.  I could definitely see that "set -x" in pg_upgrade's
test.sh might be useful as part of a verbose testing option; what
it lacks is a way to turn it off.

An entirely different big-picture point is "what the heck are we
doing using a shell script here at all?  It is useless for testing
on Windows".  Somebody should look into rewriting it in Perl,
perhaps using the facilities of PostgresNode.pm.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-11 Thread Alvaro Herrera 
Tom Lane wrote:
> Alvaro Herrera  writes:
> > Jeff Janes wrote:
> >> There was some recent discussion about making "make check-world" faster.
> >> I'm all for that, but how about making it quieter?  On both machines I've
> >> run it on (CentOS6.8 and Ubuntu 16.04.2), it dumps some gibberish to
> >> stderr, example attached.
> 
> > I think you're complaining about the test code added by commit
> > fcd15f13581f.  Together with behavior introduced by 2f227656076a, it is
> > certainly annoying.  I would vote for redirecting that output to a log
> > file which can be ignored/removed unless there is a failure.
> 
> What about just reverting 2f227656076a?

That works for me too, if we think we no longer need that level of
detail.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-10 Thread Peter Eisentraut 
On 3/10/17 19:26, Jeff Janes wrote:
> and there will be an exit code.
> 
> 
> True.  But I generally don't rely on that, unless the docs explicitly
> tell me to.
> 
> 
> If we show no output, then other people will complain that they can't
> tell whether it's hanging.
> 
> 
> Isn't that what stdout is for?

I haven't ever looked at the difference between stdout and stderr output
from the build process.  I rely on the exit code to tell me what
happened.  I don't mind changing things if it helps.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-10 Thread Tom Lane 
Alvaro Herrera  writes:
> Jeff Janes wrote:
>> There was some recent discussion about making "make check-world" faster.
>> I'm all for that, but how about making it quieter?  On both machines I've
>> run it on (CentOS6.8 and Ubuntu 16.04.2), it dumps some gibberish to
>> stderr, example attached.

> I think you're complaining about the test code added by commit
> fcd15f13581f.  Together with behavior introduced by 2f227656076a, it is
> certainly annoying.  I would vote for redirecting that output to a log
> file which can be ignored/removed unless there is a failure.

What about just reverting 2f227656076a?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-10 Thread Alvaro Herrera 
Jeff Janes wrote:
> There was some recent discussion about making "make check-world" faster.
> I'm all for that, but how about making it quieter?  On both machines I've
> run it on (CentOS6.8 and Ubuntu 16.04.2), it dumps some gibberish to
> stderr, example attached.

I think you're complaining about the test code added by commit
fcd15f13581f.  Together with behavior introduced by 2f227656076a, it is
certainly annoying.  I would vote for redirecting that output to a log
file which can be ignored/removed unless there is a failure.

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-10 Thread Jeff Janes 
On Fri, Mar 10, 2017 at 4:00 PM, Peter Eisentraut <
peter.eisentr...@2ndquadrant.com> wrote:

> On 3/10/17 15:05, Jeff Janes wrote:
> > There was some recent discussion about making "make check-world"
> > faster.  I'm all for that, but how about making it quieter?  On both
> > machines I've run it on (CentOS6.8 and Ubuntu 16.04.2), it dumps some
> > gibberish to stderr, example attached.  Which first made me wonder
> > whether the test passed or failed, and then made me give up on running
> > it altogether when I couldn't easily figure that out. Am I supposed to
> > be seeing this?  Am I supposed to understand it?
>
> Well, you are kind of showing it out of context.  Normally it will tell
> you something at the end,


"make check" doesn't say anything (to stderr) at the end, unless there are
errors.  I am expecting the same of "make check-world".


> and there will be an exit code.
>

True.  But I generally don't rely on that, unless the docs explicitly tell
me to.


> If we show no output, then other people will complain that they can't
> tell whether it's hanging.
>

Isn't that what stdout is for?

Cheers,

Jeff

Re: [HACKERS] make check-world output

2017-03-10 Thread Jeff Janes 
On Fri, Mar 10, 2017 at 12:24 PM, Alvaro Herrera 
wrote:

> Jeff Janes wrote:
>
> > Also, it runs in about 8 minutes, not the 20 minutes reported by others.
> > My system is virtualized, and not particularly fast.  I wonder if it is
> > failing early somewhere without running to completion? How would/should I
> > know?
>
> Maybe you don't have --enable-tap-tests?
>

Yes, adding that does solve the too-fast "problem".  But it doesn't solve
the "I don't know what this output to stderr means" problem.

Should --enable-tap-tests be mentioned in "32.1.3. Additional Test Suites"?
  Or at least cross-referenced from "32.4. TAP Tests"?

It was obvious to me that check-world wouldn't test plperl unless plperl
was configured.  But not so obvious that it wouldn't test anything using
the tap framework unless that was configured, until after you pointed it
out.

Thanks,

Jeff

Re: [HACKERS] make check-world output

2017-03-10 Thread Peter Eisentraut 
On 3/10/17 15:24, Alvaro Herrera wrote:
> Jeff Janes wrote:
> 
>> Also, it runs in about 8 minutes, not the 20 minutes reported by others.
>> My system is virtualized, and not particularly fast.  I wonder if it is
>> failing early somewhere without running to completion? How would/should I
>> know?
> 
> Maybe you don't have --enable-tap-tests?

Yeah, that'll cut down the time.  The recovery tests alone are 5 minutes.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-10 Thread Peter Eisentraut 
On 3/10/17 15:05, Jeff Janes wrote:
> There was some recent discussion about making "make check-world"
> faster.  I'm all for that, but how about making it quieter?  On both
> machines I've run it on (CentOS6.8 and Ubuntu 16.04.2), it dumps some
> gibberish to stderr, example attached.  Which first made me wonder
> whether the test passed or failed, and then made me give up on running
> it altogether when I couldn't easily figure that out. Am I supposed to
> be seeing this?  Am I supposed to understand it?

Well, you are kind of showing it out of context.  Normally it will tell
you something at the end, and there will be an exit code.

If we show no output, then other people will complain that they can't
tell whether it's hanging.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] make check-world output

2017-03-10 Thread Alvaro Herrera 
Jeff Janes wrote:

> Also, it runs in about 8 minutes, not the 20 minutes reported by others.
> My system is virtualized, and not particularly fast.  I wonder if it is
> failing early somewhere without running to completion? How would/should I
> know?

Maybe you don't have --enable-tap-tests?

-- 
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers