Re: [HACKERS] TAP tests - installcheck vs check

[Peter Eisentraut]

A couple of thoughts I've been having that relate to this:

The traditional meaning of "installcheck" in GNU packages is to test
against the installed code, whereas "check" tests before installation.
Our concept of testing against a running server obviously does not apply
to many kinds of software, so there is no standard name for it.

I think as our test suites expand, the concept of testing against a
running server is not going to feasible for many new and interesting
things.  I can see some marginal appeal for what what we're currently
doing, but I don't think it's useful to invest much more effort into it
to make it work in more cases and situations.

What I'm thinking going forward is:

make check - as is

make installcheck - test against installed code, but start new instances
etc. (which you currently can't do for the main test suite)

make runningcheck - test against running server, supported for certain tests

make check and installcheck would be very similar except make check
creates a temp install and sets some path variables.

A trick of sorts I saw in Python packages that might also be worth
adopting is to make the temp install use symlinks pointing into the
source tree.  Then as soon as you rebuild, the temp install is up to
date (unless you add or remove entire files).  Then the temp install is
free after the first run.

Thoughts?

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Andrew Dunstan]

On 04/25/2017 11:13 AM, Tom Lane wrote:
>
>> I'm in the process of moving all the buildfarm tests to use check
>> instead of installcheck, but in such a way that it doesn't constantly
>> generate redundant installs.
> But is that something only of interest to the buildfarm, or should we
> do something in the Makefile infrastructure to make it more generally
> available?


It already is. If you have a temp_install with all the required pieces
do "make NO_TEMP_INSTALL=1 check"

The logic I am adding to the buildfarm is essentially to install contrib
and test_modules in the temp install directory as part of their install
steps and to check that that's been done before using NO_TEMP_INSTALL.

cheers

andrew

-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Tom Lane]

Andrew Dunstan  writes:
> On 04/25/2017 10:45 AM, Robert Haas wrote:
>>> I agree entirely that it's confusing as heck. +1 for inventing a new name.

>> Yeah.  I would have expected installcheck to just skip any tests that
>> don't make sense against an already-installed cluster.  I would not
>> expect it to run those tests against some cluster other than the
>> installed cluster.  That seems super-weird.

> I'm in the process of moving all the buildfarm tests to use check
> instead of installcheck, but in such a way that it doesn't constantly
> generate redundant installs.

But is that something only of interest to the buildfarm, or should we
do something in the Makefile infrastructure to make it more generally
available?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Andrew Dunstan]

On 04/25/2017 10:45 AM, Robert Haas wrote:
> On Sun, Apr 23, 2017 at 10:57 PM, Andrew Dunstan
>  wrote:
>> On 04/23/2017 10:33 PM, Tom Lane wrote:
>>> Andrew Dunstan  writes:
 AFAICT, unlike the pg_regress checks, which in the installcheck case run
 against a running instance of postgres, for TAP tests the only
 difference is that that for the check case a temp install is done,
 possibly with some extra contrib modules. Is that correct? If is is, why
 aren't we providing an installcheck target for tests like recover. In at
 least one case (buildfarmn jacana) installs are quite expensive (2 or 3
 minutes) and if they are pointless as seems to be the case here why
 can't we just avoid them?
>>> A lot of those test cases involve setting non-default configuration
>>> parameters and/or stopping/starting the postmaster.  So I can't see how
>>> we would run them against a pre-existing live cluster, which is the usual
>>> meaning of "make installcheck".
>>>
>>> I think what you're imagining is skipping redundant builds of the
>>> "tmp_install" tree by using an installation tree with a temporary $PGDATA
>>> directory.  That seems like a fine idea, but we need another word for it.
>> That's actually the current meaning of installcheck w.r.t. TAP. See
>> Makefile.global.in. It's what we run (mostly) in the buildfarm for the
>> bin tests.
>>
>> I agree entirely that it's confusing as heck. +1 for inventing a new name.
> Yeah.  I would have expected installcheck to just skip any tests that
> don't make sense against an already-installed cluster.  I would not
> expect it to run those tests against some cluster other than the
> installed cluster.  That seems super-weird.
>



I'm in the process of moving all the buildfarm tests to use check
instead of installcheck, but in such a way that it doesn't constantly
generate redundant installs.

When I'm done testing that and we have a new buildfarm client release
and we get all those buildfarm owners updated (there are 13 animals not
controlled by me running TAP tests) I think we should just rename
installcheck. for TAP.

cheers

andrew

-- 

Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Robert Haas]

On Sun, Apr 23, 2017 at 10:57 PM, Andrew Dunstan
 wrote:
> On 04/23/2017 10:33 PM, Tom Lane wrote:
>> Andrew Dunstan  writes:
>>> AFAICT, unlike the pg_regress checks, which in the installcheck case run
>>> against a running instance of postgres, for TAP tests the only
>>> difference is that that for the check case a temp install is done,
>>> possibly with some extra contrib modules. Is that correct? If is is, why
>>> aren't we providing an installcheck target for tests like recover. In at
>>> least one case (buildfarmn jacana) installs are quite expensive (2 or 3
>>> minutes) and if they are pointless as seems to be the case here why
>>> can't we just avoid them?
>> A lot of those test cases involve setting non-default configuration
>> parameters and/or stopping/starting the postmaster.  So I can't see how
>> we would run them against a pre-existing live cluster, which is the usual
>> meaning of "make installcheck".
>>
>> I think what you're imagining is skipping redundant builds of the
>> "tmp_install" tree by using an installation tree with a temporary $PGDATA
>> directory.  That seems like a fine idea, but we need another word for it.
>
> That's actually the current meaning of installcheck w.r.t. TAP. See
> Makefile.global.in. It's what we run (mostly) in the buildfarm for the
> bin tests.
>
> I agree entirely that it's confusing as heck. +1 for inventing a new name.

Yeah.  I would have expected installcheck to just skip any tests that
don't make sense against an already-installed cluster.  I would not
expect it to run those tests against some cluster other than the
installed cluster.  That seems super-weird.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Andrew Dunstan]

On 04/23/2017 10:33 PM, Tom Lane wrote:
> Andrew Dunstan  writes:
>> AFAICT, unlike the pg_regress checks, which in the installcheck case run
>> against a running instance of postgres, for TAP tests the only
>> difference is that that for the check case a temp install is done,
>> possibly with some extra contrib modules. Is that correct? If is is, why
>> aren't we providing an installcheck target for tests like recover. In at
>> least one case (buildfarmn jacana) installs are quite expensive (2 or 3
>> minutes) and if they are pointless as seems to be the case here why
>> can't we just avoid them?
> A lot of those test cases involve setting non-default configuration
> parameters and/or stopping/starting the postmaster.  So I can't see how
> we would run them against a pre-existing live cluster, which is the usual
> meaning of "make installcheck".
>
> I think what you're imagining is skipping redundant builds of the
> "tmp_install" tree by using an installation tree with a temporary $PGDATA
> directory.  That seems like a fine idea, but we need another word for it.
>
>   


That's actually the current meaning of installcheck w.r.t. TAP. See
Makefile.global.in. It's what we run (mostly) in the buildfarm for the
bin tests.

I agree entirely that it's confusing as heck. +1 for inventing a new name.

cheers

andrew

-- 
Andrew Dunstanhttps://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Tom Lane]

Andrew Dunstan  writes:
> AFAICT, unlike the pg_regress checks, which in the installcheck case run
> against a running instance of postgres, for TAP tests the only
> difference is that that for the check case a temp install is done,
> possibly with some extra contrib modules. Is that correct? If is is, why
> aren't we providing an installcheck target for tests like recover. In at
> least one case (buildfarmn jacana) installs are quite expensive (2 or 3
> minutes) and if they are pointless as seems to be the case here why
> can't we just avoid them?

A lot of those test cases involve setting non-default configuration
parameters and/or stopping/starting the postmaster.  So I can't see how
we would run them against a pre-existing live cluster, which is the usual
meaning of "make installcheck".

I think what you're imagining is skipping redundant builds of the
"tmp_install" tree by using an installation tree with a temporary $PGDATA
directory.  That seems like a fine idea, but we need another word for it.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TAP tests - installcheck vs check

[Michael Paquier]

On Mon, Apr 24, 2017 at 7:29 AM, Andrew Dunstan
 wrote:
>
> AFAICT, unlike the pg_regress checks, which in the installcheck case run
> against a running instance of postgres, for TAP tests the only
> difference is that that for the check case a temp install is done,
> possibly with some extra contrib modules. Is that correct? If is is, why
> aren't we providing an installcheck target for tests like recover. In at
> least one case (buildfarmn jacana) installs are quite expensive (2 or 3
> minutes) and if they are pointless as seems to be the case here why
> can't we just avoid them?

install.pl deploys by default the dll of modules needed for the tests,
so no objections. Don't you think the TAP scripts in src/test/perl
should be installed as well? I think that this would make sense for
consistency with what other Nix platforms do, but there is no real
installation of PGXS there. So perhaps they could be deployed in a
different path like scripts/perl?
-- 
Michael
VMware vCenter Server
www.vmware.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers