Re: [HACKERS] regression test for extended query protocol

[Michael Paquier]

On Thu, Aug 11, 2016 at 5:33 AM, Alvaro Herrera
 wrote:
> Michael Paquier wrote:
>> On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera
>>  wrote:
>> > If somebody had some spare time to devote to this, I would suggest to
>> > implement something in core that can be used to specify a list of
>> > commands to run, and a list of files-to-be-saved-in-bf-log emitted by
>> > each command.  We could have one such base file in the core repo that
>> > specifies some tests to run (such as "make -C src/test/recovery check"),
>> > and an additional file can be given by buildfarm animals to run custom
>> > tests, without having to write BF modules for each thing.  With that,
>> > pgsql committers could simply add a new test to be run by all buildfarm
>> > animals by adding it to the list in core.
>>
>> Do you think about using a special makefile target to run those
>> commands, say in src/test? At the end we are going to need to patch
>> the buildfarm client code at least once, at least that would be worth
>> it in the long term..
>
> Sure.  Some time ago I proposed something like a JSON file, something
> like
>
> test_foo => {
>   "command" : "make -C src/test/blah check",
>   "save_output" : [ "src/test/blah/server.log", "src/test/blah/regress*.log" ]
> }
>
> as I recall, Andrew said that he didn't like JSON very much but that the
> idea made sense to him.

As long as the buildfarm client has native support to parse that, that
sounds good. I think that we had better add a TODO item now, I won't
be able to tackle that in the short term.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Alvaro Herrera]

Michael Paquier wrote:
> On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera
>  wrote:
> > If somebody had some spare time to devote to this, I would suggest to
> > implement something in core that can be used to specify a list of
> > commands to run, and a list of files-to-be-saved-in-bf-log emitted by
> > each command.  We could have one such base file in the core repo that
> > specifies some tests to run (such as "make -C src/test/recovery check"),
> > and an additional file can be given by buildfarm animals to run custom
> > tests, without having to write BF modules for each thing.  With that,
> > pgsql committers could simply add a new test to be run by all buildfarm
> > animals by adding it to the list in core.
> 
> Do you think about using a special makefile target to run those
> commands, say in src/test? At the end we are going to need to patch
> the buildfarm client code at least once, at least that would be worth
> it in the long term..

Sure.  Some time ago I proposed something like a JSON file, something
like 

test_foo => {
  "command" : "make -C src/test/blah check",
  "save_output" : [ "src/test/blah/server.log", "src/test/blah/regress*.log" ]
}

as I recall, Andrew said that he didn't like JSON very much but that the
idea made sense to him.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Robert Haas]

On Fri, Aug 5, 2016 at 10:11 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> I think it would be an interesting project for someone to try to
>> figure out how to make 'make check-extended-query-protocol' or similar
>> work.
>
> Seems like it would not be that hard to put some kind of option in psql
> to issue queries with PQexecParams not plain PQexec.

Yes, we did that.

> However, since
> that wouldn't exercise sending out-of-line parameters, it's not clear
> to me that it'd be a very interesting test.

Well, you exercise copyfuncs.c quite a bit more, if nothing else.

> I still suspect that doing this in core is mostly going to be duplicative
> effort, and we'd be better off making use of existing JDBC tests.

That's possible; I don't know much about the JDBC tests, so it's hard
for me to say how that would compare in terms of coverage to my
proposal.  Perhaps both things are worth doing.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Tom Lane]

Robert Haas  writes:
> I think it would be an interesting project for someone to try to
> figure out how to make 'make check-extended-query-protocol' or similar
> work.

Seems like it would not be that hard to put some kind of option in psql
to issue queries with PQexecParams not plain PQexec.  However, since
that wouldn't exercise sending out-of-line parameters, it's not clear
to me that it'd be a very interesting test.

I still suspect that doing this in core is mostly going to be duplicative
effort, and we'd be better off making use of existing JDBC tests.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Robert Haas]

On Tue, Aug 2, 2016 at 10:33 PM, Tatsuo Ishii  wrote:
> In my understanding, we don't have any regression test for protocol
> level prepared query (we do have SQL level prepared query tests,
> though). Shouldn't we add those tests to the regression test suites?

A few years ago, EDB had a bug that only manifested itself when using
the extended query protocol.  We hacked together something that could
run our entire regression test suite (roughly equivalent to 'make
check', but an order of magnitude larger) under the extended query
protocol and found a few more bugs of the same general flavor.  There
were a few things that fell over that, I think, were not bugs, but it
mostly worked.

I think it would be an interesting project for someone to try to
figure out how to make 'make check-extended-query-protocol' or similar
work.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Michael Paquier]

On Fri, Aug 5, 2016 at 12:21 AM, Alvaro Herrera
 wrote:
> If somebody had some spare time to devote to this, I would suggest to
> implement something in core that can be used to specify a list of
> commands to run, and a list of files-to-be-saved-in-bf-log emitted by
> each command.  We could have one such base file in the core repo that
> specifies some tests to run (such as "make -C src/test/recovery check"),
> and an additional file can be given by buildfarm animals to run custom
> tests, without having to write BF modules for each thing.  With that,
> pgsql committers could simply add a new test to be run by all buildfarm
> animals by adding it to the list in core.

Do you think about using a special makefile target to run those
commands, say in src/test? At the end we are going to need to patch
the buildfarm client code at least once, at least that would be worth
it in the long term..
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Alvaro Herrera]

Michael Paquier wrote:
> On Thu, Aug 4, 2016 at 12:14 AM, Dave Cramer  wrote:
> > We currently run tests every time a PR is created on github, but I don't
> > think there are any animals running the JDBC test suite
> >
> > We can add tests, what exactly do we want to test. Then setting up an animal
> > to run the tests would be fairly straight forward.
> 
> It may be interesting to implement that as a module in the buildfarm
> client I think that any animal could include at will. Just a thought.

Implementing things as buildfarm modules gets boring, though -- consider
for instance that src/test/recovery is only being run by hamster, and
only because you patched the buildfarm client; only Andrew can influence
buildfarm animals into running that test by integrating the module in
the client script, and even that is limited by having to request animal
caretakers to upgrade that script.

If somebody had some spare time to devote to this, I would suggest to
implement something in core that can be used to specify a list of
commands to run, and a list of files-to-be-saved-in-bf-log emitted by
each command.  We could have one such base file in the core repo that
specifies some tests to run (such as "make -C src/test/recovery check"),
and an additional file can be given by buildfarm animals to run custom
tests, without having to write BF modules for each thing.  With that,
pgsql committers could simply add a new test to be run by all buildfarm
animals by adding it to the list in core.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Michael Paquier]

On Thu, Aug 4, 2016 at 12:14 AM, Dave Cramer  wrote:
> We currently run tests every time a PR is created on github, but I don't
> think there are any animals running the JDBC test suite
>
> We can add tests, what exactly do we want to test. Then setting up an animal
> to run the tests would be fairly straight forward.

It may be interesting to implement that as a module in the buildfarm
client I think that any animal could include at will. Just a thought.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Dave Cramer]

On 3 August 2016 at 10:53, Tom Lane  wrote:

> Tatsuo Ishii  writes:
> >>> In my understanding, we don't have any regression test for protocol
> >>> level prepared query (we do have SQL level prepared query tests,
> >>> though). Shouldn't we add those tests to the regression test suites?
>
> >> I thought that ECPG was covering a portion of that. Right?
>
> > In my understanding, ECPG calls libpq, thus the test cases are limited
> > to the cases which are only possible with libpq (or it may be even
> > limited to the cases where ECPG deal with.)
>
> I do not think it's very practical for the core tests to try to cover
> any behavior that's not reachable via libpq.  Even testing stuff that's
> not reached through ecpg would require a whole slew of new infrastructure
> with no use except testing.
>
> I think realistically a better approach to this would be to get some
> buildfarm members running the JDBC regression tests (I assume there
> are some ...)
>
> regards, tom lane
>
> We currently run tests every time a PR is created on github, but I don't
think there are any animals running the JDBC test suite

We can add tests, what exactly do we want to test. Then setting up an
animal to run the tests would be fairly straight forward.

Dave Cramer

da...@postgresintl.com
www.postgresintl.com

Re: [HACKERS] regression test for extended query protocol

[Tom Lane]

Tatsuo Ishii  writes:
>>> In my understanding, we don't have any regression test for protocol
>>> level prepared query (we do have SQL level prepared query tests,
>>> though). Shouldn't we add those tests to the regression test suites?

>> I thought that ECPG was covering a portion of that. Right?

> In my understanding, ECPG calls libpq, thus the test cases are limited
> to the cases which are only possible with libpq (or it may be even
> limited to the cases where ECPG deal with.)

I do not think it's very practical for the core tests to try to cover
any behavior that's not reachable via libpq.  Even testing stuff that's
not reached through ecpg would require a whole slew of new infrastructure
with no use except testing.

I think realistically a better approach to this would be to get some
buildfarm members running the JDBC regression tests (I assume there
are some ...)

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Tatsuo Ishii]

[Sorry for a top post. For unknown reason I couldn' get mails from
pgsql-hackers since this morning and I have to check the archive]

>> In my understanding, we don't have any regression test for protocol
>> level prepared query (we do have SQL level prepared query tests,
>> though). Shouldn't we add those tests to the regression test suites?
>
> I thought that ECPG was covering a portion of that. Right?
> -- 
> Michael

In my understanding, ECPG calls libpq, thus the test cases are limited
to the cases which are only possible with libpq (or it may be even
limited to the cases where ECPG deal with.)

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] regression test for extended query protocol

[Michael Paquier]

On Wed, Aug 3, 2016 at 11:33 AM, Tatsuo Ishii  wrote:
> In my understanding, we don't have any regression test for protocol
> level prepared query (we do have SQL level prepared query tests,
> though). Shouldn't we add those tests to the regression test suites?

I thought that ECPG was covering a portion of that. Right?
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers