Re: [HACKERS] New regression test time

2013-07-03 Thread Simon Riggs
On 2 July 2013 18:43, Noah Misch n...@leadboat.com wrote: On Tue, Jul 02, 2013 at 10:17:08AM -0400, Robert Haas wrote: So I think the first question we need to answer is: Should we just reject Robins' patches en masse? If we do that, then the rest of this is moot. If we don't do that,

Re: [HACKERS] New regression test time

2013-07-03 Thread Robert Haas
On Wed, Jul 3, 2013 at 2:28 AM, Simon Riggs si...@2ndquadrant.com wrote: It's sad to simply reject meaningful automated tests on the basis of doubt that they're important enough to belong in every human-in-the-loop test run. I share the broader vision for automated testing represented by these

Re: [HACKERS] New regression test time

2013-07-03 Thread Simon Riggs
On 3 July 2013 15:43, Robert Haas robertmh...@gmail.com wrote: Let's have a new schedule called minute-check with the objective to run the common tests in 60 secs. We can continue to expand the normal schedules from here. Anybody that wants short tests can run that, everyone else can

Re: [HACKERS] New regression test time

2013-07-03 Thread Josh Berkus
On 07/03/2013 07:43 AM, Robert Haas wrote: Let's have a new schedule called minute-check with the objective to run the common tests in 60 secs. Note that we're below 60s even with assert and CLOBBER_CACHE_ALWAYS, at least on my laptop. -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] New regression test time

2013-07-03 Thread Andrew Dunstan
On 07/03/2013 02:50 PM, Josh Berkus wrote: On 07/03/2013 07:43 AM, Robert Haas wrote: Let's have a new schedule called minute-check with the objective to run the common tests in 60 secs. Note that we're below 60s even with assert and CLOBBER_CACHE_ALWAYS, at least on my laptop. I find

Re: [HACKERS] New regression test time

2013-07-02 Thread Robert Haas
Reviewing this thread, I think that the following people are in favor of adding the tests to the existing schedule: Josh Berkus, Stephen Frost, Fabien Coelho, Dann Corbit, and Jeff Janes. And I think that the following people are in favor of a new schedule: Andres Freund, Andrew Dunstan, David

Re: [HACKERS] New regression test time

2013-07-02 Thread Andrew Dunstan
On 07/02/2013 10:17 AM, Robert Haas wrote: Reviewing this thread, I think that the following people are in favor of adding the tests to the existing schedule: Josh Berkus, Stephen Frost, Fabien Coelho, Dann Corbit, and Jeff Janes. And I think that the following people are in favor of a new

Re: [HACKERS] New regression test time

2013-07-02 Thread Fabien COELHO
What is more, it's entirely possibly to invoke pg_regress with multiple --schedule arguments, so we could, for example, have a makefile target that would run both the check and some other schedule of longer running tests. I missed this fact, because I've not seen any example of multiple

Re: [HACKERS] New regression test time

2013-07-02 Thread Noah Misch
On Tue, Jul 02, 2013 at 10:17:08AM -0400, Robert Haas wrote: So I think the first question we need to answer is: Should we just reject Robins' patches en masse? If we do that, then the rest of this is moot. If we don't do that, then the second question is whether we should try to introduce a

Re: [HACKERS] New regression test time

2013-07-01 Thread Amit kapila
On Monday, July 01, 2013 8:37 AM Josh Berkus wrote: On 06/30/2013 12:33 AM, Amit kapila wrote: On Sunday, June 30, 2013 11:37 AM Fabien COELHO wrote: If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to

Re: [HACKERS] New regression test time

2013-07-01 Thread David Fetter
On Sat, Jun 29, 2013 at 02:59:35PM -0700, Josh Berkus wrote: On 06/29/2013 02:14 PM, Andrew Dunstan wrote: AIUI: They do test feature use and errors that have cropped up in the past that we need to beware of. They don't test every bug we've ever had, nor do they exercise every piece of

Re: [HACKERS] New regression test time

2013-07-01 Thread Andres Freund
On 2013-07-01 07:14:23 -0700, David Fetter wrote: If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either. With utmost respect, this just isn't true. There is a make coverage

Re: [HACKERS] New regression test time

2013-06-30 Thread Fabien COELHO
If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either. I have done a POC. See: https://commitfest.postgresql.org/action/patch_view?id=1170 What I have not done is to decide how to

Re: [HACKERS] New regression test time

2013-06-30 Thread Amit kapila
On Sunday, June 30, 2013 11:37 AM Fabien COELHO wrote: If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either. I have done a POC. See:

Re: [HACKERS] New regression test time

2013-06-30 Thread Fabien COELHO
https://commitfest.postgresql.org/action/patch_view?id=1170 I think it is better to submit for next commit fest which is at below link: https://commitfest.postgresql.org/action/commitfest_view?id=19 I put it there as the discussion whether to accept or not Robins patches because of their

Re: [HACKERS] New regression test time

2013-06-30 Thread Robins Tharakan
On 30 June 2013 02:33, Amit kapila amit.kap...@huawei.com wrote: On Sunday, June 30, 2013 11:37 AM Fabien COELHO wrote: If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either. I

Re: [HACKERS] New regression test time

2013-06-30 Thread Jeff Janes
On Sat, Jun 29, 2013 at 3:43 PM, Andrew Dunstan and...@dunslane.net wrote: On 06/29/2013 05:59 PM, Josh Berkus wrote: Maybe there is a good case for these last two in a different set of tests. If we had a different set of tests, that would be a valid argument. But we don't, so it's not.

Re: [HACKERS] New regression test time

2013-06-30 Thread Josh Berkus
On 06/30/2013 12:33 AM, Amit kapila wrote: On Sunday, June 30, 2013 11:37 AM Fabien COELHO wrote: If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either. I have done a POC. See:

Re: [HACKERS] New regression test time

2013-06-29 Thread Robert Haas
On Jun 29, 2013, at 12:36 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: I see two problems with this report: 1. it creates a new installation for each run, But that's the normal way of running the tests anyway, isn't it? 2. it only uses the serial schedule. make check uses the parallel

Re: [HACKERS] New regression test time

2013-06-29 Thread Josh Berkus
I see two problems with this report: 1. it creates a new installation for each run, Yes, I'm running make check 2. it only uses the serial schedule. Um, no: parallel group (19 tests): limit prepare copy2 plancache xml returning conversion rowtypes largeobject temp truncate polymorphism

Re: [HACKERS] New regression test time

2013-06-29 Thread Andrew Dunstan
On 06/29/2013 03:57 PM, Josh Berkus wrote: I see two problems with this report: 1. it creates a new installation for each run, Yes, I'm running make check 2. it only uses the serial schedule. Um, no: parallel group (19 tests): limit prepare copy2 plancache xml returning conversion

Re: [HACKERS] New regression test time

2013-06-29 Thread Josh Berkus
On 06/29/2013 02:14 PM, Andrew Dunstan wrote: AIUI: They do test feature use and errors that have cropped up in the past that we need to beware of. They don't test every bug we've ever had, nor do they exercise every piece of code. If we don't have a test for it, then we can break it in the

Re: [HACKERS] New regression test time

2013-06-29 Thread Dann Corbit
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Josh Berkus Sent: Saturday, June 29, 2013 3:00 PM To: Andrew Dunstan Cc: Alvaro Herrera; pgsql-hackers@postgresql.org; Robins Tharakan Subject: Re: [HACKERS] New regression

Re: [HACKERS] New regression test time

2013-06-29 Thread Andrew Dunstan
On 06/29/2013 05:59 PM, Josh Berkus wrote: Maybe there is a good case for these last two in a different set of tests. If we had a different set of tests, that would be a valid argument. But we don't, so it's not. And nobody has offered to write a feature to split our tests either. I have

Re: [HACKERS] New regression test time

2013-06-29 Thread Josh Berkus
Dividing the tests into different sections is as simple as creating one schedule file per section. Oh? Huh. I'd thought it would be much more complicated. Well, by all means, let's do it then. I'm not personally convinced that the existing regression tests all belong in the default

Re: [HACKERS] New regression test time

2013-06-29 Thread Claudio Freire
On Sat, Jun 29, 2013 at 7:58 PM, Josh Berkus j...@agliodbs.com wrote: Dividing the tests into different sections is as simple as creating one schedule file per section. Oh? Huh. I'd thought it would be much more complicated. Well, by all means, let's do it then. I think I should point

Re: [HACKERS] New regression test time

2013-06-29 Thread Stephen Frost
Josh, * Josh Berkus (j...@agliodbs.com) wrote: If we don't have a test for it, then we can break it in the future and not know we've broken it until .0 is released. Is that really a direction we're happy going in? To be fair, AIUI anyway, certain companies have much larger regression suites

Re: [HACKERS] New regression test time

2013-06-28 Thread Andres Freund
On 2013-06-28 14:01:23 -0700, Josh Berkus wrote: Per discussion on these tests, I ran make check against 9.4 head, applied all of the regression tests other than DISCARD. Time for 3 make check runs without new tests: 65.9s Time for 3 make check runs with new tests: 71.7s So that's an

Re: [HACKERS] New regression test time

2013-06-28 Thread Josh Berkus
How did you evaluate that coverage increased greatly? I am not generally against these tests but I'd be surprised if the overall test coverage improved noticeably by this. Which makes 10% runtime overhead pretty hefty if the goal is to actually achieve a high coverage. I was relying on

Re: [HACKERS] New regression test time

2013-06-28 Thread Andres Freund
On 2013-06-28 14:46:10 -0700, Josh Berkus wrote: How did you evaluate that coverage increased greatly? I am not generally against these tests but I'd be surprised if the overall test coverage improved noticeably by this. Which makes 10% runtime overhead pretty hefty if the goal is to

Re: [HACKERS] New regression test time

2013-06-28 Thread Stephen Frost
* Josh Berkus (j...@agliodbs.com) wrote: So that's an increase of about 10% in test runtime (or 2 seconds per run on my laptop), in order to greatly improve regression test coverage. I'd say that splitting the tests is not warranted, and that we should go ahead with these tests on their

Re: [HACKERS] New regression test time

2013-06-28 Thread Fabien COELHO
How did you evaluate that coverage increased greatly? I am not generally against these tests but I'd be surprised if the overall test coverage improved noticeably by this. Which makes 10% runtime overhead pretty hefty if the goal is to actually achieve a high coverage. I was relying on

Re: [HACKERS] New regression test time

2013-06-28 Thread Alvaro Herrera
Josh Berkus escribió: Hackers, Per discussion on these tests, I ran make check against 9.4 head, applied all of the regression tests other than DISCARD. Time for 3 make check runs without new tests: 65.9s Time for 3 make check runs with new tests: 71.7s So that's an increase of about