On Tue, Nov 2, 2010 at 12:05 PM, Gary Poster <[email protected]> wrote:
>
> On Nov 2, 2010, at 9:59 AM, Deryck Hodge wrote:
>
>> What do you all think?
>
> I think your argument has some merit, and I very much appreciate the reasoned
> way you approached it.
>
> Personally, though, I disagree with the proposal to throw the tests out.
Thanks for the feedback, Gary. I understand this is a huge change,
and I appreciate this kind of response.
>
> * Regression tests pay back over months and years, not suddenly.
> * I am skeptical of the idea of "temporarily" or experimentally throwing
> them out.
> - I believe that without maintenance, they will die.
> - I don't believe that the proposed metrics ("regression" tags in bugs
> after tossing out tests for two or three months) for evaluating the
> experiment convey their value.
This echoes my own concern for Robert's proposal -- that it was hard
to verify value -- so I understand your feelings. i.e. that we are
only verifying the absence of negative impact, not value of the thing
in question.
However, I think regression bugs will allow us to measure something
about the value of the tests, even if it is an incomplete picture.
This is the position I arrived at with Robert's proposal. I'm also
skeptical that we can determine the pure value of review from his
experiment, but I think the data from that experiment will still be
useful. I feel the same about my own proposal -- it's not a perfect
experiment, but it brings some value and at least gets us making an
assessment of the pain of parts of our test suite.
> * While Windmill pain has been recurring, the proposal comes right now
> because of immediate pain that is caused by unusually large changes to the
> way tests are run. Maris is trying to address that pain (while he
> simultaneously has other, more scheduled irons in the fire).
>
> Because we don't have a metric for measuring the tests' value, nor a way to
> quantitatively compare their value to their cost, this is a matter of
> opinion. This is mine.
>
FWIW, while the pain of Windmill has been acute lately, I don't recall
a month without some form of Windmill pain since I've been working on
Launchpad. I feel the same about page tests.
Again, thanks for the feedback. I very much value your thoughts on this.
Cheers,
deryck
--
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-dev
Post to : [email protected]
Unsubscribe : https://launchpad.net/~launchpad-dev
More help : https://help.launchpad.net/ListHelp
