On Sat, Aug 22, 2009 at 4:29 PM, Jeremy Orlow <jor...@chromium.org> wrote:
> On Sat, Aug 22, 2009 at 4:00 PM, Peter Kasting <pkast...@chromium.org>wrote:
>
>> On Fri, Aug 21, 2009 at 8:07 PM, Jeremy Orlow <jor...@chromium.org>wrote:
>>
>>>
>>>> 1) We don't have notes on why tests are failing. => Why not annotate
>>>> the tests in test_lists? That's what I've always done.
>>>
>>>
>>> Once again, we don't want to add more state to the test_expectations.
>>> How may people looked up the tests they were supposed to rebaseline in this
>>> file to see if there were notes? I kind of doubt anyone.
>>>
>>
>> Um... this makes no sense to me. You can't rebaseline a test without
>> modifying test_expectations. In modifying it, you *have* to look at it.
>> It's pretty difficult to miss comments above tests as you're trying to
>> write "REBASELINE" or delete the line.
>>
>> If you somehow managed to not see any comments in this file, I think
>> you're an outlier.
>>
>
> I was talking about the rebaselining teams, not the act of actually
> rebaselining. If someone's rebaselining a test, then it means we now
> believe it's passing. At that point, the notes are not very interesting,
> right? Are you saying that you looked at all the tests' notes before you
> looked through the results to determine if they should be rebaselined?
>
We're trying to leave all comments in the bugs now, rather than in the
test_expectations file, so there's only one point of contact. We used to
leave extensive comments in the file, but they always grew stale. And yes, I
looked at the bug for every test that I thought was correct, usually to
write "tests A, B and C are still bad, but D was actually correct and is
being re-baselined".
>
>
>>
>> There are different reasons for failing. A layout test could be failing
>>> because of a known bug and then start failing in a different way (later) due
>>> to a regression. When a bug fails in a new way, it's worth taking a quick
>>> look, I think.
>>>
>>
>> Why? Unless the earlier failure has been fixed we can't rebaseline the
>> test. (I ran into a number of tests like this when doing my rebaselining
>> pass.) What is the point of looking again?
>>
>
> In case the new failure is more serious than the earlier one.
>
True. But I don't think this will happen often, and I'd rather devote the
time to fixing the tests.
- Pam
--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---
