Re: [lldb-dev] BasicResultsFormatter - new test results summary

Hi, thanks a lot for fixing the timeout issue on such a short notice. I didn't think I'd find myself defending them, as I remember being quite upset when they went in, but they have proven useful in stabilising the build bots, and I think it's likely you may need them as well. I'll try to now add

Re: [lldb-dev] BasicResultsFormatter - new test results summary

On 14 December 2015 at 16:19, Todd Fiala wrote: >> We would lose the ability to individually expect "failures" and >> "timeouts", but I don't think that is really necessary, and I think it >> will be worth the extra maintainability we get from the fact of having >> fewer

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Todd, I've had to disable the new result formatter as it was not working with the expected timeout logic we have for the old one. The old XTIMEOUT code is a massive hack and I will be extremely glad when we get rid of it, but we can't keep our buildbot red until then, so I've switched it off. I

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Merging threads. > The concept is not there to protect against timeouts, which are caused by processes being too slow, for these we have been increasing timeouts where necessary. Okay, I see. If that's the intent, then expected timeout sounds reasonable. (My abhorrence was against the idea of

Re: [lldb-dev] BasicResultsFormatter - new test results summary

On Fri, Dec 11, 2015 at 3:26 AM, Pavel Labath wrote: > Todd, I've had to disable the new result formatter as it was not > working with the expected timeout logic we have for the old one. The > old XTIMEOUT code is a massive hack and I will be extremely glad when > we get rid

Re: [lldb-dev] BasicResultsFormatter - new test results summary

On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote: > Hi Tamas, > > > > On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer > wrote: > >> HI Todd, >> >> You changed the way the test failure list is printed in a way that now we >> only print the name of

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Sure, I can do that. Tamas, okay to give more detail on -v? I'll give it a shot to see what else comes out if we do that. -Todd On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner wrote: > > > On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala wrote: > >> Hi

Re: [lldb-dev] BasicResultsFormatter - new test results summary

HI Todd, You changed the way the test failure list is printed in a way that now we only print the name of the test function failing with the name of the test file in parenthesis. Can we add back the name of the test class to this list? There are 2 reason I am asking for it: * To run only a

Re: [lldb-dev] BasicResultsFormatter - new test results summary

That's a good point, Tamas. I use (so I claim) the same all upper-case markers for the test result details. Including, not using XPASS but rather UNEXPECTED SUCCESS for unexpected successes. (The former would trigger the lit script IIRC to parse that as a failing-style result). The intent is

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Specifically, the markers for issue details are: FAIL ERROR UNEXPECTED SUCCESS TIMEOUT (These are the fourth field in the array entries (lines 275 - 290) of packages/Python/lldbsuite/test/basic_results_formatter.py). -Todd On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala

Re: [lldb-dev] BasicResultsFormatter - new test results summary

If it's not too much work, I think the extra bit of noise will not be a problem. But I don't think it is really necessary either. I assume the actual flip will be a small change that we can back out easily if we notice troubles... After a sufficient grace period we can remove the old formatter

Re: [lldb-dev] BasicResultsFormatter - new test results summary

These went in as: r255130 - turn it on by default r255131 - create known issues. This one is to be reverted if all 3 types show up properly. On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala wrote: > It is a small change. > > I almost have all the trial tests ready, so I'll

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Thank you for making the experiment. It looks reasonable. For the ERROR the buildbot detected it and it will fail the build but it isn't listed in the list of failing tests what should be fixed. After this experiment I think it is fine to change the default output formatter from our side. Tamas

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Great, thanks Tamas! I left the default turned on, and just essentially removed the issues by parking them as .py.parked files. That way we can flip them on in the future if we want to verify a testbot's detection of these. I will be going back to the xUnit Results formatter and making sure it

Re: [lldb-dev] BasicResultsFormatter - new test results summary

The reports look good at the test level: http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294 I'd say the buildbot reflection script missed the ERROR, so that is something maybe Ying can look at (the summary line in the build run), but that is unrelated AFAICT. I'm going

Re: [lldb-dev] BasicResultsFormatter - new test results summary

Verification tests parked (i.e. disabled) here: r255134 I decided to leave them in the repo so it is faster/easier to do this in the future. -Todd On Wed, Dec 9, 2015 at 10:26 AM, Todd Fiala wrote: > The reports look good at the test level: > > >