Re: [lldb-dev] New test summary results formatter

2015-12-06 Thread Todd Fiala via lldb-dev
Hi all, r254890 moves the test summary counts to the end. It also greatly cleans up the issue detail line to be: ISSUE_TYPE: test_method_name (test relative path) I put a sample output in the revision comment. I think it looks much cleaner with the tweaks we discussed, and I really like the

Re: [lldb-dev] New test summary results formatter

2015-12-04 Thread Todd Fiala via lldb-dev
One thing I excluded from the newer test results detail info is the architecture. I personally haven't ever needed that. I'd be happy to leave that out until we find someone who really needs it, just to keep it shorter. On Thu, Dec 3, 2015 at 5:14 PM, Todd Fiala wrote: >

Re: [lldb-dev] New test summary results formatter

2015-12-03 Thread Pavel Labath via lldb-dev
There is already code that enforces unique names (see dotest.py:1255). I added this a while back because we were getting test flakyness because of that. I don't remember the exact reason but I think it had something to do with the log file names clashes, as all logs are placed in the same folder.

Re: [lldb-dev] New test summary results formatter

2015-12-03 Thread Zachary Turner via lldb-dev
On Wed, Dec 2, 2015 at 10:20 PM Todd Fiala wrote: > On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner wrote: > >> >> >> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala wrote: >> >>> >>> and the classname could be dropped (there's only

Re: [lldb-dev] New test summary results formatter

2015-12-03 Thread Zachary Turner via lldb-dev
Ahh I read further and see this was already mentioned by Pavel. On Thu, Dec 3, 2015 at 10:06 AM Zachary Turner wrote: > On Wed, Dec 2, 2015 at 10:20 PM Todd Fiala wrote: > >> On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner >> wrote:

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
Also, all the text in the summary is fixed-width lined up nicely, which may not show in the commit message description if you're using a variable-width font. On a terminal it looks nice. On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala wrote: > > > On Wed, Dec 2, 2015 at 10:57

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
I think it might be already - let me check. The longer-term goal would be to get this without specifying anything (i.e. does what we want by default). If stdout is not already being used by default when a formatter is specified, that would be an easy fix. Checking now... On Wed, Dec 2, 2015 at

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
Can --results-file=stdout be the default so that we don't have to specify that? On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev < lldb-dev@lists.llvm.org> wrote: > Also, all the text in the summary is fixed-width lined up nicely, which > may not show in the commit message description if

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
Also another stylistic suggestion. I've been thinking about how to more logically organize all the source files now that we have a package. So it makes sense conceptually to group all of the different result formatters under a subpackage called formatters. So right now you've got

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
Yeah I'd be good with that. I can change that as well. -Todd On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner wrote: > Also another stylistic suggestion. I've been thinking about how to more > logically organize all the source files now that we have a package. So it >

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
I'll also move those couple counts (total tests run and the rerun count) under the summary. I missed that when evolving the code. On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala wrote: > Yeah I'd be good with that. I can change that as well. > > -Todd > > On Wed, Dec 2, 2015

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
Oh yea, I made up that decorator idea because I didn't know all the formatters were derived from a common base. But your idea is better if everything is derived from a common base. To be honest you could even just generate an error if there are two ResultsFormatter derived classes in the same

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala wrote: > Hi all, > > I just put up an optional test results formatter that is a prototype of > what we may move towards for our default test summary results. It went in > here: > > r254530 > > and you can try it out with

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala wrote: > Yeah I'd be good with that. I can change that as well. > > -Todd > > On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner > wrote: > >> Also another stylistic suggestion. I've been thinking about how to

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
When I run this under Python 3 I get "A bytes object is used like a string" on Line 1033 of test_results.py. I'm going to dig into it a little bit, but maybe you know off the top of your head the right way to fix it. On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner wrote: >

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
Is there any way to force the single process test runner to use this same system? I'm trying to debug the problem, but this codepath doesn't execute in the single process test runner, and it executes in the child process in the multiprocess test runner. Basically I need the following callstack

[lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
Hi all, I just put up an optional test results formatter that is a prototype of what we may move towards for our default test summary results. It went in here: r254530 and you can try it out with something like: time test/dotest.py --executable `pwd`/build/Debug/lldb --results-formatter

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
I think the best solution is going to be to use struct.pack. Like this on the writing side: import struct msg = cPickle.dumps(test_event) packet = struct.pack("!I%ds" % len(msg), len(msg), msg) self.out_file.send(packet) and like this on the reading side:

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
BTW, this is what I do in the swig service, and I think it's the idiomatic way of sending variable length data over a socket. On Wed, Dec 2, 2015 at 2:08 PM Zachary Turner wrote: > I think the best solution is going to be to use struct.pack. Like this on > the writing side:

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
On Wed, Dec 2, 2015 at 11:04 AM, Zachary Turner wrote: > Can --results-file=stdout be the default so that we don't have to specify > that? > > I've adjusted the code here: r254546 to support dropping the --results-file=stdout part if a --results-formatter is specified and no

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
Those fixes are up here: r254550 Let me know what you see after that, Zachary. On Wed, Dec 2, 2015 at 1:45 PM, Todd Fiala wrote: > > > On Wed, Dec 2, 2015 at 1:42 PM, Todd Fiala wrote: > >> Can you try making those changes in the other spots?

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
I think I can fix the issue without you debugging. Getting the single pass test runner to use it isn't impossible but will take some work. Can you direct-send me the backtrace from the point of failure from your system? Thanks! -Todd On Wed, Dec 2, 2015 at 12:34 PM, Zachary Turner

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
Can you try this? diff --git a/packages/Python/lldbsuite/test/test_results.py b/packages/Python/lldbsuite/test/test_results.py index 12fb2e5..547603e 100644 --- a/packages/Python/lldbsuite/test/test_results.py +++ b/packages/Python/lldbsuite/test/test_results.py @@ -1029,8 +1029,8 @@ class

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala wrote: > > >> and the classname could be dropped (there's only one class per file >> anyway, so the classname is just wasted space) >> > > Part of the reason I included that is I've hit several times where copy > and paste errors

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner wrote: > > > On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala wrote: > >> >> >>> and the classname could be dropped (there's only one class per file >>> anyway, so the classname is just wasted space) >>> >> >>

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
Thanks, Zachary. On Wed, Dec 2, 2015 at 3:59 PM, Zachary Turner wrote: > Now that it's working in Python 3 and I've had a chance to see the output, > I think I like it much better. These are the things I noticed (good and > bad): > > 1) Output of each test is printed on the

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
All good. Yeah Python 2/3 string/byte handling is one of the trickier areas of cross-Python version compatibility. On Wed, Dec 2, 2015 at 2:18 PM, Zachary Turner wrote: > think I got something working, I'll upload a patch in a few. > > On Wed, Dec 2, 2015 at 2:10 PM

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Zachary Turner via lldb-dev
Now that it's working in Python 3 and I've had a chance to see the output, I think I like it much better. These are the things I noticed (good and bad): 1) Output of each test is printed on the fly as the test is run. Much better than staring at a black screen for 5 minutes and then seeing a

Re: [lldb-dev] New test summary results formatter

2015-12-02 Thread Todd Fiala via lldb-dev
On Wed, Dec 2, 2015 at 1:42 PM, Todd Fiala wrote: > Can you try making those changes in the other spots? There's a handful of > code you have probably not ever run if you haven't selected running with a > test results formatter. > > I'm actually going to make the ibuffer