On 3/12/15 6:06 PM, Zachary Turner wrote:
Wasn't really trying to get into an extended discussion about this, but
FWIW I definitely realize that lldb's tests are more complicated than
what lit currently supports.  But that's why I said "even if it meant
extending lit".  It was mostly just a general comment about how it's
nice if everyone is focused on making one thing better instead of
everyone having different things.

As for specifics, my understanding is that lit parallelizes better (so
running tests is faster), understands how to build programs (so doesn't
require makefiles), and has a richer language for specifying how and
under what circumstances different tests should be run.  It's also
familiar to other LLVM developers (so encourages cross-collaboration),
and allows one to write self-contained tests with the program to test
and the check in a single file (less maintenance).

I don't know much about lldb's testing needs, but I thought I'd chime in and mention that there's a lot of magic in the lit TestFormat (which testsuites can define themselves), and that it's waaaaay more flexible than it looks.

LLVM uses the ShTest test format a lot, and that one is well-suited for the process Jim describes: building shell script pipelines with checkers at the end, where the test file itself serves as both the input to the beginning of the pipeline, and the input to the checker at the end.

Libcxx and Libcxxabi have their own LibcxxTestFormat that behaves a little differently: they make the assumption that every test is a c/c++ file which needs to be compiled, executed, and that an exit status of 0 is a pass.

From a quick look at the lldb testsuite, I imagine if you wanted to LIT-ify it, it would make sense to have yet another TestFormat. This test format would know that it needs to look for *.py files, invoke the Makefile next to them (or put metadata in comments at the top of the *.py that says how to build it), then run the *.py file. I think the changes to the tests themselves would be minimal, as would writing the lit.cfg to glue it all together.


Cheers,

Jon

In any case, I'm really not an expert on lit, so +bogner and +chandlerc
in case they want to chime in.  I do think it's at least worth thinking
about whether lit *could* be extended to meet LLDB's needs -- if nothing
else as a thought exercise, and maybe learning more about how it works
would give us some ideas to make our own test suite better.


On Thu, Mar 12, 2015 at 4:39 PM <jing...@apple.com
<mailto:jing...@apple.com>> wrote:


     > On Mar 12, 2015, at 4:08 PM, Zachary Turner <ztur...@google.com
    <mailto:ztur...@google.com>> wrote:
     >
     > Oh I'm all for reusing as much of the existing mechanism as
    possible.  Was just stating how the CMake worked as a discussion
    point.  Another possibility would be to just have the Xcode project
    build one executable that pulls in sources recursively from the
    entire subtree.  Is this as easy in Xcode as just adding all sources
    from a subfolder to a single target?

     >
     > One day far off in the future it would be nice if all of LLDB's
    tests were ported to lit (even if that meant extending lit to make
    it do what we needed it to do),

    Why would this be nice?  It looks like lit is a good test runner for
    tests that have some input, do something with the input, produce an
    output and check that output is matches some pattern.  That is not
    at all what the lldb tests look like.  They often have to do complex
    dances - for instance depending on how the line tables come out
    there are many "correct" ways to step through code.  If you are
    going to test this you've got to do "step, if I got to a close
    bracket, step again, if I got past it don't.  Etc...

    I see no benefit in extending a simple runner like lit to do the
    complex dances the lldb testsuite sometimes has to do.  I'm all for
    sharing, but it is also okay to have two implementations of some
    functionality if the two uses are sufficiently different, and this
    certainly seems like one of those cases.

     > so I can definitely see some value in hooking lit up to the Xcode
    build so it does everything the CMake build does.  I'll have to look
    into exactly what steps the CMake and/or autoconf build are taking,
    but I suspect it's going to involve running CMake from a script, so
    not very desirable.  I'm still learning a lot of this stuff though,
    so there may be a better way.  Either way, I'll have to look into it
    a little bit.


    Jim


     >
     > In the meantime, if running unit tests from Xcode is not part of
    anyone's usual workflow, can I remove it for now?
     >
     > On Thu, Mar 12, 2015 at 4:01 PM <jing...@apple.com
    <mailto:jing...@apple.com>> wrote:
     > I'm not sure if this is what you meant, but I don't see a lot of
    value in making an Xcode project that has targets for each of the
    gtest binaries, and then tries to run the tests.  Seems to me it
    would be better if the gtest project just invokes whatever mechanism
    the cmake build would do to run the tests.  That's just another set
    of things to keep in sync.
     >
     > It is sufficient to have a target that just does whatever steps
    cmake/lit do to build the gtests & run them, if that is possible.  I
    guess if you can't do this without running cmake in the lldb
    top-level directory that would be a problem.  But it still seems
    better to me to wire that up, than to have to add tests to both
    Xcode & cmake.
     >
     > Jim
     >
     >
     >
     >
     >
     > > On Mar 12, 2015, at 3:46 PM, Zachary Turner <ztur...@google.com
    <mailto:ztur...@google.com>> wrote:
     > >
     > > So I'm guessing the scheme runs do-gtest.py.  I'd like to
    delete that file as well as all the Makefiles in the directory if
    possible.  It seems like these files should be built using the
    normal Xcode build system the same way the rest of LLDB is built.
     > >
     > > The way the CMake does it is that each test folder generates a
    new executable.  So right now it will build HostTests.exe,
    ProcessLinuxTests.exe, and UtilityTests.exe.  And then CMake will
    invoke lit (the LLVM test runner) to run each of the executables one
    by one and print the output.
     > >
     > > I'm not sure if that's easy or feasible to do in the Xcode
    build.  I kind of don't want to leave this do-gtest.py and Makefiles
    in the build though, because the more of this stuff we have the more
    maintenance it is, and things tend to rot.
     > >
     > >
     > > On Thu, Mar 12, 2015 at 3:23 PM <jing...@apple.com
    <mailto:jing...@apple.com>> wrote:
     > > Xcode has "projects" and then "workspaces" and "schemes".
    Workspaces aggregate projects.  Schemes exist in both workspaces and
    projects and are the way to say "do something with some of the stuff
    referred to by this project/workspace."  So the way to do this
    formally is to have the gtest scheme build & run the tests from the
    gtest project.
     > >
     > > The lldb.xcworkspace file does reference the gtest xcode
    project, and it has a scheme for the gtest.
     > >
     > > Not sure what the scheme does yet, I'll look in a few minutes
    if nobody beats me to it, I'm in the middle of things right now.
     > >
     > > Jim
     > >
     > >
     > >
     > > > On Mar 12, 2015, at 2:41 PM, Zachary Turner
    <ztur...@google.com <mailto:ztur...@google.com>> wrote:
     > > >
     > > > In lldb/gtest there is a gtest.xcodeproj folder with what I
    guess is an Xcode project.  If I understand the way Xcode works, the
    way to use this is by opening this in another instance of Xcode
    separate from your normal LLDB project, and then building it.  Is
    this right?
     > > >
     > > > I have a patch that moves some files around, and if nobody is
    using this Xcode project, I would like to delete it.  Then, after I
    get the tests up and running in the CMake build, we can add it to
    the "real" Xcode project as a separate target similar to how you
    currently run the LLDB Test suite.
     > > >
     > > > Any objections to deleting the Xcode project?
     > >
     >



_______________________________________________
lldb-dev mailing list
lldb-dev@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev


--
Jon Roelofs
jonat...@codesourcery.com
CodeSourcery / Mentor Embedded
_______________________________________________
lldb-dev mailing list
lldb-dev@cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev

Reply via email to