As of r212660, lldb tests run cleanly on MacOSX 10.9.4 for x86_64 (i.e. -A x86_64). I've marked as XFAIL all tests I found that consistently fail, and marked the two tests I found intermittent there as skip on MacOSX. I filed bugs for all tests that were marked XFAIL or skip.
Tests also run reasonably fast on MacOSX. So - if you are on OSX, please run the tests, and if any of them fail, you can be reasonably assured that there is a decent chance that the pending change may be introducing it. I'm about to make one more pass on the Linux side for the same goal: so that test failures mean new breakages. (Note we still need to go back and address all the bugs filed against XFAIL/skipped tests, but at least new breakages can be detected much more readily --- any test failure should raise suspicion). Happy days! -Todd On Tue, Jul 8, 2014 at 1:50 PM, Todd Fiala <[email protected]> wrote: > FWIW I got the same set of 11 errors on MacOSX 10.10 (14A283o) - Yosemite > Preview Update 3 - and Xcode Version 6.0 (6A254o) - Xcode 6 Beta 3. > > The first set was from OS X 10.9.3 with Xcode 6 Beta 2. > > > On Tue, Jul 8, 2014 at 10:42 AM, Todd Fiala <[email protected]> wrote: > >> Here's how I run the tests: >> >> Make sure your Xcode project "locations" setting is set to "relative", so >> that the DerivedData is stored relative to your lldb source directory. >> (Above, that will be /my/workspace-root/lldb/DerivedData). >> >> I have the canonical MacOSX Xcode source layout: >> >> /my/workspace-root/ >> |- lldb/ >> |- llvm/ >> |- tools/clang/ >> >> $ cd /my/workspace-root >> $ mkdir test-run >> $ cd test-run >> # {save attached do-tests.sh file here} >> $ chmod a+x do-tests.sh >> >> # Now run it - this will run the tests using one parallel test per core. >> Tests are chunked up in units of directories, so all tests in a given >> directory are serialized. >> >> $ ./do-tests.sh >> >> The script currently only runs x86_64 - just adjust the -A value if you >> want to run more/different architectures in the tests. It also assumes >> that it will put test output into the test-run/lldb-test-results directory. >> >> The behavior is no different from the previous multithreaded behavior on >> Linux/FreeBSD test runs. >> >> Let me know if you run into any issues with that. >> >> >> On Tue, Jul 8, 2014 at 9:50 AM, Todd Fiala <[email protected]> wrote: >> >>> For the record, these are the set of tests that currently fail for me on >>> MacOSX 10.9.3 as of svn r212548 on MacOSX: >>> >>> Ran 297 tests. >>> Failing Tests (11) >>> FAIL: LLDB (suite) :: TestDataFormatterObjC.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestProcessLaunch.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestRegisterVariables.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestObjCMethods.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestHiddenIvars.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestObjCDynamicSBType.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestObjCDynamicValue.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestRealDefinition.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> FAIL: LLDB (suite) :: TestTargetAPI.py (Darwin >>> tfiala-macpro.mtv.corp.google.com 13.2.0 Darwin Kernel Version 13.2.0: >>> Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 >>> i386) >>> >>> >>> Also, there were no performance regressions detected on Linux >>> multithreaded test run for the change in implementation of the multi-core >>> testing. What I changed was implementing the worker threads via Python's >>> multiprocessing.Pool class instead of using the threading module's threads. >>> All the MacOSX threads were locked on the Python global interpreter lock. >>> Not entirely sure what the diff was that prevented that in Linux and >>> FreeBSD, but no matter. The Pool implementation is slightly cleaner anyway. >>> >>> -Todd >>> >>> >>> On Tue, Jul 8, 2014 at 9:46 AM, Todd Fiala <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> I've updated the test runner for lldb so it now runs multithreaded on >>>> MacOSX with similar performance gains to the Linux/FreeBSD test speedups. >>>> >>>> On my MacBookPro (mid-2012 Retina), test runs when from ~28 minutes to >>>> 7.5 minutes. >>>> On my MacPro (late-2013), test runs went from ~25 minutes to 3.5 >>>> minutes. >>>> >>>> It's totally dependent on the number of cores, so YMMV, but it is >>>> faster. >>>> >>>> Also, I've been discussing ways of mitigating tests that are >>>> load-sensitive (which multithreaded test running exposes). I'm looking at >>>> adding a new test declaration that lists a test as load sensitive >>>> (@load_sensitive_test), and somehow allowing that to fail in the >>>> multithreaded pass without failing the test run. Then, run all the >>>> load-sensitive tests in a follow-up single-threaded pass if they failed >>>> under load. Then, only mark them as failed if they fail under the >>>> single-threaded pass. I'll have more to say on that when I get some time >>>> behind it. If we do something like that, it should eliminate the tests >>>> that are hard to simplify and work under load, without forcing us to run >>>> all tests in a *much longer* single-threaded manner. >>>> >>>> -- >>>> Todd Fiala | Software Engineer | [email protected] | 650-943-3180 >>>> >>> >>> >>> >>> -- >>> Todd Fiala | Software Engineer | [email protected] | 650-943-3180 >>> >>> _______________________________________________ >>> lldb-dev mailing list >>> [email protected] >>> http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev >>> >>> >> >> >> -- >> -Todd >> > > > > -- > Todd Fiala | Software Engineer | [email protected] | 650-943-3180 > -- -Todd
_______________________________________________ lldb-dev mailing list [email protected] http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev
