Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 18, 2018, at 5:07 AM, Pavel Labath wrote: > > Looks like I missed a party. :) > > I'll try to give my thoughts on some of the things that were said here: > >> make -C > I don't think make -C does what you think it does. "make -C foo" is > basically equivalent to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On 18 January 2018 at 17:52, Adrian Prantl wrote: > > >> On Jan 18, 2018, at 5:07 AM, Pavel Labath wrote: >> >> Looks like I missed a party. :) >> >> I'll try to give my thoughts on some of the things that were said here: >> >>> make -C >> I don't think make

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 18, 2018, at 5:07 AM, Pavel Labath wrote: > > Looks like I missed a party. :) > > I'll try to give my thoughts on some of the things that were said here: > >> make -C > I don't think make -C does what you think it does. "make -C foo" is > basically equivalent to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

Looks like I missed a party. :) I'll try to give my thoughts on some of the things that were said here: > make -C I don't think make -C does what you think it does. "make -C foo" is basically equivalent to "cd foo && make", which is what we are doing now already. Of course, you can make this

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:26 PM, Greg Clayton via lldb-dev > wrote: > > Everything sounds good on this thread. My two cents: > > We should add some post verification after each test that looks for file that > are left over after the "clean" phase. This can help us

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:47 PM, Zachary Turner wrote: > > > > On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl wrote: > >> On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: >> >> I don't know what would be involved in getting the

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:50 PM, Zachary Turner wrote: > > > > On Wed, Jan 17, 2018 at 3:44 PM Jim Ingham wrote: > > I don't see any of these operations that can't be done in a make file, after > all you can run arbitrary commands there. We do make

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 3:44 PM Jim Ingham wrote: > > I don't see any of these operations that can't be done in a make file, > after all you can run arbitrary commands there. We do make directories, > dylibs, move and strip files, etc in some of the makefiles in the test

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl wrote: > > On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: > > I don't know what would be involved in getting the tests building out of > tree with Make. But I do know it would be simple with CMake. I'm

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:32 PM, Zachary Turner wrote: > > On Wed, Jan 17, 2018 at 3:26 PM Greg Clayton wrote: > I don't see why we would go with a lit based system that manually specifies > the arguments. Seems like a pain to get the right compiler

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: > > I don't know what would be involved in getting the tests building out of tree > with Make. But I do know it would be simple with CMake. I'm sure it's > probably not terrible with Make either, I just don't know

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 3:26 PM Greg Clayton wrote: > I don't see why we would go with a lit based system that manually > specifies the arguments. Seems like a pain to get the right compiler flags > for creating shared libs on different systems (or executables, frameworks, >

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: > > I don't know what would be involved in getting the tests building out of tree > with Make. But I do know it would be simple with CMake. I'm sure it's > probably not terrible with Make either, I just don't know

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

I don't know what would be involved in getting the tests building out of tree with Make. But I do know it would be simple with CMake. I'm sure it's probably not terrible with Make either, I just don't know enough about it to say. One thing that I do like about CMake is that it can be integrated

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

Everything sounds good on this thread. My two cents: We should add some post verification after each test that looks for file that are left over after the "clean" phase. This can help us catch the tests that aren't cleaning up after themselves. This will help us weed out the bad tests and fix

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 3:05 PM, Zachary Turner wrote: > > Note that we're going off topic from the original goal, and I just want to > re-iterate that I fully support smaller, incremental changes. Indeed. So just to close the loop on this, it sounds like everybody is in

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

Yeah, w.r.t. the actual builder part, it seems to me any option is going to be sufficiently simple to use that it would be hard for the incremental benefits to lldb developers to ever amortize the cost of switching. The only compelling reason to me is if one or the other tool made it much

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

pra...@apple.com> Cc: LLDB <lldb-dev@lists.llvm.org> Subject: Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability I don't think new test authors really need to add CMake any more so than they currently need to understand Make. Which is to say, not very much. Mos

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl wrote: > > On the other hand: > - everybody already knows make > I'm not sold on this particular reason. Make is not the LLVM build system, CMake is. "I don't know the build system of the project I actually work on, but I do

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

Note that we're going off topic from the original goal, and I just want to re-iterate that I fully support smaller, incremental changes. But since I like talking about lit so much, I can't help but chime in :) If we *did* want to move to a lit based system for the end to end based tests, the

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 2:56 PM, Jim Ingham wrote: > >> >> On Jan 17, 2018, at 2:55 PM, Adrian Prantl wrote: >> >> >> >>> On Jan 17, 2018, at 2:50 PM, Jim Ingham wrote: >>> >>> >>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

Lit provides some helpful utilities which make it easier to write portable tests. E.g %t, for temporary test directories, and portable replacements for things like `diff -r`. This is how compiler-rt's end-to-end tests are structured, and we haven't needed any build-system like functionality

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 2:55 PM, Adrian Prantl wrote: > > > >> On Jan 17, 2018, at 2:50 PM, Jim Ingham wrote: >> >> >> >>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: >>> >>> >>> On Jan 17, 2018, at 2:31 PM, Zachary

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

As a general rule, I support moving towards explicit run lines and lit-style tests, but some care has to be taken. If you examine the common Makefiles, you'll see that there's already a lot of special logic for different platforms and compilers. It might be hard to maintain that if we go back to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 2:50 PM, Jim Ingham wrote: > > > >> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: >> >> >> >>> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: >>> >>> I don't think new test authors really need to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: > > > >> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: >> >> I don't think new test authors really need to add CMake any more so than >> they currently need to understand Make. Which is to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: > > I don't think new test authors really need to add CMake any more so than they > currently need to understand Make. Which is to say, not very much. Most > Makefiles are currently 1-2 lines of code that simply does

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 1:45 PM, Vedant Kumar wrote: > > I would prefer having all of our test dependencies tracked by CMake for all > the reasons Zach brought up, but I think we should defer that undertaking > until after the bots are more stable. We have some immediate

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 1:13 PM, Davide Italiano wrote: > > On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano > wrote: >> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev >> wrote: >>> Hi lldb-dev! >>> >>> I've

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

I would prefer having all of our test dependencies tracked by CMake for all the reasons Zach brought up, but I think we should defer that undertaking until after the bots are more stable. We have some immediate problems caused by stale in-tree test artifacts. As a first milestone, it'd be great

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 12:41 PM, Zachary Turner wrote: > > If we're going to be making any significant changes to the way inferiors are > compiled, why not use cmake? Make clean is already not implemented correctly > in many places, leading to lots of remnants left over

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano wrote: > On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev > wrote: >> Hi lldb-dev! >> >> I've been investigating some spurious LLDB test suite failures on >>

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 1:02 PM, Davide Italiano via lldb-dev > wrote: > > On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev > wrote: >> Hi lldb-dev! >> >> I've been investigating some spurious LLDB test suite failures on >>

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev wrote: > Hi lldb-dev! > > I've been investigating some spurious LLDB test suite failures on > http://green.lab.llvm.org/green/ that had to do with build artifacts from > previous runs lying around in the test

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

> On Jan 17, 2018, at 12:32 PM, Adrian Prantl wrote: > > Hi lldb-dev! > > I've been investigating some spurious LLDB test suite failures on > http://green.lab.llvm.org/green/ that had to do with build artifacts from > previous runs lying around in the test directories and

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev wrote: > Hi lldb-dev! > > I've been investigating some spurious LLDB test suite failures on > http://green.lab.llvm.org/green/ that had to do with build artifacts from > previous runs lying around in the test

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

If we're going to be making any significant changes to the way inferiors are compiled, why not use cmake? Make clean is already not implemented correctly in many places, leading to lots of remnants left over in the source tree after test runs. Furthermore, make is run every single time