Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-18 Thread Adrian Prantl via lldb-dev
> On Jan 18, 2018, at 5:07 AM, Pavel Labath wrote: > > Looks like I missed a party. :) > > I'll try to give my thoughts on some of the things that were said here: > >> make -C > I don't think make -C does what you think it does. "make -C foo" is > basically equivalent to "cd foo && make", whi

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-18 Thread Pavel Labath via lldb-dev
On 18 January 2018 at 17:52, Adrian Prantl wrote: > > >> On Jan 18, 2018, at 5:07 AM, Pavel Labath wrote: >> >> Looks like I missed a party. :) >> >> I'll try to give my thoughts on some of the things that were said here: >> >>> make -C >> I don't think make -C does what you think it does. "make

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-18 Thread Adrian Prantl via lldb-dev
> On Jan 18, 2018, at 5:07 AM, Pavel Labath wrote: > > Looks like I missed a party. :) > > I'll try to give my thoughts on some of the things that were said here: > >> make -C > I don't think make -C does what you think it does. "make -C foo" is > basically equivalent to "cd foo && make", whi

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-18 Thread Pavel Labath via lldb-dev
Looks like I missed a party. :) I'll try to give my thoughts on some of the things that were said here: > make -C I don't think make -C does what you think it does. "make -C foo" is basically equivalent to "cd foo && make", which is what we are doing now already. Of course, you can make this work

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 3:26 PM, Greg Clayton via lldb-dev > wrote: > > Everything sounds good on this thread. My two cents: > > We should add some post verification after each test that looks for file that > are left over after the "clean" phase. This can help us catch the tests that > aren'

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 3:47 PM, Zachary Turner wrote: > > > > On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl wrote: > >> On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: >> >> I don't know what would be involved in getting the tests building out of >> tree with Make. But I do know it wo

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 3:50 PM, Zachary Turner wrote: > > > > On Wed, Jan 17, 2018 at 3:44 PM Jim Ingham wrote: > > I don't see any of these operations that can't be done in a make file, after > all you can run arbitrary commands there. We do make directories, dylibs, > move and strip fil

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
On Wed, Jan 17, 2018 at 3:44 PM Jim Ingham wrote: > > I don't see any of these operations that can't be done in a make file, > after all you can run arbitrary commands there. We do make directories, > dylibs, move and strip files, etc in some of the makefiles in the test case. > Make doesn't hav

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl wrote: > > On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: > > I don't know what would be involved in getting the tests building out of > tree with Make. But I do know it would be simple with CMake. I'm sure > it's probably not terrible with Mak

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 3:32 PM, Zachary Turner wrote: > > On Wed, Jan 17, 2018 at 3:26 PM Greg Clayton wrote: > I don't see why we would go with a lit based system that manually specifies > the arguments. Seems like a pain to get the right compiler flags for creating > shared libs on differen

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: > > I don't know what would be involved in getting the tests building out of tree > with Make. But I do know it would be simple with CMake. I'm sure it's > probably not terrible with Make either, I just don't know enough about it to > sa

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
On Wed, Jan 17, 2018 at 3:26 PM Greg Clayton wrote: > I don't see why we would go with a lit based system that manually > specifies the arguments. Seems like a pain to get the right compiler flags > for creating shared libs on different systems (or executables, frameworks, > etc). Seems like cmak

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 3:25 PM, Zachary Turner wrote: > > I don't know what would be involved in getting the tests building out of tree > with Make. But I do know it would be simple with CMake. I'm sure it's > probably not terrible with Make either, I just don't know enough about it to > sa

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
I don't know what would be involved in getting the tests building out of tree with Make. But I do know it would be simple with CMake. I'm sure it's probably not terrible with Make either, I just don't know enough about it to say. One thing that I do like about CMake is that it can be integrated

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Greg Clayton via lldb-dev
Everything sounds good on this thread. My two cents: We should add some post verification after each test that looks for file that are left over after the "clean" phase. This can help us catch the tests that aren't cleaning up after themselves. This will help us weed out the bad tests and fix t

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 3:05 PM, Zachary Turner wrote: > > Note that we're going off topic from the original goal, and I just want to > re-iterate that I fully support smaller, incremental changes. Indeed. So just to close the loop on this, it sounds like everybody is in agreement that runnin

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
Yeah, w.r.t. the actual builder part, it seems to me any option is going to be sufficiently simple to use that it would be hard for the incremental benefits to lldb developers to ever amortize the cost of switching. The only compelling reason to me is if one or the other tool made it much easie

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Ted Woodward via lldb-dev
Subject: Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability I don't think new test authors really need to add CMake any more so than they currently need to understand Make. Which is to say, not very much. Most Makefiles are currently 1-2 lines of code that s

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl wrote: > > On the other hand: > - everybody already knows make > I'm not sold on this particular reason. Make is not the LLVM build system, CMake is. "I don't know the build system of the project I actually work on, but I do know this other build s

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
Note that we're going off topic from the original goal, and I just want to re-iterate that I fully support smaller, incremental changes. But since I like talking about lit so much, I can't help but chime in :) If we *did* want to move to a lit based system for the end to end based tests, the firs

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 2:56 PM, Jim Ingham wrote: > >> >> On Jan 17, 2018, at 2:55 PM, Adrian Prantl wrote: >> >> >> >>> On Jan 17, 2018, at 2:50 PM, Jim Ingham wrote: >>> >>> >>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: Question: how would you feel about conve

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Vedant Kumar via lldb-dev
Lit provides some helpful utilities which make it easier to write portable tests. E.g %t, for temporary test directories, and portable replacements for things like `diff -r`. This is how compiler-rt's end-to-end tests are structured, and we haven't needed any build-system like functionality ther

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 2:55 PM, Adrian Prantl wrote: > > > >> On Jan 17, 2018, at 2:50 PM, Jim Ingham wrote: >> >> >> >>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: >>> >>> >>> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: I don't think new test authors real

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
As a general rule, I support moving towards explicit run lines and lit-style tests, but some care has to be taken. If you examine the common Makefiles, you'll see that there's already a lot of special logic for different platforms and compilers. It might be hard to maintain that if we go back to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 2:50 PM, Jim Ingham wrote: > > > >> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: >> >> >> >>> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: >>> >>> I don't think new test authors really need to add CMake any more so than >>> they currently need to unders

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 2:49 PM, Adrian Prantl wrote: > > > >> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: >> >> I don't think new test authors really need to add CMake any more so than >> they currently need to understand Make. Which is to say, not very much. >> Most Makefiles are

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 2:31 PM, Zachary Turner wrote: > > I don't think new test authors really need to add CMake any more so than they > currently need to understand Make. Which is to say, not very much. Most > Makefiles are currently 1-2 lines of code that simply does nothing other than >

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
I don't think new test authors really need to add CMake any more so than they currently need to understand Make. Which is to say, not very much. Most Makefiles are currently 1-2 lines of code that simply does nothing other than include the common Makefile. On the other hand, CMake defines a lot o

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 1:45 PM, Vedant Kumar wrote: > > I would prefer having all of our test dependencies tracked by CMake for all > the reasons Zach brought up, but I think we should defer that undertaking > until after the bots are more stable. We have some immediate problems caused > by s

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 1:13 PM, Davide Italiano wrote: > > On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano > wrote: >> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev >> wrote: >>> Hi lldb-dev! >>> >>> I've been investigating some spurious LLDB test suite failures on >>> http:

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Vedant Kumar via lldb-dev
I would prefer having all of our test dependencies tracked by CMake for all the reasons Zach brought up, but I think we should defer that undertaking until after the bots are more stable. We have some immediate problems caused by stale in-tree test artifacts. As a first milestone, it'd be great

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
> On Jan 17, 2018, at 12:41 PM, Zachary Turner wrote: > > If we're going to be making any significant changes to the way inferiors are > compiled, why not use cmake? Make clean is already not implemented correctly > in many places, leading to lots of remnants left over in the source tree >

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Davide Italiano via lldb-dev
On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano wrote: > On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev > wrote: >> Hi lldb-dev! >> >> I've been investigating some spurious LLDB test suite failures on >> http://green.lab.llvm.org/green/ that had to do with build artifacts from >>

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 1:02 PM, Davide Italiano via lldb-dev > wrote: > > On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev > wrote: >> Hi lldb-dev! >> >> I've been investigating some spurious LLDB test suite failures on >> http://green.lab.llvm.org/green/ that had to do with buil

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Davide Italiano via lldb-dev
On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev wrote: > Hi lldb-dev! > > I've been investigating some spurious LLDB test suite failures on > http://green.lab.llvm.org/green/ that had to do with build artifacts from > previous runs lying around in the test directories and this promp

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Jim Ingham via lldb-dev
> On Jan 17, 2018, at 12:32 PM, Adrian Prantl wrote: > > Hi lldb-dev! > > I've been investigating some spurious LLDB test suite failures on > http://green.lab.llvm.org/green/ that had to do with build artifacts from > previous runs lying around in the test directories and this prompted me to

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Davide Italiano via lldb-dev
On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev wrote: > Hi lldb-dev! > > I've been investigating some spurious LLDB test suite failures on > http://green.lab.llvm.org/green/ that had to do with build artifacts from > previous runs lying around in the test directories and this promp

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Zachary Turner via lldb-dev
If we're going to be making any significant changes to the way inferiors are compiled, why not use cmake? Make clean is already not implemented correctly in many places, leading to lots of remnants left over in the source tree after test runs. Furthermore, make is run every single time currently,

[lldb-dev] Questions about the LLDB testsuite and improving its reliability

2018-01-17 Thread Adrian Prantl via lldb-dev
Hi lldb-dev! I've been investigating some spurious LLDB test suite failures on http://green.lab.llvm.org/green/ that had to do with build artifacts from previous runs lying around in the test directories and this prompted me to ask a couple of general noob questions about the LLDB testsuite. M