Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 18, 2018, at 5:07 AM, Pavel Labath  wrote:
> 
> Looks like I missed a party. :)
> 
> I'll try to give my thoughts on some of the things that were said here:
> 
>> make -C
> I don't think make -C does what you think it does. "make -C foo" is
> basically equivalent to "cd foo && make", which is what we are doing
> now already. Of course, you can make this work, but you would have to
> pass an extra OUTDIR=... argument to make and then modify the
> Makefiles to reference $(OUTDIR) for its outputs:
> $(OUTDIR)/a.out: main.cc
>  $(CC) -o $(OUTDIR)/a.out main.cc ...
> 
> The standard way of doing an out-of-tree build with make is to have
> the Makefile in the build-directory and to set the magic VPATH
> variable in the Makefile (or as a part of make invocation). VPATH
> alters make's search path, so when searching for a dependency foo and
> the foo is not present in the current (build) directory, it will go
> searching for it in the VPATH (source) directory. You still need to be
> careful about paths in the command line (generally this means using
> make variables like $@ and $< instead of bare file names), but our
> makefiles are generally pretty good at this already. We even have a
> couple of makefiles using VPATH already (see TestConcurrentEvents) --
> Todd added this to speed up the build by spreading out tests over
> different folders while sharing sources (the serial execution
> problem).
> 
> I still fully support being able to build the tests out of tree, I
> just think it may be a bit more involved than you realise.

I uploaded my first attempt of implementing something along these lines to 
https://reviews.llvm.org/D42281 . Feedback of all kinds is very welcome!

-- adrian


___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Pavel Labath via lldb-dev]

On 18 January 2018 at 17:52, Adrian Prantl  wrote:
>
>
>> On Jan 18, 2018, at 5:07 AM, Pavel Labath  wrote:
>>
>> Looks like I missed a party. :)
>>
>> I'll try to give my thoughts on some of the things that were said here:
>>
>>> make -C
>> I don't think make -C does what you think it does. "make -C foo" is
>> basically equivalent to "cd foo && make", which is what we are doing
>> now already. Of course, you can make this work, but you would have to
>> pass an extra OUTDIR=... argument to make and then modify the
>> Makefiles to reference $(OUTDIR) for its outputs:
>> $(OUTDIR)/a.out: main.cc
>>  $(CC) -o $(OUTDIR)/a.out main.cc ...
>>
>> The standard way of doing an out-of-tree build with make is to have
>> the Makefile in the build-directory and to set the magic VPATH
>> variable in the Makefile (or as a part of make invocation). VPATH
>> alters make's search path, so when searching for a dependency foo and
>> the foo is not present in the current (build) directory, it will go
>> searching for it in the VPATH (source) directory. You still need to be
>> careful about paths in the command line (generally this means using
>> make variables like $@ and $< instead of bare file names), but our
>> makefiles are generally pretty good at this already. We even have a
>> couple of makefiles using VPATH already (see TestConcurrentEvents) --
>> Todd added this to speed up the build by spreading out tests over
>> different folders while sharing sources (the serial execution
>> problem).
>>
> This is of course correct. Thanks for pointing this out and for outlining all 
> the necessary changes!
>
>> I still fully support being able to build the tests out of tree, I
>> just think it may be a bit more involved than you realise.
>
> Sounds good.
>
>>
>>> cmake
>> I agree that using cmake for building tests would some things simpler.
>> Building fully working executables is fairly tricky, especially when
>> you're cross-compiling. Take test/testcases/Android.rules for example.
>> This is basically a reimplementation of the android cmake toolchain
>> file distributed with the android ndk. If we had cmake for building
>> tests we could delete all of that and replace it with
>> -DCMAKE_TOOLCHAIN_FILE=$(ANDROID_NDK_HOME)/android.toolchain.cmake.
>> However, I only had to write Android.rules just once, so it's not that
>> big of a deal for me.
>>
>>> explicit RUN lines:
>> Yes, it's true that all you need is custom CXXFLAGS (and LDFLAGS), but
>> those CXX flags could be quite complex. I'm not really opposed to
>> that, but I don't see any immediate benefit either (the only impact
>> for me would be that I'd have to translate Android.rules to python).
>>
>>> running clean after every test
>> Currently we must run "make clean" after every test because make does
>> not track the changes in it's arguments. So, if you first run "make
>> MAKE_DWO=NO" and then "make MAKE_DWO=YES", make will happily declare
>> that it has nothing to do without building the DWO binary. (This will
>> go away if we run each test variant in a separate directory).
>>
>> I don't have any data, but I would expect that running make upfront
>> would make a significant performance improvement on windows (spinning
>> up tons of processes and creating/deleting a bunch of files tends to
>> be much slower there), but to have no noticable difference on other
>> platforms.
>>
>
> I have not thought of Windows being a possible bottleneck. That sounds 
> plausible. I'm still wondering how useful incremental builds for the 
> testsuite are going to be. When I'm looking at our bots, almost all incoming 
> commits are not in LLDB, but in LLVM or Clang. If we are modeling 
> dependencies correctly, then each of these commits that changes the compiler 
> should still trigger a full rebuild of all testcases, even with CMake. The 
> only situation where incremental builds would be useful are for a 
> configuration where we are building the LLDB testsuite against a fixed 
> compiler, such as a released/stable version of clang or gcc. That's not to 
> say that that isn't an important use-case, too, of course, but it's not how 
> the bots on green dragon are configured at the moment.
>
> -- adrian

It wouldn't speed up the bots, but it may make the workflow of an lldb
developer (who  changes only lldb code most of the time) faster.
However, that doesn't matter I guess, as that's not the direction
we're going now.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 18, 2018, at 5:07 AM, Pavel Labath  wrote:
> 
> Looks like I missed a party. :)
> 
> I'll try to give my thoughts on some of the things that were said here:
> 
>> make -C
> I don't think make -C does what you think it does. "make -C foo" is
> basically equivalent to "cd foo && make", which is what we are doing
> now already. Of course, you can make this work, but you would have to
> pass an extra OUTDIR=... argument to make and then modify the
> Makefiles to reference $(OUTDIR) for its outputs:
> $(OUTDIR)/a.out: main.cc
>  $(CC) -o $(OUTDIR)/a.out main.cc ...
> 
> The standard way of doing an out-of-tree build with make is to have
> the Makefile in the build-directory and to set the magic VPATH
> variable in the Makefile (or as a part of make invocation). VPATH
> alters make's search path, so when searching for a dependency foo and
> the foo is not present in the current (build) directory, it will go
> searching for it in the VPATH (source) directory. You still need to be
> careful about paths in the command line (generally this means using
> make variables like $@ and $< instead of bare file names), but our
> makefiles are generally pretty good at this already. We even have a
> couple of makefiles using VPATH already (see TestConcurrentEvents) --
> Todd added this to speed up the build by spreading out tests over
> different folders while sharing sources (the serial execution
> problem).
> 
This is of course correct. Thanks for pointing this out and for outlining all 
the necessary changes!

> I still fully support being able to build the tests out of tree, I
> just think it may be a bit more involved than you realise.

Sounds good.

> 
>> cmake
> I agree that using cmake for building tests would some things simpler.
> Building fully working executables is fairly tricky, especially when
> you're cross-compiling. Take test/testcases/Android.rules for example.
> This is basically a reimplementation of the android cmake toolchain
> file distributed with the android ndk. If we had cmake for building
> tests we could delete all of that and replace it with
> -DCMAKE_TOOLCHAIN_FILE=$(ANDROID_NDK_HOME)/android.toolchain.cmake.
> However, I only had to write Android.rules just once, so it's not that
> big of a deal for me.
> 
>> explicit RUN lines:
> Yes, it's true that all you need is custom CXXFLAGS (and LDFLAGS), but
> those CXX flags could be quite complex. I'm not really opposed to
> that, but I don't see any immediate benefit either (the only impact
> for me would be that I'd have to translate Android.rules to python).
> 
>> running clean after every test
> Currently we must run "make clean" after every test because make does
> not track the changes in it's arguments. So, if you first run "make
> MAKE_DWO=NO" and then "make MAKE_DWO=YES", make will happily declare
> that it has nothing to do without building the DWO binary. (This will
> go away if we run each test variant in a separate directory).
> 
> I don't have any data, but I would expect that running make upfront
> would make a significant performance improvement on windows (spinning
> up tons of processes and creating/deleting a bunch of files tends to
> be much slower there), but to have no noticable difference on other
> platforms.
> 

I have not thought of Windows being a possible bottleneck. That sounds 
plausible. I'm still wondering how useful incremental builds for the testsuite 
are going to be. When I'm looking at our bots, almost all incoming commits are 
not in LLDB, but in LLVM or Clang. If we are modeling dependencies correctly, 
then each of these commits that changes the compiler should still trigger a 
full rebuild of all testcases, even with CMake. The only situation where 
incremental builds would be useful are for a configuration where we are 
building the LLDB testsuite against a fixed compiler, such as a released/stable 
version of clang or gcc. That's not to say that that isn't an important 
use-case, too, of course, but it's not how the bots on green dragon are 
configured at the moment.

-- adrian
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Pavel Labath via lldb-dev]

Looks like I missed a party. :)

I'll try to give my thoughts on some of the things that were said here:

> make -C
I don't think make -C does what you think it does. "make -C foo" is
basically equivalent to "cd foo && make", which is what we are doing
now already. Of course, you can make this work, but you would have to
pass an extra OUTDIR=... argument to make and then modify the
Makefiles to reference $(OUTDIR) for its outputs:
$(OUTDIR)/a.out: main.cc
  $(CC) -o $(OUTDIR)/a.out main.cc ...

The standard way of doing an out-of-tree build with make is to have
the Makefile in the build-directory and to set the magic VPATH
variable in the Makefile (or as a part of make invocation). VPATH
alters make's search path, so when searching for a dependency foo and
the foo is not present in the current (build) directory, it will go
searching for it in the VPATH (source) directory. You still need to be
careful about paths in the command line (generally this means using
make variables like $@ and $< instead of bare file names), but our
makefiles are generally pretty good at this already. We even have a
couple of makefiles using VPATH already (see TestConcurrentEvents) --
Todd added this to speed up the build by spreading out tests over
different folders while sharing sources (the serial execution
problem).

I still fully support being able to build the tests out of tree, I
just think it may be a bit more involved than you realise.

> cmake
I agree that using cmake for building tests would some things simpler.
Building fully working executables is fairly tricky, especially when
you're cross-compiling. Take test/testcases/Android.rules for example.
This is basically a reimplementation of the android cmake toolchain
file distributed with the android ndk. If we had cmake for building
tests we could delete all of that and replace it with
-DCMAKE_TOOLCHAIN_FILE=$(ANDROID_NDK_HOME)/android.toolchain.cmake.
However, I only had to write Android.rules just once, so it's not that
big of a deal for me.

> explicit RUN lines:
Yes, it's true that all you need is custom CXXFLAGS (and LDFLAGS), but
those CXX flags could be quite complex. I'm not really opposed to
that, but I don't see any immediate benefit either (the only impact
for me would be that I'd have to translate Android.rules to python).

> running clean after every test
Currently we must run "make clean" after every test because make does
not track the changes in it's arguments. So, if you first run "make
MAKE_DWO=NO" and then "make MAKE_DWO=YES", make will happily declare
that it has nothing to do without building the DWO binary. (This will
go away if we run each test variant in a separate directory).

I don't have any data, but I would expect that running make upfront
would make a significant performance improvement on windows (spinning
up tons of processes and creating/deleting a bunch of files tends to
be much slower there), but to have no noticable difference on other
platforms.



On 18 January 2018 at 01:45, Adrian Prantl via lldb-dev
 wrote:
>
>
>> On Jan 17, 2018, at 3:26 PM, Greg Clayton via lldb-dev 
>>  wrote:
>>
>> Everything sounds good on this thread. My two cents:
>>
>> We should add some post verification after each test that looks for file 
>> that are left over after the "clean" phase. This can help us catch the tests 
>> that aren't cleaning up after themselves. This will help us weed out the bad 
>> tests and fix this ASAP. This can be done both for in tree and out of tree 
>> solutions to verify there is no source polution.
>>
>> We can easily move build artifacts out of the source tree. Running the test 
>> suite remotely via "lldb-server platform" has code that creates directories 
>> for each test in the platform working directory. If the test runs fine and 
>> passes, it cleans up the entire numbered test directory, else it leaves the 
>> numbered directory there so we can debug any issues. Shouldn't be hard to 
>> enable this.
>
> For completeness, I looked at this and it doesn't look like that is how it 
> works. My understanding (and keep in mind that this is the first time I am 
> looking at this code so I might be misunderstanding something here) is that 
> the remote platform support works by building the test on the host in-tree 
> and then _RemoteProcess.launch() ships over only the binary when called from 
> Base.spawnSubprocess().
>
> That's not a big deal, though. I will introduce the concept of a build 
> directory (which has to be separate from the remote platform working 
> directory) and find a way to pass the source directory to runBuildCommands().
>
> -- adrian
>
>>
>> I like the current default of having a new directory with the time and data 
>> with results inside as it allows comparison of one test suite run to another.
>>
>> Switching build systems to cmake is fine if someone has the energy to do it, 
>> that would be great. I don't see why we would go with a lit based 

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 3:26 PM, Greg Clayton via lldb-dev 
>  wrote:
> 
> Everything sounds good on this thread. My two cents:
> 
> We should add some post verification after each test that looks for file that 
> are left over after the "clean" phase. This can help us catch the tests that 
> aren't cleaning up after themselves. This will help us weed out the bad tests 
> and fix this ASAP. This can be done both for in tree and out of tree 
> solutions to verify there is no source polution.
> 
> We can easily move build artifacts out of the source tree. Running the test 
> suite remotely via "lldb-server platform" has code that creates directories 
> for each test in the platform working directory. If the test runs fine and 
> passes, it cleans up the entire numbered test directory, else it leaves the 
> numbered directory there so we can debug any issues. Shouldn't be hard to 
> enable this.

For completeness, I looked at this and it doesn't look like that is how it 
works. My understanding (and keep in mind that this is the first time I am 
looking at this code so I might be misunderstanding something here) is that the 
remote platform support works by building the test on the host in-tree and then 
_RemoteProcess.launch() ships over only the binary when called from 
Base.spawnSubprocess().

That's not a big deal, though. I will introduce the concept of a build 
directory (which has to be separate from the remote platform working directory) 
and find a way to pass the source directory to runBuildCommands().

-- adrian

> 
> I like the current default of having a new directory with the time and data 
> with results inside as it allows comparison of one test suite run to another.
> 
> Switching build systems to cmake is fine if someone has the energy to do it, 
> that would be great. I don't see why we would go with a lit based system that 
> manually specifies the arguments. Seems like a pain to get the right compiler 
> flags for creating shared libs on different systems (or executables, 
> frameworks, etc). Seems like cmake is good at that and very simple.
> 
> 
>> On Jan 17, 2018, at 3:18 PM, Jim Ingham via lldb-dev 
>>  wrote:
>> 
>> Yeah, w.r.t. the actual builder part, it seems to me any option is going to 
>> be sufficiently simple to use that it would be hard for the incremental 
>> benefits to lldb developers to ever amortize the cost of switching.  The 
>> only compelling reason to me is if one or the other tool made it much easier 
>> to get building the test cases out of tree working, but that seems unlikely.
>> 
>> Jim
>> 
>> 
>>> On Jan 17, 2018, at 3:07 PM, Zachary Turner  wrote:
>>> 
>>> 
>>> 
>>> On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  wrote:
>>> 
>>> On the other hand:
>>> - everybody already knows make
>>> 
>>> I'm not sold on this particular reason.  Make is not the LLVM build system, 
>>> CMake is.  "I don't know the build system of the project I actually work 
>>> on, but I do know this other build system" is not a compelling argument. 
>>> 
>>> (As an aside, not every knows Make that well, but it doesn't actually 
>>> matter because the amount of actual Make code is negligibly small, i.e. 1-2 
>>> lines per test in a vast majority of cases)
>> 
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 3:47 PM, Zachary Turner  wrote:
> 
> 
> 
> On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl  wrote:
> 
>> On Jan 17, 2018, at 3:25 PM, Zachary Turner  wrote:
>> 
>> I don't know what would be involved in getting the tests building out of 
>> tree with Make.  But I do know it would be simple with CMake.  I'm sure it's 
>> probably not terrible with Make either, I just don't know enough about it to 
>> say.
>> 
>> One thing that I do like about CMake is that it can be integrated into the 
>> existing LLDB build configuration step, which already uses CMake, to build 
>> inferiors up front.  This has the potential to speed up the test suite by an 
>> order of magnitude.
> 
> Since the tests in the LLDB testsuite are typically very small and don't 
> import a lot of headers I'm not convinced that an incremental build of the 
> tests will have a very big impact on the running time of the testsuite, but 
> to be honest I also haven't benchmarked it to see where the time is really 
> spent.
> 
> It actually is pretty significant.  Part of this is due to the fact that a 
> single .py file has multiple tests, and the compile happens for every one of 
> those tests, even though it just produces the exact same output every time.

Ack, that doesn't seem necessary, right?  The clean should happen as part of 
the test case object cleanup, and then make can figure out what needs to be 
built.  This would have to be done with a little care, since it puts the 
responsibility on any test that mucks with the built product to clean as a test 
cleanup, but I bet very few tests do that.

Again, only worth while after we actually measure the time spent in the various 
parts of the test run.  But this seems like not that hard to fix.

Jim


___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 3:50 PM, Zachary Turner  wrote:
> 
> 
> 
> On Wed, Jan 17, 2018 at 3:44 PM Jim Ingham  wrote:
> 
> I don't see any of these operations that can't be done in a make file, after 
> all you can run arbitrary commands there.  We do make directories, dylibs, 
> move and strip files, etc in some of the makefiles in the test case.
> Make doesn't have builtin substitutions like %t or even better, ones you can 
> define yourself in Python.  Make can obviously do anything (it can even run a 
> shell script!) but when you have a DSL, then it becomes much easier to do 
> domain specific things.
> 

Sure but I can't imagine anything we want to do here that that easiness delta 
is going to be significant, and if you do want to do complex things, python 
also has the system command.

Jim

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

On Wed, Jan 17, 2018 at 3:44 PM Jim Ingham  wrote:

>
> I don't see any of these operations that can't be done in a make file,
> after all you can run arbitrary commands there.  We do make directories,
> dylibs, move and strip files, etc in some of the makefiles in the test case.
>
Make doesn't have builtin substitutions like %t or even better, ones you
can define yourself in Python.  Make can obviously do anything (it can even
run a shell script!) but when you have a DSL, then it becomes much easier
to do domain specific things.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

On Wed, Jan 17, 2018 at 3:39 PM Adrian Prantl  wrote:

>
> On Jan 17, 2018, at 3:25 PM, Zachary Turner  wrote:
>
> I don't know what would be involved in getting the tests building out of
> tree with Make.  But I do know it would be simple with CMake.  I'm sure
> it's probably not terrible with Make either, I just don't know enough about
> it to say.
>
> One thing that I do like about CMake is that it can be integrated into the
> existing LLDB build configuration step, which already uses CMake, to build
> inferiors up front.  This has the potential to speed up the test suite by
> an order of magnitude.
>
>
> Since the tests in the LLDB testsuite are typically very small and don't
> import a lot of headers I'm not convinced that an incremental build of the
> tests will have a very big impact on the running time of the testsuite, but
> to be honest I also haven't benchmarked it to see where the time is really
> spent.
>

It actually is pretty significant.  Part of this is due to the fact that a
single .py file has multiple tests, and the compile happens for every one
of those tests, even though it just produces the exact same output every
time.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 3:32 PM, Zachary Turner  wrote:
> 
> On Wed, Jan 17, 2018 at 3:26 PM Greg Clayton  wrote:
> I don't see why we would go with a lit based system that manually specifies 
> the arguments. Seems like a pain to get the right compiler flags for creating 
> shared libs on different systems (or executables, frameworks, etc). Seems 
> like cmake is good at that and very simple.
> 
> One reason it's nice is because you can specify more than just a compiler 
> command line.  You can take your input from an assembly file, for example and 
> compile it with llc
> 
>  You can mess around with files and stick them into an archive or you can 
> compile a dll, then run a tool on it to strip something out of it.  You can 
> copy files around to set up a build directory a certain way.  Only limited by 
> your imagination.
> 
> When the compiler invocation is "just another command", you can easily create 
> test cases that are a lot more interesting than just "make an executable from 
> this source code, and debug it".
> 
> I think it can be done, and would be valuable if done right, but I do think 
> getting it right would take some care.

I don't see any of these operations that can't be done in a make file, after 
all you can run arbitrary commands there.  We do make directories, dylibs, move 
and strip files, etc in some of the makefiles in the test case.

OTOH, it is pretty common to have a test directory that has a Test.py with a 
bunch of test cases that all build the same thing.  If we use a command-style 
driver for building, each of these tests will do a full rebuild, whereas now 
make figures out what needs to be done, and the build only actually happens 
once.  If we think avoiding extra compiles is important then you do want the 
build tool to be able to compute dependencies.

Jim


___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 3:25 PM, Zachary Turner  wrote:
> 
> I don't know what would be involved in getting the tests building out of tree 
> with Make.  But I do know it would be simple with CMake.  I'm sure it's 
> probably not terrible with Make either, I just don't know enough about it to 
> say.
> 
> One thing that I do like about CMake is that it can be integrated into the 
> existing LLDB build configuration step, which already uses CMake, to build 
> inferiors up front.  This has the potential to speed up the test suite by an 
> order of magnitude.

Since the tests in the LLDB testsuite are typically very small and don't import 
a lot of headers I'm not convinced that an incremental build of the tests will 
have a very big impact on the running time of the testsuite, but to be honest I 
also haven't benchmarked it to see where the time is really spent.

-- adrian

> 
> Can we get that same effect with a Make-based solution?
> 
> On Wed, Jan 17, 2018 at 3:18 PM Jim Ingham  > wrote:
> Yeah, w.r.t. the actual builder part, it seems to me any option is going to 
> be sufficiently simple to use that it would be hard for the incremental 
> benefits to lldb developers to ever amortize the cost of switching.  The only 
> compelling reason to me is if one or the other tool made it much easier to 
> get building the test cases out of tree working, but that seems unlikely.
> 
> Jim
> 
> 
> > On Jan 17, 2018, at 3:07 PM, Zachary Turner  > > wrote:
> >
> >
> >
> > On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  > > wrote:
> >
> > On the other hand:
> > - everybody already knows make
> >
> > I'm not sold on this particular reason.  Make is not the LLVM build system, 
> > CMake is.  "I don't know the build system of the project I actually work 
> > on, but I do know this other build system" is not a compelling argument.
> >
> > (As an aside, not every knows Make that well, but it doesn't actually 
> > matter because the amount of actual Make code is negligibly small, i.e. 1-2 
> > lines per test in a vast majority of cases)
> 

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

On Wed, Jan 17, 2018 at 3:26 PM Greg Clayton  wrote:

> I don't see why we would go with a lit based system that manually
> specifies the arguments. Seems like a pain to get the right compiler flags
> for creating shared libs on different systems (or executables, frameworks,
> etc). Seems like cmake is good at that and very simple.
>

One reason it's nice is because you can specify more than just a compiler
command line.  You can take your input from an assembly file, for example
and compile it with llc

 You can mess around with files and stick them into an archive or you can
compile a dll, then run a tool on it to strip something out of it.  You can
copy files around to set up a build directory a certain way.  Only limited
by your imagination.

When the compiler invocation is "just another command", you can easily
create test cases that are a lot more interesting than just "make an
executable from this source code, and debug it".

I think it can be done, and would be valuable if done right, but I do think
getting it right would take some care.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 3:25 PM, Zachary Turner  wrote:
> 
> I don't know what would be involved in getting the tests building out of tree 
> with Make.  But I do know it would be simple with CMake.  I'm sure it's 
> probably not terrible with Make either, I just don't know enough about it to 
> say.
> 
> One thing that I do like about CMake is that it can be integrated into the 
> existing LLDB build configuration step, which already uses CMake, to build 
> inferiors up front.  This has the potential to speed up the test suite by an 
> order of magnitude.
> 
> Can we get that same effect with a Make-based solution?

You are going to muck with a bunch of tests to get this to work.  For instance, 
only dotest currently knows what debug variants we are building for (they are 
often specified in the test file itself.)  Also a number of tests - for things 
like rebuild & rerun and others build several times in the test.  The test for 
basic types has a single source file with the basic type in a define that runs 
the build & debug once for each basic type, supplying the appropriate define 
each time.

So for reasons not related to make vrs. cmake I think this is a harder thing 
than you are thinking it is.

Jim


> 
> On Wed, Jan 17, 2018 at 3:18 PM Jim Ingham  wrote:
> Yeah, w.r.t. the actual builder part, it seems to me any option is going to 
> be sufficiently simple to use that it would be hard for the incremental 
> benefits to lldb developers to ever amortize the cost of switching.  The only 
> compelling reason to me is if one or the other tool made it much easier to 
> get building the test cases out of tree working, but that seems unlikely.
> 
> Jim
> 
> 
> > On Jan 17, 2018, at 3:07 PM, Zachary Turner  wrote:
> >
> >
> >
> > On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  wrote:
> >
> > On the other hand:
> > - everybody already knows make
> >
> > I'm not sold on this particular reason.  Make is not the LLVM build system, 
> > CMake is.  "I don't know the build system of the project I actually work 
> > on, but I do know this other build system" is not a compelling argument.
> >
> > (As an aside, not every knows Make that well, but it doesn't actually 
> > matter because the amount of actual Make code is negligibly small, i.e. 1-2 
> > lines per test in a vast majority of cases)
> 

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

I don't know what would be involved in getting the tests building out of
tree with Make.  But I do know it would be simple with CMake.  I'm sure
it's probably not terrible with Make either, I just don't know enough about
it to say.

One thing that I do like about CMake is that it can be integrated into the
existing LLDB build configuration step, which already uses CMake, to build
inferiors up front.  This has the potential to speed up the test suite by
an order of magnitude.

Can we get that same effect with a Make-based solution?

On Wed, Jan 17, 2018 at 3:18 PM Jim Ingham  wrote:

> Yeah, w.r.t. the actual builder part, it seems to me any option is going
> to be sufficiently simple to use that it would be hard for the incremental
> benefits to lldb developers to ever amortize the cost of switching.  The
> only compelling reason to me is if one or the other tool made it much
> easier to get building the test cases out of tree working, but that seems
> unlikely.
>
> Jim
>
>
> > On Jan 17, 2018, at 3:07 PM, Zachary Turner  wrote:
> >
> >
> >
> > On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  wrote:
> >
> > On the other hand:
> > - everybody already knows make
> >
> > I'm not sold on this particular reason.  Make is not the LLVM build
> system, CMake is.  "I don't know the build system of the project I actually
> work on, but I do know this other build system" is not a compelling
> argument.
> >
> > (As an aside, not every knows Make that well, but it doesn't actually
> matter because the amount of actual Make code is negligibly small, i.e. 1-2
> lines per test in a vast majority of cases)
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Greg Clayton via lldb-dev]

Everything sounds good on this thread. My two cents:

We should add some post verification after each test that looks for file that 
are left over after the "clean" phase. This can help us catch the tests that 
aren't cleaning up after themselves. This will help us weed out the bad tests 
and fix this ASAP. This can be done both for in tree and out of tree solutions 
to verify there is no source polution.

We can easily move build artifacts out of the source tree. Running the test 
suite remotely via "lldb-server platform" has code that creates directories for 
each test in the platform working directory. If the test runs fine and passes, 
it cleans up the entire numbered test directory, else it leaves the numbered 
directory there so we can debug any issues. Shouldn't be hard to enable this.

I like the current default of having a new directory with the time and data 
with results inside as it allows comparison of one test suite run to another.

Switching build systems to cmake is fine if someone has the energy to do it, 
that would be great. I don't see why we would go with a lit based system that 
manually specifies the arguments. Seems like a pain to get the right compiler 
flags for creating shared libs on different systems (or executables, 
frameworks, etc). Seems like cmake is good at that and very simple.


> On Jan 17, 2018, at 3:18 PM, Jim Ingham via lldb-dev 
>  wrote:
> 
> Yeah, w.r.t. the actual builder part, it seems to me any option is going to 
> be sufficiently simple to use that it would be hard for the incremental 
> benefits to lldb developers to ever amortize the cost of switching.  The only 
> compelling reason to me is if one or the other tool made it much easier to 
> get building the test cases out of tree working, but that seems unlikely.
> 
> Jim
> 
> 
>> On Jan 17, 2018, at 3:07 PM, Zachary Turner  wrote:
>> 
>> 
>> 
>> On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  wrote:
>> 
>> On the other hand:
>> - everybody already knows make
>> 
>> I'm not sold on this particular reason.  Make is not the LLVM build system, 
>> CMake is.  "I don't know the build system of the project I actually work on, 
>> but I do know this other build system" is not a compelling argument. 
>> 
>> (As an aside, not every knows Make that well, but it doesn't actually matter 
>> because the amount of actual Make code is negligibly small, i.e. 1-2 lines 
>> per test in a vast majority of cases)
> 
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 3:05 PM, Zachary Turner  wrote:
> 
> Note that we're going off topic from the original goal, and I just want to 
> re-iterate that I fully support smaller, incremental changes. 

Indeed. So just to close the loop on this, it sounds like everybody is in 
agreement that running the tests out-of-tree is a worthwhile goal. So I will 
focus on implementing this next. My rough idea is to generate a fresh directory 
for each test configuration in $builddir/path/to/testname.config and run "make 
-C $srcdir/path/to/testname" there. It looks like the right place to 
implemented this is probably dotest.py.

I like the idea of running dotest from lit, but I'll save this for later.

-- adrian
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

Yeah, w.r.t. the actual builder part, it seems to me any option is going to be 
sufficiently simple to use that it would be hard for the incremental benefits 
to lldb developers to ever amortize the cost of switching.  The only compelling 
reason to me is if one or the other tool made it much easier to get building 
the test cases out of tree working, but that seems unlikely.

Jim


> On Jan 17, 2018, at 3:07 PM, Zachary Turner  wrote:
> 
> 
> 
> On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  wrote:
> 
> On the other hand:
> - everybody already knows make
> 
> I'm not sold on this particular reason.  Make is not the LLVM build system, 
> CMake is.  "I don't know the build system of the project I actually work on, 
> but I do know this other build system" is not a compelling argument. 
> 
> (As an aside, not every knows Make that well, but it doesn't actually matter 
> because the amount of actual Make code is negligibly small, i.e. 1-2 lines 
> per test in a vast majority of cases)

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Ted Woodward via lldb-dev]

I disagree that understanding CMake is required to build LLVM. When I build 
top-of-tree on Linux (as opposed to a build that is Hexagon only) I make a 
build directory at the same level as my checkout, and simply run “cmake 
../llvm”. I don’t need to know anything.

 

--

Qualcomm Innovation Center, Inc.

The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux 
Foundation Collaborative Project

 

From: lldb-dev [mailto:lldb-dev-boun...@lists.llvm.org] On Behalf Of Zachary 
Turner via lldb-dev
Sent: Wednesday, January 17, 2018 4:31 PM
To: Adrian Prantl <apra...@apple.com>
Cc: LLDB <lldb-dev@lists.llvm.org>
Subject: Re: [lldb-dev] Questions about the LLDB testsuite and improving its 
reliability

 

I don't think new test authors really need to add CMake any more so than they 
currently need to understand Make.  Which is to say, not very much.  Most 
Makefiles are currently 1-2 lines of code that simply does nothing other than 
include the common Makefile.

 

On the other hand, CMake defines a lot of constructs designed to support 
portable builds, so actually writing and maintaining that common CMake build 
file would be much easier.  The existing Makefile-based system already doesn't 
require you to understand the specific compiler invocations you want.  Here's 3 
random Makefiles, which is hopefully representative given that I pulled them 
completely at random.

 

breakpoint-commands/Makefile:

LEVEL = ../../../make

CXX_SOURCES := nested.cpp

include $(LEVEL)/Makefile.rules

 

functionalities/inferior-assert:

LEVEL = ../../make

C_SOURCES := main.c

include $(LEVEL)/Makefile.rules

 

 

types:

LEVEL = ../make

# Example:

#

# CXX_SOURCES := int.cpp

include $(LEVEL)/Makefile.rules

 

None of this is particularly interesting.  There are a very few tests that need 
to do something weird.  I opened 10 other random Makefiles and still didn't 
find any.  I don't believe it would be hard to support those cases.

 

So now instead of "understand Make" it becomes "understand CMake".  Whic is 
already a requirement of building LLVM.

 

If our test suite was lit-based where you actually had to write compiler 
invocations into the test files, I would feel differently, but that isn't what 
we have today.  We have something that is almost a direct mapping to using 
CMake.

 

On Wed, Jan 17, 2018 at 2:22 PM Adrian Prantl <apra...@apple.com 
<mailto:apra...@apple.com> > wrote:



> On Jan 17, 2018, at 1:45 PM, Vedant Kumar <v...@apple.com 
> <mailto:v...@apple.com> > wrote:
>
> I would prefer having all of our test dependencies tracked by CMake for all 
> the reasons Zach brought up, but I think we should defer that undertaking 
> until after the bots are more stable. We have some immediate problems caused 
> by stale in-tree test artifacts. As a first milestone, it'd be great to not 
> have to run `git clean -fdx` anymore.

I'm pretty sure I do not want to go all the way to CMake right now, but I am 
curious about your motivation: Why do you think that using CMake to build the 
tests in the testsuite is a good idea? In my opinion this adds a layer of 
complexity to the tests that makes it harder to understand what exactly is 
happening and test authors now need to understand CMake *and* the compiler 
invocations they want to execute in their tests. Do you also share Zachary's 
point of view that the tests should be build artifacts that should be kept 
after an incremental build?

-- adrian

>
>
>> On Jan 17, 2018, at 1:13 PM, Davide Italiano via lldb-dev 
>> <lldb-dev@lists.llvm.org <mailto:lldb-dev@lists.llvm.org> > wrote:
>>
>> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano <dccitali...@gmail.com 
>> <mailto:dccitali...@gmail.com> > wrote:
>>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>>> <lldb-dev@lists.llvm.org <mailto:lldb-dev@lists.llvm.org> > wrote:
>>>> Hi lldb-dev!
>>>>
>>>> I've been investigating some spurious LLDB test suite failures on 
>>>> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
>>>> previous runs lying around in the test directories and this prompted me to 
>>>> ask a couple of general noob questions about the LLDB testsuite.
>>>>
>>>> My understanding is that all execution tests are compiled using using 
>>>> `make` in-tree. I.e.: the test driver (dotest.py) effectively executes 
>>>> something equivalent to `cd $srcdir/packages/.../mytest && make`. And it 
>>>> does this in a serial fashion for all configurations (dwarf, dSYM, dwo, 
>>>> ...) and relies on the `clean` target to be implemented correctly.
>>>>
>>>> I don't understand all the design decisions that went i

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

On Wed, Jan 17, 2018 at 3:04 PM Adrian Prantl  wrote:

>
> On the other hand:
> - everybody already knows make
>

I'm not sold on this particular reason.  Make is not the LLVM build system,
CMake is.  "I don't know the build system of the project I actually work
on, but I do know this other build system" is not a compelling argument.

(As an aside, not every knows Make that well, but it doesn't actually
matter because the amount of actual Make code is negligibly small, i.e. 1-2
lines per test in a vast majority of cases)
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

Note that we're going off topic from the original goal, and I just want to
re-iterate that I fully support smaller, incremental changes.  But since I
like talking about lit so much, I can't help but chime in :)

If we *did* want to move to a lit based system for the end to end based
tests, the first step would be to make an LLDBTestFormat and teach it to
literally just call dotest.py in single-process mode with the path to a
specific test file.

That would be a *great* first step, and also very manageable and bite
sized. I think we would see a measurable reduction in flakiness just from
this change.

On the other side though, I still think it is very important to increase
test coverage of *non* end-to-end tests.  When you increase the granularity
of the tests you're able to write, a lot more things become possible.  For
non end-to-end based tests, we can still use a more traditional llvm style
lit test based on lldb-test (which is still new, but the foundation is
there).

On Wed, Jan 17, 2018 at 2:59 PM Vedant Kumar  wrote:

> Lit provides some helpful utilities which make it easier to write portable
> tests. E.g %t, for temporary test directories, and portable replacements
> for things like `diff -r`. This is how compiler-rt's end-to-end tests are
> structured, and we haven't needed any build-system like functionality there.
>
> vedant
>
> > On Jan 17, 2018, at 2:56 PM, Jim Ingham  wrote:
> >
> >
> >> On Jan 17, 2018, at 2:55 PM, Adrian Prantl  wrote:
> >>
> >>
> >>
> >>> On Jan 17, 2018, at 2:50 PM, Jim Ingham  wrote:
> >>>
> >>>
> >>>
>  On Jan 17, 2018, at 2:49 PM, Adrian Prantl  wrote:
> 
> 
> 
> > On Jan 17, 2018, at 2:31 PM, Zachary Turner 
> wrote:
> >
> > I don't think new test authors really need to add CMake any more so
> than they currently need to understand Make.  Which is to say, not very
> much.  Most Makefiles are currently 1-2 lines of code that simply does
> nothing other than include the common Makefile.
> >
> > On the other hand, CMake defines a lot of constructs designed to
> support portable builds, so actually writing and maintaining that common
> CMake build file would be much easier.  The existing Makefile-based system
> already doesn't require you to understand the specific compiler invocations
> you want.  Here's 3 random Makefiles, which is hopefully representative
> given that I pulled them completely at random.
> >
> > breakpoint-commands/Makefile:
> > LEVEL = ../../../make
> > CXX_SOURCES := nested.cpp
> > include $(LEVEL)/Makefile.rules
> >
> > functionalities/inferior-assert:
> > LEVEL = ../../make
> > C_SOURCES := main.c
> > include $(LEVEL)/Makefile.rules
> >
> >
> > types:
> > LEVEL = ../make
> > # Example:
> > #
> > # CXX_SOURCES := int.cpp
> > include $(LEVEL)/Makefile.rules
> >
> > None of this is particularly interesting.  There are a very few
> tests that need to do something weird.  I opened 10 other random Makefiles
> and still didn't find any.  I don't believe it would be hard to support
> those cases.
> >
> > So now instead of "understand Make" it becomes "understand CMake".
> Whic is already a requirement of building LLVM.
> 
>  Fair point. I would suggest that I'll try to make LLDB's testsuite
> build out-of-tree using the existing Makefile system. That should be a
> generally useful first step. After doing this I will hopefully have a much
> better understanding of the requirements of the Makefiles and then we can
> revisit this idea with me actually knowing what I'm talking about :-)
> 
> > If our test suite was lit-based where you actually had to write
> compiler invocations into the test files, I would feel differently, but
> that isn't what we have today.  We have something that is almost a direct
> mapping to using CMake.
> 
>  Question: how would you feel about converting the Makefiles to
> LIT-style .test files with very explicit RUN-lines?
> >>>
> >>> I'm not sure what you mean by this.
> >>
> >> Instead of using a build system at all to build the tests, we would
> hard-code the compiler and linker invocations without encoding any
> dependencies. Because we still need this to be configurable, it would
> probably look something like this:
> >>
> >> RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
> >> RUN: %test_driver test.exe mytest.py
> >
> > I'm worried we'd back into building another make system over time.  What
> advantage would we get from this.
> >
> > Jim
> >
> >
> >
> >>
> >> -- adrian
> >>
> >>>
> >>> Jim
> >>>
> >>>
> 
>  -- adrian
> >
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 2:56 PM, Jim Ingham  wrote:
> 
>> 
>> On Jan 17, 2018, at 2:55 PM, Adrian Prantl  wrote:
>> 
>> 
>> 
>>> On Jan 17, 2018, at 2:50 PM, Jim Ingham  wrote:
>>> 
>>> 
>>> 
 On Jan 17, 2018, at 2:49 PM, Adrian Prantl  wrote:
 
 Question: how would you feel about converting the Makefiles to LIT-style 
 .test files with very explicit RUN-lines?
>>> 
>>> I'm not sure what you mean by this.
>> 
>> Instead of using a build system at all to build the tests, we would 
>> hard-code the compiler and linker invocations without encoding any 
>> dependencies. Because we still need this to be configurable, it would 
>> probably look something like this:
>> 
>> RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
>> RUN: %test_driver test.exe mytest.py
> 
> I'm worried we'd back into building another make system over time.  What 
> advantage would we get from this.

(It's possible that this isn't the right trade-off, I'm just exploring ideas 
here)

Some advantages would be:
- remove the dependency on Make
- possibly easier top debug testcases because all actions are explicit
- potentially slightly faster build times because we don't need to spawn make, 
but note that it also means that these actions will always be unconditionally 
executed
- uniformity with other LLVM projects and thus a smaller cognitive burden for 
developers that touch both clang and lldb

On the other hand:
- everybody already knows make
- maybe we do want a full build system to allow incremental builds of testcases

-- adrian
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Vedant Kumar via lldb-dev]

Lit provides some helpful utilities which make it easier to write portable 
tests. E.g %t, for temporary test directories, and portable replacements for 
things like `diff -r`. This is how compiler-rt's end-to-end tests are 
structured, and we haven't needed any build-system like functionality there.

vedant

> On Jan 17, 2018, at 2:56 PM, Jim Ingham  wrote:
> 
> 
>> On Jan 17, 2018, at 2:55 PM, Adrian Prantl  wrote:
>> 
>> 
>> 
>>> On Jan 17, 2018, at 2:50 PM, Jim Ingham  wrote:
>>> 
>>> 
>>> 
 On Jan 17, 2018, at 2:49 PM, Adrian Prantl  wrote:
 
 
 
> On Jan 17, 2018, at 2:31 PM, Zachary Turner  wrote:
> 
> I don't think new test authors really need to add CMake any more so than 
> they currently need to understand Make.  Which is to say, not very much.  
> Most Makefiles are currently 1-2 lines of code that simply does nothing 
> other than include the common Makefile.
> 
> On the other hand, CMake defines a lot of constructs designed to support 
> portable builds, so actually writing and maintaining that common CMake 
> build file would be much easier.  The existing Makefile-based system 
> already doesn't require you to understand the specific compiler 
> invocations you want.  Here's 3 random Makefiles, which is hopefully 
> representative given that I pulled them completely at random.
> 
> breakpoint-commands/Makefile:
> LEVEL = ../../../make
> CXX_SOURCES := nested.cpp
> include $(LEVEL)/Makefile.rules
> 
> functionalities/inferior-assert:
> LEVEL = ../../make
> C_SOURCES := main.c
> include $(LEVEL)/Makefile.rules
> 
> 
> types:
> LEVEL = ../make
> # Example:
> #
> # CXX_SOURCES := int.cpp
> include $(LEVEL)/Makefile.rules
> 
> None of this is particularly interesting.  There are a very few tests 
> that need to do something weird.  I opened 10 other random Makefiles and 
> still didn't find any.  I don't believe it would be hard to support those 
> cases.
> 
> So now instead of "understand Make" it becomes "understand CMake".  Whic 
> is already a requirement of building LLVM.
 
 Fair point. I would suggest that I'll try to make LLDB's testsuite build 
 out-of-tree using the existing Makefile system. That should be a generally 
 useful first step. After doing this I will hopefully have a much better 
 understanding of the requirements of the Makefiles and then we can revisit 
 this idea with me actually knowing what I'm talking about :-)
 
> If our test suite was lit-based where you actually had to write compiler 
> invocations into the test files, I would feel differently, but that isn't 
> what we have today.  We have something that is almost a direct mapping to 
> using CMake.
 
 Question: how would you feel about converting the Makefiles to LIT-style 
 .test files with very explicit RUN-lines?
>>> 
>>> I'm not sure what you mean by this.
>> 
>> Instead of using a build system at all to build the tests, we would 
>> hard-code the compiler and linker invocations without encoding any 
>> dependencies. Because we still need this to be configurable, it would 
>> probably look something like this:
>> 
>> RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
>> RUN: %test_driver test.exe mytest.py
> 
> I'm worried we'd back into building another make system over time.  What 
> advantage would we get from this.
> 
> Jim
> 
> 
> 
>> 
>> -- adrian
>> 
>>> 
>>> Jim
>>> 
>>> 
 
 -- adrian
> 

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 2:55 PM, Adrian Prantl  wrote:
> 
> 
> 
>> On Jan 17, 2018, at 2:50 PM, Jim Ingham  wrote:
>> 
>> 
>> 
>>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl  wrote:
>>> 
>>> 
>>> 
 On Jan 17, 2018, at 2:31 PM, Zachary Turner  wrote:
 
 I don't think new test authors really need to add CMake any more so than 
 they currently need to understand Make.  Which is to say, not very much.  
 Most Makefiles are currently 1-2 lines of code that simply does nothing 
 other than include the common Makefile.
 
 On the other hand, CMake defines a lot of constructs designed to support 
 portable builds, so actually writing and maintaining that common CMake 
 build file would be much easier.  The existing Makefile-based system 
 already doesn't require you to understand the specific compiler 
 invocations you want.  Here's 3 random Makefiles, which is hopefully 
 representative given that I pulled them completely at random.
 
 breakpoint-commands/Makefile:
 LEVEL = ../../../make
 CXX_SOURCES := nested.cpp
 include $(LEVEL)/Makefile.rules
 
 functionalities/inferior-assert:
 LEVEL = ../../make
 C_SOURCES := main.c
 include $(LEVEL)/Makefile.rules
 
 
 types:
 LEVEL = ../make
 # Example:
 #
 # CXX_SOURCES := int.cpp
 include $(LEVEL)/Makefile.rules
 
 None of this is particularly interesting.  There are a very few tests that 
 need to do something weird.  I opened 10 other random Makefiles and still 
 didn't find any.  I don't believe it would be hard to support those cases.
 
 So now instead of "understand Make" it becomes "understand CMake".  Whic 
 is already a requirement of building LLVM.
>>> 
>>> Fair point. I would suggest that I'll try to make LLDB's testsuite build 
>>> out-of-tree using the existing Makefile system. That should be a generally 
>>> useful first step. After doing this I will hopefully have a much better 
>>> understanding of the requirements of the Makefiles and then we can revisit 
>>> this idea with me actually knowing what I'm talking about :-)
>>> 
 If our test suite was lit-based where you actually had to write compiler 
 invocations into the test files, I would feel differently, but that isn't 
 what we have today.  We have something that is almost a direct mapping to 
 using CMake.
>>> 
>>> Question: how would you feel about converting the Makefiles to LIT-style 
>>> .test files with very explicit RUN-lines?
>> 
>> I'm not sure what you mean by this.
> 
> Instead of using a build system at all to build the tests, we would hard-code 
> the compiler and linker invocations without encoding any dependencies. 
> Because we still need this to be configurable, it would probably look 
> something like this:
> 
>  RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
>  RUN: %test_driver test.exe mytest.py

I'm worried we'd back into building another make system over time.  What 
advantage would we get from this.

Jim



> 
> -- adrian
> 
>> 
>> Jim
>> 
>> 
>>> 
>>> -- adrian

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

As a general rule, I support moving towards explicit run lines and
lit-style tests, but some care has to be taken.  If you examine the common
Makefiles, you'll see that there's already a lot of special logic for
different platforms and compilers.  It might be hard to maintain that if we
go back to explicit run lines.  I'm sure there's a way and I'm happy to
help brainstorming ideas for how to do it.

As a first idea, maybe we could have something like a REQUIRES line but
call it COMPILER-SETTINGS instead.   And you could write something like:

COMPILER-SETTINGS: pic, dwarf, shared-library

And that would be responsible for figuring out what compiler options to put
depending on the platform and compiler.

the main challenge with using explicit run lines is going to be figuring
out how to write run lines that work across all compilers and platforms.
(Luckily we don't have to care about MSVC, mostly just clang + gcc)

On Wed, Jan 17, 2018 at 2:49 PM Adrian Prantl  wrote:

>
>
> > On Jan 17, 2018, at 2:31 PM, Zachary Turner  wrote:
> >
> > I don't think new test authors really need to add CMake any more so than
> they currently need to understand Make.  Which is to say, not very much.
> Most Makefiles are currently 1-2 lines of code that simply does nothing
> other than include the common Makefile.
> >
> > On the other hand, CMake defines a lot of constructs designed to support
> portable builds, so actually writing and maintaining that common CMake
> build file would be much easier.  The existing Makefile-based system
> already doesn't require you to understand the specific compiler invocations
> you want.  Here's 3 random Makefiles, which is hopefully representative
> given that I pulled them completely at random.
> >
> > breakpoint-commands/Makefile:
> > LEVEL = ../../../make
> > CXX_SOURCES := nested.cpp
> > include $(LEVEL)/Makefile.rules
> >
> > functionalities/inferior-assert:
> > LEVEL = ../../make
> > C_SOURCES := main.c
> > include $(LEVEL)/Makefile.rules
> >
> >
> > types:
> > LEVEL = ../make
> > # Example:
> > #
> > # CXX_SOURCES := int.cpp
> > include $(LEVEL)/Makefile.rules
> >
> > None of this is particularly interesting.  There are a very few tests
> that need to do something weird.  I opened 10 other random Makefiles and
> still didn't find any.  I don't believe it would be hard to support those
> cases.
> >
> > So now instead of "understand Make" it becomes "understand CMake".  Whic
> is already a requirement of building LLVM.
>
> Fair point. I would suggest that I'll try to make LLDB's testsuite build
> out-of-tree using the existing Makefile system. That should be a generally
> useful first step. After doing this I will hopefully have a much better
> understanding of the requirements of the Makefiles and then we can revisit
> this idea with me actually knowing what I'm talking about :-)
>
> > If our test suite was lit-based where you actually had to write compiler
> invocations into the test files, I would feel differently, but that isn't
> what we have today.  We have something that is almost a direct mapping to
> using CMake.
>
> Question: how would you feel about converting the Makefiles to LIT-style
> .test files with very explicit RUN-lines?
>
> -- adrian
>
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 2:50 PM, Jim Ingham  wrote:
> 
> 
> 
>> On Jan 17, 2018, at 2:49 PM, Adrian Prantl  wrote:
>> 
>> 
>> 
>>> On Jan 17, 2018, at 2:31 PM, Zachary Turner  wrote:
>>> 
>>> I don't think new test authors really need to add CMake any more so than 
>>> they currently need to understand Make.  Which is to say, not very much.  
>>> Most Makefiles are currently 1-2 lines of code that simply does nothing 
>>> other than include the common Makefile.
>>> 
>>> On the other hand, CMake defines a lot of constructs designed to support 
>>> portable builds, so actually writing and maintaining that common CMake 
>>> build file would be much easier.  The existing Makefile-based system 
>>> already doesn't require you to understand the specific compiler invocations 
>>> you want.  Here's 3 random Makefiles, which is hopefully representative 
>>> given that I pulled them completely at random.
>>> 
>>> breakpoint-commands/Makefile:
>>> LEVEL = ../../../make
>>> CXX_SOURCES := nested.cpp
>>> include $(LEVEL)/Makefile.rules
>>> 
>>> functionalities/inferior-assert:
>>> LEVEL = ../../make
>>> C_SOURCES := main.c
>>> include $(LEVEL)/Makefile.rules
>>> 
>>> 
>>> types:
>>> LEVEL = ../make
>>> # Example:
>>> #
>>> # CXX_SOURCES := int.cpp
>>> include $(LEVEL)/Makefile.rules
>>> 
>>> None of this is particularly interesting.  There are a very few tests that 
>>> need to do something weird.  I opened 10 other random Makefiles and still 
>>> didn't find any.  I don't believe it would be hard to support those cases.
>>> 
>>> So now instead of "understand Make" it becomes "understand CMake".  Whic is 
>>> already a requirement of building LLVM.
>> 
>> Fair point. I would suggest that I'll try to make LLDB's testsuite build 
>> out-of-tree using the existing Makefile system. That should be a generally 
>> useful first step. After doing this I will hopefully have a much better 
>> understanding of the requirements of the Makefiles and then we can revisit 
>> this idea with me actually knowing what I'm talking about :-)
>> 
>>> If our test suite was lit-based where you actually had to write compiler 
>>> invocations into the test files, I would feel differently, but that isn't 
>>> what we have today.  We have something that is almost a direct mapping to 
>>> using CMake.
>> 
>> Question: how would you feel about converting the Makefiles to LIT-style 
>> .test files with very explicit RUN-lines?
> 
> I'm not sure what you mean by this.

Instead of using a build system at all to build the tests, we would hard-code 
the compiler and linker invocations without encoding any dependencies. Because 
we still need this to be configurable, it would probably look something like 
this:

  RUN: %CXX test.cpp -O0 %CXXFLAGS -o test.exe
  RUN: %test_driver test.exe mytest.py

-- adrian

> 
> Jim
> 
> 
>> 
>> -- adrian

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 2:49 PM, Adrian Prantl  wrote:
> 
> 
> 
>> On Jan 17, 2018, at 2:31 PM, Zachary Turner  wrote:
>> 
>> I don't think new test authors really need to add CMake any more so than 
>> they currently need to understand Make.  Which is to say, not very much.  
>> Most Makefiles are currently 1-2 lines of code that simply does nothing 
>> other than include the common Makefile.
>> 
>> On the other hand, CMake defines a lot of constructs designed to support 
>> portable builds, so actually writing and maintaining that common CMake build 
>> file would be much easier.  The existing Makefile-based system already 
>> doesn't require you to understand the specific compiler invocations you 
>> want.  Here's 3 random Makefiles, which is hopefully representative given 
>> that I pulled them completely at random.
>> 
>> breakpoint-commands/Makefile:
>> LEVEL = ../../../make
>> CXX_SOURCES := nested.cpp
>> include $(LEVEL)/Makefile.rules
>> 
>> functionalities/inferior-assert:
>> LEVEL = ../../make
>> C_SOURCES := main.c
>> include $(LEVEL)/Makefile.rules
>> 
>> 
>> types:
>> LEVEL = ../make
>> # Example:
>> #
>> # CXX_SOURCES := int.cpp
>> include $(LEVEL)/Makefile.rules
>> 
>> None of this is particularly interesting.  There are a very few tests that 
>> need to do something weird.  I opened 10 other random Makefiles and still 
>> didn't find any.  I don't believe it would be hard to support those cases.
>> 
>> So now instead of "understand Make" it becomes "understand CMake".  Whic is 
>> already a requirement of building LLVM.
> 
> Fair point. I would suggest that I'll try to make LLDB's testsuite build 
> out-of-tree using the existing Makefile system. That should be a generally 
> useful first step. After doing this I will hopefully have a much better 
> understanding of the requirements of the Makefiles and then we can revisit 
> this idea with me actually knowing what I'm talking about :-)
> 
>> If our test suite was lit-based where you actually had to write compiler 
>> invocations into the test files, I would feel differently, but that isn't 
>> what we have today.  We have something that is almost a direct mapping to 
>> using CMake.
> 
> Question: how would you feel about converting the Makefiles to LIT-style 
> .test files with very explicit RUN-lines?

I'm not sure what you mean by this.

Jim


> 
> -- adrian

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 2:31 PM, Zachary Turner  wrote:
> 
> I don't think new test authors really need to add CMake any more so than they 
> currently need to understand Make.  Which is to say, not very much.  Most 
> Makefiles are currently 1-2 lines of code that simply does nothing other than 
> include the common Makefile.
> 
> On the other hand, CMake defines a lot of constructs designed to support 
> portable builds, so actually writing and maintaining that common CMake build 
> file would be much easier.  The existing Makefile-based system already 
> doesn't require you to understand the specific compiler invocations you want. 
>  Here's 3 random Makefiles, which is hopefully representative given that I 
> pulled them completely at random.
> 
> breakpoint-commands/Makefile:
> LEVEL = ../../../make
> CXX_SOURCES := nested.cpp
> include $(LEVEL)/Makefile.rules
> 
> functionalities/inferior-assert:
> LEVEL = ../../make
> C_SOURCES := main.c
> include $(LEVEL)/Makefile.rules
> 
> 
> types:
> LEVEL = ../make
> # Example:
> #
> # CXX_SOURCES := int.cpp
> include $(LEVEL)/Makefile.rules
> 
> None of this is particularly interesting.  There are a very few tests that 
> need to do something weird.  I opened 10 other random Makefiles and still 
> didn't find any.  I don't believe it would be hard to support those cases.
> 
> So now instead of "understand Make" it becomes "understand CMake".  Whic is 
> already a requirement of building LLVM.

Fair point. I would suggest that I'll try to make LLDB's testsuite build 
out-of-tree using the existing Makefile system. That should be a generally 
useful first step. After doing this I will hopefully have a much better 
understanding of the requirements of the Makefiles and then we can revisit this 
idea with me actually knowing what I'm talking about :-)

> If our test suite was lit-based where you actually had to write compiler 
> invocations into the test files, I would feel differently, but that isn't 
> what we have today.  We have something that is almost a direct mapping to 
> using CMake.

Question: how would you feel about converting the Makefiles to LIT-style .test 
files with very explicit RUN-lines?

-- adrian

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 1:45 PM, Vedant Kumar  wrote:
> 
> I would prefer having all of our test dependencies tracked by CMake for all 
> the reasons Zach brought up, but I think we should defer that undertaking 
> until after the bots are more stable. We have some immediate problems caused 
> by stale in-tree test artifacts. As a first milestone, it'd be great to not 
> have to run `git clean -fdx` anymore.

I'm pretty sure I do not want to go all the way to CMake right now, but I am 
curious about your motivation: Why do you think that using CMake to build the 
tests in the testsuite is a good idea? In my opinion this adds a layer of 
complexity to the tests that makes it harder to understand what exactly is 
happening and test authors now need to understand CMake *and* the compiler 
invocations they want to execute in their tests. Do you also share Zachary's 
point of view that the tests should be build artifacts that should be kept 
after an incremental build?

-- adrian

> 
> 
>> On Jan 17, 2018, at 1:13 PM, Davide Italiano via lldb-dev 
>>  wrote:
>> 
>> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano  
>> wrote:
>>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>>>  wrote:
 Hi lldb-dev!
 
 I've been investigating some spurious LLDB test suite failures on 
 http://green.lab.llvm.org/green/ that had to do with build artifacts from 
 previous runs lying around in the test directories and this prompted me to 
 ask a couple of general noob questions about the LLDB testsuite.
 
 My understanding is that all execution tests are compiled using using 
 `make` in-tree. I.e.: the test driver (dotest.py) effectively executes 
 something equivalent to `cd $srcdir/packages/.../mytest && make`. And it 
 does this in a serial fashion for all configurations (dwarf, dSYM, dwo, 
 ...) and relies on the `clean` target to be implemented correctly.
 
 I don't understand all the design decisions that went into the LLDB 
 testsuite, but my naive intuition tells me that this is sub-optimal 
 (because of the serialization of the configurations) and dangerous 
 (because it relies on make clean being implemented correctly). It seems to 
 me that a better approach would be to create a separate build directory 
 for each test variant and then invoke something like `cd 
 $builddir/test/mytest.dwarf && make -C $srcdir/packages/.../mytest`. This 
 way all configurations can build in parallel, and we can simply nuke the 
 build directory afterwards and this way get rid of all custom 
 implementations of the `clean` target.
> 
> This sgtm as a starting point.
> 
> vedant
> 
 
 - Is this already possible, and/or am I misunderstanding how it works?
 - Would this be a goal that is worthwhile to pursue?
 - Is there a good reason why we are not already doing it this way?
 
>>> 
>>> As we're discussing lldb test suite changes, another detail that I
>>> find a little weird is that every time you execute the test suite you
>>> get a new build directory named after the time at which you run the
>>> test.
>>> It would be much much better IMHO to just have a `log/` generic
>>> directory where the failures are logged, and those who want to
>>> override this setting can just pass a flag.
>>> 
>> 
>> (The logs should also be moved out of tree, FWIW).
>> 
>> --
>> Davide
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 1:13 PM, Davide Italiano  wrote:
> 
> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano  
> wrote:
>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>>  wrote:
>>> Hi lldb-dev!
>>> 
>>> I've been investigating some spurious LLDB test suite failures on 
>>> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
>>> previous runs lying around in the test directories and this prompted me to 
>>> ask a couple of general noob questions about the LLDB testsuite.
>>> 
>>> My understanding is that all execution tests are compiled using using 
>>> `make` in-tree. I.e.: the test driver (dotest.py) effectively executes 
>>> something equivalent to `cd $srcdir/packages/.../mytest && make`. And it 
>>> does this in a serial fashion for all configurations (dwarf, dSYM, dwo, 
>>> ...) and relies on the `clean` target to be implemented correctly.
>>> 
>>> I don't understand all the design decisions that went into the LLDB 
>>> testsuite, but my naive intuition tells me that this is sub-optimal 
>>> (because of the serialization of the configurations) and dangerous (because 
>>> it relies on make clean being implemented correctly). It seems to me that a 
>>> better approach would be to create a separate build directory for each test 
>>> variant and then invoke something like `cd $builddir/test/mytest.dwarf && 
>>> make -C $srcdir/packages/.../mytest`. This way all configurations can build 
>>> in parallel, and we can simply nuke the build directory afterwards and this 
>>> way get rid of all custom implementations of the `clean` target.
>>> 
>>> - Is this already possible, and/or am I misunderstanding how it works?
>>> - Would this be a goal that is worthwhile to pursue?
>>> - Is there a good reason why we are not already doing it this way?
>>> 
>> 
>> As we're discussing lldb test suite changes, another detail that I
>> find a little weird is that every time you execute the test suite you
>> get a new build directory named after the time at which you run the
>> test.
>> It would be much much better IMHO to just have a `log/` generic
>> directory where the failures are logged, and those who want to
>> override this setting can just pass a flag.
>> 
> 
> (The logs should also be moved out of tree, FWIW).

If I'm going to move the test build artifacts out-of-source-tree the logs would 
naturally end up there, too. Let's discuss whether creating a timestamped log 
directory should be the default or an option in a different thread to keep 
things simple. This is entirely orthogonal.

-- adrian

> 
> --
> Davide

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Vedant Kumar via lldb-dev]

I would prefer having all of our test dependencies tracked by CMake for all the 
reasons Zach brought up, but I think we should defer that undertaking until 
after the bots are more stable. We have some immediate problems caused by stale 
in-tree test artifacts. As a first milestone, it'd be great to not have to run 
`git clean -fdx` anymore.


> On Jan 17, 2018, at 1:13 PM, Davide Italiano via lldb-dev 
>  wrote:
> 
> On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano  
> wrote:
>> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>>  wrote:
>>> Hi lldb-dev!
>>> 
>>> I've been investigating some spurious LLDB test suite failures on 
>>> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
>>> previous runs lying around in the test directories and this prompted me to 
>>> ask a couple of general noob questions about the LLDB testsuite.
>>> 
>>> My understanding is that all execution tests are compiled using using 
>>> `make` in-tree. I.e.: the test driver (dotest.py) effectively executes 
>>> something equivalent to `cd $srcdir/packages/.../mytest && make`. And it 
>>> does this in a serial fashion for all configurations (dwarf, dSYM, dwo, 
>>> ...) and relies on the `clean` target to be implemented correctly.
>>> 
>>> I don't understand all the design decisions that went into the LLDB 
>>> testsuite, but my naive intuition tells me that this is sub-optimal 
>>> (because of the serialization of the configurations) and dangerous (because 
>>> it relies on make clean being implemented correctly). It seems to me that a 
>>> better approach would be to create a separate build directory for each test 
>>> variant and then invoke something like `cd $builddir/test/mytest.dwarf && 
>>> make -C $srcdir/packages/.../mytest`. This way all configurations can build 
>>> in parallel, and we can simply nuke the build directory afterwards and this 
>>> way get rid of all custom implementations of the `clean` target.

This sgtm as a starting point.

vedant

>>> 
>>> - Is this already possible, and/or am I misunderstanding how it works?
>>> - Would this be a goal that is worthwhile to pursue?
>>> - Is there a good reason why we are not already doing it this way?
>>> 
>> 
>> As we're discussing lldb test suite changes, another detail that I
>> find a little weird is that every time you execute the test suite you
>> get a new build directory named after the time at which you run the
>> test.
>> It would be much much better IMHO to just have a `log/` generic
>> directory where the failures are logged, and those who want to
>> override this setting can just pass a flag.
>> 
> 
> (The logs should also be moved out of tree, FWIW).
> 
> --
> Davide
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Adrian Prantl via lldb-dev]

> On Jan 17, 2018, at 12:41 PM, Zachary Turner  wrote:
> 
> If we're going to be making any significant changes to the way inferiors are 
> compiled, why not use cmake?  Make clean is already not implemented correctly 
> in many places, leading to lots of remnants left over in the source tree 
> after test runs.  Furthermore, make is run every single time currently, 
> leading to hundreds (if not thousands) of unnecessary compilations.  Seems to 
> me like all the inferiors should be compiled one time, up front, as part of 
> the configure step, and into the build directory.  This is nice because it 
> already integrates perfectly into the existing LLVM "way" of building things.

To be honest, I have not considered the the tests to be part of the build. 
Doing so is an interesting idea that I haven't thought about so far, here are 
some thoughts about this:
- CMake is more a replacement for autoconf than for make and I'm not sure if we 
need a better tool for the "configuration" part of the testsuite.
- Some of the tests purposefully do weird stuff, such as deleting or damaging 
one .o file to test LLDB's abilities to cope with incomplete debug info. To 
implement this we will need to micro-manage things at the "make" level that 
could be hard to express in CMake.
- The LLVM/CFE way of building *tests* is to hard-code the commands to 
recompile the testcases every time you run the testsuite.
- We probably want to be able to run the LLDB testsuite using many different 
compilers. It could be possible that CMake is helpful for this, but then again 
I doubt that we need much more than setting a custom CC/CXX/CFLAGs to support 
this.

I don't want to immediately shoot this idea down, but I think it is one step 
further than I would like to go at this time. Thanks for pointing it out though!

-- adrian
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Davide Italiano via lldb-dev]

On Wed, Jan 17, 2018 at 1:02 PM, Davide Italiano  wrote:
> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>  wrote:
>> Hi lldb-dev!
>>
>> I've been investigating some spurious LLDB test suite failures on 
>> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
>> previous runs lying around in the test directories and this prompted me to 
>> ask a couple of general noob questions about the LLDB testsuite.
>>
>> My understanding is that all execution tests are compiled using using `make` 
>> in-tree. I.e.: the test driver (dotest.py) effectively executes something 
>> equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in 
>> a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies 
>> on the `clean` target to be implemented correctly.
>>
>> I don't understand all the design decisions that went into the LLDB 
>> testsuite, but my naive intuition tells me that this is sub-optimal (because 
>> of the serialization of the configurations) and dangerous (because it relies 
>> on make clean being implemented correctly). It seems to me that a better 
>> approach would be to create a separate build directory for each test variant 
>> and then invoke something like `cd $builddir/test/mytest.dwarf && make -C 
>> $srcdir/packages/.../mytest`. This way all configurations can build in 
>> parallel, and we can simply nuke the build directory afterwards and this way 
>> get rid of all custom implementations of the `clean` target.
>>
>> - Is this already possible, and/or am I misunderstanding how it works?
>> - Would this be a goal that is worthwhile to pursue?
>> - Is there a good reason why we are not already doing it this way?
>>
>
> As we're discussing lldb test suite changes, another detail that I
> find a little weird is that every time you execute the test suite you
> get a new build directory named after the time at which you run the
> test.
> It would be much much better IMHO to just have a `log/` generic
> directory where the failures are logged, and those who want to
> override this setting can just pass a flag.
>

(The logs should also be moved out of tree, FWIW).

--
Davide
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 1:02 PM, Davide Italiano via lldb-dev 
>  wrote:
> 
> On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
>  wrote:
>> Hi lldb-dev!
>> 
>> I've been investigating some spurious LLDB test suite failures on 
>> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
>> previous runs lying around in the test directories and this prompted me to 
>> ask a couple of general noob questions about the LLDB testsuite.
>> 
>> My understanding is that all execution tests are compiled using using `make` 
>> in-tree. I.e.: the test driver (dotest.py) effectively executes something 
>> equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in 
>> a serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies 
>> on the `clean` target to be implemented correctly.
>> 
>> I don't understand all the design decisions that went into the LLDB 
>> testsuite, but my naive intuition tells me that this is sub-optimal (because 
>> of the serialization of the configurations) and dangerous (because it relies 
>> on make clean being implemented correctly). It seems to me that a better 
>> approach would be to create a separate build directory for each test variant 
>> and then invoke something like `cd $builddir/test/mytest.dwarf && make -C 
>> $srcdir/packages/.../mytest`. This way all configurations can build in 
>> parallel, and we can simply nuke the build directory afterwards and this way 
>> get rid of all custom implementations of the `clean` target.
>> 
>> - Is this already possible, and/or am I misunderstanding how it works?
>> - Would this be a goal that is worthwhile to pursue?
>> - Is there a good reason why we are not already doing it this way?
>> 
> 
> As we're discussing lldb test suite changes, another detail that I
> find a little weird is that every time you execute the test suite you
> get a new build directory named after the time at which you run the
> test.
> It would be much much better IMHO to just have a `log/` generic
> directory where the failures are logged, and those who want to
> override this setting can just pass a flag.

I often use the fact that the past couple run's logs are preserved, and often 
find this valuable after the fact, so I'd like it to continue to be the 
default.  

If you want to change it, you can pass "-s log" or whatever you like to 
dotest.py.

Jim

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Davide Italiano via lldb-dev]

On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
 wrote:
> Hi lldb-dev!
>
> I've been investigating some spurious LLDB test suite failures on 
> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
> previous runs lying around in the test directories and this prompted me to 
> ask a couple of general noob questions about the LLDB testsuite.
>
> My understanding is that all execution tests are compiled using using `make` 
> in-tree. I.e.: the test driver (dotest.py) effectively executes something 
> equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a 
> serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on 
> the `clean` target to be implemented correctly.
>
> I don't understand all the design decisions that went into the LLDB 
> testsuite, but my naive intuition tells me that this is sub-optimal (because 
> of the serialization of the configurations) and dangerous (because it relies 
> on make clean being implemented correctly). It seems to me that a better 
> approach would be to create a separate build directory for each test variant 
> and then invoke something like `cd $builddir/test/mytest.dwarf && make -C 
> $srcdir/packages/.../mytest`. This way all configurations can build in 
> parallel, and we can simply nuke the build directory afterwards and this way 
> get rid of all custom implementations of the `clean` target.
>
> - Is this already possible, and/or am I misunderstanding how it works?
> - Would this be a goal that is worthwhile to pursue?
> - Is there a good reason why we are not already doing it this way?
>

As we're discussing lldb test suite changes, another detail that I
find a little weird is that every time you execute the test suite you
get a new build directory named after the time at which you run the
test.
It would be much much better IMHO to just have a `log/` generic
directory where the failures are logged, and those who want to
override this setting can just pass a flag.

--
Davide
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Jim Ingham via lldb-dev]

> On Jan 17, 2018, at 12:32 PM, Adrian Prantl  wrote:
> 
> Hi lldb-dev!
> 
> I've been investigating some spurious LLDB test suite failures on 
> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
> previous runs lying around in the test directories and this prompted me to 
> ask a couple of general noob questions about the LLDB testsuite.
> 
> My understanding is that all execution tests are compiled using using `make` 
> in-tree. I.e.: the test driver (dotest.py) effectively executes something 
> equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a 
> serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on 
> the `clean` target to be implemented correctly.
> 
> I don't understand all the design decisions that went into the LLDB 
> testsuite, but my naive intuition tells me that this is sub-optimal (because 
> of the serialization of the configurations) and dangerous (because it relies 
> on make clean being implemented correctly). It seems to me that a better 
> approach would be to create a separate build directory for each test variant 
> and then invoke something like `cd $builddir/test/mytest.dwarf && make -C 
> $srcdir/packages/.../mytest`. This way all configurations can build in 
> parallel, and we can simply nuke the build directory afterwards and this way 
> get rid of all custom implementations of the `clean` target.
> 
> - Is this already possible, and/or am I misunderstanding how it works?
> - Would this be a goal that is worthwhile to pursue?
> - Is there a good reason why we are not already doing it this way?

It would be really great to get all the binaries that you need for tests 
building outside of the test directory.  It was done in tree originally for 
expediency - the tests need to know where their binaries are, and that task is 
simple if they are in CWD of the test.  But it is annoying, both because it 
relies on each test to clean up after itself, and because you can't preserve 
one test run's results while making another, or preserve the debug variants.  
Plus, jamming stuff willy-nilly into the source tree is not something you 
should do.

It shouldn't be that hard to make a parallel hierarchy for the tests in the 
build directory, and pass that to the test as the root for products.  That 
would be a valuable project!

Jim

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Davide Italiano via lldb-dev]

On Wed, Jan 17, 2018 at 12:32 PM, Adrian Prantl via lldb-dev
 wrote:
> Hi lldb-dev!
>
> I've been investigating some spurious LLDB test suite failures on 
> http://green.lab.llvm.org/green/ that had to do with build artifacts from 
> previous runs lying around in the test directories and this prompted me to 
> ask a couple of general noob questions about the LLDB testsuite.
>
> My understanding is that all execution tests are compiled using using `make` 
> in-tree. I.e.: the test driver (dotest.py) effectively executes something 
> equivalent to `cd $srcdir/packages/.../mytest && make`. And it does this in a 
> serial fashion for all configurations (dwarf, dSYM, dwo, ...) and relies on 
> the `clean` target to be implemented correctly.
>
> I don't understand all the design decisions that went into the LLDB 
> testsuite, but my naive intuition tells me that this is sub-optimal (because 
> of the serialization of the configurations) and dangerous (because it relies 
> on make clean being implemented correctly). It seems to me that a better 
> approach would be to create a separate build directory for each test variant 
> and then invoke something like `cd $builddir/test/mytest.dwarf && make -C 
> $srcdir/packages/.../mytest`. This way all configurations can build in 
> parallel, and we can simply nuke the build directory afterwards and this way 
> get rid of all custom implementations of the `clean` target.
>

I think this is a much better strategy. FWIW, I wouldn't object if you
want to switch to cmake entirely as LLVM is using it as its only true
build system, but that seems a much larger change.
In any case, whatever gets decided, happy to help you with that.

--
Davide
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Questions about the LLDB testsuite and improving its reliability

[Zachary Turner via lldb-dev]

If we're going to be making any significant changes to the way inferiors
are compiled, why not use cmake?  Make clean is already not implemented
correctly in many places, leading to lots of remnants left over in the
source tree after test runs.  Furthermore, make is run every single time
currently, leading to hundreds (if not thousands) of unnecessary
compilations.  Seems to me like all the inferiors should be compiled one
time, up front, as part of the configure step, and into the build
directory.  This is nice because it already integrates perfectly into the
existing LLVM "way" of building things.

On Wed, Jan 17, 2018 at 12:32 PM Adrian Prantl via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Hi lldb-dev!
>
> I've been investigating some spurious LLDB test suite failures on
> http://green.lab.llvm.org/green/ that had to do with build artifacts from
> previous runs lying around in the test directories and this prompted me to
> ask a couple of general noob questions about the LLDB testsuite.
>
> My understanding is that all execution tests are compiled using using
> `make` in-tree. I.e.: the test driver (dotest.py) effectively executes
> something equivalent to `cd $srcdir/packages/.../mytest && make`. And it
> does this in a serial fashion for all configurations (dwarf, dSYM, dwo,
> ...) and relies on the `clean` target to be implemented correctly.
>
> I don't understand all the design decisions that went into the LLDB
> testsuite, but my naive intuition tells me that this is sub-optimal
> (because of the serialization of the configurations) and dangerous (because
> it relies on make clean being implemented correctly). It seems to me that a
> better approach would be to create a separate build directory for each test
> variant and then invoke something like `cd $builddir/test/mytest.dwarf &&
> make -C $srcdir/packages/.../mytest`. This way all configurations can build
> in parallel, and we can simply nuke the build directory afterwards and this
> way get rid of all custom implementations of the `clean` target.
>
> - Is this already possible, and/or am I misunderstanding how it works?
> - Would this be a goal that is worthwhile to pursue?
> - Is there a good reason why we are not already doing it this way?
>
> thanks,
> adrian
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev