Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I have been thinking about something similar. Ildb-mi specifically I have
serious concerns about because the code itself is a bit of an abomination.
It has its own entire test suite, which also doesn't work very well.
Using a tool like lldb mi is very similar in spirit to how llvm already
uses lit though. There is a collection of very specific tools that exist
just for testing, and the output of those is run through lit)
Lldb-mi was developed independently, had no review, and was essentially a
large code drop. and it also was not developed with testing in mind. I
agree we could probably make it work, but I've been in that code a few
times and I have to say, I don't want to go back.
In any case, i think we agree in principle that from a high level, we could
get a lot of benefits from this style of test.
I'll have more specifics on my thoughts in a few weeks
On Fri, Sep 16, 2016 at 4:07 PM Jason Molenda wrote:
> I thought about this more overnight and I'm more convinced that lit and
> lldb-mi make a great pair. lldb-mi is a programmatic text format that
> isn't subject to the whims of command-line UI design over the years; well
> tests written in terms of MI will be resilient and stable. lldb-mi MUCH
> more closely matches a non-lldb developer's view of how a debugger works.
> The syntax is different from what they use, but it isn't hard to figure
> out. I haven't touched MI in four or five years and it only took me a
> couple minutes to figure out how to do run to a breakpoint and print a
> variable:
>
> -file-exec-and-symbols "/tmp/a.out"
>
> -break-insert a.c:10
>
> =breakpoint-modified,bkpt={number="6",type="breakpoint",disp="keep",enabled="y",addr="0x00010f70",func="main",file="a.c",fullname="/tmp/a.c",line="10",times="0",original-location="a.c:10"}
>
> -exec-run
>
> -thread-info
>
> -exec-next
>
> -var-create var1 * c
>
> ^done,name="var1",numchild="0",value="12",type="int",thread-id="1",has_more="0"
>
>
> (I omitted the responses from all the command except -break-insert and
> -var-create to make it easier to read, but go run lldb-mi and play with it
> yourself or read the gdb MI documentation on the web. It's a really simple
> format to work in.)
>
>
> Several of the proposal have been special commands in the lldb command
> line driver that have non-changing output styles or the like. Why
> re-invent something that already exists? This makes a ton more sense on
> every front. Yes, it means you can't copy and paste your lldb debug
> session into a test case --- but we've all agreed that that's not something
> we want anyone doing anyway.
>
> I'm in favor of lit + lldb-mi and think this would add a lot of value as
> an additional way for people to write test cases.
>
>
> J
>
>
>
> > On Sep 15, 2016, at 7:40 PM, Zachary Turner wrote:
> >
> > One thing I wonder about. It seems like everyone is mostly on the same
> page about command line output .
> >
> > What about input? Would anyone have objections to a test which ran a few
> commands to get the debugger into a particular state before doing something
> to verify the output? Let's assume I'm waving my hands about this last step
> but that it doesn't involve scraping the output of a debugger command
> >
> > Maybe this will never even be an issue, but I just want to make sure
> everyone is on the same page and that the objections are:
> >
> > a) specific to command OUTPUT, not input (i.e. it's ok to have a test
> run "break set -n main")
> > b) specific to *non trivial* output (checking that "p 5" displays 5 is
> ok)
> > c) specific to the the output of the *user* command line interface, so
> that some hypothetical other command line interface (which again I'm being
> hand wavy about) would not be subject to the same objections.
> >
> >
> > On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda
> wrote:
> >
> > > On Sep 15, 2016, at 8:02 AM, Zachary Turner
> wrote:
> > >
> > >
> > > It sounds like your goal is also "tests have to use the SB API and no
> other API", which if so I think that's counterproductive. More
> productive, IMO, would be being open to any alternative that addresses the
> concerns you have with command-line tests. There are more than 2 ways to
> skin a cat, so to speak.
> >
> > Thinking about this a bit, another approach would be to do lit tests on
> top of lldb-mi. MI is a structured format for the debugger and a UI to
> communicate back and forth with a simple text markup language (it would be
> JSON if it were being done today, but it was added to gdb eighteen years
> ago, so it's not). The commands correspond to the commands a debugger user
> would think to use -- no need to understand the structure of how lldb is
> implemented, like with the SB API. The format is a little unusual for a
> human to type, but back when we supported gdb at Apple we worked in MI all
> the time (it was used to communicate between Xcode, our IDE, and gdb) by
> hand when testing and it was fine. "-exec-run" instead of run, that kin
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I thought about this more overnight and I'm more convinced that lit and lldb-mi
make a great pair. lldb-mi is a programmatic text format that isn't subject to
the whims of command-line UI design over the years; well tests written in terms
of MI will be resilient and stable. lldb-mi MUCH more closely matches a
non-lldb developer's view of how a debugger works. The syntax is different
from what they use, but it isn't hard to figure out. I haven't touched MI in
four or five years and it only took me a couple minutes to figure out how to do
run to a breakpoint and print a variable:
-file-exec-and-symbols "/tmp/a.out"
-break-insert a.c:10
=breakpoint-modified,bkpt={number="6",type="breakpoint",disp="keep",enabled="y",addr="0x00010f70",func="main",file="a.c",fullname="/tmp/a.c",line="10",times="0",original-location="a.c:10"}
-exec-run
-thread-info
-exec-next
-var-create var1 * c
^done,name="var1",numchild="0",value="12",type="int",thread-id="1",has_more="0"
(I omitted the responses from all the command except -break-insert and
-var-create to make it easier to read, but go run lldb-mi and play with it
yourself or read the gdb MI documentation on the web. It's a really simple
format to work in.)
Several of the proposal have been special commands in the lldb command line
driver that have non-changing output styles or the like. Why re-invent
something that already exists? This makes a ton more sense on every front.
Yes, it means you can't copy and paste your lldb debug session into a test case
--- but we've all agreed that that's not something we want anyone doing anyway.
I'm in favor of lit + lldb-mi and think this would add a lot of value as an
additional way for people to write test cases.
J
> On Sep 15, 2016, at 7:40 PM, Zachary Turner wrote:
>
> One thing I wonder about. It seems like everyone is mostly on the same page
> about command line output .
>
> What about input? Would anyone have objections to a test which ran a few
> commands to get the debugger into a particular state before doing something
> to verify the output? Let's assume I'm waving my hands about this last step
> but that it doesn't involve scraping the output of a debugger command
>
> Maybe this will never even be an issue, but I just want to make sure everyone
> is on the same page and that the objections are:
>
> a) specific to command OUTPUT, not input (i.e. it's ok to have a test run
> "break set -n main")
> b) specific to *non trivial* output (checking that "p 5" displays 5 is ok)
> c) specific to the the output of the *user* command line interface, so that
> some hypothetical other command line interface (which again I'm being hand
> wavy about) would not be subject to the same objections.
>
>
> On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda wrote:
>
> > On Sep 15, 2016, at 8:02 AM, Zachary Turner wrote:
> >
> >
> > It sounds like your goal is also "tests have to use the SB API and no other
> > API", which if so I think that's counterproductive. More productive, IMO,
> > would be being open to any alternative that addresses the concerns you have
> > with command-line tests. There are more than 2 ways to skin a cat, so to
> > speak.
>
> Thinking about this a bit, another approach would be to do lit tests on top
> of lldb-mi. MI is a structured format for the debugger and a UI to
> communicate back and forth with a simple text markup language (it would be
> JSON if it were being done today, but it was added to gdb eighteen years ago,
> so it's not). The commands correspond to the commands a debugger user would
> think to use -- no need to understand the structure of how lldb is
> implemented, like with the SB API. The format is a little unusual for a
> human to type, but back when we supported gdb at Apple we worked in MI all
> the time (it was used to communicate between Xcode, our IDE, and gdb) by hand
> when testing and it was fine. "-exec-run" instead of run, that kind of thing.
> I think there are four dozens different commands.
>
> lldb-mi itself uses SB API. And the concerns about hardcoding command line
> UI don't apply, it's a key-value format intended for computers, no one is
> going to add an extra space character to anything -- the most it changes is
> that new key-value pairs are added to responses.
>
>
> I agree there are many acceptable ways to write lit tests that don't involve
> lldb command line scraping, and I'm all in favor of using lit with tests that
> don't do that. Of course the patch we're discussing has lit test examples
> that contradict our own policy on writing tests. Once lit is supported in
> lldb, are we going to reject any testsuite additions that depend on the text
> output from the command line lldb? If we're all on the same page here, then
> I have no reservations.
>
> Just to say out loud the future I can easily see: We add lit, then we have
> people helpfully write a few dozen tests in lit that dep
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
One thing I wonder about. It seems like everyone is mostly on the same page about command line output . What about input? Would anyone have objections to a test which ran a few commands to get the debugger into a particular state before doing something to verify the output? Let's assume I'm waving my hands about this last step but that it doesn't involve scraping the output of a debugger command Maybe this will never even be an issue, but I just want to make sure everyone is on the same page and that the objections are: a) specific to command OUTPUT, not input (i.e. it's ok to have a test run "break set -n main") b) specific to *non trivial* output (checking that "p 5" displays 5 is ok) c) specific to the the output of the *user* command line interface, so that some hypothetical other command line interface (which again I'm being hand wavy about) would not be subject to the same objections. On Thu, Sep 15, 2016 at 7:28 PM Jason Molenda wrote: > > > On Sep 15, 2016, at 8:02 AM, Zachary Turner wrote: > > > > > > It sounds like your goal is also "tests have to use the SB API and no > other API", which if so I think that's counterproductive. More > productive, IMO, would be being open to any alternative that addresses the > concerns you have with command-line tests. There are more than 2 ways to > skin a cat, so to speak. > > Thinking about this a bit, another approach would be to do lit tests on > top of lldb-mi. MI is a structured format for the debugger and a UI to > communicate back and forth with a simple text markup language (it would be > JSON if it were being done today, but it was added to gdb eighteen years > ago, so it's not). The commands correspond to the commands a debugger user > would think to use -- no need to understand the structure of how lldb is > implemented, like with the SB API. The format is a little unusual for a > human to type, but back when we supported gdb at Apple we worked in MI all > the time (it was used to communicate between Xcode, our IDE, and gdb) by > hand when testing and it was fine. "-exec-run" instead of run, that kind of > thing. I think there are four dozens different commands. > > lldb-mi itself uses SB API. And the concerns about hardcoding command > line UI don't apply, it's a key-value format intended for computers, no one > is going to add an extra space character to anything -- the most it changes > is that new key-value pairs are added to responses. > > > I agree there are many acceptable ways to write lit tests that don't > involve lldb command line scraping, and I'm all in favor of using lit with > tests that don't do that. Of course the patch we're discussing has lit > test examples that contradict our own policy on writing tests. Once lit is > supported in lldb, are we going to reject any testsuite additions that > depend on the text output from the command line lldb? If we're all on the > same page here, then I have no reservations. > > Just to say out loud the future I can easily see: We add lit, then we > have people helpfully write a few dozen tests in lit that depend on the > command line debugger output. Those patches have to be rejected. > > J ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
> On Sep 15, 2016, at 8:02 AM, Zachary Turner wrote: > > > It sounds like your goal is also "tests have to use the SB API and no other > API", which if so I think that's counterproductive. More productive, IMO, > would be being open to any alternative that addresses the concerns you have > with command-line tests. There are more than 2 ways to skin a cat, so to > speak. Thinking about this a bit, another approach would be to do lit tests on top of lldb-mi. MI is a structured format for the debugger and a UI to communicate back and forth with a simple text markup language (it would be JSON if it were being done today, but it was added to gdb eighteen years ago, so it's not). The commands correspond to the commands a debugger user would think to use -- no need to understand the structure of how lldb is implemented, like with the SB API. The format is a little unusual for a human to type, but back when we supported gdb at Apple we worked in MI all the time (it was used to communicate between Xcode, our IDE, and gdb) by hand when testing and it was fine. "-exec-run" instead of run, that kind of thing. I think there are four dozens different commands. lldb-mi itself uses SB API. And the concerns about hardcoding command line UI don't apply, it's a key-value format intended for computers, no one is going to add an extra space character to anything -- the most it changes is that new key-value pairs are added to responses. I agree there are many acceptable ways to write lit tests that don't involve lldb command line scraping, and I'm all in favor of using lit with tests that don't do that. Of course the patch we're discussing has lit test examples that contradict our own policy on writing tests. Once lit is supported in lldb, are we going to reject any testsuite additions that depend on the text output from the command line lldb? If we're all on the same page here, then I have no reservations. Just to say out loud the future I can easily see: We add lit, then we have people helpfully write a few dozen tests in lit that depend on the command line debugger output. Those patches have to be rejected. J ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
This revision was automatically updated to reflect the committed changes.
Closed by commit rL281651: [LIT] First pass of LLDB LIT support (authored by
cbieneman).
Changed prior to commit:
https://reviews.llvm.org/D24591?vs=71537&id=71546#toc
Repository:
rL LLVM
https://reviews.llvm.org/D24591
Files:
lldb/trunk/lit/CMakeLists.txt
lldb/trunk/lit/Expr/Inputs/anonymous-struct.cpp
lldb/trunk/lit/Expr/Inputs/call-function.cpp
lldb/trunk/lit/Expr/TestCallStdStringFunction.test
lldb/trunk/lit/Expr/TestCallStopAndContinue.test
lldb/trunk/lit/Expr/TestCallUserAnonTypedef.test
lldb/trunk/lit/Expr/TestCallUserDefinedFunction.test
lldb/trunk/lit/Expr/lit.local.cfg
lldb/trunk/lit/Unit/lit.cfg
lldb/trunk/lit/Unit/lit.site.cfg.in
lldb/trunk/lit/lit.cfg
lldb/trunk/lit/lit.site.cfg.in
Index: lldb/trunk/lit/lit.cfg
===
--- lldb/trunk/lit/lit.cfg
+++ lldb/trunk/lit/lit.cfg
@@ -112,17 +112,80 @@
lit_config.load_config(config, site_cfg)
raise SystemExit
+# Register substitutions
+config.substitutions.append(('%python', config.python_executable))
+
+debugserver = lit.util.which('debugserver', llvm_tools_dir)
+lldb = lit.util.which('lldb', llvm_tools_dir)
+
+if not os.path.exists(config.cc):
+config.cc = lit.util.which(config.cc, llvm_tools_dir)
+
+if not os.path.exists(config.cxx):
+config.cxx = lit.util.which(config.cxx, llvm_tools_dir)
+
+if platform.system() in ['Darwin']:
+try:
+out = lit.util.capture(['xcrun', '--show-sdk-path']).strip()
+res = 0
+except OSError:
+res = -1
+if res == 0 and out:
+sdk_path = out
+lit_config.note('using SDKROOT: %r' % sdk_path)
+config.cc += " -isysroot %s" % sdk_path
+config.cxx += " -isysroot %s" % sdk_path
+
+config.substitutions.append(('%cc', config.cc))
+config.substitutions.append(('%cxx', config.cxx))
+
+config.substitutions.append(('%lldb', lldb))
+config.substitutions.append(('%debugserver', debugserver))
+
+for pattern in [r"\bFileCheck\b",
+r"\| \bnot\b"]:
+tool_match = re.match(r"^(\\)?((\| )?)\W+b([0-9A-Za-z-_]+)\\b\W*$",
+ pattern)
+tool_pipe = tool_match.group(2)
+tool_name = tool_match.group(4)
+tool_path = lit.util.which(tool_name, config.llvm_tools_dir)
+if not tool_path:
+# Warn, but still provide a substitution.
+lit_config.note(
+'Did not find ' + tool_name + ' in ' + config.llvm_tools_dir)
+config.substitutions.append((pattern, tool_pipe + tool_path))
+
# Shell execution
if platform.system() not in ['Windows'] or lit_config.getBashPath() != '':
config.available_features.add('shell')
# Running on Darwin OS
if platform.system() in ['Darwin']:
+config.available_features.add('darwin')
config.available_features.add('system-linker-mach-o')
# Running on ELF based *nix
if platform.system() in ['FreeBSD', 'Linux']:
config.available_features.add('system-linker-elf')
+if platform.system() in ['FreeBSD']:
+config.available_features.add('freebsd')
+else:
+config.available_features.add('linux')
+
+if platform.system() in ['Windows']:
+config.available_features.add('windows')
+
+if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
+config.available_features.add("armhf-linux")
+
+if re.match(r'icc', config.cc):
+config.available_features.add("compiler-icc")
+elif re.match(r'clang', config.cc):
+config.available_features.add("compiler-clang")
+elif re.match(r'gcc', config.cc):
+config.available_features.add("compiler-gcc")
+elif re.match(r'cl', config.cc):
+config.available_features.add("compiler-msvc")
# llvm-config knows whether it is compiled with asserts (and)
# whether we are operating in release/debug mode.
Index: lldb/trunk/lit/Unit/lit.site.cfg.in
===
--- lldb/trunk/lit/Unit/lit.site.cfg.in
+++ lldb/trunk/lit/Unit/lit.site.cfg.in
@@ -1,5 +1,6 @@
@LIT_SITE_CFG_IN_HEADER@
+config.test_exec_root = "@LLVM_BINARY_DIR@"
config.llvm_src_root = "@LLVM_SOURCE_DIR@"
config.llvm_obj_root = "@LLVM_BINARY_DIR@"
config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
Index: lldb/trunk/lit/Unit/lit.cfg
===
--- lldb/trunk/lit/Unit/lit.cfg
+++ lldb/trunk/lit/Unit/lit.cfg
@@ -6,6 +6,19 @@
import lit.formats
+# Check that the object root is known.
+if config.test_exec_root is None:
+# Otherwise, we haven't loaded the site specific configuration (the user is
+# probably trying to run on a test file directly, and either the site
+# configuration hasn't been created by the build system, or we are in an
+# out-of-tree build situation).
+
+# Check for 'llvm_unit_site_config' user parameter, and use that if available.
+site_cfg = lit_config.params.get('lldb_unit_site_config', None
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz updated this revision to Diff 71537.
beanz added a comment.
- Removed the XFAIL for icc
- Added support for LLDB_TEST__COMPILER options
Assuming there are no objections, since the patch has been approved by two
reviewers I'll commit it this afternoon.
Thanks,
-Chris
https://reviews.llvm.org/D24591
Files:
lit/CMakeLists.txt
lit/Expr/Inputs/anonymous-struct.cpp
lit/Expr/Inputs/call-function.cpp
lit/Expr/TestCallStdStringFunction.test
lit/Expr/TestCallStopAndContinue.test
lit/Expr/TestCallUserAnonTypedef.test
lit/Expr/TestCallUserDefinedFunction.test
lit/Expr/lit.local.cfg
lit/Unit/lit.cfg
lit/Unit/lit.site.cfg.in
lit/lit.cfg
lit/lit.site.cfg.in
Index: lit/lit.site.cfg.in
===
--- lit/lit.site.cfg.in
+++ lit/lit.site.cfg.in
@@ -8,6 +8,22 @@
config.lldb_obj_root = "@LLDB_BINARY_DIR@"
config.target_triple = "@TARGET_TRIPLE@"
config.python_executable = "@PYTHON_EXECUTABLE@"
+config.cc = "@CMAKE_C_COMPILER@"
+config.cxx = "@CMAKE_CXX_COMPILER@"
+
+test_c_compiler = "@LLDB_TEST_C_COMPILER@"
+test_cxx_compiler = "@LLDB_TEST_CXX_COMPILER@"
+test_clang = "@LLDB_TEST_CLANG@".lower()
+test_clang = test_clang == "on" or test_clang == "true" or test_clang == "1"
+
+if len(test_c_compiler) > 0:
+ config.cc = test_c_compiler
+if len(test_c_compiler) > 0:
+ config.cxx = test_cxx_compiler
+
+if test_clang:
+ config.cc = 'clang'
+ config.cxx = 'clang++'
# Support substitution of the tools and libs dirs with user parameters. This is
# used when we can't determine the tool dir at configuration time.
Index: lit/lit.cfg
===
--- lit/lit.cfg
+++ lit/lit.cfg
@@ -112,17 +112,80 @@
lit_config.load_config(config, site_cfg)
raise SystemExit
+# Register substitutions
+config.substitutions.append(('%python', config.python_executable))
+
+debugserver = lit.util.which('debugserver', llvm_tools_dir)
+lldb = lit.util.which('lldb', llvm_tools_dir)
+
+if not os.path.exists(config.cc):
+config.cc = lit.util.which(config.cc, llvm_tools_dir)
+
+if not os.path.exists(config.cxx):
+config.cxx = lit.util.which(config.cxx, llvm_tools_dir)
+
+if platform.system() in ['Darwin']:
+try:
+out = lit.util.capture(['xcrun', '--show-sdk-path']).strip()
+res = 0
+except OSError:
+res = -1
+if res == 0 and out:
+sdk_path = out
+lit_config.note('using SDKROOT: %r' % sdk_path)
+config.cc += " -isysroot %s" % sdk_path
+config.cxx += " -isysroot %s" % sdk_path
+
+config.substitutions.append(('%cc', config.cc))
+config.substitutions.append(('%cxx', config.cxx))
+
+config.substitutions.append(('%lldb', lldb))
+config.substitutions.append(('%debugserver', debugserver))
+
+for pattern in [r"\bFileCheck\b",
+r"\| \bnot\b"]:
+tool_match = re.match(r"^(\\)?((\| )?)\W+b([0-9A-Za-z-_]+)\\b\W*$",
+ pattern)
+tool_pipe = tool_match.group(2)
+tool_name = tool_match.group(4)
+tool_path = lit.util.which(tool_name, config.llvm_tools_dir)
+if not tool_path:
+# Warn, but still provide a substitution.
+lit_config.note(
+'Did not find ' + tool_name + ' in ' + config.llvm_tools_dir)
+config.substitutions.append((pattern, tool_pipe + tool_path))
+
# Shell execution
if platform.system() not in ['Windows'] or lit_config.getBashPath() != '':
config.available_features.add('shell')
# Running on Darwin OS
if platform.system() in ['Darwin']:
+config.available_features.add('darwin')
config.available_features.add('system-linker-mach-o')
# Running on ELF based *nix
if platform.system() in ['FreeBSD', 'Linux']:
config.available_features.add('system-linker-elf')
+if platform.system() in ['FreeBSD']:
+config.available_features.add('freebsd')
+else:
+config.available_features.add('linux')
+
+if platform.system() in ['Windows']:
+config.available_features.add('windows')
+
+if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
+config.available_features.add("armhf-linux")
+
+if re.match(r'icc', config.cc):
+config.available_features.add("compiler-icc")
+elif re.match(r'clang', config.cc):
+config.available_features.add("compiler-clang")
+elif re.match(r'gcc', config.cc):
+config.available_features.add("compiler-gcc")
+elif re.match(r'cl', config.cc):
+config.available_features.add("compiler-msvc")
# llvm-config knows whether it is compiled with asserts (and)
# whether we are operating in release/debug mode.
Index: lit/Unit/lit.site.cfg.in
===
--- lit/Unit/lit.site.cfg.in
+++ lit/Unit/lit.site.cfg.in
@@ -1,5 +1,6 @@
@LIT_SITE_CFG_IN_HEADER@
+config.test_exec_root = "@LLVM_BINARY_DIR@"
config.llvm_src_root = "@LLVM_SOURCE_DIR@"
config.llvm_obj_root = "@LLVM_BINARY_DIR@"
config.llvm_t
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
lldb-commits to bcc, lldb-dev to cc. > The biggest feature I see missing here is the ability to run tests remotely. Remote debugging is the most important use case for our team, and now we have a number of (experimental) bots running the remote test suite. We want to make sure we can debug correctly when the host and target have different architectures That is a good point. I thought about this a little bit and I believe I have a solution to most of the issues you would face when running a remote test suite, but perhaps it would be better to take that to a different thread on lldb-dev rather than on this commit. So I'm taking lldb-commits off of this thread and adding lldb-dev. Responses inline. On Thu, Sep 15, 2016 at 4:02 AM Pavel Labath wrote: > - the test suite needs to know how to connect to the remote target > (mostly it's just executing three commands: `platform select XXX`, > `platform connect` and `platform settings --working-dir YYY`), but it can > get a bit more complicated in some cases (if we need to install shared > libraries along with the main executable, if it involves the test > executable writing something to a file, etc.) This seems like the easiest of the problems to solve, so I'm mentioning it first. I can imagine a couple of command line options to the lldb executable itself that allow one to configure remote debugging from the command line. Then, we could provide one CMake target for each remote platform. check-lldb-remote-android, check-lldb-remote-fizzbuzz, whatever. And you could configure all of these at CMake time with one command. Like cmake -DLLDB_TEST_REMOTE_TARGETS=android,fizzbuzz,etc. Internally, each target's CMake logic could configure some appropriate variables that would be picked up by lit so that the lit substitution for running lldb would supply the appropriate command line to configure the remote target. - this includes the need to pass funny compiler switches based on the target (some android targets can only run `-pie` executables, some cannot run `-pie` executables at all). This should be handled the same way as above. Configuring the lit substitutions for the compiler invocation can depend on which cmake target you're running lit through. ninja check-lldb-remote-android could define one set of lit substitutions, and another target could define another. Is this sufficient? Is it easy to define a mapping between the set of all remote targets and the set of all command line flag combinations? - we need to be able to fully specify the toolchain used to compile the > target executables (specifically, we should not assume the host compiler) > Will you want a single run of the test suite to use multiple compilers, or can we assume there is only 1 compiler for a given cmake output directory? I think we can assume the latter since even now we only have 1 cmake variable `LLDB_TEST_COMPILER`, but let me know if this is a wrong assumption > > - we need to be careful about strong cmake integration. The same lldb > binary is capable of debugging all kinds of targets. We should not require > a fresh cmake build to run the test suite against multiple targets. If we > need to switch cmake options (`LLDB_TEST_COMPILER`, `LLDB_TEST_ARCH`, > `LLDB_TEST_URL` or similar) and it does not trigger a rebuild then it's > usable, but slightly inpractical. For us, it would be best to be able a > fire off a remote test with a single command (like we can do now). > See first response. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
On Wed, Sep 14, 2016 at 11:43 PM Jason Molenda wrote:
> If we want to add a testsuite runner which takes a source file, a place to
> put a breakpoint, the name of a variable to examine, and it runs through
> those in SB API, I'm all in favor. I don't know if that's going to add a
> lot of test coverage to lldb, but I have no problem with such a thing.
>
> My goal isn't to make test case writing hard. My goal is to make the
> testsuite a benefit to the project, instead of a boat anchor.
>
> My non-goal is what testsuite harness is used to run the SB API tests -
> whether it's lit or whatever, I don't care.
>
It sounds like your goal is also "tests have to use the SB API and no other
API", which if so I think that's counterproductive. More productive, IMO,
would be being open to any alternative that addresses the concerns you have
with command-line tests. There are more than 2 ways to skin a cat, so to
speak.
Aside from that, it sounds like we have the same goal -- make the test
suite a benefit to the project rather than a boat anchor. I guess we
differ on what constitutes a benefit and a boat anchor. TBH I think the
current test suite is a boat anchor due to its numerous stability issues
and the barrier to entry for writing new tests. lit wins by leaps and
bounds in both of these areas. If there were something else that were even
better, then I'd be all for using it (although then we'd have to question
why LLVM doesn't also use it). But the fact is -- as a test
infrastructure, lit is better.
On Wed, Sep 14, 2016 at 11:19 PM Jason Molenda wrote:
> It's great to make writing tests easier. We'd all love to have more tests.
>
> If "writing tests easier" is "command line output scraping", that's only
> hurting the project in the long term. I'm telling you this from years of
> experience on a project where we did exactly this. It's great to have
> examples where we're testing that "p 5" prints 5. That's awesome and I'm
> sure it won't break. But that's not more than a tiny fraction of what you
> need to test in a debugger.
At least 2 different brainstorm ideas have been proposed that would make
lit tests not rely on "command line output scraping". A command line is
just a particular form of an API. The SB API is another form of an API.
There are infinite numbers of ways to design an API, these aren't the only
2 APIs in the world. One could easily imagine an "API" that was itself a
single LLDB command and which you could pass some kind of argument to it
that resolved to an SB API call. Sean gave an example earlier.
(lldb) script lldb.frame.FindVariable("argc").GetValue()
'1'
(lldb) script lldb.process.GetNumThreads()
1
(lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
3514809
Assuming your test ran against this output, is this a "command line
scraping" test or an "SB API" test? Seems to be both, which is why I keep
saying that it's unproductive to talk about SB API tests and command line
scraping tests as the entire universe of options.
But even the above is not the only possibility. You could directly map
every single SB API call to an LLDB command-line independent of the normal
command line UI. One-to-one mapping
(lldb) object lldb.frame.variables["argc"].value
'1'
(lldb) object lldb.process.threads.count
1
(lldb) object lldb.process.threads[0].id
3514809
Is this really any different than the script example above?
So I would request that we stop talking about "command line scraping
tests". Let's focus on the actual issue, which is "tests that are hard to
maintain: bad. Tests that are easy to maintain: good"
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
labath added a comment. Btw, I will be in the bay area from 3rd to 7th of October. Maybe we could sit down and talk the design and requirements through in person? I've been hoping to speak to some of you in person anyway, and this could be a good first topic on the agenda... https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
labath added a comment. The biggest feature I see missing here is the ability to run tests remotely. Remote debugging is the most important use case for our team, and now we have a number of (experimental) bots running the remote test suite. We want to make sure we can debug correctly when the host and target have different architectures, operating systems, pointer sizes, etc. Some of the implications of this are: - we need to be able to fully specify the toolchain used to compile the target executables (specifically, we should not assume the host compiler) - this includes the need to pass funny compiler switches based on the target (some android targets can only run `-pie` executables, some cannot run `-pie` executables at all). - the test suite needs to know how to connect to the remote target (mostly it's just executing three commands: `platform select XXX`, `platform connect` and `platform settings --working-dir YYY`), but it can get a bit more complicated in some cases (if we need to install shared libraries along with the main executable, if it involves the test executable writing something to a file, etc.) - we need to be careful about strong cmake integration. The same lldb binary is capable of debugging all kinds of targets. We should not require a fresh cmake build to run the test suite against multiple targets. If we need to switch cmake options (`LLDB_TEST_COMPILER`, `LLDB_TEST_ARCH`, `LLDB_TEST_URL` or similar) and it does not trigger a rebuild then it's usable, but slightly inpractical. For us, it would be best to be able a fire off a remote test with a single command (like we can do now). I am not against this going in without the remote feature in the first version, but I think we should think hardly about it, as I think it could impact a number of design decisions. Comment at: lit/Expr/TestCallStdStringFunction.test:4 @@ +3,3 @@ + +# XFAIL: compiler-icc +# -> llvm.org/pr14437 I don't think anyone has tested lldb with icc in the past few years. We can avoid adding those xfails to the new tests. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
If we want to add a testsuite runner which takes a source file, a place to put a breakpoint, the name of a variable to examine, and it runs through those in SB API, I'm all in favor. I don't know if that's going to add a lot of test coverage to lldb, but I have no problem with such a thing. My goal isn't to make test case writing hard. My goal is to make the testsuite a benefit to the project, instead of a boat anchor. My non-goal is what testsuite harness is used to run the SB API tests - whether it's lit or whatever, I don't care. > On Sep 14, 2016, at 11:19 PM, Jason Molenda wrote: > > It's great to make writing tests easier. We'd all love to have more tests. > > If "writing tests easier" is "command line output scraping", that's only > hurting the project in the long term. I'm telling you this from years of > experience on a project where we did exactly this. It's great to have > examples where we're testing that "p 5" prints 5. That's awesome and I'm > sure it won't break. But that's not more than a tiny fraction of what you > need to test in a debugger. > > It's important to separate "let's use lit" and "let's write command line > scraping tests", unless lit means "command line scraping tests", in which > case it's fine to conflate the two. No one is going to be up in arms about > lit as a test harness -- but using that as a way to get lots of command line > scraping tests added to lldb is disingenuous. We've had this discussion many > times on the mailing lists, and all of us who had to work on a scraping > testsuite have pushed back hard against the siren song of these tests because > we've learned what a big mistake it is. > > Talking about python 2 v python 3 compatibility or speed of the testsuite -- > these are implementation side issues that I don't have an opinion on. If we > want to write our SB API in a different way to solve those issues -- want to > write the tests in C++? Fine, whatever. I think that would just make it > slower to write new tests. Can tests in lit use the SB API exclusively? > Then I don't care about adding lit. > > I remain firmly against command line scraping tests, it only adds debt to the > project long term. It's a lesson learned from gdb that bears not repeating. > > J > > >> On Sep 14, 2016, at 9:36 PM, Zachary Turner wrote: >> >> There's also a cost to *not* writing test cases, which is paid >> proportionally to how difficult it is to write test cases. >> >> That said, aren't the existing lldbinline tests an example of the behavior >> you're objecting to? And that is a relatively recent addition which I have >> never seen or heard of any objections to. >> >> On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda wrote: >> >> On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: >> >>> I don't blame you for being scared of command tests. I don't support their >>> use in the current LLDB test suite either, for exactly the same reasons you >>> and Jason have expressed. But I do think it's possible to come up with >>> something that a) doesn't suffer from the same problems, b) allows testing >>> a ton of extra functionality that is not currently testable through the >>> api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow :) >> >> >> I think your examples are reductive to "printing values is easy to keep >> consistent" -- but that's only a small part of what a debugger testsuite >> needs to do. >> >> If we want to use lit to drive tests written in terms of SB API, then I'm >> open to seeing how this tool can be used in lldb. >> >> If lit can only be used to write command output tests, then I believe, based >> on years and years of experience with exactly that kind of test suite in >> with gdb, that this is a very poor addition to lldb. It will only hamper >> future lldb development as we want to improve the lldb command line UI - as >> it did with gdb. Every change to the command line UI breaks hundreds of >> "easy to write" tests. And who is responsible for cleaning all those up? >> The people who wrote these easy-to-write command line output matching tests? >> No, it'll be the person trying to improve the debugger, which is a big >> disincentive to changing anything else you'll need to wade through the swamp >> of thousands of accumulated test cases. >> >> There's a cost to writing a test case. Either it is paid up front when you >> write the test in terms of SB API, or it is paid repeatedly over time as >> people change the UI to the command line tool. >> >> I have nothing against lit per se, I'm open to seeing what an SB API based >> test case in that harness looks like. I have nothing but objections to >> adding significant new additions to the testsuite that are locked to the >> command line UI behavior of lldb today. >> >> >> J > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.or
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
It's great to make writing tests easier. We'd all love to have more tests. If "writing tests easier" is "command line output scraping", that's only hurting the project in the long term. I'm telling you this from years of experience on a project where we did exactly this. It's great to have examples where we're testing that "p 5" prints 5. That's awesome and I'm sure it won't break. But that's not more than a tiny fraction of what you need to test in a debugger. It's important to separate "let's use lit" and "let's write command line scraping tests", unless lit means "command line scraping tests", in which case it's fine to conflate the two. No one is going to be up in arms about lit as a test harness -- but using that as a way to get lots of command line scraping tests added to lldb is disingenuous. We've had this discussion many times on the mailing lists, and all of us who had to work on a scraping testsuite have pushed back hard against the siren song of these tests because we've learned what a big mistake it is. Talking about python 2 v python 3 compatibility or speed of the testsuite -- these are implementation side issues that I don't have an opinion on. If we want to write our SB API in a different way to solve those issues -- want to write the tests in C++? Fine, whatever. I think that would just make it slower to write new tests. Can tests in lit use the SB API exclusively? Then I don't care about adding lit. I remain firmly against command line scraping tests, it only adds debt to the project long term. It's a lesson learned from gdb that bears not repeating. J > On Sep 14, 2016, at 9:36 PM, Zachary Turner wrote: > > There's also a cost to *not* writing test cases, which is paid proportionally > to how difficult it is to write test cases. > > That said, aren't the existing lldbinline tests an example of the behavior > you're objecting to? And that is a relatively recent addition which I have > never seen or heard of any objections to. > > On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda wrote: > > On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: > > > I don't blame you for being scared of command tests. I don't support their > > use in the current LLDB test suite either, for exactly the same reasons you > > and Jason have expressed. But I do think it's possible to come up with > > something that a) doesn't suffer from the same problems, b) allows testing > > a ton of extra functionality that is not currently testable through the > > api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow :) > > > I think your examples are reductive to "printing values is easy to keep > consistent" -- but that's only a small part of what a debugger testsuite > needs to do. > > If we want to use lit to drive tests written in terms of SB API, then I'm > open to seeing how this tool can be used in lldb. > > If lit can only be used to write command output tests, then I believe, based > on years and years of experience with exactly that kind of test suite in with > gdb, that this is a very poor addition to lldb. It will only hamper future > lldb development as we want to improve the lldb command line UI - as it did > with gdb. Every change to the command line UI breaks hundreds of "easy to > write" tests. And who is responsible for cleaning all those up? The people > who wrote these easy-to-write command line output matching tests? No, it'll > be the person trying to improve the debugger, which is a big disincentive to > changing anything else you'll need to wade through the swamp of thousands of > accumulated test cases. > > There's a cost to writing a test case. Either it is paid up front when you > write the test in terms of SB API, or it is paid repeatedly over time as > people change the UI to the command line tool. > > I have nothing against lit per se, I'm open to seeing what an SB API based > test case in that harness looks like. I have nothing but objections to > adding significant new additions to the testsuite that are locked to the > command line UI behavior of lldb today. > > > J ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
Also, sb api and command line test are neither mutually exclusive nor the only 2 options. Sean mentioned the idea of a dsl for example. I mentioned the idea of a command that is independent of the normal ui command line. There are lots of possibilities if you look. Also our current situation is far from perfect. There is flakiness on every platform. Python is slow. There's complexity in supporting both Python 2 and 3. The test suite is about 80% smaller than it should be since so much goes into writing one. I don't know if there's any buildbots at all running tests due to how fragile everything is. Using lit solves many of those problems immediately, and makes it possible to solve the rest. Maybe we've solved the problem of hard to maintain tests, which again I never experienced, but it's hard to argue that we're in a good state. On Wed, Sep 14, 2016 at 9:36 PM Zachary Turner wrote: > There's also a cost to *not* writing test cases, which is paid > proportionally to how difficult it is to write test cases. > > That said, aren't the existing lldbinline tests an example of the behavior > you're objecting to? And that is a relatively recent addition which I have > never seen or heard of any objections to. > > On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda wrote: > > > On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: > > > I don't blame you for being scared of command tests. I don't support > their use in the current LLDB test suite either, for exactly the same > reasons you and Jason have expressed. But I do think it's possible to come > up with something that a) doesn't suffer from the same problems, b) allows > testing a ton of extra functionality that is not currently testable through > the api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow > :) > > > I think your examples are reductive to "printing values is easy to keep > consistent" -- but that's only a small part of what a debugger testsuite > needs to do. > > If we want to use lit to drive tests written in terms of SB API, then I'm > open to seeing how this tool can be used in lldb. > > If lit can only be used to write command output tests, then I believe, > based on years and years of experience with exactly that kind of test suite > in with gdb, that this is a very poor addition to lldb. It will only > hamper future lldb development as we want to improve the lldb command line > UI - as it did with gdb. Every change to the command line UI breaks > hundreds of "easy to write" tests. And who is responsible for cleaning all > those up? The people who wrote these easy-to-write command line output > matching tests? No, it'll be the person trying to improve the debugger, > which is a big disincentive to changing anything else you'll need to wade > through the swamp of thousands of accumulated test cases. > > There's a cost to writing a test case. Either it is paid up front when > you write the test in terms of SB API, or it is paid repeatedly over time > as people change the UI to the command line tool. > > I have nothing against lit per se, I'm open to seeing what an SB API based > test case in that harness looks like. I have nothing but objections to > adding significant new additions to the testsuite that are locked to the > command line UI behavior of lldb today. > > > J > > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
There's also a cost to *not* writing test cases, which is paid proportionally to how difficult it is to write test cases. That said, aren't the existing lldbinline tests an example of the behavior you're objecting to? And that is a relatively recent addition which I have never seen or heard of any objections to. On Wed, Sep 14, 2016 at 9:07 PM Jason Molenda wrote: > > On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: > > > I don't blame you for being scared of command tests. I don't support > their use in the current LLDB test suite either, for exactly the same > reasons you and Jason have expressed. But I do think it's possible to come > up with something that a) doesn't suffer from the same problems, b) allows > testing a ton of extra functionality that is not currently testable through > the api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow > :) > > > I think your examples are reductive to "printing values is easy to keep > consistent" -- but that's only a small part of what a debugger testsuite > needs to do. > > If we want to use lit to drive tests written in terms of SB API, then I'm > open to seeing how this tool can be used in lldb. > > If lit can only be used to write command output tests, then I believe, > based on years and years of experience with exactly that kind of test suite > in with gdb, that this is a very poor addition to lldb. It will only > hamper future lldb development as we want to improve the lldb command line > UI - as it did with gdb. Every change to the command line UI breaks > hundreds of "easy to write" tests. And who is responsible for cleaning all > those up? The people who wrote these easy-to-write command line output > matching tests? No, it'll be the person trying to improve the debugger, > which is a big disincentive to changing anything else you'll need to wade > through the swamp of thousands of accumulated test cases. > > There's a cost to writing a test case. Either it is paid up front when > you write the test in terms of SB API, or it is paid repeatedly over time > as people change the UI to the command line tool. > > I have nothing against lit per se, I'm open to seeing what an SB API based > test case in that harness looks like. I have nothing but objections to > adding significant new additions to the testsuite that are locked to the > command line UI behavior of lldb today. > > > J ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: > I don't blame you for being scared of command tests. I don't support their > use in the current LLDB test suite either, for exactly the same reasons you > and Jason have expressed. But I do think it's possible to come up with > something that a) doesn't suffer from the same problems, b) allows testing a > ton of extra functionality that is not currently testable through the api, > and c) doesn't rely on python at all. If I'm wrong I'll eat crow :) I think your examples are reductive to "printing values is easy to keep consistent" -- but that's only a small part of what a debugger testsuite needs to do. If we want to use lit to drive tests written in terms of SB API, then I'm open to seeing how this tool can be used in lldb. If lit can only be used to write command output tests, then I believe, based on years and years of experience with exactly that kind of test suite in with gdb, that this is a very poor addition to lldb. It will only hamper future lldb development as we want to improve the lldb command line UI - as it did with gdb. Every change to the command line UI breaks hundreds of "easy to write" tests. And who is responsible for cleaning all those up? The people who wrote these easy-to-write command line output matching tests? No, it'll be the person trying to improve the debugger, which is a big disincentive to changing anything else you'll need to wade through the swamp of thousands of accumulated test cases. There's a cost to writing a test case. Either it is paid up front when you write the test in terms of SB API, or it is paid repeatedly over time as people change the UI to the command line tool. I have nothing against lit per se, I'm open to seeing what an SB API based test case in that harness looks like. I have nothing but objections to adding significant new additions to the testsuite that are locked to the command line UI behavior of lldb today. J ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner accepted this revision. zturner added a comment. lgtm from my point of view after `LLDB_TEST_CXX_COMPILER` and `LLDB_TEST_C_COMPILER` are added. One question: Currently the tests appear to run all commands until the end of the file, and then do 1 pass over the output through `FileCheck`. Is this correct? We have a lot of tests where the command to run next depends on the output of a previous command. If the test is running LLDB commands rather than using the python api, this means you need some pexpect-style solution that can read output and send input to the process as it's running. Does lit have anything like this? I'm pretty sure the answer is no, but just want to check. We can punt on supporting this type of test until later, just want to find out if it's already supported. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I actually like the idea of feeding command lines. Again, not when the command you're running prints a huge block of text and you try to match it. But for "get the debugger into the state where i can reproduce the bug", and sometimes the step to actually repro it, I think it's hands down the best. You can look at a test and tell in a few seconds how to repro it. You can grok the test almost immediately. More importantly, it gives you everything you need to repro it under a debugger. If you wanted to be fancy it could even spit out a command file so you could get the test case failing in a debugger in seconds On Wed, Sep 14, 2016 at 6:02 PM Sean Callanan wrote: > I agree completely with tackling the easy stuff first. > That said, the easy stuff (probably >50% of the testsuite) doesn't even > require command-line interaction and is just of the form "stop here, run > this one expression, maybe print this variable using 'frame variab'e'" > I would argue that we can find a simpler way to handle these cases than > feeding lldb command lines to the command-line parser. > To be clear, that simpler way is not the SB API, it's probably a > special-purpose language that knows how to stop at breakpoints and inspect > locals/run expressions. That's all it does. > The SB API / command line parser argument is for what I'd argue are the > complex cases, and I think we should deal with those later because they're > both technically hairy and seem to attract more discussion. > > Sean > > On Sep 14, 2016, at 5:57 PM, Zachary Turner wrote: > > 1. get all the easy stuff out of the way first, such as the tests in this > patch (and there are probably at least 100 more that are just as easy) > > > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I agree completely with tackling the easy stuff first. That said, the easy stuff (probably >50% of the testsuite) doesn't even require command-line interaction and is just of the form "stop here, run this one expression, maybe print this variable using 'frame variab'e'" I would argue that we can find a simpler way to handle these cases than feeding lldb command lines to the command-line parser. To be clear, that simpler way is not the SB API, it's probably a special-purpose language that knows how to stop at breakpoints and inspect locals/run expressions. That's all it does. The SB API / command line parser argument is for what I'd argue are the complex cases, and I think we should deal with those later because they're both technically hairy and seem to attract more discussion. Sean > On Sep 14, 2016, at 5:57 PM, Zachary Turner wrote: > > 1. get all the easy stuff out of the way first, such as the tests in this > patch (and there are probably at least 100 more that are just as easy) > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I agree that if all the tests look like the ones in this review there will be no way to test 100% of the things we currently test. At the same time, I see this as just a beginning. A replacement for lldbinline but one which is more extensible. To test things like you said like multi-threaded stepping behavior, a simple sequential list of actions and pattern matches won't be sufficient. So I think a path forward is something like this: 1. get all the easy stuff out of the way first, such as the tests in this patch (and there are probably at least 100 more that are just as easy) 2. Once we have a reasonable number of all easy cases ported, we do some benchmarking (at this point the original versions of these tests are still in the tree). Is it faster? Slower? Just to give us actionable data. 3. Turn on these tests on the bots, let them run for a while and see how stable they are. 4. If all goes well after a week or two, delete the ones that have been ported over and starting using this style of test whenever possible and whenever one would have previously used an lldbinline test. 5. Analyze the remaining difficult tests, and find areas where we can make simple, incremental changes to LLDB's lit test runner to support new types of tests. Some will be easier than others. At every point we're just getting low hanging fruit, small, non-controversial improvements that enable writing new tests in this style. And then just see how far we can get. I could throw out crazy sledgehammer ideas right now, like "allow people to embed SB API code directly in the lit test file", but it seems unproductive since we don't even know exactly what problems we'll encounter. On Wed, Sep 14, 2016 at 5:38 PM Jim Ingham wrote: > > > On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: > > > > I'm only saying that we should have an open mind. Obviously there are > (valid!) concerns. If we can't solve them then we can't solve them. The > goal (my goal anyway) is always to make things better, not to use X because > it's X. There's value in consistency, but that doesn't mean that the value > from consistency always outweighs the value you get from doing a custom > thing. > > > > So what I'm saying is: IF we can find a way to have one test suite > across all of LLVM and it's subprojects, AND it is sufficiently powerful to > test all the things we need to test while remaining maintainable in the > long term, we should absolutely jump on the opportunity. > > > > But this is one of those things that requires a lot of upfront time > investment before you can actually know if it can work for 100% of things. > Obviously some people don't want to invest their time that way when they're > already satisfied, and I don't blame them. > > > > But for the people who do, and who think they can solve the problem, > what's the harm? Obviously the burden is on those people to prove that > their vision can be realized. > > > > But if it is successful, then there's no denying the benefits. 1) Tests > become easier to write. 2) Tests become easier to debug. 3) Consistency > encourages people who have traditionally stayed away from LLDB to > contribute. 4) All the people pouring their effort into the custom thing > can now pour it into the shared thing, so everybody benefits. > > I disagree that one of the benefits will be #2. I worry that what will > really happen is "easy tests will become slightly easier to write, and > complex tests will be fragile when they aren't impossible" which will mean > we end up only writing easy tests. Lots of easy tests, but we won't test > complex multi-threaded stepping behavior, etc and so we'll end up breaking > that sort of thing, which we won't find out about quickly because the > failures are intermittent, and hard to write bugs about. > > Jim > > > > > > I don't blame you for being scared of command tests. I don't support > their use in the current LLDB test suite either, for exactly the same > reasons you and Jason have expressed. But I do think it's possible to come > up with something that a) doesn't suffer from the same problems, b) allows > testing a ton of extra functionality that is not currently testable through > the api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow > :) > > > > On Wed, Sep 14, 2016 at 5:00 PM Jim Ingham wrote: > > Also, w.r.t: > > > > > Aside from write imperative control flow constructs, which I see as a > positive rather than a negative. > > > > I wrote a bunch of tests to test that stepping behavior for swift and C > was reasonable. When stepping through source code, there is not one > correct way to write the line tables, and in fact clang & swiftc change how > they describe the source through the line tables all the time. So you have > to do: I stepped, and sometimes I'll get to A, sometimes to B, both are > "right" but I have to do different things in either case. If A, step again > before the next test, if B go to the next test. > > >
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
> On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote: > > I'm only saying that we should have an open mind. Obviously there are > (valid!) concerns. If we can't solve them then we can't solve them. The > goal (my goal anyway) is always to make things better, not to use X because > it's X. There's value in consistency, but that doesn't mean that the value > from consistency always outweighs the value you get from doing a custom thing. > > So what I'm saying is: IF we can find a way to have one test suite across all > of LLVM and it's subprojects, AND it is sufficiently powerful to test all the > things we need to test while remaining maintainable in the long term, we > should absolutely jump on the opportunity. > > But this is one of those things that requires a lot of upfront time > investment before you can actually know if it can work for 100% of things. > Obviously some people don't want to invest their time that way when they're > already satisfied, and I don't blame them. > > But for the people who do, and who think they can solve the problem, what's > the harm? Obviously the burden is on those people to prove that their vision > can be realized. > > But if it is successful, then there's no denying the benefits. 1) Tests > become easier to write. 2) Tests become easier to debug. 3) Consistency > encourages people who have traditionally stayed away from LLDB to contribute. > 4) All the people pouring their effort into the custom thing can now pour it > into the shared thing, so everybody benefits. I disagree that one of the benefits will be #2. I worry that what will really happen is "easy tests will become slightly easier to write, and complex tests will be fragile when they aren't impossible" which will mean we end up only writing easy tests. Lots of easy tests, but we won't test complex multi-threaded stepping behavior, etc and so we'll end up breaking that sort of thing, which we won't find out about quickly because the failures are intermittent, and hard to write bugs about. Jim > > I don't blame you for being scared of command tests. I don't support their > use in the current LLDB test suite either, for exactly the same reasons you > and Jason have expressed. But I do think it's possible to come up with > something that a) doesn't suffer from the same problems, b) allows testing a > ton of extra functionality that is not currently testable through the api, > and c) doesn't rely on python at all. If I'm wrong I'll eat crow :) > > On Wed, Sep 14, 2016 at 5:00 PM Jim Ingham wrote: > Also, w.r.t: > > > Aside from write imperative control flow constructs, which I see as a > > positive rather than a negative. > > I wrote a bunch of tests to test that stepping behavior for swift and C was > reasonable. When stepping through source code, there is not one correct way > to write the line tables, and in fact clang & swiftc change how they describe > the source through the line tables all the time. So you have to do: I > stepped, and sometimes I'll get to A, sometimes to B, both are "right" but I > have to do different things in either case. If A, step again before the next > test, if B go to the next test. > > You could "fix" that by only doing one step per test, and taking each of > these as a success. But then you wouldn't test that series of steps don't > accumulate errors, you'd only test "run to a breakpoint and step once." That > would not be good. So your positive would be very much a negative for this > kind of test. > > Traditionally the answer to this has been: we know we have to keep the > current testsuite around but we're just adding other new different ways to > write tests. Now you are saying something very different. Do you really > mean that? > > Jim > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
Also, you mentioned that this is very similar to the lldbinline tests. For that reason, I would actually propose dropping lldbinline tests in favor of this. If they are essentially the same, then it seems better to have 2 types of tests rather than 3, and it seems better for those 2 to be lit + python rather than lldbinline + python, solely for the reason that lit is shared and receives many performance and stability improvements from non LLDB developers. On Wed, Sep 14, 2016 at 5:13 PM Zachary Turner wrote: > I'm only saying that we should have an open mind. Obviously there are > (valid!) concerns. If we can't solve them then we can't solve them. The > goal (my goal anyway) is always to make things better, not to use X because > it's X. There's value in consistency, but that doesn't mean that the value > from consistency always outweighs the value you get from doing a custom > thing. > > So what I'm saying is: IF we can find a way to have one test suite across > all of LLVM and it's subprojects, AND it is sufficiently powerful to test > all the things we need to test while remaining maintainable in the long > term, we should absolutely jump on the opportunity. > > But this is one of those things that requires a lot of upfront time > investment before you can actually know if it can work for 100% of things. > Obviously some people don't want to invest their time that way when they're > already satisfied, and I don't blame them. > > But for the people who do, and who think they can solve the problem, > what's the harm? Obviously the burden is on those people to prove that > their vision can be realized. > > But if it is successful, then there's no denying the benefits. 1) Tests > become easier to write. 2) Tests become easier to debug. 3) Consistency > encourages people who have traditionally stayed away from LLDB to > contribute. 4) All the people pouring their effort into the custom thing > can now pour it into the shared thing, so everybody benefits. > > I don't blame you for being scared of command tests. I don't support > their use in the current LLDB test suite either, for exactly the same > reasons you and Jason have expressed. But I do think it's possible to come > up with something that a) doesn't suffer from the same problems, b) allows > testing a ton of extra functionality that is not currently testable through > the api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow > :) > > On Wed, Sep 14, 2016 at 5:00 PM Jim Ingham wrote: > > Also, w.r.t: > > > Aside from write imperative control flow constructs, which I see as a > positive rather than a negative. > > I wrote a bunch of tests to test that stepping behavior for swift and C > was reasonable. When stepping through source code, there is not one > correct way to write the line tables, and in fact clang & swiftc change how > they describe the source through the line tables all the time. So you have > to do: I stepped, and sometimes I'll get to A, sometimes to B, both are > "right" but I have to do different things in either case. If A, step again > before the next test, if B go to the next test. > > You could "fix" that by only doing one step per test, and taking each of > these as a success. But then you wouldn't test that series of steps don't > accumulate errors, you'd only test "run to a breakpoint and step once." > That would not be good. So your positive would be very much a negative for > this kind of test. > > Traditionally the answer to this has been: we know we have to keep the > current testsuite around but we're just adding other new different ways to > write tests. Now you are saying something very different. Do you really > mean that? > > Jim > > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I'm only saying that we should have an open mind. Obviously there are (valid!) concerns. If we can't solve them then we can't solve them. The goal (my goal anyway) is always to make things better, not to use X because it's X. There's value in consistency, but that doesn't mean that the value from consistency always outweighs the value you get from doing a custom thing. So what I'm saying is: IF we can find a way to have one test suite across all of LLVM and it's subprojects, AND it is sufficiently powerful to test all the things we need to test while remaining maintainable in the long term, we should absolutely jump on the opportunity. But this is one of those things that requires a lot of upfront time investment before you can actually know if it can work for 100% of things. Obviously some people don't want to invest their time that way when they're already satisfied, and I don't blame them. But for the people who do, and who think they can solve the problem, what's the harm? Obviously the burden is on those people to prove that their vision can be realized. But if it is successful, then there's no denying the benefits. 1) Tests become easier to write. 2) Tests become easier to debug. 3) Consistency encourages people who have traditionally stayed away from LLDB to contribute. 4) All the people pouring their effort into the custom thing can now pour it into the shared thing, so everybody benefits. I don't blame you for being scared of command tests. I don't support their use in the current LLDB test suite either, for exactly the same reasons you and Jason have expressed. But I do think it's possible to come up with something that a) doesn't suffer from the same problems, b) allows testing a ton of extra functionality that is not currently testable through the api, and c) doesn't rely on python at all. If I'm wrong I'll eat crow :) On Wed, Sep 14, 2016 at 5:00 PM Jim Ingham wrote: > Also, w.r.t: > > > Aside from write imperative control flow constructs, which I see as a > positive rather than a negative. > > I wrote a bunch of tests to test that stepping behavior for swift and C > was reasonable. When stepping through source code, there is not one > correct way to write the line tables, and in fact clang & swiftc change how > they describe the source through the line tables all the time. So you have > to do: I stepped, and sometimes I'll get to A, sometimes to B, both are > "right" but I have to do different things in either case. If A, step again > before the next test, if B go to the next test. > > You could "fix" that by only doing one step per test, and taking each of > these as a success. But then you wouldn't test that series of steps don't > accumulate errors, you'd only test "run to a breakpoint and step once." > That would not be good. So your positive would be very much a negative for > this kind of test. > > Traditionally the answer to this has been: we know we have to keep the > current testsuite around but we're just adding other new different ways to > write tests. Now you are saying something very different. Do you really > mean that? > > Jim > > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
> On Sep 14, 2016, at 3:51 PM, Zachary Turner wrote: > > zturner added a comment. > > Also, it occurred to me that if all tests were like this (and yes, that's a > tall order to imagine a world where not a single test was written using the > Python API), we could probably actually drop the Python 3.5 requirement on > Windows. > > Another thing that's nice about tests like this is that it makes it trivial > to see how to reproduce a failure. It's currently very hard to debug > failures because you have to first figure out where in the test it's failing > (i.e. what line of python), then attach to the python executable and try to > get a breakpoint on the native code side in the right SB API call matching up > with the place where you determined the test is failing. This is really a > pain without a debugger that supports mixed<->native transitions between > python and c++, and even with a debugger that does support it like we have on > Windows, it often doesn't work very well or exhibits flakiness. I must admit, I have never found analyzing individual test failures to be particularly difficult. The test suite tells you exactly which line failed in the test suite. Most of the time looking at the failure line will tell you how to repro the problem with just the test binary. The rest of the time I can go put a few printf's around that line, and quickly figure out what is going wrong. Python is great for this sort of iterative exploration. Once you know what fails, run "dotest -d" attach, break at the SB API that is causing the problem and start debugging. Jim > > > https://reviews.llvm.org/D24591 > > > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
Also, w.r.t: > Aside from write imperative control flow constructs, which I see as a > positive rather than a negative. I wrote a bunch of tests to test that stepping behavior for swift and C was reasonable. When stepping through source code, there is not one correct way to write the line tables, and in fact clang & swiftc change how they describe the source through the line tables all the time. So you have to do: I stepped, and sometimes I'll get to A, sometimes to B, both are "right" but I have to do different things in either case. If A, step again before the next test, if B go to the next test. You could "fix" that by only doing one step per test, and taking each of these as a success. But then you wouldn't test that series of steps don't accumulate errors, you'd only test "run to a breakpoint and step once." That would not be good. So your positive would be very much a negative for this kind of test. Traditionally the answer to this has been: we know we have to keep the current testsuite around but we're just adding other new different ways to write tests. Now you are saying something very different. Do you really mean that? Jim ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
You can get thing 1, thing 2, and thing 3 in declarative ways with a command line api too. Like the example I gave earlier: (lldb) print-dev threads[3].id 0x1234 (lldb) print-dev threads[4].id 0x2345 You're getting one value at a time, not a space separated list. Although I think we could take it one step further and say that all developer commands like this must either return one value or a comma separated list with spaces. In any case, we're not talking about the output of arbitrary commands here, we're talking about the output of a very controlled command which we can restrict any way we like. And we can even have tests that validate that this special command's output always matches a particular simple regex to ensure that it's never something that will be confusing to parse. Furthermore, we're talking about a command independent of the normal user command api. There would be zero risk associated with changing the output format of a given command, because it would be completely independent of the command the tests would be using. On Wed, Sep 14, 2016 at 4:51 PM Jim Ingham wrote: > If the internal thing is three things in a list, you get back thing one, > thing two and thing three in declarative ways rather than "I got three > space separated things, maybe they had spaces in them, maybe there were > quotes, etc, and don't ever change this now or you'll have to go fix all > the tests... I really don't see how going back to irregular text output > grubbing is a step forward. > > Also, the SBInternal API's would be much simpler to write, (a) since you > wouldn't have to concern yourself with how the other end was going to parse > up what you were outputting and (b) because individual API calls are much > easier to write than LLDB commands/sub-commands. > > You could have your special lldb command that produced JSON and then the > test engine had some language to pick fields out, then you could only read > this output with confidence, and you're back to the same fragility when > testing the results of regular commands. You could change all the regular > commands to optionally produce some structured output, but that's a lot of > work to reproduce something that already works pretty well. > > Jim > > > > > On Sep 14, 2016, at 4:43 PM, Zachary Turner wrote: > > > > > > > > On Wed, Sep 14, 2016 at 4:39 PM Jim Ingham wrote: > > > > > 4. You can test a LOT more things when you are able to use an api that > doesn't have to be stable. > > > > When I mentioned that API, I had in mind an internal Python module that > you could use to grub into internals. I'm all for that. I'm not so much > in favor of "I have an ad hoc command that prints out various bits of > texts, and we'll use that for testing." > > > > Is it any different though? Take the API you're imagining to grub for > internals. Now imagine the EXACT same API as an lldb command. What's the > difference? What could you do with the Python API that you couldn't do > with the command API? Aside from write imperative control flow constructs, > which I see as a positive rather than a negative. > > ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
> On Sep 14, 2016, at 3:56 PM, Chris Bieneman wrote: > > @jingham and @zturner, we can also take advantage of FileCheck's use of > regular expressions to write robust matchers. In general LLVM has managed to > change text output formats many times in radical ways, and LIT's testing has > still suited the project well. > The gdb testsuite used expect for all its pattern matching. And yet it still ended up both being fragile and making it pretty much impossible to change some aspects of the command output because nobody would ever want to go fix all the tests. Jim ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
If the internal thing is three things in a list, you get back thing one, thing two and thing three in declarative ways rather than "I got three space separated things, maybe they had spaces in them, maybe there were quotes, etc, and don't ever change this now or you'll have to go fix all the tests... I really don't see how going back to irregular text output grubbing is a step forward. Also, the SBInternal API's would be much simpler to write, (a) since you wouldn't have to concern yourself with how the other end was going to parse up what you were outputting and (b) because individual API calls are much easier to write than LLDB commands/sub-commands. You could have your special lldb command that produced JSON and then the test engine had some language to pick fields out, then you could only read this output with confidence, and you're back to the same fragility when testing the results of regular commands. You could change all the regular commands to optionally produce some structured output, but that's a lot of work to reproduce something that already works pretty well. Jim > On Sep 14, 2016, at 4:43 PM, Zachary Turner wrote: > > > > On Wed, Sep 14, 2016 at 4:39 PM Jim Ingham wrote: > > > 4. You can test a LOT more things when you are able to use an api that > > doesn't have to be stable. > > When I mentioned that API, I had in mind an internal Python module that you > could use to grub into internals. I'm all for that. I'm not so much in > favor of "I have an ad hoc command that prints out various bits of texts, and > we'll use that for testing." > > Is it any different though? Take the API you're imagining to grub for > internals. Now imagine the EXACT same API as an lldb command. What's the > difference? What could you do with the Python API that you couldn't do with > the command API? Aside from write imperative control flow constructs, which > I see as a positive rather than a negative. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
On Wed, Sep 14, 2016 at 4:39 PM Jim Ingham wrote: > > > 4. You can test a LOT more things when you are able to use an api that > doesn't have to be stable. > > When I mentioned that API, I had in mind an internal Python module that > you could use to grub into internals. I'm all for that. I'm not so much > in favor of "I have an ad hoc command that prints out various bits of > texts, and we'll use that for testing." > Is it any different though? Take the API you're imagining to grub for internals. Now imagine the EXACT same API as an lldb command. What's the difference? What could you do with the Python API that you couldn't do with the command API? Aside from write imperative control flow constructs, which I see as a positive rather than a negative. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
> On Sep 14, 2016, at 4:00 PM, Zachary Turner wrote:
>
> On Wed, Sep 14, 2016 at 3:51 PM Sean Callanan wrote:
> How different is that really from
>
> (lldb) script lldb.frame.FindVariable("argc").GetValue()
> '1'
> (lldb) script lldb.process.GetNumThreads()
> 1
> (lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
> 3514809
>
> ? If it's developer-only, then this is even fairly well-documented using
> e.g. "script dir(lldb.process)"
>
> Admittedly I hadn't thought of doing that. That said I can think of a couple
> of differences:
>
> 1. Writing it using script commands means you have to be familiar with the
> python api, somewhat raising the barrier to entry for writing new tests.
The lldb api is pretty simple, especially for basic stuff like checking
variable values. Plus, if you wanted to write tests like "I have a funky new
type and I want to check that all its children are right" the bit of time you
spent looking at how SBValues works will pay off many-fold in how much more
accurate you can make these tests, and how much more easily you can get exactly
what you want to test, when you aren't trying to parse text output of the value
printing. I think this is really a false tradeoff.
And if you are testing something simple, you can just use Sean's inline tests,
they are not really any more difficult to write than the lit tests (except that
using the SB API's are the natural language, but I see that as a positive, not
a negative.)
>
> 2. It depends on proper behavior of the script command, which means that if
> the script command itself is broken, every single test will be broken.
> 3. It means that building the lldb python module is required to run tests.
> Wouldn't it be awesome if there was a way to run tests even with
> LLDB_DISABLE_PYTHON=1? Then script could just be a "feature" (as in, you
> could limit tests which use the script command to tests whose purpose is
> explicitly to test where the scripting system works, and for those you could
> just add "REQUIRES: python" at the top of the lit file. And I imagine that
> not invoking python during the running of the test suite would drastically
> speed up its execution.
>
I would find that very not awesome, because it would mean ALL our tests are
scraping text output, and I lived through that once already and would rather
not do it again.
> 4. You can test a LOT more things when you are able to use an api that
> doesn't have to be stable.
When I mentioned that API, I had in mind an internal Python module that you
could use to grub into internals. I'm all for that. I'm not so much in favor
of "I have an ad hoc command that prints out various bits of texts, and we'll
use that for testing."
Jim
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added inline comments. Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + beanz wrote: > zturner wrote: > > beanz wrote: > > > Disallowing setting both seems reasonable to me. I'm not entirely sure > > > how to connect `LLDB_TEST_COMPILER` up into the lit suite because we > > > really want something that more matches the CMake style of > > > `...__COMPILER` so that we could override multiple compilers. > > What if you added a parallel of `LLDB_TEST_COMPILER` directly in this file, > > that is used specifically for lit tests? Like `LLDB_LIT_TEST_COMPILER` or > > `LLDB_LIT_CLANG_PATH`? > > > > The only reason I'm harping on this is because as it stands, this won't > > work on windows. (The host compiler is MSVC, which uses a completely > > different command line syntax, and the in-tree clang is going to be a debug > > one when doing a debug build, which is going to be unacceptably slow). > We could add parallels. How about instead of trying to match the existing > model though we go with something more like `LLDB_TEST_C_COMPILER` and > `LLDB_TEST_CXX_COMPILER` as values that replace the `%cc` and `%cxx` > substitutions respectively? Works for me. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz added a comment. @zturner, we can absolutely add flags into the substitutions. Other projects do that too. Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + zturner wrote: > beanz wrote: > > Disallowing setting both seems reasonable to me. I'm not entirely sure how > > to connect `LLDB_TEST_COMPILER` up into the lit suite because we really > > want something that more matches the CMake style of `...__COMPILER` > > so that we could override multiple compilers. > What if you added a parallel of `LLDB_TEST_COMPILER` directly in this file, > that is used specifically for lit tests? Like `LLDB_LIT_TEST_COMPILER` or > `LLDB_LIT_CLANG_PATH`? > > The only reason I'm harping on this is because as it stands, this won't work > on windows. (The host compiler is MSVC, which uses a completely different > command line syntax, and the in-tree clang is going to be a debug one when > doing a debug build, which is going to be unacceptably slow). We could add parallels. How about instead of trying to match the existing model though we go with something more like `LLDB_TEST_C_COMPILER` and `LLDB_TEST_CXX_COMPILER` as values that replace the `%cc` and `%cxx` substitutions respectively? https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
On Wed, Sep 14, 2016 at 4:28 PM Jason Molenda wrote: > I have to concur with Jim's point -- writing & maintaining the gdb > testsuite for years, which was based on commands & expected output like > these lit tests, was a huge drag on everyone's productivity as the debugger > changed over time. This style of test looks wonderfully easy to read & > write & debug, but that's only if you ignore long-term maintainability, > then you quickly find that it was a huge mistake. I have nothing against > lit, this is a very nice way of writing commands & expected output tests, > but having lived through the consequences of that style testsuite, I > couldn't imagine making that choice again, especially when we've avoided it > so far. It's a bit more work to read & write & debug the SB API testsuites > that we have today, but it's vastly more maintainable long-term. Isn't that solvable though by having commands which return single values instead of large blocks of test, so that you don't have to worry about pattern matching and formatting issues? The only reason the SB API solves this problem is because it allows you to query for specific values instead of for formatted blocks of text. If you can query for the same specific values without the SB API, then the problem still seems to be solved. It's just a different API. I would say it's significantly more work to read, write, and debug the SB API tests, and I'm speculating that that's in part why there's so few. Granted I have no familiarity with GDB's test suite, but I can't imagine it not being possible to solve this. As long as you can query the debugger for data about a sufficiently granular state, I don't foresee this problem being an issue in practice. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added a comment. One more question: Is there a way in lit that we can append command line flags to the run lines even if the user doesn't specify them? For example in the substitution? For example, if someone writes `# RUN: %cxx %p/Inputs/call-function.cpp -g -o %t && %lldb -b -s %s -- %t | FileCheck %s` then this is always going to run `clang function.cpp -g -o`. But we need to manipulate the command line on different platforms. Like on Windows we will need to add `-fms-compatibility` `-fuse-ld=lld`, and various other things. And on other platforms there are other specific things that always have to be added. Is this possible somehow? (You don't have to address it in this patch, just curious) Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + beanz wrote: > Disallowing setting both seems reasonable to me. I'm not entirely sure how to > connect `LLDB_TEST_COMPILER` up into the lit suite because we really want > something that more matches the CMake style of `...__COMPILER` so that > we could override multiple compilers. What if you added a parallel of `LLDB_TEST_COMPILER` directly in this file, that is used specifically for lit tests? Like `LLDB_LIT_TEST_COMPILER` or `LLDB_LIT_CLANG_PATH`? The only reason I'm harping on this is because as it stands, this won't work on windows. (The host compiler is MSVC, which uses a completely different command line syntax, and the in-tree clang is going to be a debug one when doing a debug build, which is going to be unacceptably slow). https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
I have to concur with Jim's point -- writing & maintaining the gdb testsuite for years, which was based on commands & expected output like these lit tests, was a huge drag on everyone's productivity as the debugger changed over time. This style of test looks wonderfully easy to read & write & debug, but that's only if you ignore long-term maintainability, then you quickly find that it was a huge mistake. I have nothing against lit, this is a very nice way of writing commands & expected output tests, but having lived through the consequences of that style testsuite, I couldn't imagine making that choice again, especially when we've avoided it so far. It's a bit more work to read & write & debug the SB API testsuites that we have today, but it's vastly more maintainable long-term. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz added inline comments. Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + Disallowing setting both seems reasonable to me. I'm not entirely sure how to connect `LLDB_TEST_COMPILER` up into the lit suite because we really want something that more matches the CMake style of `...__COMPILER` so that we could override multiple compilers. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added inline comments. Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + beanz wrote: > The `LLDB_TEST_COMPILER` option doesn't signify that it is using an in-tree > or out-of-tree compiler which is significant if we're going to tie the test > target to depending on the clang target. We could support that option in > addition to this one, but I see them as distinctly different. At the same time, setting `LLDB_TEST_COMPILER` to something and `LLDB_TEST_CLANG` seem to be incompatible with each other. Should we error if both of them are set? Another possibility is to allow one to set `LLDB_TEST_COMPILER` to be set to a magic value like `` https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added a comment. Todd probably knows about this since he added the gtest target to the Xcode project. I imagine there's a python test suite target and a gtest target. Isn't it possible to have a third target which when you build it, builds the other two targets (which would run the test suites)? If that's not possible, then having a target which invokes a command line would work as beanz mentioned. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz added a comment. @granata.enrico, it seems to me that this is a problem the people maintaining the Xcode project should address. I know it is possible to create targets in Xcode projects that call shell commands or makefiles (CMake does this). It is therefore possible that the Xcode project could have a "check" target that runs all the tests, and I really don't think we should limit our testing options based on solvable problems. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz updated this revision to Diff 71460.
beanz added a comment.
- Use lit.util.which instead of constructing paths, which adds exe suffixes and
uses proper path seperators
- Fixed spelling error in list(APPEND ...) command
https://reviews.llvm.org/D24591
Files:
lit/CMakeLists.txt
lit/Expr/Inputs/anonymous-struct.cpp
lit/Expr/Inputs/call-function.cpp
lit/Expr/TestCallStdStringFunction.test
lit/Expr/TestCallStopAndContinue.test
lit/Expr/TestCallUserAnonTypedef.test
lit/Expr/TestCallUserDefinedFunction.test
lit/Expr/lit.local.cfg
lit/Unit/lit.cfg
lit/Unit/lit.site.cfg.in
lit/lit.cfg
lit/lit.site.cfg.in
Index: lit/lit.site.cfg.in
===
--- lit/lit.site.cfg.in
+++ lit/lit.site.cfg.in
@@ -8,6 +8,12 @@
config.lldb_obj_root = "@LLDB_BINARY_DIR@"
config.target_triple = "@TARGET_TRIPLE@"
config.python_executable = "@PYTHON_EXECUTABLE@"
+config.host_cc = "@CMAKE_C_COMPILER@"
+config.host_cxx = "@CMAKE_CXX_COMPILER@"
+
+test_clang = "@LLDB_TEST_CLANG@".lower()
+
+config.test_clang = test_clang == "on" or test_clang == "true" or test_clang == "1"
# Support substitution of the tools and libs dirs with user parameters. This is
# used when we can't determine the tool dir at configuration time.
Index: lit/lit.cfg
===
--- lit/lit.cfg
+++ lit/lit.cfg
@@ -112,17 +112,83 @@
lit_config.load_config(config, site_cfg)
raise SystemExit
+# Register substitutions
+config.substitutions.append(('%python', config.python_executable))
+config.substitutions.append(('%host_cc', config.host_cc))
+config.substitutions.append(('%host_cxx', config.host_cxx))
+
+cc = config.host_cc
+cxx = config.host_cxx
+debugserver = lit.util.which('debugserver', llvm_tools_dir)
+lldb = lit.util.which('lldb', llvm_tools_dir)
+
+if config.test_clang:
+cc = lit.util.which('clang', llvm_tools_dir)
+cxx = lit.util.which('clang++', llvm_tools_dir)
+
+if platform.system() in ['Darwin']:
+try:
+out = lit.util.capture(['xcrun', '--show-sdk-path']).strip()
+res = 0
+except OSError:
+res = -1
+if res == 0 and out:
+sdk_path = out
+lit_config.note('using SDKROOT: %r' % sdk_path)
+cxx += " -isysroot %s" % sdk_path
+debugserver = '%s/Library/Frameworks/LLDB.frameworks/Resources/debugserver' % config.llvm_obj_root
+
+
+config.substitutions.append(('%cc', cc))
+config.substitutions.append(('%cxx', cxx))
+
+config.substitutions.append(('%lldb', lldb))
+config.substitutions.append(('%debugserver', debugserver))
+
+for pattern in [r"\bFileCheck\b",
+r"\| \bnot\b"]:
+tool_match = re.match(r"^(\\)?((\| )?)\W+b([0-9A-Za-z-_]+)\\b\W*$",
+ pattern)
+tool_pipe = tool_match.group(2)
+tool_name = tool_match.group(4)
+tool_path = lit.util.which(tool_name, config.llvm_tools_dir)
+if not tool_path:
+# Warn, but still provide a substitution.
+lit_config.note(
+'Did not find ' + tool_name + ' in ' + config.llvm_tools_dir)
+config.substitutions.append((pattern, tool_pipe + tool_path))
+
# Shell execution
if platform.system() not in ['Windows'] or lit_config.getBashPath() != '':
config.available_features.add('shell')
# Running on Darwin OS
if platform.system() in ['Darwin']:
+config.available_features.add('darwin')
config.available_features.add('system-linker-mach-o')
# Running on ELF based *nix
if platform.system() in ['FreeBSD', 'Linux']:
config.available_features.add('system-linker-elf')
+if platform.system() in ['FreeBSD']:
+config.available_features.add('freebsd')
+else:
+config.available_features.add('linux')
+
+if platform.system() in ['Windows']:
+config.available_features.add('windows')
+
+if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
+config.available_features.add("armhf-linux")
+
+if re.match(r'icc', cc):
+config.available_features.add("compiler-icc")
+elif re.match(r'clang', cc):
+config.available_features.add("compiler-clang")
+elif re.match(r'gcc', cc):
+config.available_features.add("compiler-gcc")
+elif re.match(r'cl', cc):
+config.available_features.add("compiler-msvc")
# llvm-config knows whether it is compiled with asserts (and)
# whether we are operating in release/debug mode.
Index: lit/Unit/lit.site.cfg.in
===
--- lit/Unit/lit.site.cfg.in
+++ lit/Unit/lit.site.cfg.in
@@ -1,5 +1,6 @@
@LIT_SITE_CFG_IN_HEADER@
+config.test_exec_root = "@LLVM_BINARY_DIR@"
config.llvm_src_root = "@LLVM_SOURCE_DIR@"
config.llvm_obj_root = "@LLVM_BINARY_DIR@"
config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
Index: lit/Unit/lit.cfg
===
--- lit/Unit/lit.cfg
+++ lit/Unit/lit.cfg
@@ -6,6 +6,19 @@
import lit.formats
+# Chec
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
On Wed, Sep 14, 2016 at 4:00 PM Zachary Turner wrote: > > 4. You can test a LOT more things when you are able to use an api that > doesn't have to be stable. > Regarding this point specifically, I think in the past Jim has even floated the idea of having a "private API" that doesn't have to be stable and coudl be used for testing. Only after seeing this review though did I have the idea that such an API could actually just be a single LLDB command. ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
On Wed, Sep 14, 2016 at 3:51 PM Sean Callanan wrote:
> How different is that really from
>
> (lldb) script lldb.frame.FindVariable("argc").GetValue()
> '1'
> (lldb) script lldb.process.GetNumThreads()
> 1
> (lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
> 3514809
>
> ? If it's developer-only, then this is even fairly well-documented using
> e.g. "script dir(lldb.process)"
>
Admittedly I hadn't thought of doing that. That said I can think of a
couple of differences:
1. Writing it using script commands means you have to be familiar with the
python api, somewhat raising the barrier to entry for writing new tests.
2. It depends on proper behavior of the script command, which means that if
the script command itself is broken, every single test will be broken.
3. It means that building the lldb python module is required to run tests.
Wouldn't it be awesome if there was a way to run tests even with
LLDB_DISABLE_PYTHON=1? Then script could just be a "feature" (as in, you
could limit tests which use the script command to tests whose purpose is
explicitly to test where the scripting system works, and for those you
could just add "REQUIRES: python" at the top of the lit file. And I
imagine that not invoking python during the running of the test suite would
drastically speed up its execution.
4. You can test a LOT more things when you are able to use an api that
doesn't have to be stable.
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
granata.enrico added a comment. In https://reviews.llvm.org/D24591#543277, @beanz wrote: > @granata.enrico, we could migrate the existing tests into being executed by > lit even if they aren't using lit's features, so if that direction is desired > we could get everything in lit. That said, you shouldn't ever really have > multiple incantations. Once we have reliable lit testing that is useful it > should be connected to the "check-lldb" and "check-all" targets > appropriately. Just because it runs more than one type of test doesn't mean > you need multiple incantations, and more and varied testing is generally > better for the quality of the product. The problem is that some of us at Apple build LLDB in Xcode, and then test by saying $ ./dotest.py which means no amount of CMake magic will do anything for us. If Xcode builds are supposed to be deprecated and not-to-be-used, that's a possible path forward (but one I am hearing about for the first time...) Mind you, I would not be opposed to having dotest.py also run lit tests and unit tests - or lit run Python and unit tests as well if that's the preferred direction I am just worried about the multiplication of solutions we see in LLDB (Xcode build vs. Cmake build, unit tests, vs lit tests vs Python tests, ...) - it would be nice to streamline those at some point. Your patch is just getting friendly fire because it adds one more axis to this space is all :) https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz added a comment. @zturner, on many of your comments I need to do some research and get back to you. Particularly I need to understand how lit works on Windows better. I do have one inline response. @granata.enrico, we could migrate the existing tests into being executed by lit even if they aren't using lit's features, so if that direction is desired we could get everything in lit. That said, you shouldn't ever really have multiple incantations. Once we have reliable lit testing that is useful it should be connected to the "check-lldb" and "check-all" targets appropriately. Just because it runs more than one type of test doesn't mean you need multiple incantations, and more and varied testing is generally better for the quality of the product. @jingham and @zturner, we can also take advantage of FileCheck's use of regular expressions to write robust matchers. In general LLVM has managed to change text output formats many times in radical ways, and LIT's testing has still suited the project well. And to echo @zturner's last comment, one huge benefit to LIT is that reproducing failures is very simple. LIT failures log simple shell commands that can be executed to reproduce. Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + The `LLDB_TEST_COMPILER` option doesn't signify that it is using an in-tree or out-of-tree compiler which is significant if we're going to tie the test target to depending on the clang target. We could support that option in addition to this one, but I see them as distinctly different. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added a comment. Also, it occurred to me that if all tests were like this (and yes, that's a tall order to imagine a world where not a single test was written using the Python API), we could probably actually drop the Python 3.5 requirement on Windows. Another thing that's nice about tests like this is that it makes it trivial to see how to reproduce a failure. It's currently very hard to debug failures because you have to first figure out where in the test it's failing (i.e. what line of python), then attach to the python executable and try to get a breakpoint on the native code side in the right SB API call matching up with the place where you determined the test is failing. This is really a pain without a debugger that supports mixed<->native transitions between python and c++, and even with a debugger that does support it like we have on Windows, it often doesn't work very well or exhibits flakiness. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
How different is that really from
(lldb) script lldb.frame.FindVariable("argc").GetValue()
'1'
(lldb) script lldb.process.GetNumThreads()
1
(lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
3514809
? If it's developer-only, then this is even fairly well-documented using e.g.
"script dir(lldb.process)"
> On Sep 14, 2016, at 3:43 PM, Zachary Turner wrote:
>
> (lldb) print-dev stack-frame[0].params[argc]
> 3
> (lldb) print-dev threads.count
> 7
> (lldb) print-dev --hex threads[6].id
> 0x1234
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added a comment. In https://reviews.llvm.org/D24591#543242, @jingham wrote: > Writing tests this way means we're going back to testing the command line > commands. That was what gdb did for the most part, and you ended up > terrified of making changes that might effect command output, not because the > change was hard but because knew you would have to go waste a day or your > life making annoying fixes to a whole bunch of tests. This happened in lldb > with the command line tests we did have, specifically with the breakpoint > reporting. So I had to go make functions for setting breakpoints and making > sure they got set correctly - and change all the tests to use them - so I > could change just one place when I added output to breakpoint reporting. > > We made a choice a while ago to favor tests using the SB API's for this > reason. Is the virtue of lit sufficient that we really want to go back on > this decision? I had been thinking about this the other day as well. An example that springs to mind that I personally encountered is when a test on Windows was trying to verify that the value of argc was 3. So it was trying to match "argc=3" in stack trace. Seems reasonable, except that the code was actually completely broken on Windows, and argc was pointing to junk memory, which was something like "argc=3239082340982", and the test was passing because this value happened to start with a 3. One idea might be to add a developer command to LLDB that is able to drill down much deeper to return specific values of interest instead of large blocks of text. For example, imagine commands like this: (lldb) print-dev stack-frame[0].params[argc] 3 (lldb) print-dev threads.count 7 (lldb) print-dev --hex threads[6].id 0x1234 Now imagine you can get every little nook and cranny of the process's state through a syntax such as this. As you know I've been a big supporter of testing the API since the beginning for the same reason you mention -- that it makes people too afraid to change the output format -- but to answer your question: I do think the virtue of lit is high enough that we find a way to solve the problem. The above idea is one possibility for reducing the fragility of command line tests, but if we brainstorm we can probably come up with others as well. It's certainly a lot of work, but I do think the benefits are worth it. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
jingham added a comment. Writing tests this way means we're going back to testing the command line commands. That was what gdb did for the most part, and you ended up terrified of making changes that might effect command output, not because the change was hard but because knew you would have to go waste a day or your life making annoying fixes to a whole bunch of tests. This happened in lldb with the command line tests we did have, specifically with the breakpoint reporting. So I had to go make functions for setting breakpoints and making sure they got set correctly - and change all the tests to use them - so I could change just one place when I added output to breakpoint reporting. We made a choice a while ago to favor tests using the SB API's for this reason. Is the virtue of lit sufficient that we really want to go back on this decision? https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
granata.enrico added a subscriber: granata.enrico. granata.enrico added a comment. One piece of concern that I have is the multiplication of things that "testing LLDB" means. We have unit tests, Python tests, and now LIT tests. That means, for each change I want to commit, if I want confidence that I am not breaking anything (which seems like a thing one could conceivably want :-) now I need to know and perform three unrelated incantations instead of one. If that is the temporary state of affairs while we transition to a better new world, then so be it. But I would definitely want for there to eventually be one true incantation that runs all tests, no matter their flavor. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
beanz updated this revision to Diff 71452.
beanz added a comment.
Added lldb-server as a test dependency based on feedback from @tfiala.
https://reviews.llvm.org/D24591
Files:
lit/CMakeLists.txt
lit/Expr/Inputs/anonymous-struct.cpp
lit/Expr/Inputs/call-function.cpp
lit/Expr/TestCallStdStringFunction.test
lit/Expr/TestCallStopAndContinue.test
lit/Expr/TestCallUserAnonTypedef.test
lit/Expr/TestCallUserDefinedFunction.test
lit/Expr/lit.local.cfg
lit/Unit/lit.cfg
lit/Unit/lit.site.cfg.in
lit/lit.cfg
lit/lit.site.cfg.in
Index: lit/lit.site.cfg.in
===
--- lit/lit.site.cfg.in
+++ lit/lit.site.cfg.in
@@ -8,6 +8,12 @@
config.lldb_obj_root = "@LLDB_BINARY_DIR@"
config.target_triple = "@TARGET_TRIPLE@"
config.python_executable = "@PYTHON_EXECUTABLE@"
+config.host_cc = "@CMAKE_C_COMPILER@"
+config.host_cxx = "@CMAKE_CXX_COMPILER@"
+
+test_clang = "@LLDB_TEST_CLANG@".lower()
+
+config.test_clang = test_clang == "on" or test_clang == "true" or test_clang == "1"
# Support substitution of the tools and libs dirs with user parameters. This is
# used when we can't determine the tool dir at configuration time.
Index: lit/lit.cfg
===
--- lit/lit.cfg
+++ lit/lit.cfg
@@ -112,17 +112,82 @@
lit_config.load_config(config, site_cfg)
raise SystemExit
+# Register substitutions
+config.substitutions.append(('%python', config.python_executable))
+config.substitutions.append(('%host_cc', config.host_cc))
+config.substitutions.append(('%host_cxx', config.host_cxx))
+
+cc = config.host_cc
+cxx = config.host_cxx
+debugserver = '%s/debugserver' % llvm_tools_dir
+
+if config.test_clang:
+cc = '%s/clang' % llvm_tools_dir
+cxx = '%s/clang++' % llvm_tools_dir
+
+if platform.system() in ['Darwin']:
+try:
+out = lit.util.capture(['xcrun', '--show-sdk-path']).strip()
+res = 0
+except OSError:
+res = -1
+if res == 0 and out:
+sdk_path = out
+lit_config.note('using SDKROOT: %r' % sdk_path)
+cxx += " -isysroot %s" % sdk_path
+debugserver = '%s/Library/Frameworks/LLDB.frameworks/Resources/debugserver' % config.llvm_obj_root
+
+
+config.substitutions.append(('%cc', cc))
+config.substitutions.append(('%cxx', cxx))
+
+config.substitutions.append(('%lldb', '%s/lldb' % llvm_tools_dir))
+config.substitutions.append(('%debugserver', debugserver))
+
+for pattern in [r"\bFileCheck\b",
+r"\| \bnot\b"]:
+tool_match = re.match(r"^(\\)?((\| )?)\W+b([0-9A-Za-z-_]+)\\b\W*$",
+ pattern)
+tool_pipe = tool_match.group(2)
+tool_name = tool_match.group(4)
+tool_path = config.llvm_tools_dir + '/' + tool_name
+if not tool_path:
+# Warn, but still provide a substitution.
+lit_config.note(
+'Did not find ' + tool_name + ' in ' + config.llvm_tools_dir)
+config.substitutions.append((pattern, tool_pipe + tool_path))
+
# Shell execution
if platform.system() not in ['Windows'] or lit_config.getBashPath() != '':
config.available_features.add('shell')
# Running on Darwin OS
if platform.system() in ['Darwin']:
+config.available_features.add('darwin')
config.available_features.add('system-linker-mach-o')
# Running on ELF based *nix
if platform.system() in ['FreeBSD', 'Linux']:
config.available_features.add('system-linker-elf')
+if platform.system() in ['FreeBSD']:
+config.available_features.add('freebsd')
+else:
+config.available_features.add('linux')
+
+if platform.system() in ['Windows']:
+config.available_features.add('windows')
+
+if re.match(r'^arm(hf.*-linux)|(.*-linux-gnuabihf)', config.target_triple):
+config.available_features.add("armhf-linux")
+
+if re.match(r'icc', cc):
+config.available_features.add("compiler-icc")
+elif re.match(r'clang', cc):
+config.available_features.add("compiler-clang")
+elif re.match(r'gcc', cc):
+config.available_features.add("compiler-gcc")
+elif re.match(r'cl', cc):
+config.available_features.add("compiler-msvc")
# llvm-config knows whether it is compiled with asserts (and)
# whether we are operating in release/debug mode.
Index: lit/Unit/lit.site.cfg.in
===
--- lit/Unit/lit.site.cfg.in
+++ lit/Unit/lit.site.cfg.in
@@ -1,5 +1,6 @@
@LIT_SITE_CFG_IN_HEADER@
+config.test_exec_root = "@LLVM_BINARY_DIR@"
config.llvm_src_root = "@LLVM_SOURCE_DIR@"
config.llvm_obj_root = "@LLVM_BINARY_DIR@"
config.llvm_tools_dir = "@LLVM_TOOLS_DIR@"
Index: lit/Unit/lit.cfg
===
--- lit/Unit/lit.cfg
+++ lit/Unit/lit.cfg
@@ -6,6 +6,19 @@
import lit.formats
+# Check that the object root is known.
+if config.test_exec_root is None:
+# Otherwise, we haven't loaded the site specific configuration (the user is
+# pro
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
tfiala added a comment. One other items we discussed that is sure to come up: - Right now this is geared towards one compile per .test file (similar to something Zachary brought up before). One way we could get the multiple debug info formats handled is to have a .test for each of the formats, but have each of them pull in the common test checks from a shared file in the same directory. We'll have to play with that. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added a comment.
Couple questions:
1. What is the status of lit and Python 3? Running the test suite on Windows
**requires** Python 3.5+, so if we want this to work on Windows, we will need
to make sure the lit infrastructure is compatible with Python 3.
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
LLDB's CMake already has an option called `LLDB_TEST_COMPILER`. Is it possible
to re-use that?
We have situations where we want to run the test suite using neither the
in-tree clang nor the host compiler, so I think we need to retain this
flexibility to specify a path to the compiler.
Comment at: lit/CMakeLists.txt:33
@@ +32,3 @@
+if(LLDB_TEST_CLANG)
+ list(APPELND LLDB_TEST_DEPS clang)
+endif()
`s/APPELND/APPEND/`
Comment at: lit/lit.cfg:125-126
@@ +124,4 @@
+if config.test_clang:
+cc = '%s/clang' % llvm_tools_dir
+cxx = '%s/clang++' % llvm_tools_dir
+
Can you use `os.path.join` instead of hardcoding a forward slash? This might
not apply though given my earlier comment about `LLDB_TEST_COMPILER`.
Comment at: lit/lit.cfg:144
@@ +143,3 @@
+
+config.substitutions.append(('%lldb', '%s/lldb' % llvm_tools_dir))
+config.substitutions.append(('%debugserver', debugserver))
`os.path.join` here, and on Windows you will need to add `.exe`
Comment at: lit/lit.cfg:189-190
@@ -126,1 +188,4 @@
+config.available_features.add("compiler-gcc")
+elif re.match(r'cl', cc):
+config.available_features.add("compiler-msvc")
I'm ok with removing this branch. Currently on Windows we require `clang.exe`
as the test compiler, and we use it in cl driver mode, so command lines are
mostly interchangeable across platforms. Having to support an entirely
different command line syntax would fragment the tests too much.
https://reviews.llvm.org/D24591
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
spyffe added a subscriber: spyffe. spyffe added a comment. I like this concept a lot, and I think it's great for testcases that actually need to interact with the command line. I'm concerned, though, that the separation of input from command files makes tests more complex to write. One thing that's very nice about the `inline` tests in LLDB is that the test and the source for the debugged program are both in the same file. If the goal is to port compiler tests across (or just to have tests be more understandable for compiler engineers), I would argue for choosing that model as the default. It would mean a slightly customized LLDB driver but that wouldn't be insanely hard. (We've already done it in Python!) https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
tfiala accepted this revision. tfiala added a comment. This revision is now accepted and ready to land. Assuming this doesn't break anybody, this LGTM. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
tfiala added inline comments. Comment at: lit/CMakeLists.txt:14 @@ -13,1 +13,3 @@ +option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off) + tfiala wrote: > The macOS bots (and all the Swift ones) build with in-tree clang. I'm glad > to have this option. > The macOS bots (and all the Swift ones) build with in-tree clang. I'm glad to > have this option. ... test with the in-tree clang, rather. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
tfiala added a comment.
Looks like a nice start. Thanks for pulling this together, Chris!
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
The macOS bots (and all the Swift ones) build with in-tree clang. I'm glad to
have this option.
Comment at: lit/CMakeLists.txt:26
@@ -23,1 +25,3 @@
+ FileCheck
+ debugserver
LLDBUnitTests
Not entirely sure what exactly this is checking, but for non-Apple platforms
(and maybe Apple in the future), we'll be using an lldb-server binary rather
than debugserver. Anything we need to do here to handle that?
Comment at: lit/lit.cfg:190
@@ -126,1 +189,3 @@
+elif re.match(r'cl', cc):
+config.available_features.add("compiler-msvc")
As we discussed earlier, it'll be helpful to be able to parse out the compiler
versions so we can do some kind of conditional check on specific version or
version ranges for known bugs (or features).
https://reviews.llvm.org/D24591
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
Re: [Lldb-commits] [PATCH] D24591: [LIT] First pass of LLDB LIT support
zturner added a comment. Huh... This is a surprise. Looking forward to reviewing this. https://reviews.llvm.org/D24591 ___ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
