Re: [lldb-dev] skip some tests with "check-lldb"
When I am working on some changes and there are tests I know I've broken but don't want to fix just yet, I just edit the test file and skip it using the skip decorators. You have to remember not to check in the skips when you check in whatever change you are making - but presumably by the time you are ready to submit something you will have fixed the tests anyway. That's much simpler than trying to monkey with the test runner to do your bidding. Jim > On Sep 20, 2018, at 5:25 AM, Gábor Márton via lldb-dev > wrote: > > Hi Jonas, > > Thanks for the clarification. > Finally I could skip some of the tests by using a negative lookahead > regex passed to --filter. I wouldn't say that this is so convenient > and easy to overview, but works. > For example, to skip the TestCalculatorMode.py and TestPrintArray.py > tests I used the following argument (appended to the normal lit > command line reported by "ninja check-lldb -v" ) : > --filter '^((?!(TestCalculatorMode\.py|TestPrintArray\.py)).)*$' > Would be beneficial though to have a --exclude besides --filter to > avoid using a complex negative lookahead regex. > > Cheers, > Gabor > On Thu, Sep 20, 2018 at 12:53 PM Jonas Devlieghere > wrote: >> >> When using lit as the driver (which is the case for check-lldb), the test >> granularity is at file-level: lit invokes dotest.py for every test file. You >> should be able to specify files to skip with lit's --filter. >> >> With regards to threading, lit will schedule one instance of dotest.py on >> every >> thread, which processes one test file. Unless you passed specific options for >> dotest.py, the latter will run the different tests within that one file on >> different threads as well. (IIRC) >> >> On Sep 19, 2018, at 7:20 PM, Gábor Márton via lldb-dev >> wrote: >> >> That's okay, but is it possible to skip a few tests, when using lit? I was >> thinking about moving the test files I want to skip, but that has obvious >> drawbacks. Also --filter does not seem so useful in this case. >> >> On Wed, 19 Sep 2018, 18:46 , wrote: >>> >>> Unless you pass --no-multiprocess to dotest, it should detect how many >>> cores your system has and use them. >>> >>> -- >>> Ted Woodward >>> Qualcomm Innovation Center, Inc. >>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux >>> Foundation Collaborative Project >>> >>> -Original Message- >>> From: lldb-dev On Behalf Of Gábor Márton >>> via lldb-dev >>> Sent: Wednesday, September 19, 2018 11:04 AM >>> To: lldb-dev@lists.llvm.org >>> Subject: Re: [lldb-dev] skip some tests with "check-lldb" >>> >>> I just realized that `dotest.py` has a --thread option. Is that the one >>> which is used during the lit test (`ninja check-lldb`) ? >>> >>> On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton wrote: >>>> >>>> Hi, >>>> >>>> I'd like to skip some tests when I run "ninja check-lldb", because they >>>> fail. >>>> I am on release_70 branch. >>>> I know I could use dotest.py directly, but that would exercise only one >>>> thread. >>>> Is there a way to execute the tests parallel on all cores and in the >>>> same time skip some of the tests? >>>> >>>> Thanks, >>>> Gabor >>> ___ >>> lldb-dev mailing list >>> lldb-dev@lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>> >> ___ >> lldb-dev mailing list >> lldb-dev@lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >> >> > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] skip some tests with "check-lldb"
> On Sep 20, 2018, at 2:25 PM, Gábor Márton wrote: > > Hi Jonas, > > Thanks for the clarification. > Finally I could skip some of the tests by using a negative lookahead > regex passed to --filter. I wouldn't say that this is so convenient > and easy to overview, but works. > For example, to skip the TestCalculatorMode.py and TestPrintArray.py > tests I used the following argument (appended to the normal lit > command line reported by "ninja check-lldb -v" ) : > --filter '^((?!(TestCalculatorMode\.py|TestPrintArray\.py)).)*$' > Would be beneficial though to have a --exclude besides --filter to > avoid using a complex negative lookahead regex. I have to agree that this looks overly complex. Maybe it's something to bring up on the llvm-dev mailing list as a general improvement for lit? > Cheers, > Gabor > On Thu, Sep 20, 2018 at 12:53 PM Jonas Devlieghere > wrote: >> >> When using lit as the driver (which is the case for check-lldb), the test >> granularity is at file-level: lit invokes dotest.py for every test file. You >> should be able to specify files to skip with lit's --filter. >> >> With regards to threading, lit will schedule one instance of dotest.py on >> every >> thread, which processes one test file. Unless you passed specific options for >> dotest.py, the latter will run the different tests within that one file on >> different threads as well. (IIRC) >> >> On Sep 19, 2018, at 7:20 PM, Gábor Márton via lldb-dev >> wrote: >> >> That's okay, but is it possible to skip a few tests, when using lit? I was >> thinking about moving the test files I want to skip, but that has obvious >> drawbacks. Also --filter does not seem so useful in this case. >> >> On Wed, 19 Sep 2018, 18:46 , wrote: >>> >>> Unless you pass --no-multiprocess to dotest, it should detect how many >>> cores your system has and use them. >>> >>> -- >>> Ted Woodward >>> Qualcomm Innovation Center, Inc. >>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux >>> Foundation Collaborative Project >>> >>> -Original Message- >>> From: lldb-dev On Behalf Of Gábor Márton >>> via lldb-dev >>> Sent: Wednesday, September 19, 2018 11:04 AM >>> To: lldb-dev@lists.llvm.org >>> Subject: Re: [lldb-dev] skip some tests with "check-lldb" >>> >>> I just realized that `dotest.py` has a --thread option. Is that the one >>> which is used during the lit test (`ninja check-lldb`) ? >>> >>> On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton wrote: >>>> >>>> Hi, >>>> >>>> I'd like to skip some tests when I run "ninja check-lldb", because they >>>> fail. >>>> I am on release_70 branch. >>>> I know I could use dotest.py directly, but that would exercise only one >>>> thread. >>>> Is there a way to execute the tests parallel on all cores and in the >>>> same time skip some of the tests? >>>> >>>> Thanks, >>>> Gabor >>> ___ >>> lldb-dev mailing list >>> lldb-dev@lists.llvm.org >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>> >> ___ >> lldb-dev mailing list >> lldb-dev@lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >> >> ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] skip some tests with "check-lldb"
Hi Jonas, Thanks for the clarification. Finally I could skip some of the tests by using a negative lookahead regex passed to --filter. I wouldn't say that this is so convenient and easy to overview, but works. For example, to skip the TestCalculatorMode.py and TestPrintArray.py tests I used the following argument (appended to the normal lit command line reported by "ninja check-lldb -v" ) : --filter '^((?!(TestCalculatorMode\.py|TestPrintArray\.py)).)*$' Would be beneficial though to have a --exclude besides --filter to avoid using a complex negative lookahead regex. Cheers, Gabor On Thu, Sep 20, 2018 at 12:53 PM Jonas Devlieghere wrote: > > When using lit as the driver (which is the case for check-lldb), the test > granularity is at file-level: lit invokes dotest.py for every test file. You > should be able to specify files to skip with lit's --filter. > > With regards to threading, lit will schedule one instance of dotest.py on > every > thread, which processes one test file. Unless you passed specific options for > dotest.py, the latter will run the different tests within that one file on > different threads as well. (IIRC) > > On Sep 19, 2018, at 7:20 PM, Gábor Márton via lldb-dev > wrote: > > That's okay, but is it possible to skip a few tests, when using lit? I was > thinking about moving the test files I want to skip, but that has obvious > drawbacks. Also --filter does not seem so useful in this case. > > On Wed, 19 Sep 2018, 18:46 , wrote: >> >> Unless you pass --no-multiprocess to dotest, it should detect how many >> cores your system has and use them. >> >> -- >> Ted Woodward >> Qualcomm Innovation Center, Inc. >> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux >> Foundation Collaborative Project >> >> -Original Message- >> From: lldb-dev On Behalf Of Gábor Márton >> via lldb-dev >> Sent: Wednesday, September 19, 2018 11:04 AM >> To: lldb-dev@lists.llvm.org >> Subject: Re: [lldb-dev] skip some tests with "check-lldb" >> >> I just realized that `dotest.py` has a --thread option. Is that the one >> which is used during the lit test (`ninja check-lldb`) ? >> >> On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton wrote: >> > >> > Hi, >> > >> > I'd like to skip some tests when I run "ninja check-lldb", because they >> > fail. >> > I am on release_70 branch. >> > I know I could use dotest.py directly, but that would exercise only one >> > thread. >> > Is there a way to execute the tests parallel on all cores and in the >> > same time skip some of the tests? >> > >> > Thanks, >> > Gabor >> ___ >> lldb-dev mailing list >> lldb-dev@lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >> > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > > ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] skip some tests with "check-lldb"
When using lit as the driver (which is the case for check-lldb), the test granularity is at file-level: lit invokes dotest.py for every test file. You should be able to specify files to skip with lit's --filter. With regards to threading, lit will schedule one instance of dotest.py on every thread, which processes one test file. Unless you passed specific options for dotest.py, the latter will run the different tests within that one file on different threads as well. (IIRC) > On Sep 19, 2018, at 7:20 PM, Gábor Márton via lldb-dev > wrote: > > That's okay, but is it possible to skip a few tests, when using lit? I was > thinking about moving the test files I want to skip, but that has obvious > drawbacks. Also --filter does not seem so useful in this case. > > On Wed, 19 Sep 2018, 18:46 , <mailto:ted.woodw...@codeaurora.org>> wrote: > Unless you pass --no-multiprocess to dotest, it should detect how many cores > your system has and use them. > > -- > Ted Woodward > Qualcomm Innovation Center, Inc. > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux > Foundation Collaborative Project > > -Original Message- > From: lldb-dev <mailto:lldb-dev-boun...@lists.llvm.org>> On Behalf Of Gábor Márton via > lldb-dev > Sent: Wednesday, September 19, 2018 11:04 AM > To: lldb-dev@lists.llvm.org <mailto:lldb-dev@lists.llvm.org> > Subject: Re: [lldb-dev] skip some tests with "check-lldb" > > I just realized that `dotest.py` has a --thread option. Is that the one which > is used during the lit test (`ninja check-lldb`) ? > > On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton <mailto:martongab...@gmail.com>> wrote: > > > > Hi, > > > > I'd like to skip some tests when I run "ninja check-lldb", because they > > fail. > > I am on release_70 branch. > > I know I could use dotest.py directly, but that would exercise only one > > thread. > > Is there a way to execute the tests parallel on all cores and in the > > same time skip some of the tests? > > > > Thanks, > > Gabor > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org <mailto:lldb-dev@lists.llvm.org> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > <http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev> > > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] skip some tests with "check-lldb"
That's okay, but is it possible to skip a few tests, when using lit? I was thinking about moving the test files I want to skip, but that has obvious drawbacks. Also --filter does not seem so useful in this case. On Wed, 19 Sep 2018, 18:46 , wrote: > Unless you pass --no-multiprocess to dotest, it should detect how many > cores your system has and use them. > > -- > Ted Woodward > Qualcomm Innovation Center, Inc. > Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux > Foundation Collaborative Project > > -Original Message- > From: lldb-dev On Behalf Of Gábor > Márton via lldb-dev > Sent: Wednesday, September 19, 2018 11:04 AM > To: lldb-dev@lists.llvm.org > Subject: Re: [lldb-dev] skip some tests with "check-lldb" > > I just realized that `dotest.py` has a --thread option. Is that the one > which is used during the lit test (`ninja check-lldb`) ? > > On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton > wrote: > > > > Hi, > > > > I'd like to skip some tests when I run "ninja check-lldb", because they > fail. > > I am on release_70 branch. > > I know I could use dotest.py directly, but that would exercise only one > thread. > > Is there a way to execute the tests parallel on all cores and in the > > same time skip some of the tests? > > > > Thanks, > > Gabor > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > > ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] skip some tests with "check-lldb"
Unless you pass --no-multiprocess to dotest, it should detect how many cores your system has and use them. -- Ted Woodward Qualcomm Innovation Center, Inc. Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project -Original Message- From: lldb-dev On Behalf Of Gábor Márton via lldb-dev Sent: Wednesday, September 19, 2018 11:04 AM To: lldb-dev@lists.llvm.org Subject: Re: [lldb-dev] skip some tests with "check-lldb" I just realized that `dotest.py` has a --thread option. Is that the one which is used during the lit test (`ninja check-lldb`) ? On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton wrote: > > Hi, > > I'd like to skip some tests when I run "ninja check-lldb", because they fail. > I am on release_70 branch. > I know I could use dotest.py directly, but that would exercise only one > thread. > Is there a way to execute the tests parallel on all cores and in the > same time skip some of the tests? > > Thanks, > Gabor ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] skip some tests with "check-lldb"
I just realized that `dotest.py` has a --thread option. Is that the one which is used during the lit test (`ninja check-lldb`) ? On Wed, Sep 19, 2018 at 6:00 PM Gábor Márton wrote: > > Hi, > > I'd like to skip some tests when I run "ninja check-lldb", because they fail. > I am on release_70 branch. > I know I could use dotest.py directly, but that would exercise only one > thread. > Is there a way to execute the tests parallel on all cores and in the > same time skip some of the tests? > > Thanks, > Gabor ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] skip some tests with "check-lldb"
Hi, I'd like to skip some tests when I run "ninja check-lldb", because they fail. I am on release_70 branch. I know I could use dotest.py directly, but that would exercise only one thread. Is there a way to execute the tests parallel on all cores and in the same time skip some of the tests? Thanks, Gabor ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
