Re: [gem5-dev] [EXT] Re: testing framework

[Gabe Black]

Also, it would be nice if the unit tests were more unit-ey and didn't
require linking in all of gem5 or even building all of gem5 to run. I don't
know how feasible that is in reality though. I think that's normally done
by stubbing out the external stuff they'd call into with mocks. That would
make them a lot quicker to build, and more self contained so that you'd
know a failure of xyztest meant xyz was broken, not that some other thing
it depended on wasn't behaving properly. This is a bit of a learning
experience for me too, so I'm working my way through this as a I go to some
extent.

Gabe

On Wed, Nov 22, 2017 at 12:10 PM, Gabe Black <gabebl...@google.com> wrote:

> Declaring unit tests in the SConscript file just tells scons they exist,
> but if you don't ask it to build them then it won't since they aren't a
> dependency of, for example, gem5.opt. It's annoying to tell it to just
> build all the unit tests since you could do that based on directory, and as
> discussed in another thread that would build all 5 versions of each.
> Otherwise you have to ask scons to build each unit test binary by name
> which is very cumbersome. I'm actively thinking about and working on making
> the unit tests work and easy (or even mandatory) to run, so hopefully this
> will get a lot better soon.
>
> Gabe
>
> On Wed, Nov 22, 2017 at 9:20 AM, Potter, Brandon <brandon.pot...@amd.com>
> wrote:
>
>> Is there any documentation on how to run the current unit tests? I notice
>> that there's a unit test directory in the X86 build folder, but nothing
>> inside is executable. I do not see a way of generating output from SCons
>> related to unittests either.
>>
>> From the src/SConscript file, I see that the unittests should get created
>> inside the makeEnv function, but it doesn't appear that this does anything
>> (at least for X86).
>>
>> -Brandon
>>
>> -Original Message-
>> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas
>> Sandberg
>> Sent: Wednesday, November 22, 2017 3:35 AM
>> To: gem5 Developer List <gem5-dev@gem5.org>; Gabe Black <
>> gabebl...@google.com>
>> Subject: Re: [gem5-dev] [EXT] Re: testing framework
>>
>> One issue that is preventing me from running the unit tests is that they
>> require separate scons targets. One solution that I was considering at one
>> point was to make the unit tests available as Python exports in the main
>> build. This would allow us to run a magic config that calls unit tests
>> instead of running the simulator proper. I think it would work for
>> all(?) current unit tests, but I don't know how it would interact with
>> other test unit test frameworks.
>>
>> Cheers,
>> Andreas
>>
>> On 21/11/2017 01:37, Gabe Black wrote:
>> > What I would minimally like to see is for people to run the unit tests
>> > that already exist. Clearly that hasn't been happening since one of
>> > them hasn't compiled since 2014 (now fixed). We need our tests to test
>> > something specific, say whether or not it worked (not leave
>> > interpretation up to the reader), and for people to actually run them.
>> >
>> > It's not at all clear what to do to actually build the unit tests from
>> > scons, or how to run them, or what their results mean. I'm going to
>> > try cleaning these up a bit. Also, they shouldn't be something that
>> > lives off on the side or in their own directory, out of sight and out
>> > of mind, they should be alongside the code they test.
>> >
>> > It might even make sense to associate a unit test with a particular
>> > source file and make compilation fail if the unit test doesn't pass.
>> > That might be a bit overboard, but would at least ensure that the tests
>> get run.
>> >
>> > Gabe
>> >
>> > On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon
>> > <brandon.pot...@amd.com>
>> > wrote:
>> >
>> >> I also support adding unit tests (and possibly a code coverage
>> >> checker to identify weak points). Ideally, we'd add unit tests for
>> >> the entire simulator, but we should probably put something into place
>> >> for new code requiring some type of unit tests. Otherwise, only a
>> >> subset of the submissions will contain unit tests.
>> >>
>> >> Regards,
>> >> Brandon
>> >>
>> >> -Original Message-
>> >> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Jason
>> >> Lowe-Power
>> >> Sent: Monday, November 20, 2017 11:22 AM

Re: [gem5-dev] [EXT] Re: testing framework

[Gabe Black]

Declaring unit tests in the SConscript file just tells scons they exist,
but if you don't ask it to build them then it won't since they aren't a
dependency of, for example, gem5.opt. It's annoying to tell it to just
build all the unit tests since you could do that based on directory, and as
discussed in another thread that would build all 5 versions of each.
Otherwise you have to ask scons to build each unit test binary by name
which is very cumbersome. I'm actively thinking about and working on making
the unit tests work and easy (or even mandatory) to run, so hopefully this
will get a lot better soon.

Gabe

On Wed, Nov 22, 2017 at 9:20 AM, Potter, Brandon <brandon.pot...@amd.com>
wrote:

> Is there any documentation on how to run the current unit tests? I notice
> that there's a unit test directory in the X86 build folder, but nothing
> inside is executable. I do not see a way of generating output from SCons
> related to unittests either.
>
> From the src/SConscript file, I see that the unittests should get created
> inside the makeEnv function, but it doesn't appear that this does anything
> (at least for X86).
>
> -Brandon
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas
> Sandberg
> Sent: Wednesday, November 22, 2017 3:35 AM
> To: gem5 Developer List <gem5-dev@gem5.org>; Gabe Black <
> gabebl...@google.com>
> Subject: Re: [gem5-dev] [EXT] Re: testing framework
>
> One issue that is preventing me from running the unit tests is that they
> require separate scons targets. One solution that I was considering at one
> point was to make the unit tests available as Python exports in the main
> build. This would allow us to run a magic config that calls unit tests
> instead of running the simulator proper. I think it would work for
> all(?) current unit tests, but I don't know how it would interact with
> other test unit test frameworks.
>
> Cheers,
> Andreas
>
> On 21/11/2017 01:37, Gabe Black wrote:
> > What I would minimally like to see is for people to run the unit tests
> > that already exist. Clearly that hasn't been happening since one of
> > them hasn't compiled since 2014 (now fixed). We need our tests to test
> > something specific, say whether or not it worked (not leave
> > interpretation up to the reader), and for people to actually run them.
> >
> > It's not at all clear what to do to actually build the unit tests from
> > scons, or how to run them, or what their results mean. I'm going to
> > try cleaning these up a bit. Also, they shouldn't be something that
> > lives off on the side or in their own directory, out of sight and out
> > of mind, they should be alongside the code they test.
> >
> > It might even make sense to associate a unit test with a particular
> > source file and make compilation fail if the unit test doesn't pass.
> > That might be a bit overboard, but would at least ensure that the tests
> get run.
> >
> > Gabe
> >
> > On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon
> > <brandon.pot...@amd.com>
> > wrote:
> >
> >> I also support adding unit tests (and possibly a code coverage
> >> checker to identify weak points). Ideally, we'd add unit tests for
> >> the entire simulator, but we should probably put something into place
> >> for new code requiring some type of unit tests. Otherwise, only a
> >> subset of the submissions will contain unit tests.
> >>
> >> Regards,
> >> Brandon
> >>
> >> -Original Message-
> >> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Jason
> >> Lowe-Power
> >> Sent: Monday, November 20, 2017 11:22 AM
> >> To: gem5 Developer List <gem5-dev@gem5.org>
> >> Subject: Re: [gem5-dev] [EXT] Re: testing framework
> >>
> >> I'm all for unit tests. GTest seems fine, but I'm not a huge fan of
> >> its syntax. Although, that shouldn't stop us from having unit tests!!
> >>
> >> What's the state of getting whimsy pushed in? From my side, it's
> >> ready to go. The only thing on my to do list that I would have liked
> >> to finish up before pushing is better integration with the current test
> infrastructure.
> >> It would be good for whoever is running with the old test
> >> infrastructure to be able to run both the old-style and new-style tests
> with one command.
> >> Unfortunately, I don't have the time to dig into this for the
> >> foreseeable future.
> >>
> >> Jason
> >>
> >> On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
> >

Re: [gem5-dev] [EXT] Re: testing framework

[Potter, Brandon]

Is there any documentation on how to run the current unit tests? I notice that 
there's a unit test directory in the X86 build folder, but nothing inside is 
executable. I do not see a way of generating output from SCons related to 
unittests either.

From the src/SConscript file, I see that the unittests should get created 
inside the makeEnv function, but it doesn't appear that this does anything (at 
least for X86).

-Brandon

-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Andreas Sandberg
Sent: Wednesday, November 22, 2017 3:35 AM
To: gem5 Developer List <gem5-dev@gem5.org>; Gabe Black <gabebl...@google.com>
Subject: Re: [gem5-dev] [EXT] Re: testing framework

One issue that is preventing me from running the unit tests is that they 
require separate scons targets. One solution that I was considering at one 
point was to make the unit tests available as Python exports in the main build. 
This would allow us to run a magic config that calls unit tests instead of 
running the simulator proper. I think it would work for
all(?) current unit tests, but I don't know how it would interact with other 
test unit test frameworks.

Cheers,
Andreas

On 21/11/2017 01:37, Gabe Black wrote:
> What I would minimally like to see is for people to run the unit tests 
> that already exist. Clearly that hasn't been happening since one of 
> them hasn't compiled since 2014 (now fixed). We need our tests to test 
> something specific, say whether or not it worked (not leave 
> interpretation up to the reader), and for people to actually run them.
>
> It's not at all clear what to do to actually build the unit tests from 
> scons, or how to run them, or what their results mean. I'm going to 
> try cleaning these up a bit. Also, they shouldn't be something that 
> lives off on the side or in their own directory, out of sight and out 
> of mind, they should be alongside the code they test.
>
> It might even make sense to associate a unit test with a particular 
> source file and make compilation fail if the unit test doesn't pass. 
> That might be a bit overboard, but would at least ensure that the tests get 
> run.
>
> Gabe
>
> On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon 
> <brandon.pot...@amd.com>
> wrote:
>
>> I also support adding unit tests (and possibly a code coverage 
>> checker to identify weak points). Ideally, we'd add unit tests for 
>> the entire simulator, but we should probably put something into place 
>> for new code requiring some type of unit tests. Otherwise, only a 
>> subset of the submissions will contain unit tests.
>>
>> Regards,
>> Brandon
>>
>> -Original Message-
>> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Jason 
>> Lowe-Power
>> Sent: Monday, November 20, 2017 11:22 AM
>> To: gem5 Developer List <gem5-dev@gem5.org>
>> Subject: Re: [gem5-dev] [EXT] Re: testing framework
>>
>> I'm all for unit tests. GTest seems fine, but I'm not a huge fan of 
>> its syntax. Although, that shouldn't stop us from having unit tests!!
>>
>> What's the state of getting whimsy pushed in? From my side, it's 
>> ready to go. The only thing on my to do list that I would have liked 
>> to finish up before pushing is better integration with the current test 
>> infrastructure.
>> It would be good for whoever is running with the old test 
>> infrastructure to be able to run both the old-style and new-style tests with 
>> one command.
>> Unfortunately, I don't have the time to dig into this for the 
>> foreseeable future.
>>
>> Jason
>>
>> On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) < 
>> prosenf...@micron.com> wrote:
>>
>>> Even without the complexities of setting up SystemC+gem5 it's pretty 
>>> painful to "unit" test SystemC because the SystemC runtime has a lot 
>>> of global state that comes along with it (for example: you can't "uncall"
>>> sc_start() to go back and re-do elaboration with different modules).
>>> The only (not very elegant) workaround we have discussed for unit 
>>> testing SystemC modules is to just to build each set of related 
>>> SystemC unit tests into their own independent GTest binaries.
>>>
>>> On the whole, though, GTest is pretty great.
>>>
>>> -Original Message-
>>> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe 
>>> Black
>>> Sent: Friday, November 17, 2017 5:51 PM
>>> To: Matthias Jung <jun...@eit.uni-kl.de>
>>> Cc: gem5 Developer List <gem5-dev@gem5.org>
>>> Subject: [EXT] Re: [gem5-dev] 

Re: [gem5-dev] [EXT] Re: testing framework

[Andreas Sandberg]

One issue that is preventing me from running the unit tests is that they
require separate scons targets. One solution that I was considering at
one point was to make the unit tests available as Python exports in the
main build. This would allow us to run a magic config that calls unit
tests instead of running the simulator proper. I think it would work for
all(?) current unit tests, but I don't know how it would interact with
other test unit test frameworks.

Cheers,
Andreas

On 21/11/2017 01:37, Gabe Black wrote:

What I would minimally like to see is for people to run the unit tests that
already exist. Clearly that hasn't been happening since one of them hasn't
compiled since 2014 (now fixed). We need our tests to test something
specific, say whether or not it worked (not leave interpretation up to the
reader), and for people to actually run them.

It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to try
cleaning these up a bit. Also, they shouldn't be something that lives off
on the side or in their own directory, out of sight and out of mind, they
should be alongside the code they test.

It might even make sense to associate a unit test with a particular source
file and make compilation fail if the unit test doesn't pass. That might be
a bit overboard, but would at least ensure that the tests get run.

Gabe

On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon <brandon.pot...@amd.com>
wrote:


I also support adding unit tests (and possibly a code coverage checker to
identify weak points). Ideally, we'd add unit tests for the entire
simulator, but we should probably put something into place for new code
requiring some type of unit tests. Otherwise, only a subset of the
submissions will contain unit tests.

Regards,
Brandon

-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Jason
Lowe-Power
Sent: Monday, November 20, 2017 11:22 AM
To: gem5 Developer List <gem5-dev@gem5.org>
Subject: Re: [gem5-dev] [EXT] Re: testing framework

I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its
syntax. Although, that shouldn't stop us from having unit tests!!

What's the state of getting whimsy pushed in? From my side, it's ready to
go. The only thing on my to do list that I would have liked to finish up
before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure
to be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable
future.

Jason

On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
prosenf...@micron.com> wrote:


Even without the complexities of setting up SystemC+gem5 it's pretty
painful to "unit" test SystemC because the SystemC runtime has a lot
of global state that comes along with it (for example: you can't "uncall"
sc_start() to go back and re-do elaboration with different modules).
The only (not very elegant) workaround we have discussed for unit
testing SystemC modules is to just to build each set of related
SystemC unit tests into their own independent GTest binaries.

On the whole, though, GTest is pretty great.

-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
Black
Sent: Friday, November 17, 2017 5:51 PM
To: Matthias Jung <jun...@eit.uni-kl.de>
Cc: gem5 Developer List <gem5-dev@gem5.org>
Subject: [EXT] Re: [gem5-dev] testing framework

As far as a high level framework, I think whimsy should help there
since it's a bit more decoupled and will let you run arbitrary things
as tests, at least as far as I can tell. Since the systemC integration
is just that, an integration of two systems, I don't think unit tests
would be as applicable generally speaking. You could have one for some
things though, like the ports that translate between gem5's and

systemC's memory protocol.

GTest claims to be BSD, so the license should be compatible.

Gabe

On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung <jun...@eit.uni-kl.de>
wrote:


Hey,

It would be also nice if we have something that we could also use
for test the SystemC stuff. Right now this is not possible with the
current framework. Because we have to build gem5 without python as a
library for the SystemC stuff... and then build again with linking
of

the gem5 library.

Regards
Matthias


Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <

andreas.sandb...@arm.com>:

I'm generally in favour of switching to an existing unit test

framework.

I haven't surveyed the C++ test framework landscape, but judging
by the users (Chromium, LLVM, ...) GTest seems pretty solid.

As far as I can tell, GTest is all BSD. Is that correct?

Cheers,
Andreas


On 16/11/2017 05:04, Gabe Black wrote:
Hi folks. How is the test framework integration co

Re: [gem5-dev] [EXT] Re: testing framework

[Gabe Black]

What I would minimally like to see is for people to run the unit tests that
already exist. Clearly that hasn't been happening since one of them hasn't
compiled since 2014 (now fixed). We need our tests to test something
specific, say whether or not it worked (not leave interpretation up to the
reader), and for people to actually run them.

It's not at all clear what to do to actually build the unit tests from
scons, or how to run them, or what their results mean. I'm going to try
cleaning these up a bit. Also, they shouldn't be something that lives off
on the side or in their own directory, out of sight and out of mind, they
should be alongside the code they test.

It might even make sense to associate a unit test with a particular source
file and make compilation fail if the unit test doesn't pass. That might be
a bit overboard, but would at least ensure that the tests get run.

Gabe

On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon <brandon.pot...@amd.com>
wrote:

> I also support adding unit tests (and possibly a code coverage checker to
> identify weak points). Ideally, we'd add unit tests for the entire
> simulator, but we should probably put something into place for new code
> requiring some type of unit tests. Otherwise, only a subset of the
> submissions will contain unit tests.
>
> Regards,
> Brandon
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Jason
> Lowe-Power
> Sent: Monday, November 20, 2017 11:22 AM
> To: gem5 Developer List <gem5-dev@gem5.org>
> Subject: Re: [gem5-dev] [EXT] Re: testing framework
>
> I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its
> syntax. Although, that shouldn't stop us from having unit tests!!
>
> What's the state of getting whimsy pushed in? From my side, it's ready to
> go. The only thing on my to do list that I would have liked to finish up
> before pushing is better integration with the current test infrastructure.
> It would be good for whoever is running with the old test infrastructure
> to be able to run both the old-style and new-style tests with one command.
> Unfortunately, I don't have the time to dig into this for the foreseeable
> future.
>
> Jason
>
> On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
> prosenf...@micron.com> wrote:
>
> > Even without the complexities of setting up SystemC+gem5 it's pretty
> > painful to "unit" test SystemC because the SystemC runtime has a lot
> > of global state that comes along with it (for example: you can't "uncall"
> > sc_start() to go back and re-do elaboration with different modules).
> > The only (not very elegant) workaround we have discussed for unit
> > testing SystemC modules is to just to build each set of related
> > SystemC unit tests into their own independent GTest binaries.
> >
> > On the whole, though, GTest is pretty great.
> >
> > -Original Message-
> > From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe
> > Black
> > Sent: Friday, November 17, 2017 5:51 PM
> > To: Matthias Jung <jun...@eit.uni-kl.de>
> > Cc: gem5 Developer List <gem5-dev@gem5.org>
> > Subject: [EXT] Re: [gem5-dev] testing framework
> >
> > As far as a high level framework, I think whimsy should help there
> > since it's a bit more decoupled and will let you run arbitrary things
> > as tests, at least as far as I can tell. Since the systemC integration
> > is just that, an integration of two systems, I don't think unit tests
> > would be as applicable generally speaking. You could have one for some
> > things though, like the ports that translate between gem5's and
> systemC's memory protocol.
> >
> > GTest claims to be BSD, so the license should be compatible.
> >
> > Gabe
> >
> > On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung <jun...@eit.uni-kl.de>
> > wrote:
> >
> > > Hey,
> > >
> > > It would be also nice if we have something that we could also use
> > > for test the SystemC stuff. Right now this is not possible with the
> > > current framework. Because we have to build gem5 without python as a
> > > library for the SystemC stuff... and then build again with linking
> > > of
> > the gem5 library.
> > >
> > > Regards
> > > Matthias
> > >
> > > > Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
> > > andreas.sandb...@arm.com>:
> > > >
> > > > I'm generally in favour of switching to an existing unit test
> > framework.
> > > > I haven't surveyed the C++ test framework landscape, but judging
> > &g

Re: [gem5-dev] [EXT] Re: testing framework

[Potter, Brandon]

I also support adding unit tests (and possibly a code coverage checker to 
identify weak points). Ideally, we'd add unit tests for the entire simulator, 
but we should probably put something into place for new code requiring some 
type of unit tests. Otherwise, only a subset of the submissions will contain 
unit tests.

Regards,
Brandon

-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Jason Lowe-Power
Sent: Monday, November 20, 2017 11:22 AM
To: gem5 Developer List <gem5-dev@gem5.org>
Subject: Re: [gem5-dev] [EXT] Re: testing framework

I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its syntax. 
Although, that shouldn't stop us from having unit tests!!

What's the state of getting whimsy pushed in? From my side, it's ready to go. 
The only thing on my to do list that I would have liked to finish up before 
pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure to be 
able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable 
future.

Jason

On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) < 
prosenf...@micron.com> wrote:

> Even without the complexities of setting up SystemC+gem5 it's pretty 
> painful to "unit" test SystemC because the SystemC runtime has a lot 
> of global state that comes along with it (for example: you can't "uncall"
> sc_start() to go back and re-do elaboration with different modules). 
> The only (not very elegant) workaround we have discussed for unit 
> testing SystemC modules is to just to build each set of related 
> SystemC unit tests into their own independent GTest binaries.
>
> On the whole, though, GTest is pretty great.
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe 
> Black
> Sent: Friday, November 17, 2017 5:51 PM
> To: Matthias Jung <jun...@eit.uni-kl.de>
> Cc: gem5 Developer List <gem5-dev@gem5.org>
> Subject: [EXT] Re: [gem5-dev] testing framework
>
> As far as a high level framework, I think whimsy should help there 
> since it's a bit more decoupled and will let you run arbitrary things 
> as tests, at least as far as I can tell. Since the systemC integration 
> is just that, an integration of two systems, I don't think unit tests 
> would be as applicable generally speaking. You could have one for some 
> things though, like the ports that translate between gem5's and systemC's 
> memory protocol.
>
> GTest claims to be BSD, so the license should be compatible.
>
> Gabe
>
> On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung <jun...@eit.uni-kl.de>
> wrote:
>
> > Hey,
> >
> > It would be also nice if we have something that we could also use 
> > for test the SystemC stuff. Right now this is not possible with the 
> > current framework. Because we have to build gem5 without python as a 
> > library for the SystemC stuff... and then build again with linking 
> > of
> the gem5 library.
> >
> > Regards
> > Matthias
> >
> > > Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
> > andreas.sandb...@arm.com>:
> > >
> > > I'm generally in favour of switching to an existing unit test
> framework.
> > > I haven't surveyed the C++ test framework landscape, but judging 
> > > by the users (Chromium, LLVM, ...) GTest seems pretty solid.
> > >
> > > As far as I can tell, GTest is all BSD. Is that correct?
> > >
> > > Cheers,
> > > Andreas
> > >
> > >> On 16/11/2017 05:04, Gabe Black wrote:
> > >> Hi folks. How is the test framework integration coming along? Is 
> > >> there anything I can do to help?
> > >>
> > >> Also, looking at the whimsy documentation and talking to some 
> > >> other
> > folks
> > >> at Google, it looks like the framework is good at running tests, 
> > >> but doesn't itself really implement, for instance, unit tests 
> > >> which have
> > their
> > >> own C++ main function, etc.
> > >>
> > >> Does anyone have an opinion about using the google gtest library 
> > >> for writing C++ unit tests? I've only worked with it a little bit 
> > >> superficially, but so far it seems pretty nice and seems like it 
> > >> would
> > fit
> > >> into whimsy fairly well.
> > >>
> > >> Gabe
> > >> ___
> > >> gem5-dev mailing list

Re: [gem5-dev] [EXT] Re: testing framework

[Jason Lowe-Power]

I'm all for unit tests. GTest seems fine, but I'm not a huge fan of its
syntax. Although, that shouldn't stop us from having unit tests!!

What's the state of getting whimsy pushed in? From my side, it's ready to
go. The only thing on my to do list that I would have liked to finish up
before pushing is better integration with the current test infrastructure.
It would be good for whoever is running with the old test infrastructure to
be able to run both the old-style and new-style tests with one command.
Unfortunately, I don't have the time to dig into this for the foreseeable
future.

Jason

On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) <
prosenf...@micron.com> wrote:

> Even without the complexities of setting up SystemC+gem5 it's pretty
> painful to "unit" test SystemC because the SystemC runtime has a lot of
> global state that comes along with it (for example: you can't "uncall"
> sc_start() to go back and re-do elaboration with different modules). The
> only (not very elegant) workaround we have discussed for unit testing
> SystemC modules is to just to build each set of related SystemC unit tests
> into their own independent GTest binaries.
>
> On the whole, though, GTest is pretty great.
>
> -Original Message-
> From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black
> Sent: Friday, November 17, 2017 5:51 PM
> To: Matthias Jung 
> Cc: gem5 Developer List 
> Subject: [EXT] Re: [gem5-dev] testing framework
>
> As far as a high level framework, I think whimsy should help there since
> it's a bit more decoupled and will let you run arbitrary things as tests,
> at least as far as I can tell. Since the systemC integration is just that,
> an integration of two systems, I don't think unit tests would be as
> applicable generally speaking. You could have one for some things though,
> like the ports that translate between gem5's and systemC's memory protocol.
>
> GTest claims to be BSD, so the license should be compatible.
>
> Gabe
>
> On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung 
> wrote:
>
> > Hey,
> >
> > It would be also nice if we have something that we could also use for
> > test the SystemC stuff. Right now this is not possible with the
> > current framework. Because we have to build gem5 without python as a
> > library for the SystemC stuff... and then build again with linking of
> the gem5 library.
> >
> > Regards
> > Matthias
> >
> > > Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
> > andreas.sandb...@arm.com>:
> > >
> > > I'm generally in favour of switching to an existing unit test
> framework.
> > > I haven't surveyed the C++ test framework landscape, but judging by
> > > the users (Chromium, LLVM, ...) GTest seems pretty solid.
> > >
> > > As far as I can tell, GTest is all BSD. Is that correct?
> > >
> > > Cheers,
> > > Andreas
> > >
> > >> On 16/11/2017 05:04, Gabe Black wrote:
> > >> Hi folks. How is the test framework integration coming along? Is
> > >> there anything I can do to help?
> > >>
> > >> Also, looking at the whimsy documentation and talking to some other
> > folks
> > >> at Google, it looks like the framework is good at running tests,
> > >> but doesn't itself really implement, for instance, unit tests which
> > >> have
> > their
> > >> own C++ main function, etc.
> > >>
> > >> Does anyone have an opinion about using the google gtest library
> > >> for writing C++ unit tests? I've only worked with it a little bit
> > >> superficially, but so far it seems pretty nice and seems like it
> > >> would
> > fit
> > >> into whimsy fairly well.
> > >>
> > >> Gabe
> > >> ___
> > >> gem5-dev mailing list
> > >> gem5-dev@gem5.org
> > >> http://m5sim.org/mailman/listinfo/gem5-dev
> > >
> > > IMPORTANT NOTICE: The contents of this email and any attachments are
> > confidential and may also be privileged. If you are not the intended
> > recipient, please notify the sender immediately and do not disclose
> > the contents to any other person, use it for any purpose, or store or
> > copy the information in any medium. Thank you.
> > > ___
> > > gem5-dev mailing list
> > > gem5-dev@gem5.org
> > > http://m5sim.org/mailman/listinfo/gem5-dev
> >
> >
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
> ___
> gem5-dev mailing list
> gem5-dev@gem5.org
> http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev

Re: [gem5-dev] [EXT] Re: testing framework

[Paul Rosenfeld (prosenfeld)]

Even without the complexities of setting up SystemC+gem5 it's pretty painful to 
"unit" test SystemC because the SystemC runtime has a lot of global state that 
comes along with it (for example: you can't "uncall" sc_start() to go back and 
re-do elaboration with different modules). The only (not very elegant) 
workaround we have discussed for unit testing SystemC modules is to just to 
build each set of related SystemC unit tests into their own independent GTest 
binaries. 

On the whole, though, GTest is pretty great. 

-Original Message-
From: gem5-dev [mailto:gem5-dev-boun...@gem5.org] On Behalf Of Gabe Black
Sent: Friday, November 17, 2017 5:51 PM
To: Matthias Jung 
Cc: gem5 Developer List 
Subject: [EXT] Re: [gem5-dev] testing framework

As far as a high level framework, I think whimsy should help there since it's a 
bit more decoupled and will let you run arbitrary things as tests, at least as 
far as I can tell. Since the systemC integration is just that, an integration 
of two systems, I don't think unit tests would be as applicable generally 
speaking. You could have one for some things though, like the ports that 
translate between gem5's and systemC's memory protocol.

GTest claims to be BSD, so the license should be compatible.

Gabe

On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung 
wrote:

> Hey,
>
> It would be also nice if we have something that we could also use for 
> test the SystemC stuff. Right now this is not possible with the 
> current framework. Because we have to build gem5 without python as a 
> library for the SystemC stuff... and then build again with linking of the 
> gem5 library.
>
> Regards
> Matthias
>
> > Am 17.11.2017 um 17:43 schrieb Andreas Sandberg <
> andreas.sandb...@arm.com>:
> >
> > I'm generally in favour of switching to an existing unit test framework.
> > I haven't surveyed the C++ test framework landscape, but judging by 
> > the users (Chromium, LLVM, ...) GTest seems pretty solid.
> >
> > As far as I can tell, GTest is all BSD. Is that correct?
> >
> > Cheers,
> > Andreas
> >
> >> On 16/11/2017 05:04, Gabe Black wrote:
> >> Hi folks. How is the test framework integration coming along? Is 
> >> there anything I can do to help?
> >>
> >> Also, looking at the whimsy documentation and talking to some other
> folks
> >> at Google, it looks like the framework is good at running tests, 
> >> but doesn't itself really implement, for instance, unit tests which 
> >> have
> their
> >> own C++ main function, etc.
> >>
> >> Does anyone have an opinion about using the google gtest library 
> >> for writing C++ unit tests? I've only worked with it a little bit 
> >> superficially, but so far it seems pretty nice and seems like it 
> >> would
> fit
> >> into whimsy fairly well.
> >>
> >> Gabe
> >> ___
> >> gem5-dev mailing list
> >> gem5-dev@gem5.org
> >> http://m5sim.org/mailman/listinfo/gem5-dev
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended 
> recipient, please notify the sender immediately and do not disclose 
> the contents to any other person, use it for any purpose, or store or 
> copy the information in any medium. Thank you.
> > ___
> > gem5-dev mailing list
> > gem5-dev@gem5.org
> > http://m5sim.org/mailman/listinfo/gem5-dev
>
>
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev
___
gem5-dev mailing list
gem5-dev@gem5.org
http://m5sim.org/mailman/listinfo/gem5-dev