Also, it would be nice if the unit tests were more unit-ey and didn't require linking in all of gem5 or even building all of gem5 to run. I don't know how feasible that is in reality though. I think that's normally done by stubbing out the external stuff they'd call into with mocks. That would make them a lot quicker to build, and more self contained so that you'd know a failure of xyztest meant xyz was broken, not that some other thing it depended on wasn't behaving properly. This is a bit of a learning experience for me too, so I'm working my way through this as a I go to some extent.
Gabe On Wed, Nov 22, 2017 at 12:10 PM, Gabe Black <[email protected]> wrote: > Declaring unit tests in the SConscript file just tells scons they exist, > but if you don't ask it to build them then it won't since they aren't a > dependency of, for example, gem5.opt. It's annoying to tell it to just > build all the unit tests since you could do that based on directory, and as > discussed in another thread that would build all 5 versions of each. > Otherwise you have to ask scons to build each unit test binary by name > which is very cumbersome. I'm actively thinking about and working on making > the unit tests work and easy (or even mandatory) to run, so hopefully this > will get a lot better soon. > > Gabe > > On Wed, Nov 22, 2017 at 9:20 AM, Potter, Brandon <[email protected]> > wrote: > >> Is there any documentation on how to run the current unit tests? I notice >> that there's a unit test directory in the X86 build folder, but nothing >> inside is executable. I do not see a way of generating output from SCons >> related to unittests either. >> >> From the src/SConscript file, I see that the unittests should get created >> inside the makeEnv function, but it doesn't appear that this does anything >> (at least for X86). >> >> -Brandon >> >> -----Original Message----- >> From: gem5-dev [mailto:[email protected]] On Behalf Of Andreas >> Sandberg >> Sent: Wednesday, November 22, 2017 3:35 AM >> To: gem5 Developer List <[email protected]>; Gabe Black < >> [email protected]> >> Subject: Re: [gem5-dev] [EXT] Re: testing framework >> >> One issue that is preventing me from running the unit tests is that they >> require separate scons targets. One solution that I was considering at one >> point was to make the unit tests available as Python exports in the main >> build. This would allow us to run a magic config that calls unit tests >> instead of running the simulator proper. I think it would work for >> all(?) current unit tests, but I don't know how it would interact with >> other test unit test frameworks. >> >> Cheers, >> Andreas >> >> On 21/11/2017 01:37, Gabe Black wrote: >> > What I would minimally like to see is for people to run the unit tests >> > that already exist. Clearly that hasn't been happening since one of >> > them hasn't compiled since 2014 (now fixed). We need our tests to test >> > something specific, say whether or not it worked (not leave >> > interpretation up to the reader), and for people to actually run them. >> > >> > It's not at all clear what to do to actually build the unit tests from >> > scons, or how to run them, or what their results mean. I'm going to >> > try cleaning these up a bit. Also, they shouldn't be something that >> > lives off on the side or in their own directory, out of sight and out >> > of mind, they should be alongside the code they test. >> > >> > It might even make sense to associate a unit test with a particular >> > source file and make compilation fail if the unit test doesn't pass. >> > That might be a bit overboard, but would at least ensure that the tests >> get run. >> > >> > Gabe >> > >> > On Mon, Nov 20, 2017 at 3:14 PM, Potter, Brandon >> > <[email protected]> >> > wrote: >> > >> >> I also support adding unit tests (and possibly a code coverage >> >> checker to identify weak points). Ideally, we'd add unit tests for >> >> the entire simulator, but we should probably put something into place >> >> for new code requiring some type of unit tests. Otherwise, only a >> >> subset of the submissions will contain unit tests. >> >> >> >> Regards, >> >> Brandon >> >> >> >> -----Original Message----- >> >> From: gem5-dev [mailto:[email protected]] On Behalf Of Jason >> >> Lowe-Power >> >> Sent: Monday, November 20, 2017 11:22 AM >> >> To: gem5 Developer List <[email protected]> >> >> Subject: Re: [gem5-dev] [EXT] Re: testing framework >> >> >> >> I'm all for unit tests. GTest seems fine, but I'm not a huge fan of >> >> its syntax. Although, that shouldn't stop us from having unit tests!! >> >> >> >> What's the state of getting whimsy pushed in? From my side, it's >> >> ready to go. The only thing on my to do list that I would have liked >> >> to finish up before pushing is better integration with the current >> test infrastructure. >> >> It would be good for whoever is running with the old test >> >> infrastructure to be able to run both the old-style and new-style >> tests with one command. >> >> Unfortunately, I don't have the time to dig into this for the >> >> foreseeable future. >> >> >> >> Jason >> >> >> >> On Fri, Nov 17, 2017 at 3:19 PM Paul Rosenfeld (prosenfeld) < >> >> [email protected]> wrote: >> >> >> >>> Even without the complexities of setting up SystemC+gem5 it's pretty >> >>> painful to "unit" test SystemC because the SystemC runtime has a lot >> >>> of global state that comes along with it (for example: you can't >> "uncall" >> >>> sc_start() to go back and re-do elaboration with different modules). >> >>> The only (not very elegant) workaround we have discussed for unit >> >>> testing SystemC modules is to just to build each set of related >> >>> SystemC unit tests into their own independent GTest binaries. >> >>> >> >>> On the whole, though, GTest is pretty great. >> >>> >> >>> -----Original Message----- >> >>> From: gem5-dev [mailto:[email protected]] On Behalf Of Gabe >> >>> Black >> >>> Sent: Friday, November 17, 2017 5:51 PM >> >>> To: Matthias Jung <[email protected]> >> >>> Cc: gem5 Developer List <[email protected]> >> >>> Subject: [EXT] Re: [gem5-dev] testing framework >> >>> >> >>> As far as a high level framework, I think whimsy should help there >> >>> since it's a bit more decoupled and will let you run arbitrary >> >>> things as tests, at least as far as I can tell. Since the systemC >> >>> integration is just that, an integration of two systems, I don't >> >>> think unit tests would be as applicable generally speaking. You >> >>> could have one for some things though, like the ports that translate >> >>> between gem5's and >> >> systemC's memory protocol. >> >>> GTest claims to be BSD, so the license should be compatible. >> >>> >> >>> Gabe >> >>> >> >>> On Fri, Nov 17, 2017 at 10:14 AM, Matthias Jung >> >>> <[email protected]> >> >>> wrote: >> >>> >> >>>> Hey, >> >>>> >> >>>> It would be also nice if we have something that we could also use >> >>>> for test the SystemC stuff. Right now this is not possible with the >> >>>> current framework. Because we have to build gem5 without python as >> >>>> a library for the SystemC stuff... and then build again with >> >>>> linking of >> >>> the gem5 library. >> >>>> Regards >> >>>> Matthias >> >>>> >> >>>>> Am 17.11.2017 um 17:43 schrieb Andreas Sandberg < >> >>>> [email protected]>: >> >>>>> I'm generally in favour of switching to an existing unit test >> >>> framework. >> >>>>> I haven't surveyed the C++ test framework landscape, but judging >> >>>>> by the users (Chromium, LLVM, ...) GTest seems pretty solid. >> >>>>> >> >>>>> As far as I can tell, GTest is all BSD. Is that correct? >> >>>>> >> >>>>> Cheers, >> >>>>> Andreas >> >>>>> >> >>>>>> On 16/11/2017 05:04, Gabe Black wrote: >> >>>>>> Hi folks. How is the test framework integration coming along? Is >> >>>>>> there anything I can do to help? >> >>>>>> >> >>>>>> Also, looking at the whimsy documentation and talking to some >> >>>>>> other >> >>>> folks >> >>>>>> at Google, it looks like the framework is good at running tests, >> >>>>>> but doesn't itself really implement, for instance, unit tests >> >>>>>> which have >> >>>> their >> >>>>>> own C++ main function, etc. >> >>>>>> >> >>>>>> Does anyone have an opinion about using the google gtest library >> >>>>>> for writing C++ unit tests? I've only worked with it a little bit >> >>>>>> superficially, but so far it seems pretty nice and seems like it >> >>>>>> would >> >>>> fit >> >>>>>> into whimsy fairly well. >> >>>>>> >> >>>>>> Gabe >> >>>>>> _______________________________________________ >> >>>>>> gem5-dev mailing list >> >>>>>> [email protected] >> >>>>>> http://m5sim.org/mailman/listinfo/gem5-dev >> >>>>> IMPORTANT NOTICE: The contents of this email and any attachments >> >>>>> are >> >>>> confidential and may also be privileged. If you are not the >> >>>> intended recipient, please notify the sender immediately and do not >> >>>> disclose the contents to any other person, use it for any purpose, >> >>>> or store or copy the information in any medium. Thank you. >> >>>>> _______________________________________________ >> >>>>> gem5-dev mailing list >> >>>>> [email protected] >> >>>>> http://m5sim.org/mailman/listinfo/gem5-dev >> >>>> >> >>> _______________________________________________ >> >>> gem5-dev mailing list >> >>> [email protected] >> >>> http://m5sim.org/mailman/listinfo/gem5-dev >> >>> _______________________________________________ >> >>> gem5-dev mailing list >> >>> [email protected] >> >>> http://m5sim.org/mailman/listinfo/gem5-dev >> >> _______________________________________________ >> >> gem5-dev mailing list >> >> [email protected] >> >> http://m5sim.org/mailman/listinfo/gem5-dev >> >> _______________________________________________ >> >> gem5-dev mailing list >> >> [email protected] >> >> http://m5sim.org/mailman/listinfo/gem5-dev >> >> >> > _______________________________________________ >> > gem5-dev mailing list >> > [email protected] >> > http://m5sim.org/mailman/listinfo/gem5-dev >> >> IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> _______________________________________________ >> gem5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/listinfo/gem5-dev >> > > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
