Re: [Openstack] [nova-testing] Efforts for Essex

2011-12-05 Thread Duncan McGreggor
On 30 Nov 2011 - 11:07, Duncan McGreggor wrote: On Tue, Nov 29, 2011 at 12:21 PM, Soren Hansen so...@linux2go.dk wrote: It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there

Re: [Openstack] [nova-testing] Efforts for Essex

2011-12-01 Thread Duncan McGreggor
On 30 Nov 2011 - 11:07, Duncan McGreggor wrote: On Tue, Nov 29, 2011 at 12:21 PM, Soren Hansen so...@linux2go.dk wrote: It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-30 Thread Duncan McGreggor
On Tue, Nov 29, 2011 at 12:21 PM, Soren Hansen so...@linux2go.dk wrote: It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there that aren't unit tests. We've also discussed my fake

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-30 Thread Chris Behrens
I need to catch up a bit with this thread, but I wanted to mention I have a huge patch coming that refactors almost all of the scheduler tests into true unit tests. I'd started this for other reasons and I hope it jives with the plans here. But if anyone is looking at the scheduler tests, we

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-30 Thread Duncan McGreggor
On 30 Nov 2011 - 19:26, Chris Behrens wrote: I need to catch up a bit with this thread, but I wanted to mention I have a huge patch coming that refactors almost all of the scheduler tests into true unit tests. Nice! I'd started this for other reasons and I hope it jives with the plans here.

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-30 Thread Chris Behrens
It'll be a couple days yet. I was refactoring a few things in the scheduler and while re-doing some tests, I ended up going down this rabbit hole of re-doing all of the tests. It's turned into a 6500 line diff so far... :) which is a bit much for just the refactoring that I need to get in

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-29 Thread Soren Hansen
It's been a bit over a week since I started this thread. So far we've agreed that running the test suite is too slow, mostly because there are too many things in there that aren't unit tests. We've also discussed my fake db implementation at length. I think we've generally agreed that it isn't

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-29 Thread Nachi Ueno
Hi folks Jay Thank you for your pointing this!:) Hey OpenStackers,please help forward-porting :P Soren Anything else around unit testing anyone wants to get into The Great Big Plan[tm]? And also, we should have a policy for unit test. Something like this - New code should have a concrete

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-28 Thread Trey Morris
I retract any concerns I brought up earlier as Soren gracefully answered them in last week's meeting. +1 from me. I look forward to the review. -tr3buchet On Thu, Nov 24, 2011 at 3:30 AM, Soren Hansen so...@linux2go.dk wrote: 2011/11/24 Sandy Walsh sandy.wa...@rackspace.com: haha ... worse

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-23 Thread Sandy Walsh
: [Openstack] [nova-testing] Efforts for Essex 2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I suspect the problem is coming in with our definition of unit tests. I don't think a unit test should be calling out of the method being tested at all. So anything beyond stubbing out the methods within

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-23 Thread Soren Hansen
2011/11/23 Sandy Walsh sandy.wa...@rackspace.com: Thanks Soren, I see what you're doing now and it makes perfect sense. It'll be a nice helper class. Cool. My only snipe would be that mox is generic to any library and this fake only gives the benefit to db operations. We have to remember

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-23 Thread Sandy Walsh
:) yeah, you're completely misunderstanding me. So, you've made a much better StubOutWithMock() and slightly better stubs.Set() by (essentially) ignoring the method parameter checks and just focusing on the return type. Using your example: def test_something(self): def

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-23 Thread Sandy Walsh
I understand what you're proposing, but I'm backtracking a little. (my kingdom for you and a whiteboard in the same room :) I think that you could have a hybrid of your db.do_something(desired_return_value) and self.stubs.Set(nova.db, 'instance_get_by_uuid', fake_instance_get) (which I don't

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-23 Thread Soren Hansen
2011/11/23 Sandy Walsh sandy.wa...@rackspace.com: I understand what you're proposing, but I'm backtracking a little. (my kingdom for you and a whiteboard in the same room :) Well, IRC would be a good start. :) I haven't seen you on IRC for days? I think that you could have a hybrid of your

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-23 Thread Sandy Walsh
: [Openstack] [nova-testing] Efforts for Essex 2011/11/23 Sandy Walsh sandy.wa...@rackspace.com: I understand what you're proposing, but I'm backtracking a little. (my kingdom for you and a whiteboard in the same room :) Well, IRC would be a good start. :) I haven't seen you on IRC for days? I think

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
2011/11/22 Rohit Karajgi rohit.kara...@vertex.co.in: Also I really  should not feel the need to install a DB on my box to run a unit-test suite. None, but the DB API tests should have the need to perform any database operations and validations. It slows down the overall execution. Such

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Sandy Walsh
To: openstack@lists.launchpad.net Subject: [Openstack] [nova-testing] Efforts for Essex Hi, guys. We're scattered across enough different timezones to make real-time communication really awkward, so let's see if we can get by using e-mail instead. A good test suite will let you keep up the pace

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Trey Morris
, November 21, 2011 8:24 AM To: openstack@lists.launchpad.net Subject: [Openstack] [nova-testing] Efforts for Essex Hi, guys. We're scattered across enough different timezones to make real-time communication really awkward, so let's see if we can get by using e-mail instead. A good test suite

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Jay Pipes
On Tue, Nov 22, 2011 at 8:31 AM, Soren Hansen so...@linux2go.dk wrote: 2011/11/22 Rohit Karajgi rohit.kara...@vertex.co.in: Also I really  should not feel the need to install a DB on my box to run a unit-test suite. None, but the DB API tests should have the need to perform any database

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
Ok, this seems like a good time to repeat what I posted to nova-database the other day. tl;dr: I'm adding a fake DB driver as well as a DB test suite that we can run against any of the backends to verify that they act the same. This should address all the concerns I've heard so far.

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Sandy Walsh
: openstack@lists.launchpad.net Subject: Re: [Openstack] [nova-testing] Efforts for Essex Ok, this seems like a good time to repeat what I posted to nova-database the other day. tl;dr: I'm adding a fake DB driver as well as a DB test suite that we can run against any of the backends to verify

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I'm not a big fan of faking a database, not only for the reasons outlined already, but because it makes the tests harder to understand. Can you give me an example? I find the opposite to be true, so I'd love to see counterexamples. Most of the

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I suppose there is a matter of preference here. I prefer to look in the setup() and teardown() methods of my test suite to find out how everything hangs together. Otherwise I have to check nova.TestCase when things break. The closer my test

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Mark Washenberger
I'm tending to agree with Sandy's comments. I think we all agree that we have a mess with the database stubbing that is going on. And I'm confident that the db fake would make that mess more manageable. But the way I see the mess, it comes from having a giant flat db interface and really

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
2011/11/22 Soren Hansen so...@linux2go.dk: (real) Unit tests are our documentation and having to jump around to find out how the plumbing works doesn't make for good documentation. I agree. That's exactly why I *don't* want mocks (for this) in the unit tests. It's been pointed out to me that

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
2011/11/22 Mark Washenberger mark.washenber...@rackspace.com: I think we all agree that we have a mess with the database stubbing that is going on. And I'm confident that the db fake would make that mess more manageable. But the way I see the mess, it comes from having a giant flat db

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Sandy Walsh
@lists.launchpad.net Subject: Re: [Openstack] [nova-testing] Efforts for Essex 2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I suppose there is a matter of preference here. I prefer to look in the setup() and teardown() methods of my test suite to find out how everything hangs together. Otherwise I

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-22 Thread Soren Hansen
2011/11/22 Sandy Walsh sandy.wa...@rackspace.com: I suspect the problem is coming in with our definition of unit tests. I don't think a unit test should be calling out of the method being tested at all. So anything beyond stubbing out the methods within the method being tested seems like noise

[Openstack] [nova-testing] Efforts for Essex

2011-11-21 Thread Soren Hansen
Hi, guys. We're scattered across enough different timezones to make real-time communication really awkward, so let's see if we can get by using e-mail instead. A good test suite will let you keep up the pace of development. It will offer confidence that your change didn't break any expectations,

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-21 Thread Soren Hansen
2011/11/21 Duncan McGreggor dun...@dreamhost.com: Soren, I'm delighted to see this email come across the list -- thanks for jump-starting such a discussion. It probably goes without saying that I completely agree with your assessments above :-) Great! :) The one thing that has bothered me

Re: [Openstack] [nova-testing] Efforts for Essex

2011-11-21 Thread Rohit Karajgi
: [Openstack] [nova-testing] Efforts for Essex On Mon, 2011-11-21 at 13:24 +0100, Soren Hansen wrote: = Speed up the test suite = +1; it can take about 6 minutes for the full suite to run on my dev box, which can definitely slow me down. = Review of existing tests = = Test coverage = One other