Re: full functional tests for API methods

2014-07-23 Thread Eric Snow
Cool. I'll have a look. :) -eric On Tue, Jul 22, 2014 at 10:30 PM, John Meinel wrote: > https://docs.google.com/a/canonical.com/document/d/1hXAOBVpM8o--9L7GOb4r6NUaloZQ8BsWj8W-Bqpznpw/edit?usp=sharing > is about how CI is set up today, and how to add more tests there. > > John > =:-> > > > On T

Re: full functional tests for API methods

2014-07-22 Thread John Meinel
https://docs.google.com/a/canonical.com/document/d/1hXAOBVpM8o--9L7GOb4r6NUaloZQ8BsWj8W-Bqpznpw/edit?usp=sharing is about how CI is set up today, and how to add more tests there. John =:-> On Tue, Jul 22, 2014 at 7:10 PM, Eric Snow wrote: > On Tue, Jul 22, 2014 at 8:17 AM, Eric Snow > wrote:

Re: full functional tests for API methods

2014-07-22 Thread Eric Snow
On Tue, Jul 22, 2014 at 8:17 AM, Eric Snow wrote: > Yep. For my case (backup/restore) we have a "good enough" solution already. Incidentally, do we have documentation somewhere on CI tests (e.g. backup/restore)? Specifically, where are they, how do they work, and how do you install them? -eric

Re: full functional tests for API methods

2014-07-22 Thread Eric Snow
On Mon, Jul 21, 2014 at 3:08 PM, Eric Snow wrote: > Ideally the > test would create an LXC into which it would bootstrap. Then the test > would set up the environment however it needed before running the API > client method against that environment. I imagine that this could be > addressed with

Re: full functional tests for API methods

2014-07-22 Thread Eric Snow
On Mon, Jul 21, 2014 at 5:56 PM, Mark Ramm-Christensen (Canonical.com) wrote: > I would like to propose that we *tag* tests as small, medium, or large. > > Small tests are measured in millisecionds, medium are measured in seconds up > to a minute, and large are measured in minutes or hours > +1

Re: full functional tests for API methods

2014-07-22 Thread Nate Finch
FYI, it's trivial to add build tags to test files and put slower tests in separate files that only build when requested. At the top of files with large tests add // +build large At the top of files with medium tests, add // +build medium large All the rest of the tests need no changes. Then, if y

Re: full functional tests for API methods

2014-07-21 Thread Mark Ramm-Christensen (Canonical.com)
I would like to propose that we *tag* tests as small, medium, or large. Small tests are measured in millisecionds, medium are measured in seconds up to a minute, and large are measured in minutes or hours We should run small tests locally, all the time. Medium tests should gate commits. Large

full functional tests for API methods

2014-07-21 Thread Eric Snow
We've been working on adding a backup method to the API client. One test I wanted to add was a functional test that did no patching anywhere. Such an end-to-end test would give us more confidence in the API method. We currently have something like that for backup/restore, but only as a CI test r