On 01/07/2016 06:12 PM, Ben Meyer wrote:
On 01/07/2016 03:32 PM, Jay Pipes wrote:
On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
Hi all,
A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic,
including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.
Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out
that
when tests are failing, people may end up digging into mimic or
twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.
So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r
change
until we decide one way or another.
// jim
[1] https://review.openstack.org/#/c/220268/
What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.
Requests-mock debugging when things go wrong seems a bit simpler.
This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.
-Sean
We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.
I don't really see the point of a separate project like Mimic that has
a whole bunch of reimplementations (mocked out) of all sorts of
OpenStack (and RAX-specific) API services. It's just a great way to
introduce a larger surface area for bugs to creep in -- since you have
to keep the Mimic interfaces up to date with the real interfaces.
Better to keep something like this -- if it is TRULY needed -- in-tree
with the API service itself, so that the chances of divergence are
reduced. This is similar to the fakevirt driver in Nova. It's in tree
for good reason: when someone changes the virt driver interface, the
fakevirt driver goes boom and needs to be changed in a corresponding
fashion in the same patch.
A tool like my OpenStackInABox could certainly benefit from the models
or services being provided by each project - aside from the complexity
that that adds to the installation of installing all the dependencies
related instead of just implementing a simple model with a file and
sqlite backend that has minimal dependencies...but I digress as I
haven't looked at how nova's fakevirt driver installs so may be that's
not as big an issue. I'll certainly look at it for use in
OpenStackInABox, but even there I'm aiming for a more complete scenario
where you can interact with Keystone, Nova, Swift, etc...(e.g auth
against a fake Keystone, use the token with the fake nova which
validates it against the same fake Keystone instance, same with a fake
Swift...).
But your fake Keystone wouldn't be "authing" anything at all. Your fake
Nova wouldn't be "validating" anything at all.
You aren't *functionally* testing anything of importance above if the
things you are testing aren't doing what they are supposed to do.
The only things you are functionally testing are the *clients* to those
fake HTTP services. And what you are *actually* validating the client
code for isn't *actually* the real HTTP API service -- it's a fake which
can have its own surface area for bugginess -- which takes me back to my
original question: what value does one get from *functional* tests of a
client that calls a faked-out HTTP API versus *unit* tests of said
client that simply uses requests-mock (or similar) to set the returned
value of the HTTP API service to some expected value?
What value does a functional test against an HTTP API service that
does nothing (other than introduce greater surface area for bugs)
actually offer over unit tests anyway?
So to use Nova's fakevirt your project is limited to the same language
as nova - python, correct?
Nova's fakevirt driver simply does NOOP calls for most things and some
light in-memory bookkeeping of resources that a normal virt driver
would, in reality, consume from the host. It's a way we have to "test"
launching many instances in Nova without actually having those instances
consume resource on the host.
The main advantage of mimic is that it is language agnostic and multiple
moving parts (f.e taskflow) can interact with it.
See above point about what value this has over unit testing with mocked
response payloads. If you're looking for speed of testing a client, you
aren't going to beat mocking out the HTTP response payload.
Best,
-jay
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
