Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Thu, Jan 14, 2016 at 11:00:16AM +1300, Robert Collins wrote:
> On 8 January 2016 at 08:09, Jim Rollenhagen  wrote:
> > Hi all,
> >
> > A change to global-requirements[1] introduces mimic, which is an http
> > server that can mock various APIs, including nova and ironic, including
> > control of error codes and timeouts. The ironic team plans to use this
> > for testing python-ironicclient without standing up a full ironic
> > environment.
> 
> Whee that was a bit of a thread. Sorry for kicking this off on IRC and
> not chiming in until now.
> 
> As I read it the following points were made:
>   - this is a new an different testing thing
>   - which has the same interface skew issues as regular mocks
>   - and devs may need to hack twisted to fix issues with it in future
>   - but it is being used to target local devs, not gate jobs [today]
> 
> Interface skew can be handled a couple of ways. I'm a big fan of a
> technique (that I *think* came out of Google) of having a test fake
> layer which is substitutable by real things, so you can always run it
> in 'slow mode' to make sure things still work with real components.
> It's my understanding that with the way mimic is used, this can be
> done: run the same tests with real e.g. devstack brought up
> components. I think that that is good insurance, particularly if this
> is being done periodically rather than
> when-someone-remembers-or-has-an-issue.

Indeed, this is mimic's intent, and the way it's been used elsewhere
(e.g. by autoscale QA folks).

> Another way to mitigate interface skew is to have the publisher of an
> interface also publish a fast test fake for folk to use. I am also a
> fan of doing that, but as we haven't done so yet, I think its ok to go
> with what we have. If we couldn't run the same things with a real
> implementation, I would be more worried - because mocks of moving
> interfaces are fragile (e.g. mocking out oslo.config is fragile vs
> mocking out os.unlink).

So this made me think a bit and poke Glyph - turns out out-of-tree
plugins are being worked on by mimic; might be worth investigating
putting the mimic plugin for ironic in ironic's tree itself.

> Personally, I'm fine hacking on Twisted code - I quite like it
> (particularly good Twisted code :)) - and since folk seem ok with that
> aspect, great - thank you for raising the discussion here.

Totally; thanks for chiming in :)

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Robert Collins]

On 8 January 2016 at 08:09, Jim Rollenhagen  wrote:
> Hi all,
>
> A change to global-requirements[1] introduces mimic, which is an http
> server that can mock various APIs, including nova and ironic, including
> control of error codes and timeouts. The ironic team plans to use this
> for testing python-ironicclient without standing up a full ironic
> environment.

Whee that was a bit of a thread. Sorry for kicking this off on IRC and
not chiming in until now.

As I read it the following points were made:
  - this is a new an different testing thing
  - which has the same interface skew issues as regular mocks
  - and devs may need to hack twisted to fix issues with it in future
  - but it is being used to target local devs, not gate jobs [today]

Interface skew can be handled a couple of ways. I'm a big fan of a
technique (that I *think* came out of Google) of having a test fake
layer which is substitutable by real things, so you can always run it
in 'slow mode' to make sure things still work with real components.
It's my understanding that with the way mimic is used, this can be
done: run the same tests with real e.g. devstack brought up
components. I think that that is good insurance, particularly if this
is being done periodically rather than
when-someone-remembers-or-has-an-issue.

Another way to mitigate interface skew is to have the publisher of an
interface also publish a fast test fake for folk to use. I am also a
fan of doing that, but as we haven't done so yet, I think its ok to go
with what we have. If we couldn't run the same things with a real
implementation, I would be more worried - because mocks of moving
interfaces are fragile (e.g. mocking out oslo.config is fragile vs
mocking out os.unlink).

Personally, I'm fine hacking on Twisted code - I quite like it
(particularly good Twisted code :)) - and since folk seem ok with that
aspect, great - thank you for raising the discussion here.

I look forward to hearing how it goes for Ironic.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Sean Dague]

On 01/08/2016 03:52 PM, Jim Rollenhagen wrote:
> On Fri, Jan 08, 2016 at 03:39:51PM -0500, Jay Pipes wrote:
>>
>> No, nothing nefarious there. Sorry for letting my personal frustrations
>> bubble over into this.
>>
>> I am not blocking anything from going forward and I definitely am not asking
>> for a revert of any g-r patch. Nor am I trying to obstruct you in your
>> governance of Ironic.
>>
>> I was just raising my concerns as an OpenStack citizen and getting my
>> opinion out on paper.
> 
> As you should; thanks for doing that. I'm eager to see how this goes. :)

I think there is also a huge difference between how an effort like this
goes in the first 6 months, and what it looks like in 2 or 3 years once
none of the people originally working on it still are. Especially when
you start talking about more complex API flows that require there to be
state behind the scenes. The internal state associated with flows is
where this tends to get into the weeds.

An alternative approach would be making a fake / simulator driver like
Jay suggested, and spawning all the ironic services in a single process
using the oslo.messaging in memory driver and in mem sqlite db. This
kind of approach is what we use in Nova functional tests to setup a
whole stack fake cloud inside a single test process.

At the end of the day it's the choice of the Ironic team. Just something
to think really hard about.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Ben Meyer]

On 01/08/2016 12:56 PM, Jim Rollenhagen wrote:
> On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote:
>> Jim Rollenhagen wrote:
>>> [...]
>>> Here's the catch - mimic is built on twisted. I know twisted was
>>> previously removed from OpenStack (or at least people said "pls no", I
>>> don't know the full history). We didn't intend to stealth-introduce
>>> twisted back into g-r, but it was pointed out to me that it may appear
>>> this way, so here I am letting everyone know. lifeless pointed out that
>>> when tests are failing, people may end up digging into mimic or twisted
>>> code, which most people in this community aren't familiar with AFAIK,
>>> which is a valid point though I hope it isn't required often.
>> A bit of history with Twisted.
>>
>> Back in 2010 we decided we could not afford asking OpenStack developers to
>> be familiar with multiple service architecture frameworks, and eventlet was
>> chosen as the simplest framework to learn and debug. The best reference I
>> found on this is still visible in the wiki:
>>
>> https://wiki.openstack.org/wiki/UnifiedServiceArchitecture
>>
>>> So, the primary question here is: do folks have a problem with adding
>>> twisted here? We're holding off on Ironic changes that depend on this
>>> until this discussion has happened, but aren't reverting the g-r change
>>> until we decide one way or another.
>> The only friction I see is how many developers would be expected to need to
>> learn Twisted in order to complete their jobs. My understanding is that
>> Twisted expertise could be needed to debug python-ironicclient functional
>> tests, which makes the cost relatively limited. So if Mimic brings in a
>> clear and significant benefit, I don't think its Twisted dependence should
>> play that much against it.
>>
>> However, I agree with Sean and Jay that the benefit is unclear -- the few
>> features that Mimic brings seem to be outweighed by the increased risk of
>> introducing a delta between the implementation and the mock. If the main
>> benefit is that it's used in other Rackspace projects for testing (like Ben
>> said), I'm not sure that makes a compelling argument for the rest of the
>> community...
> No, that is not the main benefit, at all. Ben isn't involved in Ironic
> and until now has had nothing to do with this work to add mimic here, so
> I'm not sure where he got that impression from, or why he's speaking on
> our behalf as to the goals here.
I never claimed involvement in Ironic and from the outset noted my
status on this list. I've only commented in so far as trying to show the
value of mimic and similar tools.

I'm sorry if there was any confusion in that with respect to ironic.
That was certainly not my intent.

> As pointed out before, the risk of a delta between the mocks in mimic
> and reality is identical to the risk of a delta between the mocks in a
> client's unit tests and reality, so I don't see a particular downside
> there.
Agreed.

Ben

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[gord chung]

On 08/01/16 04:49 AM, Julien Danjou wrote:

On Thu, Jan 07 2016, Jay Pipes wrote:


OK, I just watched that. Sorry, still don't see the value that Mimic provides
over unit testing the client interfaces and mocking out the HTTP payloads so
you have strict control over the expectations.

The problem that Glyph noted in the video about the unit test that mocked out
os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
find it odd that that example was used in discussing the value of Mimic. Bad
mocks are just that: bad mocks. The same false positive failure could easily be
introduced with a typo in the "metadata injection" that Mimic does to inject
failures into the system.

Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing, and
see my response to Ben, the surface area it introduces for bugs in the test
platform itself in my opinion outweigh the multi-language value it might have.

I completely agree with what Jay points out here.

To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.

This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)

¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh



this happens with aodhclient as well. the one caveat i would mention is 
that it doesn't really allow for test isolation which requires you to 
put a bit more thought into your tests.


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote:
> Jim Rollenhagen wrote:
> >[...]
> >Here's the catch - mimic is built on twisted. I know twisted was
> >previously removed from OpenStack (or at least people said "pls no", I
> >don't know the full history). We didn't intend to stealth-introduce
> >twisted back into g-r, but it was pointed out to me that it may appear
> >this way, so here I am letting everyone know. lifeless pointed out that
> >when tests are failing, people may end up digging into mimic or twisted
> >code, which most people in this community aren't familiar with AFAIK,
> >which is a valid point though I hope it isn't required often.
> 
> A bit of history with Twisted.
> 
> Back in 2010 we decided we could not afford asking OpenStack developers to
> be familiar with multiple service architecture frameworks, and eventlet was
> chosen as the simplest framework to learn and debug. The best reference I
> found on this is still visible in the wiki:
> 
> https://wiki.openstack.org/wiki/UnifiedServiceArchitecture
> 
> >So, the primary question here is: do folks have a problem with adding
> >twisted here? We're holding off on Ironic changes that depend on this
> >until this discussion has happened, but aren't reverting the g-r change
> >until we decide one way or another.
> 
> The only friction I see is how many developers would be expected to need to
> learn Twisted in order to complete their jobs. My understanding is that
> Twisted expertise could be needed to debug python-ironicclient functional
> tests, which makes the cost relatively limited. So if Mimic brings in a
> clear and significant benefit, I don't think its Twisted dependence should
> play that much against it.
> 
> However, I agree with Sean and Jay that the benefit is unclear -- the few
> features that Mimic brings seem to be outweighed by the increased risk of
> introducing a delta between the implementation and the mock. If the main
> benefit is that it's used in other Rackspace projects for testing (like Ben
> said), I'm not sure that makes a compelling argument for the rest of the
> community...

No, that is not the main benefit, at all. Ben isn't involved in Ironic
and until now has had nothing to do with this work to add mimic here, so
I'm not sure where he got that impression from, or why he's speaking on
our behalf as to the goals here.

As pointed out before, the risk of a delta between the mocks in mimic
and reality is identical to the risk of a delta between the mocks in a
client's unit tests and reality, so I don't see a particular downside
there.

Again, I think "benefit is unclear" isn't a valid reason to block this,
so unless someone posts a revert we're going to move forward with this.

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Fri, Jan 08, 2016 at 03:39:51PM -0500, Jay Pipes wrote:
> On 01/08/2016 02:52 PM, Jim Rollenhagen wrote:
> >On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> >>On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> >snippity snip snip
> >
> >>>We haven't made it a dep for anything yet, only added to g-r.
> >>
> >>According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> >>makes functionally any difference, though (pun intended).
> >
> >Both. https://review.openstack.org/#/c/220268/
> >
> >This thread was originally about twisted, which is added to u-c with the
> >introduction of mimic.
> 
> Got it, thanks.
> 
> >>>However, now that you mention that, a really ambitious goal would be to
> >>>add a rabbit interface to mimic, and functionally test the API server
> >>>(that it sends the right messages, etc). Another would be to mimic
> >>>(Neutron, Glance) and test Ironic by itself.
> >>
> >>So you would reimplement AMQP communication protocols using an in-memory
> >>data store for queues. Sounds like an even greater surface area for bugs to
> >>be introduced.
> >>
> >>>Last, I frankly don't understand why there's
> >>>such heavy opposition to the ironic team using an additional tool for
> >>>testing.
> >>
> >>Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> >>though.
> >>
> >>a) Because it fractures the testing and QA processes used by upstream
> >>contributors that work on OpenStack projects by requiring them to learn
> >>another system -- and one that potentially would require them to understand
> >>a whole new surface area for potential bugs
> >
> >I don't think there's a large risk of needing to dig deep into mimic,
> >and especially twisted. If this does prove to be a problem, I'm happy to
> >remove it. However, we can't even explore what it would be like, or how
> >hard we would depend on mimic, without mimic being in g-r.
> 
> Sure, fair enough.
> 
> >>b) Because it represents yet another RAX-driven divergence in the QA space.
> >>CloudCafe took essentially all of the RAX folks that were at one point
> >>working on Tempest and upstream QA and siloed them into a totally different
> >>organization, in true RAX fashion. Instead of pulling the OpenStack QA
> >>community along together, RAX QA continues to just do its own thing and
> >>there's still bitterness on the tips of tongues.
> >
> >So, this isn't trying to replace anything. This is adding a different
> >way to run functional tests, that is *much* faster than standing up a
> >full ironic environment. This is helpful for developers that want to
> >quickly run tests before posting them to gerrit, people that need to
> >test in constrained environments, etc.
> 
> I recognize it is much faster than standing up a real environment. And I
> recognize that running faster client tests is a useful thing -- as long as
> we can be confident that what is tested does not suffer from some of the
> issues I identified earlier (syncing with real API and introduction of
> greater surface area for bugs in the test platform itself).
> 
> >I'm 100% against doing things like Rackspace did with tempest and
> >cloudcafe, and I wouldn't be supporting this effort if I felt it was
> >similar. Here's how this went:
> >
> >* Lekha started working on OnMetal QA, with a goal of doing some amount
> >   of upstream work as well.
> >
> >* She's previously worked on projects (like autoscale) that interact
> >   with OpenStack APIs, and seeing the need to test without a full nova
> >   environment, built mimic.
> >
> >* In talking with some of the other Ironic folks working on QA (from
> >   Intel, IBM, more), she presented mimic as something that may be useful
> >   for testing the client (and more). They (and I) agreed it was a neat
> >   idea worth trying.
> >
> >* Jay offered to help with the global-requirements patch as it's
> >   something he's done before, and did the review grinding here.
> >
> >* It finally landed, and lifeless asked me to bring up the Twisted
> >   conversation on the list. Note that this is not the "is mimic
> >   useful" conversation we're having now. Nobody remotely voiced
> >   concerns about using a new test tool until this thread happened.
> >
> >Please do let me know if any of that seems nefarious; I hope it doesn't.
> 
> No, nothing nefarious there. Sorry for letting my personal frustrations
> bubble over into this.
> 
> I am not blocking anything from going forward and I definitely am not asking
> for a revert of any g-r patch. Nor am I trying to obstruct you in your
> governance of Ironic.
> 
> I was just raising my concerns as an OpenStack citizen and getting my
> opinion out on paper.

As you should; thanks for doing that. I'm eager to see how this goes. :)

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2016-01-07 11:09:32 -0800:
> Hi all,
> 
> A change to global-requirements[1] introduces mimic, which is an http
> server that can mock various APIs, including nova and ironic, including
> control of error codes and timeouts. The ironic team plans to use this
> for testing python-ironicclient without standing up a full ironic
> environment.
> 
> Here's the catch - mimic is built on twisted. I know twisted was
> previously removed from OpenStack (or at least people said "pls no", I
> don't know the full history). We didn't intend to stealth-introduce
> twisted back into g-r, but it was pointed out to me that it may appear
> this way, so here I am letting everyone know. lifeless pointed out that
> when tests are failing, people may end up digging into mimic or twisted
> code, which most people in this community aren't familiar with AFAIK,
> which is a valid point though I hope it isn't required often.
> 
> So, the primary question here is: do folks have a problem with adding
> twisted here? We're holding off on Ironic changes that depend on this
> until this discussion has happened, but aren't reverting the g-r change
> until we decide one way or another.
> 
> // jim
> 
> [1] https://review.openstack.org/#/c/220268/
> 

I looked through the Ironic specs for anything mentioning "mimic" (using
the search box) and didn't find anything. Is there a plan or something
describing what's happening that I can read as background for this?

Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
snippity snip snip

> >We haven't made it a dep for anything yet, only added to g-r.
> 
> According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> makes functionally any difference, though (pun intended).

Both. https://review.openstack.org/#/c/220268/

This thread was originally about twisted, which is added to u-c with the
introduction of mimic.

> >However, now that you mention that, a really ambitious goal would be to
> >add a rabbit interface to mimic, and functionally test the API server
> >(that it sends the right messages, etc). Another would be to mimic
> >(Neutron, Glance) and test Ironic by itself.
> 
> So you would reimplement AMQP communication protocols using an in-memory
> data store for queues. Sounds like an even greater surface area for bugs to
> be introduced.
> 
> >Last, I frankly don't understand why there's
> >such heavy opposition to the ironic team using an additional tool for
> >testing.
> 
> Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> though.
> 
> a) Because it fractures the testing and QA processes used by upstream
> contributors that work on OpenStack projects by requiring them to learn
> another system -- and one that potentially would require them to understand
> a whole new surface area for potential bugs

I don't think there's a large risk of needing to dig deep into mimic,
and especially twisted. If this does prove to be a problem, I'm happy to
remove it. However, we can't even explore what it would be like, or how
hard we would depend on mimic, without mimic being in g-r.

> b) Because it represents yet another RAX-driven divergence in the QA space.
> CloudCafe took essentially all of the RAX folks that were at one point
> working on Tempest and upstream QA and siloed them into a totally different
> organization, in true RAX fashion. Instead of pulling the OpenStack QA
> community along together, RAX QA continues to just do its own thing and
> there's still bitterness on the tips of tongues.

So, this isn't trying to replace anything. This is adding a different
way to run functional tests, that is *much* faster than standing up a
full ironic environment. This is helpful for developers that want to
quickly run tests before posting them to gerrit, people that need to
test in constrained environments, etc.

I'm 100% against doing things like Rackspace did with tempest and
cloudcafe, and I wouldn't be supporting this effort if I felt it was
similar. Here's how this went:

* Lekha started working on OnMetal QA, with a goal of doing some amount
  of upstream work as well.

* She's previously worked on projects (like autoscale) that interact
  with OpenStack APIs, and seeing the need to test without a full nova
  environment, built mimic.

* In talking with some of the other Ironic folks working on QA (from
  Intel, IBM, more), she presented mimic as something that may be useful
  for testing the client (and more). They (and I) agreed it was a neat
  idea worth trying.

* Jay offered to help with the global-requirements patch as it's
  something he's done before, and did the review grinding here.

* It finally landed, and lifeless asked me to bring up the Twisted
  conversation on the list. Note that this is not the "is mimic
  useful" conversation we're having now. Nobody remotely voiced
  concerns about using a new test tool until this thread happened.

Please do let me know if any of that seems nefarious; I hope it doesn't.

> 
> 
> 
> >There was *more* than sufficient time for "I don't think this is useful"
> >to be posted on the review.
> 
> Sorry, this was the first I'd heard of it all.
> 
> > If anyone has a hard technical reason why
> >mimic should not be used to test an OpenStack project, I'm happy to
> >listen. Otherwise I'd like to work on actually getting things done.
> 
> I've listed my hard technical reason: it introduces, IMHO, too large of a
> surface area for bugs to creep in to the testing process versus the benefit
> it provides.

Again, I don't think that's any larger of a surface area than mocks in client
unit tests being incorrect, and we really won't know whether it causes
problems in reality.

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800:
> On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> snippity snip snip
> 
> > >We haven't made it a dep for anything yet, only added to g-r.
> > 
> > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> > makes functionally any difference, though (pun intended).
> 
> Both. https://review.openstack.org/#/c/220268/
> 
> This thread was originally about twisted, which is added to u-c with the
> introduction of mimic.
> 
> > >However, now that you mention that, a really ambitious goal would be to
> > >add a rabbit interface to mimic, and functionally test the API server
> > >(that it sends the right messages, etc). Another would be to mimic
> > >(Neutron, Glance) and test Ironic by itself.
> > 
> > So you would reimplement AMQP communication protocols using an in-memory
> > data store for queues. Sounds like an even greater surface area for bugs to
> > be introduced.
> > 
> > >Last, I frankly don't understand why there's
> > >such heavy opposition to the ironic team using an additional tool for
> > >testing.
> > 
> > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> > though.
> > 
> > a) Because it fractures the testing and QA processes used by upstream
> > contributors that work on OpenStack projects by requiring them to learn
> > another system -- and one that potentially would require them to understand
> > a whole new surface area for potential bugs
> 
> I don't think there's a large risk of needing to dig deep into mimic,
> and especially twisted. If this does prove to be a problem, I'm happy to
> remove it. However, we can't even explore what it would be like, or how
> hard we would depend on mimic, without mimic being in g-r.
> 
> > b) Because it represents yet another RAX-driven divergence in the QA space.
> > CloudCafe took essentially all of the RAX folks that were at one point
> > working on Tempest and upstream QA and siloed them into a totally different
> > organization, in true RAX fashion. Instead of pulling the OpenStack QA
> > community along together, RAX QA continues to just do its own thing and
> > there's still bitterness on the tips of tongues.
> 
> So, this isn't trying to replace anything. This is adding a different
> way to run functional tests, that is *much* faster than standing up a
> full ironic environment. This is helpful for developers that want to
> quickly run tests before posting them to gerrit, people that need to
> test in constrained environments, etc.

So it won't be used in the gate, just on local developer systems?

Doug

> 
> I'm 100% against doing things like Rackspace did with tempest and
> cloudcafe, and I wouldn't be supporting this effort if I felt it was
> similar. Here's how this went:
> 
> * Lekha started working on OnMetal QA, with a goal of doing some amount
>   of upstream work as well.
> 
> * She's previously worked on projects (like autoscale) that interact
>   with OpenStack APIs, and seeing the need to test without a full nova
>   environment, built mimic.
> 
> * In talking with some of the other Ironic folks working on QA (from
>   Intel, IBM, more), she presented mimic as something that may be useful
>   for testing the client (and more). They (and I) agreed it was a neat
>   idea worth trying.
> 
> * Jay offered to help with the global-requirements patch as it's
>   something he's done before, and did the review grinding here.
> 
> * It finally landed, and lifeless asked me to bring up the Twisted
>   conversation on the list. Note that this is not the "is mimic
>   useful" conversation we're having now. Nobody remotely voiced
>   concerns about using a new test tool until this thread happened.
> 
> Please do let me know if any of that seems nefarious; I hope it doesn't.
> 
> > 
> > 
> > 
> > >There was *more* than sufficient time for "I don't think this is useful"
> > >to be posted on the review.
> > 
> > Sorry, this was the first I'd heard of it all.
> > 
> > > If anyone has a hard technical reason why
> > >mimic should not be used to test an OpenStack project, I'm happy to
> > >listen. Otherwise I'd like to work on actually getting things done.
> > 
> > I've listed my hard technical reason: it introduces, IMHO, too large of a
> > surface area for bugs to creep in to the testing process versus the benefit
> > it provides.
> 
> Again, I don't think that's any larger of a surface area than mocks in client
> unit tests being incorrect, and we really won't know whether it causes
> problems in reality.
> 
> // jim
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Fri, Jan 08, 2016 at 03:13:15PM -0500, Doug Hellmann wrote:
> Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800:
> > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> > snippity snip snip
> > 
> > > >We haven't made it a dep for anything yet, only added to g-r.
> > > 
> > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> > > makes functionally any difference, though (pun intended).
> > 
> > Both. https://review.openstack.org/#/c/220268/
> > 
> > This thread was originally about twisted, which is added to u-c with the
> > introduction of mimic.
> > 
> > > >However, now that you mention that, a really ambitious goal would be to
> > > >add a rabbit interface to mimic, and functionally test the API server
> > > >(that it sends the right messages, etc). Another would be to mimic
> > > >(Neutron, Glance) and test Ironic by itself.
> > > 
> > > So you would reimplement AMQP communication protocols using an in-memory
> > > data store for queues. Sounds like an even greater surface area for bugs 
> > > to
> > > be introduced.
> > > 
> > > >Last, I frankly don't understand why there's
> > > >such heavy opposition to the ironic team using an additional tool for
> > > >testing.
> > > 
> > > Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
> > > though.
> > > 
> > > a) Because it fractures the testing and QA processes used by upstream
> > > contributors that work on OpenStack projects by requiring them to learn
> > > another system -- and one that potentially would require them to 
> > > understand
> > > a whole new surface area for potential bugs
> > 
> > I don't think there's a large risk of needing to dig deep into mimic,
> > and especially twisted. If this does prove to be a problem, I'm happy to
> > remove it. However, we can't even explore what it would be like, or how
> > hard we would depend on mimic, without mimic being in g-r.
> > 
> > > b) Because it represents yet another RAX-driven divergence in the QA 
> > > space.
> > > CloudCafe took essentially all of the RAX folks that were at one point
> > > working on Tempest and upstream QA and siloed them into a totally 
> > > different
> > > organization, in true RAX fashion. Instead of pulling the OpenStack QA
> > > community along together, RAX QA continues to just do its own thing and
> > > there's still bitterness on the tips of tongues.
> > 
> > So, this isn't trying to replace anything. This is adding a different
> > way to run functional tests, that is *much* faster than standing up a
> > full ironic environment. This is helpful for developers that want to
> > quickly run tests before posting them to gerrit, people that need to
> > test in constrained environments, etc.
> 
> So it won't be used in the gate, just on local developer systems?

Initially, yes. I'd like to also add it to the gate at some point to
make sure we don't break those interactions, though I could deal with a
non-voting job just in case.

// jim

> 
> Doug
> 
> > 
> > I'm 100% against doing things like Rackspace did with tempest and
> > cloudcafe, and I wouldn't be supporting this effort if I felt it was
> > similar. Here's how this went:
> > 
> > * Lekha started working on OnMetal QA, with a goal of doing some amount
> >   of upstream work as well.
> > 
> > * She's previously worked on projects (like autoscale) that interact
> >   with OpenStack APIs, and seeing the need to test without a full nova
> >   environment, built mimic.
> > 
> > * In talking with some of the other Ironic folks working on QA (from
> >   Intel, IBM, more), she presented mimic as something that may be useful
> >   for testing the client (and more). They (and I) agreed it was a neat
> >   idea worth trying.
> > 
> > * Jay offered to help with the global-requirements patch as it's
> >   something he's done before, and did the review grinding here.
> > 
> > * It finally landed, and lifeless asked me to bring up the Twisted
> >   conversation on the list. Note that this is not the "is mimic
> >   useful" conversation we're having now. Nobody remotely voiced
> >   concerns about using a new test tool until this thread happened.
> > 
> > Please do let me know if any of that seems nefarious; I hope it doesn't.
> > 
> > > 
> > > 
> > > 
> > > >There was *more* than sufficient time for "I don't think this is useful"
> > > >to be posted on the review.
> > > 
> > > Sorry, this was the first I'd heard of it all.
> > > 
> > > > If anyone has a hard technical reason why
> > > >mimic should not be used to test an OpenStack project, I'm happy to
> > > >listen. Otherwise I'd like to work on actually getting things done.
> > > 
> > > I've listed my hard technical reason: it introduces, IMHO, too large of a
> > > surface area for bugs to creep in to the testing process versus the 
> > > benefit
> > > it provides.
> > 
> > Again, I don't think that's any larger of a surface area than mocks in 
> 

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2016-01-08 12:30:33 -0800:
> On Fri, Jan 08, 2016 at 03:13:15PM -0500, Doug Hellmann wrote:
> > Excerpts from Jim Rollenhagen's message of 2016-01-08 11:52:51 -0800:
> > > On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:
> > > > On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
> > > snippity snip snip
> > > 
> > > > >We haven't made it a dep for anything yet, only added to g-r.
> > > > 
> > > > According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
> > > > makes functionally any difference, though (pun intended).
> > > 
> > > Both. https://review.openstack.org/#/c/220268/
> > > 
> > > This thread was originally about twisted, which is added to u-c with the
> > > introduction of mimic.
> > > 
> > > > >However, now that you mention that, a really ambitious goal would be to
> > > > >add a rabbit interface to mimic, and functionally test the API server
> > > > >(that it sends the right messages, etc). Another would be to mimic
> > > > >(Neutron, Glance) and test Ironic by itself.
> > > > 
> > > > So you would reimplement AMQP communication protocols using an in-memory
> > > > data store for queues. Sounds like an even greater surface area for 
> > > > bugs to
> > > > be introduced.
> > > > 
> > > > >Last, I frankly don't understand why there's
> > > > >such heavy opposition to the ironic team using an additional tool for
> > > > >testing.
> > > > 
> > > > Since you asked, I'll be blunt. This isn't a personal attack on you, 
> > > > Jim,
> > > > though.
> > > > 
> > > > a) Because it fractures the testing and QA processes used by upstream
> > > > contributors that work on OpenStack projects by requiring them to learn
> > > > another system -- and one that potentially would require them to 
> > > > understand
> > > > a whole new surface area for potential bugs
> > > 
> > > I don't think there's a large risk of needing to dig deep into mimic,
> > > and especially twisted. If this does prove to be a problem, I'm happy to
> > > remove it. However, we can't even explore what it would be like, or how
> > > hard we would depend on mimic, without mimic being in g-r.
> > > 
> > > > b) Because it represents yet another RAX-driven divergence in the QA 
> > > > space.
> > > > CloudCafe took essentially all of the RAX folks that were at one point
> > > > working on Tempest and upstream QA and siloed them into a totally 
> > > > different
> > > > organization, in true RAX fashion. Instead of pulling the OpenStack QA
> > > > community along together, RAX QA continues to just do its own thing and
> > > > there's still bitterness on the tips of tongues.
> > > 
> > > So, this isn't trying to replace anything. This is adding a different
> > > way to run functional tests, that is *much* faster than standing up a
> > > full ironic environment. This is helpful for developers that want to
> > > quickly run tests before posting them to gerrit, people that need to
> > > test in constrained environments, etc.
> > 
> > So it won't be used in the gate, just on local developer systems?
> 
> Initially, yes. I'd like to also add it to the gate at some point to
> make sure we don't break those interactions, though I could deal with a
> non-voting job just in case.

OK, if that's the case then although I share Jay's concerns about bug
surface area and keeping the APIs in sync, I think the risk is low
enough that it would only annoy developers and not actually allow bugs
to slip through testing. So, I'm OK with continuing with the experiment
in Ironic.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jay Pipes]

On 01/08/2016 02:52 PM, Jim Rollenhagen wrote:

On Fri, Jan 08, 2016 at 02:08:04PM -0500, Jay Pipes wrote:

On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:

snippity snip snip


We haven't made it a dep for anything yet, only added to g-r.


According to Dims, not to g-r, but to u-c, right Dims? Not sure if that
makes functionally any difference, though (pun intended).


Both. https://review.openstack.org/#/c/220268/

This thread was originally about twisted, which is added to u-c with the
introduction of mimic.


Got it, thanks.


However, now that you mention that, a really ambitious goal would be to
add a rabbit interface to mimic, and functionally test the API server
(that it sends the right messages, etc). Another would be to mimic
(Neutron, Glance) and test Ironic by itself.


So you would reimplement AMQP communication protocols using an in-memory
data store for queues. Sounds like an even greater surface area for bugs to
be introduced.


Last, I frankly don't understand why there's
such heavy opposition to the ironic team using an additional tool for
testing.


Since you asked, I'll be blunt. This isn't a personal attack on you, Jim,
though.

a) Because it fractures the testing and QA processes used by upstream
contributors that work on OpenStack projects by requiring them to learn
another system -- and one that potentially would require them to understand
a whole new surface area for potential bugs


I don't think there's a large risk of needing to dig deep into mimic,
and especially twisted. If this does prove to be a problem, I'm happy to
remove it. However, we can't even explore what it would be like, or how
hard we would depend on mimic, without mimic being in g-r.


Sure, fair enough.


b) Because it represents yet another RAX-driven divergence in the QA space.
CloudCafe took essentially all of the RAX folks that were at one point
working on Tempest and upstream QA and siloed them into a totally different
organization, in true RAX fashion. Instead of pulling the OpenStack QA
community along together, RAX QA continues to just do its own thing and
there's still bitterness on the tips of tongues.


So, this isn't trying to replace anything. This is adding a different
way to run functional tests, that is *much* faster than standing up a
full ironic environment. This is helpful for developers that want to
quickly run tests before posting them to gerrit, people that need to
test in constrained environments, etc.


I recognize it is much faster than standing up a real environment. And I 
recognize that running faster client tests is a useful thing -- as long 
as we can be confident that what is tested does not suffer from some of 
the issues I identified earlier (syncing with real API and introduction 
of greater surface area for bugs in the test platform itself).



I'm 100% against doing things like Rackspace did with tempest and
cloudcafe, and I wouldn't be supporting this effort if I felt it was
similar. Here's how this went:

* Lekha started working on OnMetal QA, with a goal of doing some amount
   of upstream work as well.

* She's previously worked on projects (like autoscale) that interact
   with OpenStack APIs, and seeing the need to test without a full nova
   environment, built mimic.

* In talking with some of the other Ironic folks working on QA (from
   Intel, IBM, more), she presented mimic as something that may be useful
   for testing the client (and more). They (and I) agreed it was a neat
   idea worth trying.

* Jay offered to help with the global-requirements patch as it's
   something he's done before, and did the review grinding here.

* It finally landed, and lifeless asked me to bring up the Twisted
   conversation on the list. Note that this is not the "is mimic
   useful" conversation we're having now. Nobody remotely voiced
   concerns about using a new test tool until this thread happened.

Please do let me know if any of that seems nefarious; I hope it doesn't.


No, nothing nefarious there. Sorry for letting my personal frustrations 
bubble over into this.


I am not blocking anything from going forward and I definitely am not 
asking for a revert of any g-r patch. Nor am I trying to obstruct you in 
your governance of Ironic.


I was just raising my concerns as an OpenStack citizen and getting my 
opinion out on paper.


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Davanum Srinivas]

yes Jay, i have -W on the bot proposed merge, pending this discussion:
https://review.openstack.org/#/c/263592/

-- Dims

On Fri, Jan 8, 2016 at 2:08 PM, Jay Pipes  wrote:
> On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:
>>
>> On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
>>>
>>> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:

 On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
>
> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
>>
>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
>>>
>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

 Hi all,

 A change to global-requirements[1] introduces mimic, which is an
 http
 server that can mock various APIs, including nova and ironic,
 including
 control of error codes and timeouts. The ironic team plans to use
 this
 for testing python-ironicclient without standing up a full ironic
 environment.

 Here's the catch - mimic is built on twisted. I know twisted was
 previously removed from OpenStack (or at least people said "pls no",
 I
 don't know the full history). We didn't intend to stealth-introduce
 twisted back into g-r, but it was pointed out to me that it may
 appear
 this way, so here I am letting everyone know. lifeless pointed out
 that
 when tests are failing, people may end up digging into mimic or
 twisted
 code, which most people in this community aren't familiar with
 AFAIK,
 which is a valid point though I hope it isn't required often.

 So, the primary question here is: do folks have a problem with
 adding
 twisted here? We're holding off on Ironic changes that depend on
 this
 until this discussion has happened, but aren't reverting the g-r
 change
 until we decide one way or another.

 // jim

 [1] https://review.openstack.org/#/c/220268/
>>>
>>>
>>> What is the advantage of running another server like this over using
>>> requests-mock (which is used by other OpenStack projects for testing
>>> today)? The only difference here seems to be that you actually
>>> execute
>>> requests code in one case and not in the other.
>>>
>>> Requests-mock debugging when things go wrong seems a bit simpler.
>>>
>>> This is less about twisted and more about trying to not introduce yet
>>> another way to mock code in the tree that people need to understand.
>>>
>>> -Sean
>>
>>
>> We'd be using this for functional tests, not unit, where we can't
>> really
>> inject mocks. The idea is that we could run a full functional suite
>> against either mimic or a full ironic environment, just by changing a
>> test setting.
>
>
> I don't really see the point of a separate project like Mimic that has
> a
> whole bunch of reimplementations (mocked out) of all sorts of OpenStack
> (and
> RAX-specific) API services. It's just a great way to introduce a larger
> surface area for bugs to creep in -- since you have to keep the Mimic
> interfaces up to date with the real interfaces. Better to keep
> something
> like this -- if it is TRULY needed -- in-tree with the API service
> itself,
> so that the chances of divergence are reduced. This is similar to the
> fakevirt driver in Nova. It's in tree for good reason: when someone
> changes
> the virt driver interface, the fakevirt driver goes boom and needs to
> be
> changed in a corresponding fashion in the same patch.


 I tend to think our REST APIs are much more stable than internal python
 APIs, and so we can manage the divergence much easier.

 Also, mimic can simulate:

 * old versions (less needed with all the microversion stuff)
 * old bugs that were shipped
 * misconfigurations
 * networking errors
 * the passage of time (including timeouts)

 We probably don't want to keep a catalog of old bugs and
 misconfigurations in
 tree forever.

> What value does a functional test against an HTTP API service that does
> nothing (other than introduce greater surface area for bugs) actually
> offer
> over unit tests anyway?


 Testing the full stack of the client instead of mocking the bottom
 half of requests is a big one.

 Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
 https://www.youtube.com/watch?v=HKUUQme3dwA
>>>
>>>
>>> OK, I just watched that. Sorry, still don't see the value that Mimic
>>> provides over unit testing the client interfaces and mocking out the HTTP
>>> payloads so you have strict control over the expectations.
>>>
>>> The problem that Glyph noted 

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Fri, Jan 08, 2016 at 02:39:52PM -0500, Davanum Srinivas wrote:
> yes Jay, i have -W on the bot proposed merge, pending this discussion:
> https://review.openstack.org/#/c/263592/

Note that this doesn't actually block mimic from being in g-r, or being
used. It's already in g-r. You'd need to revert
https://review.openstack.org/#/c/220268/ .

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2016-01-08 09:56:46 -0800:
> On Fri, Jan 08, 2016 at 11:00:35AM +0100, Thierry Carrez wrote:
> > Jim Rollenhagen wrote:
> > >[...]
> > >Here's the catch - mimic is built on twisted. I know twisted was
> > >previously removed from OpenStack (or at least people said "pls no", I
> > >don't know the full history). We didn't intend to stealth-introduce
> > >twisted back into g-r, but it was pointed out to me that it may appear
> > >this way, so here I am letting everyone know. lifeless pointed out that
> > >when tests are failing, people may end up digging into mimic or twisted
> > >code, which most people in this community aren't familiar with AFAIK,
> > >which is a valid point though I hope it isn't required often.
> > 
> > A bit of history with Twisted.
> > 
> > Back in 2010 we decided we could not afford asking OpenStack developers to
> > be familiar with multiple service architecture frameworks, and eventlet was
> > chosen as the simplest framework to learn and debug. The best reference I
> > found on this is still visible in the wiki:
> > 
> > https://wiki.openstack.org/wiki/UnifiedServiceArchitecture
> > 
> > >So, the primary question here is: do folks have a problem with adding
> > >twisted here? We're holding off on Ironic changes that depend on this
> > >until this discussion has happened, but aren't reverting the g-r change
> > >until we decide one way or another.
> > 
> > The only friction I see is how many developers would be expected to need to
> > learn Twisted in order to complete their jobs. My understanding is that
> > Twisted expertise could be needed to debug python-ironicclient functional
> > tests, which makes the cost relatively limited. So if Mimic brings in a
> > clear and significant benefit, I don't think its Twisted dependence should
> > play that much against it.
> > 
> > However, I agree with Sean and Jay that the benefit is unclear -- the few
> > features that Mimic brings seem to be outweighed by the increased risk of
> > introducing a delta between the implementation and the mock. If the main
> > benefit is that it's used in other Rackspace projects for testing (like Ben
> > said), I'm not sure that makes a compelling argument for the rest of the
> > community...
> 
> No, that is not the main benefit, at all. Ben isn't involved in Ironic
> and until now has had nothing to do with this work to add mimic here, so
> I'm not sure where he got that impression from, or why he's speaking on
> our behalf as to the goals here.
> 
> As pointed out before, the risk of a delta between the mocks in mimic
> and reality is identical to the risk of a delta between the mocks in a
> client's unit tests and reality, so I don't see a particular downside
> there.
> 
> Again, I think "benefit is unclear" isn't a valid reason to block this,
> so unless someone posts a revert we're going to move forward with this.

I have not made a decision myself about whether it's right to go
ahead or not, but if we need to have a revert in place to continue
the conversation, I'll do that.  Please see
https://review.openstack.org/#/c/265416/ as the patch to revert
adding mimic.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jay Pipes]

On 01/07/2016 07:38 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:

On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:

On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:

On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/


What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.

Requests-mock debugging when things go wrong seems a bit simpler.

This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.

-Sean


We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.


I don't really see the point of a separate project like Mimic that has a
whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
RAX-specific) API services. It's just a great way to introduce a larger
surface area for bugs to creep in -- since you have to keep the Mimic
interfaces up to date with the real interfaces. Better to keep something
like this -- if it is TRULY needed -- in-tree with the API service itself,
so that the chances of divergence are reduced. This is similar to the
fakevirt driver in Nova. It's in tree for good reason: when someone changes
the virt driver interface, the fakevirt driver goes boom and needs to be
changed in a corresponding fashion in the same patch.


I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.

Also, mimic can simulate:

* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)

We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.


What value does a functional test against an HTTP API service that does
nothing (other than introduce greater surface area for bugs) actually offer
over unit tests anyway?


Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.

Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA


OK, I just watched that. Sorry, still don't see the value that Mimic
provides over unit testing the client interfaces and mocking out the HTTP
payloads so you have strict control over the expectations.

The problem that Glyph noted in the video about the unit test that mocked
out os.chdir to improperly return a value isn't solved whatsoever by Mimic,
so I find it odd that that example was used in discussing the value of
Mimic. Bad mocks are just that: bad mocks. The same false positive failure
could easily be introduced with a typo in the "metadata injection" that
Mimic does to inject failures into the system.

Mimic isn't testing anything on the server side at all so I'm not sure why
folks call it "integration testing". It isn't testing the integration of
anything at all. All it enables is multi-language client library testing,
and see my response to Ben, the surface area it introduces for bugs in the
test platform itself in my opinion outweigh the multi-language value it
might have.


FWIW, I've only been calling it functional testing. I also don't care
about the multi-language parts here. The sole purpose we have for mimic
(so far) is to functionally test python-ironicclient without mocking the
world in 

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Julien Danjou]

On Thu, Jan 07 2016, Jay Pipes wrote:

> OK, I just watched that. Sorry, still don't see the value that Mimic provides
> over unit testing the client interfaces and mocking out the HTTP payloads so
> you have strict control over the expectations.
>
> The problem that Glyph noted in the video about the unit test that mocked out
> os.chdir to improperly return a value isn't solved whatsoever by Mimic, so I
> find it odd that that example was used in discussing the value of Mimic. Bad
> mocks are just that: bad mocks. The same false positive failure could easily 
> be
> introduced with a typo in the "metadata injection" that Mimic does to inject
> failures into the system.
>
> Mimic isn't testing anything on the server side at all so I'm not sure why
> folks call it "integration testing". It isn't testing the integration of
> anything at all. All it enables is multi-language client library testing, and
> see my response to Ben, the surface area it introduces for bugs in the test
> platform itself in my opinion outweigh the multi-language value it might have.

I completely agree with what Jay points out here.

To illustrate with what we do in the Telemetry team: in order test
gnocchiclient, we pip install gnocchi¹, configure it briefly and use the
client against that freshly installed server. It works very well, and
we're sure we test against real data. That's what I would call
integration testing.
We could also easily test against older version of the Gnocchi server by
pip install-ing an older version if we wanted.

This has way more value than mocking the whole HTTP server and crossing
fingers we did a good job mimicking it. Oh, and it's also actually less
work. :)

¹  https://github.com/openstack/python-gnocchiclient/blob/master/setup-tests.sh

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Thierry Carrez]

Jim Rollenhagen wrote:

[...]
Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.


A bit of history with Twisted.

Back in 2010 we decided we could not afford asking OpenStack developers 
to be familiar with multiple service architecture frameworks, and 
eventlet was chosen as the simplest framework to learn and debug. The 
best reference I found on this is still visible in the wiki:


https://wiki.openstack.org/wiki/UnifiedServiceArchitecture


So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.


The only friction I see is how many developers would be expected to need 
to learn Twisted in order to complete their jobs. My understanding is 
that Twisted expertise could be needed to debug python-ironicclient 
functional tests, which makes the cost relatively limited. So if Mimic 
brings in a clear and significant benefit, I don't think its Twisted 
dependence should play that much against it.


However, I agree with Sean and Jay that the benefit is unclear -- the 
few features that Mimic brings seem to be outweighed by the increased 
risk of introducing a delta between the implementation and the mock. If 
the main benefit is that it's used in other Rackspace projects for 
testing (like Ben said), I'm not sure that makes a compelling argument 
for the rest of the community...


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Ben Meyer]

On 01/07/2016 06:28 PM, Jay Pipes wrote:
> On 01/07/2016 06:12 PM, Ben Meyer wrote:
>> On 01/07/2016 03:32 PM, Jay Pipes wrote:
>>> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
 On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
>> Hi all,
>>
>> A change to global-requirements[1] introduces mimic, which is an
>> http
>> server that can mock various APIs, including nova and ironic,
>> including
>> control of error codes and timeouts. The ironic team plans to use
>> this
>> for testing python-ironicclient without standing up a full ironic
>> environment.
>>
>> Here's the catch - mimic is built on twisted. I know twisted was
>> previously removed from OpenStack (or at least people said "pls
>> no", I
>> don't know the full history). We didn't intend to stealth-introduce
>> twisted back into g-r, but it was pointed out to me that it may
>> appear
>> this way, so here I am letting everyone know. lifeless pointed out
>> that
>> when tests are failing, people may end up digging into mimic or
>> twisted
>> code, which most people in this community aren't familiar with
>> AFAIK,
>> which is a valid point though I hope it isn't required often.
>>
>> So, the primary question here is: do folks have a problem with
>> adding
>> twisted here? We're holding off on Ironic changes that depend on
>> this
>> until this discussion has happened, but aren't reverting the g-r
>> change
>> until we decide one way or another.
>>
>> // jim
>>
>> [1] https://review.openstack.org/#/c/220268/
>
> What is the advantage of running another server like this over using
> requests-mock (which is used by other OpenStack projects for testing
> today)? The only difference here seems to be that you actually
> execute
> requests code in one case and not in the other.
>
> Requests-mock debugging when things go wrong seems a bit simpler.
>
> This is less about twisted and more about trying to not introduce yet
> another way to mock code in the tree that people need to understand.
>
>  -Sean

 We'd be using this for functional tests, not unit, where we can't
 really
 inject mocks. The idea is that we could run a full functional suite
 against either mimic or a full ironic environment, just by changing a
 test setting.
>>>
>>> I don't really see the point of a separate project like Mimic that has
>>> a whole bunch of reimplementations (mocked out) of all sorts of
>>> OpenStack (and RAX-specific) API services. It's just a great way to
>>> introduce a larger surface area for bugs to creep in -- since you have
>>> to keep the Mimic interfaces up to date with the real interfaces.
>>> Better to keep something like this -- if it is TRULY needed -- in-tree
>>> with the API service itself, so that the chances of divergence are
>>> reduced. This is similar to the fakevirt driver in Nova. It's in tree
>>> for good reason: when someone changes the virt driver interface, the
>>> fakevirt driver goes boom and needs to be changed in a corresponding
>>> fashion in the same patch.
>> A tool like my OpenStackInABox could certainly benefit from the models
>> or services being provided by each project - aside from the complexity
>> that that adds to the installation of installing all the dependencies
>> related instead of just implementing a simple model with a file and
>> sqlite backend that has minimal dependencies...but I digress as I
>> haven't looked at how nova's fakevirt driver installs so may be that's
>> not as big an issue. I'll certainly look at it for use in
>> OpenStackInABox, but even there I'm aiming for a more complete scenario
>> where you can interact with Keystone, Nova, Swift, etc...(e.g auth
>> against a fake Keystone, use the token with the fake nova which
>> validates it against the same fake Keystone instance, same with a fake
>> Swift...).
>
> But your fake Keystone wouldn't be "authing" anything at all. Your
> fake Nova wouldn't be "validating" anything at all.
In my OpenStackInABox functionality, it does do authing. It generates
and maintains in the model auth tokens, and can be configured to reject
all tokens, etc. depending on your test.
Same for my swift mock - it stores the data and generates responses just
like real Swift - in accordance with the API documentation in both cases.

> You aren't *functionally* testing anything of importance above if the
> things you are testing aren't doing what they are supposed to do.
But you are. Sure, you're not going to have the same timing, but it at
least proves the code paths really well. It also reduces the necessary
code to run the tests.

> The only things you are functionally testing are the *clients* to
> those fake HTTP services. And what you are *actually* validating the
> client code for isn't *actually* 

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jay Pipes]

On 01/07/2016 06:12 PM, Ben Meyer wrote:

On 01/07/2016 03:32 PM, Jay Pipes wrote:

On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:

On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic,
including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out
that
when tests are failing, people may end up digging into mimic or
twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r
change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/


What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.

Requests-mock debugging when things go wrong seems a bit simpler.

This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.

 -Sean


We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.


I don't really see the point of a separate project like Mimic that has
a whole bunch of reimplementations (mocked out) of all sorts of
OpenStack (and RAX-specific) API services. It's just a great way to
introduce a larger surface area for bugs to creep in -- since you have
to keep the Mimic interfaces up to date with the real interfaces.
Better to keep something like this -- if it is TRULY needed -- in-tree
with the API service itself, so that the chances of divergence are
reduced. This is similar to the fakevirt driver in Nova. It's in tree
for good reason: when someone changes the virt driver interface, the
fakevirt driver goes boom and needs to be changed in a corresponding
fashion in the same patch.

A tool like my OpenStackInABox could certainly benefit from the models
or services being provided by each project - aside from the complexity
that that adds to the installation of installing all the dependencies
related instead of just implementing a simple model with a file and
sqlite backend that has minimal dependencies...but I digress as I
haven't looked at how nova's fakevirt driver installs so may be that's
not as big an issue. I'll certainly look at it for use in
OpenStackInABox, but even there I'm aiming for a more complete scenario
where you can interact with Keystone, Nova, Swift, etc...(e.g auth
against a fake Keystone, use the token with the fake nova which
validates it against the same fake Keystone instance, same with a fake
Swift...).


But your fake Keystone wouldn't be "authing" anything at all. Your fake 
Nova wouldn't be "validating" anything at all.


You aren't *functionally* testing anything of importance above if the 
things you are testing aren't doing what they are supposed to do.


The only things you are functionally testing are the *clients* to those 
fake HTTP services. And what you are *actually* validating the client 
code for isn't *actually* the real HTTP API service -- it's a fake which 
can have its own surface area for bugginess -- which takes me back to my 
original question: what value does one get from *functional* tests of a 
client that calls a faked-out HTTP API versus *unit* tests of said 
client that simply uses requests-mock (or similar) to set the returned 
value of the HTTP API service to some expected value?



What value does a functional test against an HTTP API service that
does nothing (other than introduce greater surface area for bugs)
actually offer over unit tests anyway?


So to use Nova's fakevirt your project is limited to the same language
as nova - python, correct?


Nova's fakevirt driver simply does NOOP calls for most things and some 
light in-memory bookkeeping of resources that a normal virt driver 
would, in reality, consume from the host. It's a way we have to "test" 
launching many instances in Nova without actually having those instances 
consume resource on the host.

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[David Stanek]

On Thu, Jan 7, 2016 at 3:01 PM, Jim Rollenhagen 
wrote:

> We'd be using this for functional tests, not unit, where we can't really
> inject mocks. The idea is that we could run a full functional suite
> against either mimic or a full ironic environment, just by changing a
> test setting.
>

I'm assuming that this would be for client functional tests because it
wouldn't make sense for testing a server. How is the interface created? It
seems like it would be possible for the mocked API to not match the actual
API.


-- 
David
blog: http://www.traceback.org
twitter: http://twitter.com/dstanek
www: http://dstanek.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Ben Meyer]

On 01/07/2016 03:32 PM, Jay Pipes wrote:
> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
>> On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
>>> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
 Hi all,

 A change to global-requirements[1] introduces mimic, which is an http
 server that can mock various APIs, including nova and ironic,
 including
 control of error codes and timeouts. The ironic team plans to use this
 for testing python-ironicclient without standing up a full ironic
 environment.

 Here's the catch - mimic is built on twisted. I know twisted was
 previously removed from OpenStack (or at least people said "pls no", I
 don't know the full history). We didn't intend to stealth-introduce
 twisted back into g-r, but it was pointed out to me that it may appear
 this way, so here I am letting everyone know. lifeless pointed out
 that
 when tests are failing, people may end up digging into mimic or
 twisted
 code, which most people in this community aren't familiar with AFAIK,
 which is a valid point though I hope it isn't required often.

 So, the primary question here is: do folks have a problem with adding
 twisted here? We're holding off on Ironic changes that depend on this
 until this discussion has happened, but aren't reverting the g-r
 change
 until we decide one way or another.

 // jim

 [1] https://review.openstack.org/#/c/220268/
>>>
>>> What is the advantage of running another server like this over using
>>> requests-mock (which is used by other OpenStack projects for testing
>>> today)? The only difference here seems to be that you actually execute
>>> requests code in one case and not in the other.
>>>
>>> Requests-mock debugging when things go wrong seems a bit simpler.
>>>
>>> This is less about twisted and more about trying to not introduce yet
>>> another way to mock code in the tree that people need to understand.
>>>
>>> -Sean
>>
>> We'd be using this for functional tests, not unit, where we can't really
>> inject mocks. The idea is that we could run a full functional suite
>> against either mimic or a full ironic environment, just by changing a
>> test setting.
>
> I don't really see the point of a separate project like Mimic that has
> a whole bunch of reimplementations (mocked out) of all sorts of
> OpenStack (and RAX-specific) API services. It's just a great way to
> introduce a larger surface area for bugs to creep in -- since you have
> to keep the Mimic interfaces up to date with the real interfaces.
> Better to keep something like this -- if it is TRULY needed -- in-tree
> with the API service itself, so that the chances of divergence are
> reduced. This is similar to the fakevirt driver in Nova. It's in tree
> for good reason: when someone changes the virt driver interface, the
> fakevirt driver goes boom and needs to be changed in a corresponding
> fashion in the same patch.
A tool like my OpenStackInABox could certainly benefit from the models
or services being provided by each project - aside from the complexity
that that adds to the installation of installing all the dependencies
related instead of just implementing a simple model with a file and
sqlite backend that has minimal dependencies...but I digress as I
haven't looked at how nova's fakevirt driver installs so may be that's
not as big an issue. I'll certainly look at it for use in
OpenStackInABox, but even there I'm aiming for a more complete scenario
where you can interact with Keystone, Nova, Swift, etc...(e.g auth
against a fake Keystone, use the token with the fake nova which
validates it against the same fake Keystone instance, same with a fake
Swift...).

> What value does a functional test against an HTTP API service that
> does nothing (other than introduce greater surface area for bugs)
> actually offer over unit tests anyway?

So to use Nova's fakevirt your project is limited to the same language
as nova - python, correct?

The main advantage of mimic is that it is language agnostic and multiple
moving parts (f.e taskflow) can interact with it.

$0.02

Ben


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Thu, Jan 07, 2016 at 07:18:15PM -0500, Jay Pipes wrote:
> On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:
> >On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
> >>On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
> >>>On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
> >Hi all,
> >
> >A change to global-requirements[1] introduces mimic, which is an http
> >server that can mock various APIs, including nova and ironic, including
> >control of error codes and timeouts. The ironic team plans to use this
> >for testing python-ironicclient without standing up a full ironic
> >environment.
> >
> >Here's the catch - mimic is built on twisted. I know twisted was
> >previously removed from OpenStack (or at least people said "pls no", I
> >don't know the full history). We didn't intend to stealth-introduce
> >twisted back into g-r, but it was pointed out to me that it may appear
> >this way, so here I am letting everyone know. lifeless pointed out that
> >when tests are failing, people may end up digging into mimic or twisted
> >code, which most people in this community aren't familiar with AFAIK,
> >which is a valid point though I hope it isn't required often.
> >
> >So, the primary question here is: do folks have a problem with adding
> >twisted here? We're holding off on Ironic changes that depend on this
> >until this discussion has happened, but aren't reverting the g-r change
> >until we decide one way or another.
> >
> >// jim
> >
> >[1] https://review.openstack.org/#/c/220268/
> 
> What is the advantage of running another server like this over using
> requests-mock (which is used by other OpenStack projects for testing
> today)? The only difference here seems to be that you actually execute
> requests code in one case and not in the other.
> 
> Requests-mock debugging when things go wrong seems a bit simpler.
> 
> This is less about twisted and more about trying to not introduce yet
> another way to mock code in the tree that people need to understand.
> 
>   -Sean
> >>>
> >>>We'd be using this for functional tests, not unit, where we can't really
> >>>inject mocks. The idea is that we could run a full functional suite
> >>>against either mimic or a full ironic environment, just by changing a
> >>>test setting.
> >>
> >>I don't really see the point of a separate project like Mimic that has a
> >>whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
> >>RAX-specific) API services. It's just a great way to introduce a larger
> >>surface area for bugs to creep in -- since you have to keep the Mimic
> >>interfaces up to date with the real interfaces. Better to keep something
> >>like this -- if it is TRULY needed -- in-tree with the API service itself,
> >>so that the chances of divergence are reduced. This is similar to the
> >>fakevirt driver in Nova. It's in tree for good reason: when someone changes
> >>the virt driver interface, the fakevirt driver goes boom and needs to be
> >>changed in a corresponding fashion in the same patch.
> >
> >I tend to think our REST APIs are much more stable than internal python
> >APIs, and so we can manage the divergence much easier.
> >
> >Also, mimic can simulate:
> >
> >* old versions (less needed with all the microversion stuff)
> >* old bugs that were shipped
> >* misconfigurations
> >* networking errors
> >* the passage of time (including timeouts)
> >
> >We probably don't want to keep a catalog of old bugs and misconfigurations in
> >tree forever.
> >
> >>What value does a functional test against an HTTP API service that does
> >>nothing (other than introduce greater surface area for bugs) actually offer
> >>over unit tests anyway?
> >
> >Testing the full stack of the client instead of mocking the bottom
> >half of requests is a big one.
> >
> >Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
> >https://www.youtube.com/watch?v=HKUUQme3dwA
> 
> OK, I just watched that. Sorry, still don't see the value that Mimic
> provides over unit testing the client interfaces and mocking out the HTTP
> payloads so you have strict control over the expectations.
> 
> The problem that Glyph noted in the video about the unit test that mocked
> out os.chdir to improperly return a value isn't solved whatsoever by Mimic,
> so I find it odd that that example was used in discussing the value of
> Mimic. Bad mocks are just that: bad mocks. The same false positive failure
> could easily be introduced with a typo in the "metadata injection" that
> Mimic does to inject failures into the system.
> 
> Mimic isn't testing anything on the server side at all so I'm not sure why
> folks call it "integration testing". It isn't testing the integration of
> anything at all. All it enables is multi-language client library testing,
> and see my 

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:
> On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:
> >On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
> >>On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
> >>>Hi all,
> >>>
> >>>A change to global-requirements[1] introduces mimic, which is an http
> >>>server that can mock various APIs, including nova and ironic, including
> >>>control of error codes and timeouts. The ironic team plans to use this
> >>>for testing python-ironicclient without standing up a full ironic
> >>>environment.
> >>>
> >>>Here's the catch - mimic is built on twisted. I know twisted was
> >>>previously removed from OpenStack (or at least people said "pls no", I
> >>>don't know the full history). We didn't intend to stealth-introduce
> >>>twisted back into g-r, but it was pointed out to me that it may appear
> >>>this way, so here I am letting everyone know. lifeless pointed out that
> >>>when tests are failing, people may end up digging into mimic or twisted
> >>>code, which most people in this community aren't familiar with AFAIK,
> >>>which is a valid point though I hope it isn't required often.
> >>>
> >>>So, the primary question here is: do folks have a problem with adding
> >>>twisted here? We're holding off on Ironic changes that depend on this
> >>>until this discussion has happened, but aren't reverting the g-r change
> >>>until we decide one way or another.
> >>>
> >>>// jim
> >>>
> >>>[1] https://review.openstack.org/#/c/220268/
> >>
> >>What is the advantage of running another server like this over using
> >>requests-mock (which is used by other OpenStack projects for testing
> >>today)? The only difference here seems to be that you actually execute
> >>requests code in one case and not in the other.
> >>
> >>Requests-mock debugging when things go wrong seems a bit simpler.
> >>
> >>This is less about twisted and more about trying to not introduce yet
> >>another way to mock code in the tree that people need to understand.
> >>
> >>-Sean
> >
> >We'd be using this for functional tests, not unit, where we can't really
> >inject mocks. The idea is that we could run a full functional suite
> >against either mimic or a full ironic environment, just by changing a
> >test setting.
> 
> I don't really see the point of a separate project like Mimic that has a
> whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
> RAX-specific) API services. It's just a great way to introduce a larger
> surface area for bugs to creep in -- since you have to keep the Mimic
> interfaces up to date with the real interfaces. Better to keep something
> like this -- if it is TRULY needed -- in-tree with the API service itself,
> so that the chances of divergence are reduced. This is similar to the
> fakevirt driver in Nova. It's in tree for good reason: when someone changes
> the virt driver interface, the fakevirt driver goes boom and needs to be
> changed in a corresponding fashion in the same patch.

I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.

Also, mimic can simulate:

* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)

We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.

> What value does a functional test against an HTTP API service that does
> nothing (other than introduce greater surface area for bugs) actually offer
> over unit tests anyway?

Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.

Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA

// jim



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jay Pipes]

On 01/07/2016 06:42 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 03:32:39PM -0500, Jay Pipes wrote:

On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:

On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/


What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.

Requests-mock debugging when things go wrong seems a bit simpler.

This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.

-Sean


We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.


I don't really see the point of a separate project like Mimic that has a
whole bunch of reimplementations (mocked out) of all sorts of OpenStack (and
RAX-specific) API services. It's just a great way to introduce a larger
surface area for bugs to creep in -- since you have to keep the Mimic
interfaces up to date with the real interfaces. Better to keep something
like this -- if it is TRULY needed -- in-tree with the API service itself,
so that the chances of divergence are reduced. This is similar to the
fakevirt driver in Nova. It's in tree for good reason: when someone changes
the virt driver interface, the fakevirt driver goes boom and needs to be
changed in a corresponding fashion in the same patch.


I tend to think our REST APIs are much more stable than internal python
APIs, and so we can manage the divergence much easier.

Also, mimic can simulate:

* old versions (less needed with all the microversion stuff)
* old bugs that were shipped
* misconfigurations
* networking errors
* the passage of time (including timeouts)

We probably don't want to keep a catalog of old bugs and misconfigurations in
tree forever.


What value does a functional test against an HTTP API service that does
nothing (other than introduce greater surface area for bugs) actually offer
over unit tests anyway?


Testing the full stack of the client instead of mocking the bottom
half of requests is a big one.

Lekha and Glyph gave a talk on Mimic in Paris; it may be enlightening.
https://www.youtube.com/watch?v=HKUUQme3dwA


OK, I just watched that. Sorry, still don't see the value that Mimic 
provides over unit testing the client interfaces and mocking out the 
HTTP payloads so you have strict control over the expectations.


The problem that Glyph noted in the video about the unit test that 
mocked out os.chdir to improperly return a value isn't solved whatsoever 
by Mimic, so I find it odd that that example was used in discussing the 
value of Mimic. Bad mocks are just that: bad mocks. The same false 
positive failure could easily be introduced with a typo in the "metadata 
injection" that Mimic does to inject failures into the system.


Mimic isn't testing anything on the server side at all so I'm not sure 
why folks call it "integration testing". It isn't testing the 
integration of anything at all. All it enables is multi-language client 
library testing, and see my response to Ben, the surface area it 
introduces for bugs in the test platform itself in my opinion outweigh 
the multi-language value it might have.


Final question on this... if Mimic is *only* for testing clients, why 
not make it just a dependency for python-ironicclient and not ironic itself?


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Ian Cordasco]

-Original Message-
From: David Stanek <dsta...@dstanek.com>
Reply: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Date: January 7, 2016 at 18:14:23
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject:  Re: [openstack-dev] [all] re-introducing twisted to   
global-requirements

> On Thu, Jan 7, 2016 at 3:01 PM, Jim Rollenhagen  
> wrote:
>  
> > We'd be using this for functional tests, not unit, where we can't really
> > inject mocks. The idea is that we could run a full functional suite
> > against either mimic or a full ironic environment, just by changing a
> > test setting.
> >
>  
> I'm assuming that this would be for client functional tests because it
> wouldn't make sense for testing a server. How is the interface created? It
> seems like it would be possible for the mocked API to not match the actual
> API.

I agree. It's very easy for servers intending to mimic APIs and their behaviour 
to fall out of synchronization with the server they're attempting to mimic.

--  
Ian Cordasco
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Ben Meyer]

On 01/07/2016 02:41 PM, Sean Dague wrote:
> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
>> Hi all,
>>
>> A change to global-requirements[1] introduces mimic, which is an http
>> server that can mock various APIs, including nova and ironic, including
>> control of error codes and timeouts. The ironic team plans to use this
>> for testing python-ironicclient without standing up a full ironic
>> environment.
>>
>> Here's the catch - mimic is built on twisted. I know twisted was
>> previously removed from OpenStack (or at least people said "pls no", I
>> don't know the full history). We didn't intend to stealth-introduce
>> twisted back into g-r, but it was pointed out to me that it may appear
>> this way, so here I am letting everyone know. lifeless pointed out that
>> when tests are failing, people may end up digging into mimic or twisted
>> code, which most people in this community aren't familiar with AFAIK,
>> which is a valid point though I hope it isn't required often.
>>
>> So, the primary question here is: do folks have a problem with adding
>> twisted here? We're holding off on Ironic changes that depend on this
>> until this discussion has happened, but aren't reverting the g-r change
>> until we decide one way or another.
>>
>> // jim
>>
>> [1] https://review.openstack.org/#/c/220268/
> What is the advantage of running another server like this over using
> requests-mock (which is used by other OpenStack projects for testing
> today)? The only difference here seems to be that you actually execute
> requests code in one case and not in the other.
>
> Requests-mock debugging when things go wrong seems a bit simpler.
>
> This is less about twisted and more about trying to not introduce yet
> another way to mock code in the tree that people need to understand.

As an outsider and lurker...this is a fair point. I also maintain
another project - StackInABox (and OpenStackInABox) - that resolves the
issue in a slightly different way that utilizes requests-mock to do
something similar, but closer to true unit tests.

The main advantage of mimic (https://pypi.python.org/pypi/mimic) is that
it can be independent of language framework (e.g you can use it for C++
or Java or Python2, Python3, or...) as its an independent process from
your testing, and the tasks can be handled very asynchronously so it can
mock out the responses for various tasks in a timely manner, and several
parts of a system can interact together during a test. You're also
utilizing a common mock of the entire system so multiple projects can
benefit from the same code base. It's kind of integration-test but not
quite - a good fit for projects where unit tests need multiple other
components working together, e.g services that use taskflow would
certainly benefit from using mimic.

The disadvantage of mimic from what I can tell is how the tests are
setup. But I'll leave that more to the mimic users and maintainers
(OpenStack Poppy, Rackspace Autoscale, etc) to explain.

For comparison: My StackInABox project provides a framework upon which
OpenStackInABox builds to provide a common model so multiple projects
benefit. I've used StackInABox on several of my own (non-OpenStack
projects) and my unit tests (run by tox) have been able to be nearly
complete integration tests. Aside from the slight use-case difference
(as exemplified by Rackspace AutoScale),  the main advantage of mimic
over my OpenStackInABox is the momentum behind it due to its use by
Rackspace AutoScale and OpenStack Poppy; I just don't have the resources
to build out the models in OpenStackInABox efficiently or quickly
(though I'd certainly welcome the help to do so). Advantage of the
StackInABox approach is every unit test is self-contained and you don't
have to change code to run the test in most cases (where you do, it's
minimal) - just like with using requests-mock; but you can't have two
services independently hit it - like with Rackspace AutoScale.

HTH,

Ben

P.S here's my two projects:
https://github.com/BenjamenMeyer/stackInABox ~
https://pypi.python.org/pypi/stackinabox
https://github.com/BenjamenMeyer/openstackinabox ~
https://pypi.python.org/pypi/openstackinabox

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jay Pipes]

On 01/07/2016 03:01 PM, Jim Rollenhagen wrote:

On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:

On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:

Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/


What is the advantage of running another server like this over using
requests-mock (which is used by other OpenStack projects for testing
today)? The only difference here seems to be that you actually execute
requests code in one case and not in the other.

Requests-mock debugging when things go wrong seems a bit simpler.

This is less about twisted and more about trying to not introduce yet
another way to mock code in the tree that people need to understand.

-Sean


We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.


I don't really see the point of a separate project like Mimic that has a 
whole bunch of reimplementations (mocked out) of all sorts of OpenStack 
(and RAX-specific) API services. It's just a great way to introduce a 
larger surface area for bugs to creep in -- since you have to keep the 
Mimic interfaces up to date with the real interfaces. Better to keep 
something like this -- if it is TRULY needed -- in-tree with the API 
service itself, so that the chances of divergence are reduced. This is 
similar to the fakevirt driver in Nova. It's in tree for good reason: 
when someone changes the virt driver interface, the fakevirt driver goes 
boom and needs to be changed in a corresponding fashion in the same patch.


What value does a functional test against an HTTP API service that does 
nothing (other than introduce greater surface area for bugs) actually 
offer over unit tests anyway?


-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

On Thu, Jan 07, 2016 at 02:41:12PM -0500, Sean Dague wrote:
> On 01/07/2016 02:09 PM, Jim Rollenhagen wrote:
> > Hi all,
> > 
> > A change to global-requirements[1] introduces mimic, which is an http
> > server that can mock various APIs, including nova and ironic, including
> > control of error codes and timeouts. The ironic team plans to use this
> > for testing python-ironicclient without standing up a full ironic
> > environment.
> > 
> > Here's the catch - mimic is built on twisted. I know twisted was
> > previously removed from OpenStack (or at least people said "pls no", I
> > don't know the full history). We didn't intend to stealth-introduce
> > twisted back into g-r, but it was pointed out to me that it may appear
> > this way, so here I am letting everyone know. lifeless pointed out that
> > when tests are failing, people may end up digging into mimic or twisted
> > code, which most people in this community aren't familiar with AFAIK,
> > which is a valid point though I hope it isn't required often.
> > 
> > So, the primary question here is: do folks have a problem with adding
> > twisted here? We're holding off on Ironic changes that depend on this
> > until this discussion has happened, but aren't reverting the g-r change
> > until we decide one way or another.
> > 
> > // jim
> > 
> > [1] https://review.openstack.org/#/c/220268/
> 
> What is the advantage of running another server like this over using
> requests-mock (which is used by other OpenStack projects for testing
> today)? The only difference here seems to be that you actually execute
> requests code in one case and not in the other.
> 
> Requests-mock debugging when things go wrong seems a bit simpler.
> 
> This is less about twisted and more about trying to not introduce yet
> another way to mock code in the tree that people need to understand.
> 
>   -Sean

We'd be using this for functional tests, not unit, where we can't really
inject mocks. The idea is that we could run a full functional suite
against either mimic or a full ironic environment, just by changing a
test setting.

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Jay Faulkner]

It's also worth noting that the mimic team, along with other Rackers who work 
on Twistd, all worked to get python 3 support for mimic and associated 
dependencies in order to get this into OpenStack. I think it's safe to say this 
is a very friendly upstream and will help resolve any issues we might suss out.


Thanks,

Jay Faulkner


From: Dmitry Tantsur <divius.ins...@gmail.com>
Sent: Thursday, January 7, 2016 11:35 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [all] re-introducing twisted to global-requirements

2016-01-07 20:09 GMT+01:00 Jim Rollenhagen 
<j...@jimrollenhagen.com<mailto:j...@jimrollenhagen.com>>:
Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

Btw, I've spent some amount of time (5 years?) with twisted on my previous 
jobs. While my memory is no longer fresh on it, I can definitely be pinged to 
help with it, if problems appear.


So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
--
-- Dmitry Tantsur
--
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Dmitry Tantsur]

2016-01-07 20:09 GMT+01:00 Jim Rollenhagen :

> Hi all,
>
> A change to global-requirements[1] introduces mimic, which is an http
> server that can mock various APIs, including nova and ironic, including
> control of error codes and timeouts. The ironic team plans to use this
> for testing python-ironicclient without standing up a full ironic
> environment.
>
> Here's the catch - mimic is built on twisted. I know twisted was
> previously removed from OpenStack (or at least people said "pls no", I
> don't know the full history). We didn't intend to stealth-introduce
> twisted back into g-r, but it was pointed out to me that it may appear
> this way, so here I am letting everyone know. lifeless pointed out that
> when tests are failing, people may end up digging into mimic or twisted
> code, which most people in this community aren't familiar with AFAIK,
> which is a valid point though I hope it isn't required often.
>

Btw, I've spent some amount of time (5 years?) with twisted on my previous
jobs. While my memory is no longer fresh on it, I can definitely be pinged
to help with it, if problems appear.


>
> So, the primary question here is: do folks have a problem with adding
> twisted here? We're holding off on Ironic changes that depend on this
> until this discussion has happened, but aren't reverting the g-r change
> until we decide one way or another.
>
> // jim
>
> [1] https://review.openstack.org/#/c/220268/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
--
-- Dmitry Tantsur
--
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] re-introducing twisted to global-requirements

[Jim Rollenhagen]

Hi all,

A change to global-requirements[1] introduces mimic, which is an http
server that can mock various APIs, including nova and ironic, including
control of error codes and timeouts. The ironic team plans to use this
for testing python-ironicclient without standing up a full ironic
environment.

Here's the catch - mimic is built on twisted. I know twisted was
previously removed from OpenStack (or at least people said "pls no", I
don't know the full history). We didn't intend to stealth-introduce
twisted back into g-r, but it was pointed out to me that it may appear
this way, so here I am letting everyone know. lifeless pointed out that
when tests are failing, people may end up digging into mimic or twisted
code, which most people in this community aren't familiar with AFAIK,
which is a valid point though I hope it isn't required often.

So, the primary question here is: do folks have a problem with adding
twisted here? We're holding off on Ironic changes that depend on this
until this discussion has happened, but aren't reverting the g-r change
until we decide one way or another.

// jim

[1] https://review.openstack.org/#/c/220268/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] re-introducing twisted to global-requirements

[Davanum Srinivas]

A bit more information. the dependency for twisted is not in
global-requirements.txt, it will be only added to
upper-constraints.txt when the CI job/bot proposes it next.

Thanks,
Dims

On Thu, Jan 7, 2016 at 2:09 PM, Jim Rollenhagen  wrote:
> Hi all,
>
> A change to global-requirements[1] introduces mimic, which is an http
> server that can mock various APIs, including nova and ironic, including
> control of error codes and timeouts. The ironic team plans to use this
> for testing python-ironicclient without standing up a full ironic
> environment.
>
> Here's the catch - mimic is built on twisted. I know twisted was
> previously removed from OpenStack (or at least people said "pls no", I
> don't know the full history). We didn't intend to stealth-introduce
> twisted back into g-r, but it was pointed out to me that it may appear
> this way, so here I am letting everyone know. lifeless pointed out that
> when tests are failing, people may end up digging into mimic or twisted
> code, which most people in this community aren't familiar with AFAIK,
> which is a valid point though I hope it isn't required often.
>
> So, the primary question here is: do folks have a problem with adding
> twisted here? We're holding off on Ironic changes that depend on this
> until this discussion has happened, but aren't reverting the g-r change
> until we decide one way or another.
>
> // jim
>
> [1] https://review.openstack.org/#/c/220268/
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev