Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2015-01-09 Thread Matt Riedemann



On 1/8/2015 1:28 PM, Matt Riedemann wrote:



On 4/25/2014 4:13 AM, Alexandre Levine wrote:

Joe,

In regard to your first question - yes we'll be going in this direction
very soon. It's being discussed with Randy now.
As for the second question - we'd love to participate in fixing it (in
fact we've done it for OCS already) and probably maintaining it but I'm
not sure what it takes and means to commit to this - we'll discuss it as
well.

Best regards,
   Alex Levine

24.04.2014 23:33, Joe Gordon пишет:




On Thu, Apr 24, 2014 at 10:10 AM, Alexandre Levine
alev...@cloudscaling.com mailto:alev...@cloudscaling.com wrote:

Cristopher,

FYI in regard to 


Its the sort of direction that we tried to steer the GCE
API folks in I
cehouse, though I don't know what they ended up doing



We ended up perfectly ok. The project is on Stackforge for some
time https://github.com/stackforge/gce-api. It works.
I believe that this is exactly what should be done with EC2 as
well. We even considered and tried to estimate it once.

I can tell you even more that we do have lots of AWS Tempest tests
specifically to check various compatibility issues in OpenStack.
And we've created a number of fixes for proprietary implementation
of a cloud based on OpenStack. Some of them are in EC2 layer, some
are in nova core.


Any plans to contribute this to the community?


But anyways, I'm completely convinced that:

1. Any further improvements to EC2 layer should be done after its
separation from nova.


So the fundamental problem we are having with Nova's EC2
implementation is that no one is maintaining it upstream.  If pulling
EC2 out of nova into its own repo solves this problem then wonderful.
But the status quo is untenable, Nova does not want to ship code that
we know to be broken, so we need folks interested in it to help fix it.

2. EC2 should still somehow be supported by OpenStack because as
far as I know lots of people use euca2ools to access it.


Best regards,
  Alex Levine

24.04.2014 19 tel:24.04.2014%2019:24, Christopher Yeoh пишет:

On Thu, 24 Apr 2014 09:10:19 +1000
Michael Still mi...@stillhq.com mailto:mi...@stillhq.com
wrote:

These seem like the obvious places to talk to people about
helping us
get this code maintained before we're forced to drop it.
Unfortunately
we can't compel people to work on things, but we can make
it in their
best interests.

A followup question as well -- there's a proposal to
implement the
Nova v2 API on top of the v3 API. Is something similar
possible with
EC2? Most of the details of EC2 have fallen out of my
brain, but I'd
be very interested in if such a thing is possible.

So there's sort of a couple of ways we suggested doing a V2
API on top
of V3 long term. The current most promising proposal (and I
think
Kenichi has covered this a bit in another email) is a very
thin layer
inside the Nova API code. This works well because the V2 and
V3 APIs in
many areas are very closely related anyway - so emulation is
straightforward.

However there is another alternative (which I don't think is
necessary
for V2) and that is to have a more fuller fledged type proxy
where
translation is say done between receiving V2 requests and
translating
them to native V3 API requests. Responses are similarly
translated but
in reverse. Its the sort of direction that we tried to steer
the GCE
API folks in Icehouse, though I don't know what they ended up
doing -
IIRC I think they said it would be possible.

Longer term I suspect its something we should consider if we
could do
something like that for the EC2 API and then be able to rip
out the
ec2 API specific code from the nova API part of tree. The
messiness of
any UUID or state map translation perhaps could then be
handled in a
very isolated manner from the core Nova code (though I won't
pretend to
understand the specifics of what is required here). I guess the
critical question will be if the emulation of the EC2 API is
good
enough, but as Sean points out - there are lots of existing
issues
already so it may end up not perfect, but still much better
than what we
have now.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___

Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2015-01-08 Thread Matt Riedemann



On 4/25/2014 4:13 AM, Alexandre Levine wrote:

Joe,

In regard to your first question - yes we'll be going in this direction
very soon. It's being discussed with Randy now.
As for the second question - we'd love to participate in fixing it (in
fact we've done it for OCS already) and probably maintaining it but I'm
not sure what it takes and means to commit to this - we'll discuss it as
well.

Best regards,
   Alex Levine

24.04.2014 23:33, Joe Gordon пишет:




On Thu, Apr 24, 2014 at 10:10 AM, Alexandre Levine
alev...@cloudscaling.com mailto:alev...@cloudscaling.com wrote:

Cristopher,

FYI in regard to 


Its the sort of direction that we tried to steer the GCE
API folks in I
cehouse, though I don't know what they ended up doing



We ended up perfectly ok. The project is on Stackforge for some
time https://github.com/stackforge/gce-api. It works.
I believe that this is exactly what should be done with EC2 as
well. We even considered and tried to estimate it once.

I can tell you even more that we do have lots of AWS Tempest tests
specifically to check various compatibility issues in OpenStack.
And we've created a number of fixes for proprietary implementation
of a cloud based on OpenStack. Some of them are in EC2 layer, some
are in nova core.


Any plans to contribute this to the community?


But anyways, I'm completely convinced that:

1. Any further improvements to EC2 layer should be done after its
separation from nova.


So the fundamental problem we are having with Nova's EC2
implementation is that no one is maintaining it upstream.  If pulling
EC2 out of nova into its own repo solves this problem then wonderful.
But the status quo is untenable, Nova does not want to ship code that
we know to be broken, so we need folks interested in it to help fix it.

2. EC2 should still somehow be supported by OpenStack because as
far as I know lots of people use euca2ools to access it.


Best regards,
  Alex Levine

24.04.2014 19 tel:24.04.2014%2019:24, Christopher Yeoh пишет:

On Thu, 24 Apr 2014 09:10:19 +1000
Michael Still mi...@stillhq.com mailto:mi...@stillhq.com
wrote:

These seem like the obvious places to talk to people about
helping us
get this code maintained before we're forced to drop it.
Unfortunately
we can't compel people to work on things, but we can make
it in their
best interests.

A followup question as well -- there's a proposal to
implement the
Nova v2 API on top of the v3 API. Is something similar
possible with
EC2? Most of the details of EC2 have fallen out of my
brain, but I'd
be very interested in if such a thing is possible.

So there's sort of a couple of ways we suggested doing a V2
API on top
of V3 long term. The current most promising proposal (and I think
Kenichi has covered this a bit in another email) is a very
thin layer
inside the Nova API code. This works well because the V2 and
V3 APIs in
many areas are very closely related anyway - so emulation is
straightforward.

However there is another alternative (which I don't think is
necessary
for V2) and that is to have a more fuller fledged type proxy where
translation is say done between receiving V2 requests and
translating
them to native V3 API requests. Responses are similarly
translated but
in reverse. Its the sort of direction that we tried to steer
the GCE
API folks in Icehouse, though I don't know what they ended up
doing -
IIRC I think they said it would be possible.

Longer term I suspect its something we should consider if we
could do
something like that for the EC2 API and then be able to rip
out the
ec2 API specific code from the nova API part of tree. The
messiness of
any UUID or state map translation perhaps could then be
handled in a
very isolated manner from the core Nova code (though I won't
pretend to
understand the specifics of what is required here). I guess the
critical question will be if the emulation of the EC2 API is good
enough, but as Sean points out - there are lots of existing issues
already so it may end up not perfect, but still much better
than what we
have now.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list

Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-25 Thread Alexandre Levine

Joe,

In regard to your first question - yes we'll be going in this direction 
very soon. It's being discussed with Randy now.
As for the second question - we'd love to participate in fixing it (in 
fact we've done it for OCS already) and probably maintaining it but I'm 
not sure what it takes and means to commit to this - we'll discuss it as 
well.


Best regards,
  Alex Levine

24.04.2014 23:33, Joe Gordon ?:




On Thu, Apr 24, 2014 at 10:10 AM, Alexandre Levine 
alev...@cloudscaling.com mailto:alev...@cloudscaling.com wrote:


Cristopher,

FYI in regard to 


Its the sort of direction that we tried to steer the GCE
API folks in I
cehouse, though I don't know what they ended up doing



We ended up perfectly ok. The project is on Stackforge for some
time https://github.com/stackforge/gce-api. It works.
I believe that this is exactly what should be done with EC2 as
well. We even considered and tried to estimate it once.

I can tell you even more that we do have lots of AWS Tempest tests
specifically to check various compatibility issues in OpenStack.
And we've created a number of fixes for proprietary implementation
of a cloud based on OpenStack. Some of them are in EC2 layer, some
are in nova core. 



Any plans to contribute this to the community?


But anyways, I'm completely convinced that:

1. Any further improvements to EC2 layer should be done after its
separation from nova. 



So the fundamental problem we are having with Nova's EC2 
implementation is that no one is maintaining it upstream.  If pulling 
EC2 out of nova into its own repo solves this problem then wonderful. 
But the status quo is untenable, Nova does not want to ship code that 
we know to be broken, so we need folks interested in it to help fix it.


2. EC2 should still somehow be supported by OpenStack because as
far as I know lots of people use euca2ools to access it.


Best regards,
  Alex Levine

24.04.2014 19 tel:24.04.2014%2019:24, Christopher Yeoh ?:

On Thu, 24 Apr 2014 09:10:19 +1000
Michael Still mi...@stillhq.com mailto:mi...@stillhq.com
wrote:

These seem like the obvious places to talk to people about
helping us
get this code maintained before we're forced to drop it.
Unfortunately
we can't compel people to work on things, but we can make
it in their
best interests.

A followup question as well -- there's a proposal to
implement the
Nova v2 API on top of the v3 API. Is something similar
possible with
EC2? Most of the details of EC2 have fallen out of my
brain, but I'd
be very interested in if such a thing is possible.

So there's sort of a couple of ways we suggested doing a V2
API on top
of V3 long term. The current most promising proposal (and I think
Kenichi has covered this a bit in another email) is a very
thin layer
inside the Nova API code. This works well because the V2 and
V3 APIs in
many areas are very closely related anyway - so emulation is
straightforward.

However there is another alternative (which I don't think is
necessary
for V2) and that is to have a more fuller fledged type proxy where
translation is say done between receiving V2 requests and
translating
them to native V3 API requests. Responses are similarly
translated but
in reverse. Its the sort of direction that we tried to steer
the GCE
API folks in Icehouse, though I don't know what they ended up
doing -
IIRC I think they said it would be possible.

Longer term I suspect its something we should consider if we
could do
something like that for the EC2 API and then be able to rip
out the
ec2 API specific code from the nova API part of tree. The
messiness of
any UUID or state map translation perhaps could then be
handled in a
very isolated manner from the core Nova code (though I won't
pretend to
understand the specifics of what is required here). I guess the
critical question will be if the emulation of the EC2 API is good
enough, but as Sean points out - there are lots of existing issues
already so it may end up not perfect, but still much better
than what we
have now.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Thierry Carrez
Michael Still wrote:
 On Thu, Apr 24, 2014 at 7:39 AM, Joe Gordon joe.gord...@gmail.com wrote:
 
 So no one is seriously discussing moving EC2 out of nova right now. The
 issue is that the EC2 code and tempest tests aren't being maintained are
 slowly code rotting. The goal of this thread is to get some volunteers to
 work on EC2.

 I'd like to see if there are any more people interested in keeping these
 interfaces functional (by contributing both on the nova and tempest
 sides). If so, great!
 
 Sure, but if this code continues being ignored, then I don't see how
 we have a choice long term.
 
 So a few questions:
 
  - who currently ships a private cloud with EC2 enabled?
  - who has a public cloud with EC2 enabled?

In the user survey from October 2013, about 30% of respondents claim to
have the EC2 API enabled.

It's not the first time we make the sad observation that noone actually
cares enough about the EC2 API to invest the one FTE that would be
required to maintain it in good shape. That session is up at every
Design Summit, but every time we get various resource pledges that never
pan out.

Now that we have raised the QA bar for hypervisors, plugins and backend
drivers to stay in mainline code, it's only a matter of time until the
EC2 API is held to the same standards... I think it's important that we
keep that API though, so I really hope someone will step up soon.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Sean Dague
On 04/24/2014 05:12 AM, Thierry Carrez wrote:
 Michael Still wrote:
 On Thu, Apr 24, 2014 at 7:39 AM, Joe Gordon joe.gord...@gmail.com wrote:

 So no one is seriously discussing moving EC2 out of nova right now. The
 issue is that the EC2 code and tempest tests aren't being maintained are
 slowly code rotting. The goal of this thread is to get some volunteers to
 work on EC2.

 I'd like to see if there are any more people interested in keeping these
 interfaces functional (by contributing both on the nova and tempest
 sides). If so, great!

 Sure, but if this code continues being ignored, then I don't see how
 we have a choice long term.

 So a few questions:

  - who currently ships a private cloud with EC2 enabled?
  - who has a public cloud with EC2 enabled?
 
 In the user survey from October 2013, about 30% of respondents claim to
 have the EC2 API enabled.
 
 It's not the first time we make the sad observation that noone actually
 cares enough about the EC2 API to invest the one FTE that would be
 required to maintain it in good shape. That session is up at every
 Design Summit, but every time we get various resource pledges that never
 pan out.
 
 Now that we have raised the QA bar for hypervisors, plugins and backend
 drivers to stay in mainline code, it's only a matter of time until the
 EC2 API is held to the same standards... I think it's important that we
 keep that API though, so I really hope someone will step up soon.

It's also important to realize that outside of *very* basic stuff, it
doesn't really work very well. For instance, the bug in question that I
was diving on was a simple scenario of booting a compute, attaching a
volume, detaching the volume, and bringing down the compute.

It failed at some noticable rate every day. The issue was around the
fact that in OpenStack we use a single status for volume lifecycle. EC2
uses 2. So we have to collapse our 'attaching' and 'detaching' states
into 'in-use'. We were waiting for 'in-use' before proceeding to the
volume detach. That's not sufficient. There is a second status in EC2
land that tells the attachment state. The first fix was to check for:
not it ('attaching', 'detaching'). But it turns out we can't do that,
because the EC2 implementation in Nova never actually sets those states.
It only sets the attachment state to None or 'attached'. So we changed that.

All of this was to handle the fact that volumes.detach() with boto on an
unattached volume is a 500 - Unidentified error that creates a stack
trace in n-cpu. And if you get there first, you are clearly going to
leak a volume.

Which means our test will no longer explode, and randomly fail unrelated
code (Good). However things are still pretty broken in the tear down
path (Bad). And for normal use there is still a volume leak path (Bad).

This issue has existed for a long time. In past we just turned off this
test for some period of time because no one seemed willing to dive and
debug what was actually going on. It took me a linear week to get to the
bottom of this, realistically it was probably about a day or two of
actual work on this one to unwind what was going on and figure out where
our race in the test was.

I don't intend to fix the underlying Nova issue (don't explode and don't
leak), because the EC2 paths in Nova need more than random fixes, they
need some real love. My interest was sorting out why this was impacting
unrelated code changes, and I believe we have a test case work around
for that now.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Christopher Yeoh
On Thu, 24 Apr 2014 09:10:19 +1000
Michael Still mi...@stillhq.com wrote:
 
 These seem like the obvious places to talk to people about helping us
 get this code maintained before we're forced to drop it. Unfortunately
 we can't compel people to work on things, but we can make it in their
 best interests.
 
 A followup question as well -- there's a proposal to implement the
 Nova v2 API on top of the v3 API. Is something similar possible with
 EC2? Most of the details of EC2 have fallen out of my brain, but I'd
 be very interested in if such a thing is possible.

So there's sort of a couple of ways we suggested doing a V2 API on top
of V3 long term. The current most promising proposal (and I think
Kenichi has covered this a bit in another email) is a very thin layer
inside the Nova API code. This works well because the V2 and V3 APIs in
many areas are very closely related anyway - so emulation is
straightforward.

However there is another alternative (which I don't think is necessary
for V2) and that is to have a more fuller fledged type proxy where
translation is say done between receiving V2 requests and translating
them to native V3 API requests. Responses are similarly translated but
in reverse. Its the sort of direction that we tried to steer the GCE
API folks in Icehouse, though I don't know what they ended up doing -
IIRC I think they said it would be possible.

Longer term I suspect its something we should consider if we could do
something like that for the EC2 API and then be able to rip out the
ec2 API specific code from the nova API part of tree. The messiness of
any UUID or state map translation perhaps could then be handled in a
very isolated manner from the core Nova code (though I won't pretend to
understand the specifics of what is required here). I guess the
critical question will be if the emulation of the EC2 API is good
enough, but as Sean points out - there are lots of existing issues
already so it may end up not perfect, but still much better than what we
have now.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Alexandre Levine

Cristopher,

FYI in regard to 

Its the sort of direction that we tried to steer the GCE
API folks in I
cehouse, though I don't know what they ended up doing



We ended up perfectly ok. The project is on Stackforge for some time 
https://github.com/stackforge/gce-api. It works.
I believe that this is exactly what should be done with EC2 as well. We even 
considered and tried to estimate it once.

I can tell you even more that we do have lots of AWS Tempest tests specifically 
to check various compatibility issues in OpenStack. And we've created a number 
of fixes for proprietary implementation of a cloud based on OpenStack. Some of 
them are in EC2 layer, some are in nova core.

But anyways, I'm completely convinced that:

1. Any further improvements to EC2 layer should be done after its separation 
from nova.
2. EC2 should still somehow be supported by OpenStack because as far as I know 
lots of people use euca2ools to access it.

Best regards,
  Alex Levine

24.04.2014 19:24, Christopher Yeoh пишет:

On Thu, 24 Apr 2014 09:10:19 +1000
Michael Still mi...@stillhq.com wrote:

These seem like the obvious places to talk to people about helping us
get this code maintained before we're forced to drop it. Unfortunately
we can't compel people to work on things, but we can make it in their
best interests.

A followup question as well -- there's a proposal to implement the
Nova v2 API on top of the v3 API. Is something similar possible with
EC2? Most of the details of EC2 have fallen out of my brain, but I'd
be very interested in if such a thing is possible.

So there's sort of a couple of ways we suggested doing a V2 API on top
of V3 long term. The current most promising proposal (and I think
Kenichi has covered this a bit in another email) is a very thin layer
inside the Nova API code. This works well because the V2 and V3 APIs in
many areas are very closely related anyway - so emulation is
straightforward.

However there is another alternative (which I don't think is necessary
for V2) and that is to have a more fuller fledged type proxy where
translation is say done between receiving V2 requests and translating
them to native V3 API requests. Responses are similarly translated but
in reverse. Its the sort of direction that we tried to steer the GCE
API folks in Icehouse, though I don't know what they ended up doing -
IIRC I think they said it would be possible.

Longer term I suspect its something we should consider if we could do
something like that for the EC2 API and then be able to rip out the
ec2 API specific code from the nova API part of tree. The messiness of
any UUID or state map translation perhaps could then be handled in a
very isolated manner from the core Nova code (though I won't pretend to
understand the specifics of what is required here). I guess the
critical question will be if the emulation of the EC2 API is good
enough, but as Sean points out - there are lots of existing issues
already so it may end up not perfect, but still much better than what we
have now.

Chris

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Rushi Agrawal
Thanks for bringing this up. We at Reliance are building a team completely
dedicated to make the EC2 API support in OpenStack rock-solid and complete.
Coming from block-storage background, I submitted a few blueprints[1][2][3]
for the volumes stuff, along with the code for them[4][5][6] in the
Icehouse release. Unfortunately, the reviews didn't get the attention they
required and they were pushed to Juno (probably, partly due to it being
residing in Nova but actually being related to Cinder).

I was the only person in our team for a few months to take care of efforts
in this direction. The process of adding more resources has already been
started, although I am expecting that it will take a month of time before
they start actively contributing to upstream code.

The point I am trying to make is that there are people interested in
keeping this layer functional and in a healthy state. Now that we have
promised in the writing that we will dedicate efforts, I think waiting till
end of Icehouse is a good idea. Let this cycle be our last chance to show
that we care for this piece of code :)

I will make sure I make it clear to my team that making the EC2 API
'robust' is more important than making it 'feature complete'. As you can
see from my contributions, I have been guilty of not following this myself
in the past.

Links:
1. https://blueprints.launchpad.net/nova/+spec/ec2-volume-and-snapshot-tags
2. https://blueprints.launchpad.net/nova/+spec/ec2-volume-type
3. https://blueprints.launchpad.net/nova/+spec/ec2-volume-filtering

4. [DescribeVolumes all filters](https://review.openstack.org/#/c/70085/)
5. [EC2 Volume type](https://review.openstack.org/#/c/61041/)
6. [EC2 volume tags(metadata)](https://review.openstack.org/#/c/64690/)

Regards,
Rushi Agrawal
Cloud engineer,
Reliance Jio Infocomm

On 23 April 2014 17:17, Sean Dague s...@dague.net wrote:

 I've spent a couple of days getting to the bottom of:

 Bug 1302774 - Failed to detach volume because of volume not found error
 prevents vm teardown

 This is an ec2 specific failure path, which mostly looks like a
 combination of a not very good test case and the EC2 code in nova
 collapsing the volume states in a way that seems completely incorrect
 based on what I can read on what's expected from this call.

 However, these are symptoms of a bigger issue. The EC2 paths in Nova are
 old, fragile, and error prone. The test coverage for these paths is
 minimal, and largely hasn't evolved in the last year. The last
 substantial addition to the EC2 tests in Tempest was by Burt Holtzman in
 July 2013, Burt has also been contributing to the Nova side, but beyond
 Burt, there basically aren't contributors right now.

 I really don't like shipping code in Nova that we know isn't good. With
 very few contributions in this code though, it's defacto, if not
 officially, deprecated.

 I'd like to see if there are any more people interested in keeping these
 interfaces functional (by contributing both on the nova and tempest
 sides). If so, great!

 If we get to the end of Juno in the current state, I think we need to
 consider actually deprecating the EC2 support in Nova. Because I'm
 pretty sure what we have today actually only works if you are using boto
 on the client side, and doesn't really look like EC2 at any real level
 of inspection.

 -Sean

 --
 Sean Dague
 http://dague.net


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





Regards,
Rushi Agrawal
Cloud engineer,
Reliance Jio Infocomm, India
Ph: (+91) 99 4518 4519
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Joe Gordon
On Thu, Apr 24, 2014 at 10:10 AM, Alexandre Levine alev...@cloudscaling.com
 wrote:

 Cristopher,

 FYI in regard to 


 Its the sort of direction that we tried to steer the GCE
 API folks in I
 cehouse, though I don't know what they ended up doing

 

 We ended up perfectly ok. The project is on Stackforge for some time
 https://github.com/stackforge/gce-api. It works.
 I believe that this is exactly what should be done with EC2 as well. We
 even considered and tried to estimate it once.

 I can tell you even more that we do have lots of AWS Tempest tests
 specifically to check various compatibility issues in OpenStack. And we've
 created a number of fixes for proprietary implementation of a cloud based
 on OpenStack. Some of them are in EC2 layer, some are in nova core.


Any plans to contribute this to the community?



 But anyways, I'm completely convinced that:

 1. Any further improvements to EC2 layer should be done after its
 separation from nova.


So the fundamental problem we are having with Nova's EC2 implementation is
that no one is maintaining it upstream.  If pulling EC2 out of nova into
its own repo solves this problem then wonderful. But the status quo is
untenable, Nova does not want to ship code that we know to be broken, so we
need folks interested in it to help fix it.


 2. EC2 should still somehow be supported by OpenStack because as far as I
 know lots of people use euca2ools to access it.


 Best regards,
   Alex Levine

 24.04.2014 19:24, Christopher Yeoh пишет:

  On Thu, 24 Apr 2014 09:10:19 +1000
 Michael Still mi...@stillhq.com wrote:

 These seem like the obvious places to talk to people about helping us
 get this code maintained before we're forced to drop it. Unfortunately
 we can't compel people to work on things, but we can make it in their
 best interests.

 A followup question as well -- there's a proposal to implement the
 Nova v2 API on top of the v3 API. Is something similar possible with
 EC2? Most of the details of EC2 have fallen out of my brain, but I'd
 be very interested in if such a thing is possible.

 So there's sort of a couple of ways we suggested doing a V2 API on top
 of V3 long term. The current most promising proposal (and I think
 Kenichi has covered this a bit in another email) is a very thin layer
 inside the Nova API code. This works well because the V2 and V3 APIs in
 many areas are very closely related anyway - so emulation is
 straightforward.

 However there is another alternative (which I don't think is necessary
 for V2) and that is to have a more fuller fledged type proxy where
 translation is say done between receiving V2 requests and translating
 them to native V3 API requests. Responses are similarly translated but
 in reverse. Its the sort of direction that we tried to steer the GCE
 API folks in Icehouse, though I don't know what they ended up doing -
 IIRC I think they said it would be possible.

 Longer term I suspect its something we should consider if we could do
 something like that for the EC2 API and then be able to rip out the
 ec2 API specific code from the nova API part of tree. The messiness of
 any UUID or state map translation perhaps could then be handled in a
 very isolated manner from the core Nova code (though I won't pretend to
 understand the specifics of what is required here). I guess the
 critical question will be if the emulation of the EC2 API is good
 enough, but as Sean points out - there are lots of existing issues
 already so it may end up not perfect, but still much better than what we
 have now.

 Chris

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-24 Thread Sean Dague
On 04/24/2014 03:32 PM, Rushi Agrawal wrote:
 Thanks for bringing this up. We at Reliance are building a team
 completely dedicated to make the EC2 API support in OpenStack rock-solid
 and complete. Coming from block-storage background, I submitted a few
 blueprints[1][2][3] for the volumes stuff, along with the code for
 them[4][5][6] in the Icehouse release. Unfortunately, the reviews didn't
 get the attention they required and they were pushed to Juno (probably,
 partly due to it being residing in Nova but actually being related to
 Cinder).
 
 I was the only person in our team for a few months to take care of
 efforts in this direction. The process of adding more resources has
 already been started, although I am expecting that it will take a month
 of time before they start actively contributing to upstream code.
 
 The point I am trying to make is that there are people interested in
 keeping this layer functional and in a healthy state. Now that we have
 promised in the writing that we will dedicate efforts, I think waiting
 till end of Icehouse is a good idea. Let this cycle be our last chance
 to show that we care for this piece of code :)
 
 I will make sure I make it clear to my team that making the EC2 API
 'robust' is more important than making it 'feature complete'. As you can
 see from my contributions, I have been guilty of not following this
 myself in the past.
 
 Links:
 1. https://blueprints.launchpad.net/nova/+spec/ec2-volume-and-snapshot-tags
 2. https://blueprints.launchpad.net/nova/+spec/ec2-volume-type
 3. https://blueprints.launchpad.net/nova/+spec/ec2-volume-filtering
 
 4. [DescribeVolumes all filters](https://review.openstack.org/#/c/70085/)
 5. [EC2 Volume type](https://review.openstack.org/#/c/61041/)
 6. [EC2 volume tags(metadata)](https://review.openstack.org/#/c/64690/)
 
 Regards,
 Rushi Agrawal
 Cloud engineer,
 Reliance Jio Infocomm

Great! happy to see someone stepping up here.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Joe Gordon
On Wed, Apr 23, 2014 at 4:47 AM, Sean Dague s...@dague.net wrote:

 I've spent a couple of days getting to the bottom of:

 Bug 1302774 - Failed to detach volume because of volume not found error
 prevents vm teardown

 This is an ec2 specific failure path, which mostly looks like a
 combination of a not very good test case and the EC2 code in nova
 collapsing the volume states in a way that seems completely incorrect
 based on what I can read on what's expected from this call.

 However, these are symptoms of a bigger issue. The EC2 paths in Nova are
 old, fragile, and error prone. The test coverage for these paths is
 minimal, and largely hasn't evolved in the last year. The last
 substantial addition to the EC2 tests in Tempest was by Burt Holtzman in
 July 2013, Burt has also been contributing to the Nova side, but beyond
 Burt, there basically aren't contributors right now.

 I really don't like shipping code in Nova that we know isn't good. With
 very few contributions in this code though, it's defacto, if not
 officially, deprecated.

 I'd like to see if there are any more people interested in keeping these
 interfaces functional (by contributing both on the nova and tempest
 sides). If so, great!

 If we get to the end of Juno in the current state, I think we need to
 consider actually deprecating the EC2 support in Nova. Because I'm
 pretty sure what we have today actually only works if you are using boto
 on the client side, and doesn't really look like EC2 at any real level
 of inspection.


I Agree with this general sentiment.  I would like to see EC2 support stay,
but if no one is maintaining it and we know its broken we should deprecate
it.



 -Sean

 --
 Sean Dague
 http://dague.net


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Randy Bias
Sean,


We have a bunch of tempest tests we’re ready to push back that increase 
coverage for the EC2 tests.  Also, rather than deprecating the EC2 pieces, 
which are in use by ~35% of the community, I’d like to recommend we move them 
into Stackforge as we did with the GCE APIs.  That way these can become 
optional components installed by those who want to use them.  The other 
advantage is that they can be iterated on out of band.  I would still like to 
see them all in the CI system however.


Thanks,


--Randy

Founder  CEO, Cloudscaling
Board of Directors, OpenStack Foundation
+1 (415) 787-2253 [SMS or voice] 
TWITTER: twitter.com/randybias
LINKEDIN: linkedin.com/in/randybias
CALENDAR: doodle.com/randybias









On Apr 23, 2014, at 4:47 AM, Sean Dague s...@dague.net wrote:

 I've spent a couple of days getting to the bottom of:
 
 Bug 1302774 - Failed to detach volume because of volume not found error
 prevents vm teardown
 
 This is an ec2 specific failure path, which mostly looks like a
 combination of a not very good test case and the EC2 code in nova
 collapsing the volume states in a way that seems completely incorrect
 based on what I can read on what's expected from this call.
 
 However, these are symptoms of a bigger issue. The EC2 paths in Nova are
 old, fragile, and error prone. The test coverage for these paths is
 minimal, and largely hasn't evolved in the last year. The last
 substantial addition to the EC2 tests in Tempest was by Burt Holtzman in
 July 2013, Burt has also been contributing to the Nova side, but beyond
 Burt, there basically aren't contributors right now.
 
 I really don't like shipping code in Nova that we know isn't good. With
 very few contributions in this code though, it's defacto, if not
 officially, deprecated.
 
 I'd like to see if there are any more people interested in keeping these
 interfaces functional (by contributing both on the nova and tempest
 sides). If so, great!
 
 If we get to the end of Juno in the current state, I think we need to
 consider actually deprecating the EC2 support in Nova. Because I'm
 pretty sure what we have today actually only works if you are using boto
 on the client side, and doesn't really look like EC2 at any real level
 of inspection.
 
   -Sean
 
 -- 
 Sean Dague
 http://dague.net
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Sean Dague
On 04/23/2014 02:47 PM, Randy Bias wrote:
 Sean,
 
 
 We have a bunch of tempest tests we’re ready to push back that increase
 coverage for the EC2 tests.  Also, rather than deprecating the EC2
 pieces, which are in use by ~35% of the community, I’d like to recommend
 we move them into Stackforge as we did with the GCE APIs.  That way
 these can become optional components installed by those who want to use
 them.  The other advantage is that they can be iterated on out of band.
  I would still like to see them all in the CI system however.

Assistance on Tempest would be highly appreciated. I'm a little saddened
that there is apparently a large set of out of tree tests that this is
the first we've heard of. These would have been good community
contributions during the icehouse cycle. Tempest runs like any other
part of OpenStack, so a giant dump isn't going to work, and it's going
to have to be proposed in manageable and reviewable chunks.

If people want EC2 in the main CI it needs to stay in Nova. It's either
part of the project, and supported, or it's not, and it's not.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Randy Bias
On Apr 23, 2014, at 12:04 PM, Sean Dague s...@dague.net wrote:
 We have a bunch of tempest tests we’re ready to push back that increase
 coverage for the EC2 tests.  Also, rather than deprecating the EC2
 pieces, which are in use by ~35% of the community, I’d like to recommend
 we move them into Stackforge as we did with the GCE APIs.  That way
 these can become optional components installed by those who want to use
 them.  The other advantage is that they can be iterated on out of band.
 I would still like to see them all in the CI system however.
 
 Assistance on Tempest would be highly appreciated. I'm a little saddened
 that there is apparently a large set of out of tree tests that this is
 the first we've heard of. These would have been good community
 contributions during the icehouse cycle. Tempest runs like any other
 part of OpenStack, so a giant dump isn't going to work, and it's going
 to have to be proposed in manageable and reviewable chunks.
 
 If people want EC2 in the main CI it needs to stay in Nova. It's either
 part of the project, and supported, or it's not, and it's not.


It wasn’t intentional.  More of a resource constraint issue than anything else. 
 Some of us are at small startups and it can be very challenging to prioritize 
beyond our immediate needs.  I’m in the process of standing up one of the 
largest OpenStack private clouds out there right now for a Fortune 10 company.  
That kind of takes precedence.  I’m sorry this happens.  Your email fortunately 
catalyzed me to get off my arse, not to mention Joe Gordon prodding me.

If being in mainline is required for the CI system, then sure, I prefer the EC2 
APIs stay there.  I think there should be some thought given to how to support 
StackForge projects as “nice to haves” in the CI system though.  Doesn’t really 
make sense for something to have to be suddenly thrown over the wall into 
integrated before being in some kind of CI pipeline.  StackForge, as I 
understand it, is the “official” place for incubating projects.  If so, should 
there be some kind of best effort CI system for those being incubated that have 
an intention for becoming integrated??  Maybe I just don’t understand how 
StackForge is supposed to work then.

We’ll get the new tests we have into some manageable chunks for review and 
inclusion into tempest mainline.


Thanks,


—Randy


Ps. Looping in Reliance as I believe they have skin in the game here as well.


Founder  CEO, Cloudscaling
Board of Directors, OpenStack Foundation
+1 (415) 787-2253 [SMS or voice] 
TWITTER: twitter.com/randybias
LINKEDIN: linkedin.com/in/randybias
CALENDAR: doodle.com/randybias

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Jay Pipes
On Wed, 2014-04-23 at 11:47 -0700, Randy Bias wrote:
 Sean,
 
 We have a bunch of tempest tests we’re ready to push back that
 increase coverage for the EC2 tests.  Also, rather than deprecating
 the EC2 pieces, which are in use by ~35% of the community, I’d like to
 recommend we move them into Stackforge as we did with the GCE APIs.
 That way these can become optional components installed by those who
 want to use them.  The other advantage is that they can be iterated on
 out of band.  I would still like to see them all in the CI system
 however.

+1 on all points.

-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Joe Gordon
On Wed, Apr 23, 2014 at 12:39 PM, Randy Bias ran...@cloudscaling.comwrote:

 On Apr 23, 2014, at 12:04 PM, Sean Dague s...@dague.net wrote:

 We have a bunch of tempest tests we’re ready to push back that increase
 coverage for the EC2 tests.  Also, rather than deprecating the EC2
 pieces, which are in use by ~35% of the community, I’d like to recommend
 we move them into Stackforge as we did with the GCE APIs.  That way
 these can become optional components installed by those who want to use
 them.  The other advantage is that they can be iterated on out of band.
 I would still like to see them all in the CI system however.


 Assistance on Tempest would be highly appreciated. I'm a little saddened
 that there is apparently a large set of out of tree tests that this is
 the first we've heard of. These would have been good community
 contributions during the icehouse cycle. Tempest runs like any other
 part of OpenStack, so a giant dump isn't going to work, and it's going
 to have to be proposed in manageable and reviewable chunks.

 If people want EC2 in the main CI it needs to stay in Nova. It's either
 part of the project, and supported, or it's not, and it's not.


 It wasn’t intentional.  More of a resource constraint issue than anything
 else.  Some of us are at small startups and it can be very challenging to
 prioritize beyond our immediate needs.  I’m in the process of standing up
 one of the largest OpenStack private clouds out there right now for a
 Fortune 10 company.  That kind of takes precedence.  I’m sorry this
 happens.  Your email fortunately catalyzed me to get off my arse, not to
 mention Joe Gordon prodding me.

 If being in mainline is required for the CI system, then sure, I prefer
 the EC2 APIs stay there.  I think there should be some thought given to how
 to support StackForge projects as “nice to haves” in the CI system though.
  Doesn’t really make sense for something to have to be suddenly thrown over
 the wall into integrated before being in some kind of CI pipeline.
  StackForge, as I understand it, is the “official” place for incubating
 projects.  If so, should there be some kind of best effort CI system for
 those being incubated that have an intention for becoming integrated??
  Maybe I just don’t understand how StackForge is supposed to work then.


So no one is seriously discussing moving EC2 out of nova right now. The
issue is that the EC2 code and tempest tests aren't being maintained are
slowly code rotting. The goal of this thread is to get some volunteers to
work on EC2.

I'd like to see if there are any more people interested in keeping these
interfaces functional (by contributing both on the nova and tempest
sides). If so, great!



 We’ll get the new tests we have into some manageable chunks for review and
 inclusion into tempest mainline.


Looking forward to it.




 Thanks,


 —Randy


 Ps. Looping in Reliance as I believe they have skin in the game here as
 well.


 Founder  CEO, Cloudscaling
 Board of Directors, OpenStack Foundation
 +1 (415) 787-2253 [SMS or voice]
 TWITTER: twitter.com/randybias
 LINKEDIN: linkedin.com/in/randybias
 CALENDAR: doodle.com/randybias


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Michael Still
On Thu, Apr 24, 2014 at 4:47 AM, Randy Bias ran...@cloudscaling.com wrote:
 ... I’d like to recommend we move
 them into Stackforge as we did with the GCE APIs.  That way these can become
 optional components installed by those who want to use them.  The other
 advantage is that they can be iterated on out of band.  I would still like
 to see them all in the CI system however.

How does state mapping work in the stackforge GCE API? The problem
we've had in the past is that EC2 requires we create synthetic
identifiers for many of our objects and map them to our UUIDs. That
mapping is stored in the nova database, and I would be uncomfortable
with allowing out of tree code to manipulate our database. I'm hoping
the GCE code has solved a similar problem in an interesting way.

Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Michael Still
On Thu, Apr 24, 2014 at 7:39 AM, Joe Gordon joe.gord...@gmail.com wrote:

 So no one is seriously discussing moving EC2 out of nova right now. The
 issue is that the EC2 code and tempest tests aren't being maintained are
 slowly code rotting. The goal of this thread is to get some volunteers to
 work on EC2.

 I'd like to see if there are any more people interested in keeping these
 interfaces functional (by contributing both on the nova and tempest
 sides). If so, great!

Sure, but if this code continues being ignored, then I don't see how
we have a choice long term.

So a few questions:

 - who currently ships a private cloud with EC2 enabled?
 - who has a public cloud with EC2 enabled?

These seem like the obvious places to talk to people about helping us
get this code maintained before we're forced to drop it. Unfortunately
we can't compel people to work on things, but we can make it in their
best interests.

A followup question as well -- there's a proposal to implement the
Nova v2 API on top of the v3 API. Is something similar possible with
EC2? Most of the details of EC2 have fallen out of my brain, but I'd
be very interested in if such a thing is possible.

Michael

-- 
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance

2014-04-23 Thread Kenichi Oomichi


 -Original Message-
 From: Michael Still [mailto:mi...@stillhq.com]
 Sent: Thursday, April 24, 2014 8:10 AM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [nova] [qa] EC2 status and call for assistance
 
 On Thu, Apr 24, 2014 at 7:39 AM, Joe Gordon joe.gord...@gmail.com wrote:
 
  So no one is seriously discussing moving EC2 out of nova right now. The
  issue is that the EC2 code and tempest tests aren't being maintained are
  slowly code rotting. The goal of this thread is to get some volunteers to
  work on EC2.
 
  I'd like to see if there are any more people interested in keeping these
  interfaces functional (by contributing both on the nova and tempest
  sides). If so, great!
 
 Sure, but if this code continues being ignored, then I don't see how
 we have a choice long term.
 
 So a few questions:
 
  - who currently ships a private cloud with EC2 enabled?
  - who has a public cloud with EC2 enabled?
 
 These seem like the obvious places to talk to people about helping us
 get this code maintained before we're forced to drop it. Unfortunately
 we can't compel people to work on things, but we can make it in their
 best interests.
 
 A followup question as well -- there's a proposal to implement the
 Nova v2 API on top of the v3 API. Is something similar possible with
 EC2? Most of the details of EC2 have fallen out of my brain, but I'd
 be very interested in if such a thing is possible.

The difference between v2 and v3 is not so big, and it is easy to
implement v2 API on top of v3 API. However EC2 API is very different
from v2/v3 API, so I think it is difficult to implement EC2 API on top
of v3 API.

Ex) Create a keypair
  EC2[1]:
https://ec2.amazonaws.com/?Action=CreateKeyPair
KeyName=my-key-pair
AUTHPARAMS

  v2[2]:
http://{nova-api}/v2/​{tenant_id}​/os-keypairs
{
keypair: {
name: keypair-dab428fe-6186-4a14-b3de-92131f76cd39,
public_key: ssh-rsa B3Nza[..]== Generated by Nova
}
}

I feel it would be nice to keep current EC2 implementation if necessary.


Thanks
Ken'ichi Ohmichi
---
[1]: 
http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-CreateKeyPair.html
[2]: http://api.openstack.org/api-ref-compute-v2-ext.html
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev