Re: [openstack-dev] [api] API recommendation

2014-10-20 Thread Adam Young

On 10/15/2014 11:49 AM, Kevin L. Mitchell wrote:

Now that we have an API working group forming, I'd like to kick off some
discussion over one point I'd really like to see our APIs using (and
I'll probably drop it in to the repo once that gets fully set up): the
difference between synchronous and asynchronous operations.  Using nova
as an example—right now, if you kick off a long-running operation, such
as a server create or a reboot, you watch the resource itself to
determine the status of the operation.  What I'd like to propose is that
future APIs use a separate operation resource to track status
information on the particular operation.  For instance, if we were to
rebuild the nova API with this idea in mind, booting a new server would
give you a server handle and an operation handle; querying the server
resource would give you summary information about the state of the
server (running, not running) and pending operations, while querying the
operation would give you detailed information about the status of the
operation.  As another example, issuing a reboot would give you the
operation handle; you'd see the operation in a queue on the server
resource, but the actual state of the operation itself would be listed
on that operation.  As a side effect, this would allow us (not require,
though) to queue up operations on a resource, and allow us to cancel an
operation that has not yet been started.

Thoughts?
I'd like to couple this approach with a a greater use of Keystone trusts 
for delegation of authority.  Trusts and async calls are designed to 
work together.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] API recommendation

2014-10-18 Thread Jay Pipes
Sorry for top-posting, but Salvatore and Dean, you might find my 
proposed vNext Compute API interesting, since it does what you are 
describing you'd like to see with regards to task-based APIs:


http://docs.oscomputevnext.apiary.io/

Specifically, the schemas and endpoints for tasks and subtasks:

http://docs.oscomputevnext.apiary.io/#servertask
http://docs.oscomputevnext.apiary.io/#servertaskitem

When you call to POST /servers, you can create multiple servers:

http://docs.oscomputevnext.apiary.io/#server

And the _links JSON-HAL document returned by the POST /servers API call 
contains a describedby hyperlink that points to the JSONSchema 
document for a server object, and a rel/server_tasks hyperlink that 
points to a URL that may be used to determine the tasks and task states 
for any task involved in creating the server.


Best,
-jay

On 10/16/2014 05:57 AM, Salvatore Orlando wrote:

In an analysis we recently did for managing lifecycle of neutron
resources, it also emerged that task (or operation) API are a very
useful resource. Indeed several neutron resources introduced the
(in)famous PENDING_XXX operational statuses to note the fact that an
operation is in progress and its status is changing.

This could have been easily avoided if a facility for querying active
 tasks through the API was available.

From an API guideline viewpoint, I understand that
https://review.openstack.org/#/c/86938/ proposes the introduction of
a rather simple endpoint to query active tasks and filter them by
resource uuid or state, for example. While this is hardly
questionable, I wonder if it might be worth typifying the task, ie:
adding a resource_type attribute, and/or allowing to retrieve active
tasks as a chile resource of an object, eg.: GET
/servers/server_id/tasks?state=running or if just for running tasks
GET /servers/server_id/active_tasks

The proposed approach for the multiple server create case also makes
 sense to me. Other than bulk operations there are indeed cases
where a single API operation needs to perform multiple tasks. For
instance, in Neutron, creating a port implies L2 wiring, setting up
DHCP info, and securing it on the compute node by enforcing
anti-spoof rules and security groups. This means there will be 3/4
active tasks. For this reason I wonder if it might be the case of
differentiating between the concept of operation and tasks where
the former is the activity explicitly initiated by the API consumer,
and the latter are the activities which need to complete to fulfil
it. This is where we might leverage the already proposed request_id
attribute of the task data structure.

Finally, a note on persistency. How long a completed task,
successfully or not should be stored for? Do we want to store them
until the resource they operated on is deleted? I don't think it's a
great idea to store them indefinitely in the DB. Tying their lifespan
to resources is probably a decent idea, but time-based cleanup
policies might also be considered (e.g.: destroy a task record 24
hours after its completion)

Salvatore


On 16 October 2014 08:38, Christopher Yeoh cbky...@gmail.com
mailto:cbky...@gmail.com wrote:

On Thu, Oct 16, 2014 at 7:19 AM, Kevin L. Mitchell
kevin.mitch...@rackspace.com mailto:kevin.mitch...@rackspace.com
wrote:

On Wed, 2014-10-15 at 12:39 -0400, Andrew Laski wrote:

On 10/15/2014 11:49 AM, Kevin L. Mitchell wrote:

Now that we have an API working group forming, I'd like to

kick off some

discussion over one point I'd really like to see our APIs

using (and

I'll probably drop it in to the repo once that gets fully

set up): the

difference between synchronous and asynchronous

operations.  Using nova

as an example—right now, if you kick off a long-running

operation, such

as a server create or a reboot, you watch the resource

itself to

determine the status of the operation.  What I'd like to

propose is that

future APIs use a separate operation resource to track status
information on the particular operation.  For instance, if

we were to

rebuild the nova API with this idea in mind, booting a new

server would

give you a server handle and an operation handle; querying

the server

resource would give you summary information about the state

of the

server (running, not running) and pending operations, while

querying the

operation would give you detailed information about the

status of the

operation.  As another example, issuing a reboot would give

you the

operation handle; you'd see the operation in a queue on the

server

resource, but the actual state of the operation itself

would be listed

on that operation.  As a side effect, this would allow us

(not require,

though) to queue up operations on a resource, and allow us

to cancel an

operation that has not yet been started.

Thoughts?


Something like https://review.openstack.org/#/c/86938/ ?

I know that Jay has proposed a similar thing before as well.

I would

love to get some feedback from others on this as it's

Re: [openstack-dev] [api] API recommendation

2014-10-17 Thread Peter Balland
On Oct 16, 2014 8:24 AM, Dean Troyer dtro...@gmail.com wrote:



 On Thu, Oct 16, 2014 at 4:57 AM, Salvatore Orlando sorla...@nicira.com
wrote:

 From an API guideline viewpoint, I understand that
https://review.openstack.org/#/c/86938/ proposes the introduction of a
rather simple endpoint to query active tasks and filter them by resource
uuid or state, for example.


 That review/blueprint contains one thing that I want to address in more
detail below along with Sal's comment on persistence...


 While this is hardly questionable, I wonder if it might be worth
typifying the task, ie: adding a resource_type attribute, and/or allowing
to retrieve active tasks as a chile resource of an object, eg.: GET
/servers/server_id/tasks?state=running or if just for running tasks GET
/servers/server_id/active_tasks


 I'd prefer the filter approach, but more importantly, it should be the
_same_ structure as listing resources themselves.

 To note: here is another API design detail, specifying resource types in
the URL path:

 /server/server/foo

 vs

 /server/foo

 or what we have today, for example, in compute:

 /tenant/foo

 The proposed approach for the multiple server create case also makes
sense to me. Other than bulk operations there are indeed cases where a
single API operation needs to perform multiple tasks. For instance, in
Neutron, creating a port implies L2 wiring, setting up DHCP info, and
securing it on the compute node by enforcing anti-spoof rules and security
groups. This means there will be 3/4 active tasks. For this reason I wonder
if it might be the case of differentiating between the concept of
operation and tasks where the former is the activity explicitly
initiated by the API consumer, and the latter are the activities which need
to complete to fulfil it. This is where we might leverage the already
proposed request_id attribute of the task data structure.


 I like the ability to track the fan-out, especially if I can get the
state of the entire set of tasks in a single round-trip.  This also makes
it easier to handle backout of failed requests without having to maintain a
lot of client-side state, or make a lot of round-trips.


Based on previous experience, I highly recommend maintaining separation
between tracking work at an API call level aggregate and other subtasks.
In non-provisioning scenarios, tasks may fire independent of API
operations, so there wouldn't be an API handle to query on. It is great to
manage per-API call level tasks in the framework. The other work type
tasks are *much* more complicated beasts, deserving of their own design.

 Finally, a note on persistency. How long a completed task, successfully
or not should be stored for? Do we want to store them until the resource
they operated on is deleted?
 I don't think it's a great idea to store them indefinitely in the DB.
Tying their lifespan to resources is probably a decent idea, but time-based
cleanup policies might also be considered (e.g.: destroy a task record 24
hours after its completion)


 I can envision an operator/user wanting to be able to pull a log of an
operation/task for not only cloud debugging (x failed to build, when/why?)
but also app-level debugging (concrete use case not ready at deadline).
This would require a minimum of life-of-resource + some-amount-of-time.
The time might also be variable, failed operations might actually need to
stick around longer.

 Even as an operator with access to backend logging, pulling these state
transitions out should not be hard, and should be available to the resource
owner (project).

 dt

 --

 Dean Troyer
 dtro...@gmail.com

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] API recommendation

2014-10-16 Thread Salvatore Orlando
In an analysis we recently did for managing lifecycle of neutron resources,
it also emerged that task (or operation) API are a very useful resource.
Indeed several neutron resources introduced the (in)famous PENDING_XXX
operational statuses to note the fact that an operation is in progress and
its status is changing.

This could have been easily avoided if a facility for querying active tasks
through the API was available.

From an API guideline viewpoint, I understand that
https://review.openstack.org/#/c/86938/ proposes the introduction of a
rather simple endpoint to query active tasks and filter them by resource
uuid or state, for example.
While this is hardly questionable, I wonder if it might be worth
typifying the task, ie: adding a resource_type attribute, and/or allowing
to retrieve active tasks as a chile resource of an object, eg.: GET
/servers/server_id/tasks?state=running or if just for running tasks GET
/servers/server_id/active_tasks

The proposed approach for the multiple server create case also makes sense
to me. Other than bulk operations there are indeed cases where a single
API operation needs to perform multiple tasks. For instance, in Neutron,
creating a port implies L2 wiring, setting up DHCP info, and securing it on
the compute node by enforcing anti-spoof rules and security groups. This
means there will be 3/4 active tasks. For this reason I wonder if it might
be the case of differentiating between the concept of operation and
tasks where the former is the activity explicitly initiated by the API
consumer, and the latter are the activities which need to complete to
fulfil it. This is where we might leverage the already proposed request_id
attribute of the task data structure.

Finally, a note on persistency. How long a completed task, successfully or
not should be stored for? Do we want to store them until the resource they
operated on is deleted?
I don't think it's a great idea to store them indefinitely in the DB. Tying
their lifespan to resources is probably a decent idea, but time-based
cleanup policies might also be considered (e.g.: destroy a task record 24
hours after its completion)

Salvatore


On 16 October 2014 08:38, Christopher Yeoh cbky...@gmail.com wrote:

 On Thu, Oct 16, 2014 at 7:19 AM, Kevin L. Mitchell 
 kevin.mitch...@rackspace.com wrote:

 On Wed, 2014-10-15 at 12:39 -0400, Andrew Laski wrote:
  On 10/15/2014 11:49 AM, Kevin L. Mitchell wrote:
   Now that we have an API working group forming, I'd like to kick off
 some
   discussion over one point I'd really like to see our APIs using (and
   I'll probably drop it in to the repo once that gets fully set up): the
   difference between synchronous and asynchronous operations.  Using
 nova
   as an example—right now, if you kick off a long-running operation,
 such
   as a server create or a reboot, you watch the resource itself to
   determine the status of the operation.  What I'd like to propose is
 that
   future APIs use a separate operation resource to track status
   information on the particular operation.  For instance, if we were to
   rebuild the nova API with this idea in mind, booting a new server
 would
   give you a server handle and an operation handle; querying the server
   resource would give you summary information about the state of the
   server (running, not running) and pending operations, while querying
 the
   operation would give you detailed information about the status of the
   operation.  As another example, issuing a reboot would give you the
   operation handle; you'd see the operation in a queue on the server
   resource, but the actual state of the operation itself would be listed
   on that operation.  As a side effect, this would allow us (not
 require,
   though) to queue up operations on a resource, and allow us to cancel
 an
   operation that has not yet been started.
  
   Thoughts?
 
  Something like https://review.openstack.org/#/c/86938/ ?
 
  I know that Jay has proposed a similar thing before as well.  I would
  love to get some feedback from others on this as it's something I'm
  going to propose for Nova in Kilo.

 Yep, something very much like that :)  But the idea behind my proposal
 is to make that a codified API guideline, rather than just an addition
 to Nova.


 Perhaps the best way to make this move faster is for developers not from
 Nova
 who are interested to help develop the tasks api spec Andrew pointed to.
 Its been
  on the Nova to-do list for a few cycles now and had quite a bit of
 discussion both
 at mid cycles and summit meetings.

 Once we have a nova spec approved we can extract the project common parts
 out into the API guidelines.

 I think we really want microversions up so we can make backwards
 incompatible API
 changes when we implement the API side of tasks, but that is something
 members
 of the API WG are hopefully interested in too.

 https://review.openstack.org/127127

 Regards,

 Chris

 ___
 

Re: [openstack-dev] [api] API recommendation

2014-10-16 Thread Alex Xu
2014-10-16 17:57 GMT+08:00 Salvatore Orlando sorla...@nicira.com:

 In an analysis we recently did for managing lifecycle of neutron
 resources, it also emerged that task (or operation) API are a very useful
 resource.
 Indeed several neutron resources introduced the (in)famous PENDING_XXX
 operational statuses to note the fact that an operation is in progress and
 its status is changing.

 This could have been easily avoided if a facility for querying active
 tasks through the API was available.

 From an API guideline viewpoint, I understand that
 https://review.openstack.org/#/c/86938/ proposes the introduction of a
 rather simple endpoint to query active tasks and filter them by resource
 uuid or state, for example.
 While this is hardly questionable, I wonder if it might be worth
 typifying the task, ie: adding a resource_type attribute, and/or allowing
 to retrieve active tasks as a chile resource of an object, eg.: GET
 /servers/server_id/tasks?state=running or if just for running tasks GET
 /servers/server_id/active_tasks

 The proposed approach for the multiple server create case also makes sense
 to me. Other than bulk operations there are indeed cases where a single
 API operation needs to perform multiple tasks. For instance, in Neutron,
 creating a port implies L2 wiring, setting up DHCP info, and securing it on
 the compute node by enforcing anti-spoof rules and security groups. This
 means there will be 3/4 active tasks. For this reason I wonder if it might
 be the case of differentiating between the concept of operation and
 tasks where the former is the activity explicitly initiated by the API
 consumer, and the latter are the activities which need to complete to
 fulfil it. This is where we might leverage the already proposed request_id
 attribute of the task data structure.


This sounds like sub-task. The propose from Andrew include the sub-task
concept, it just didn't implement in the first step.



 Finally, a note on persistency. How long a completed task, successfully or
 not should be stored for? Do we want to store them until the resource they
 operated on is deleted?
 I don't think it's a great idea to store them indefinitely in the DB.
 Tying their lifespan to resources is probably a decent idea, but time-based
 cleanup policies might also be considered (e.g.: destroy a task record 24
 hours after its completion)


This is good point! Task can be removed after finished except failed. And
maybe can implement plugin mechanism to add different persistency backend?



 Salvatore


 On 16 October 2014 08:38, Christopher Yeoh cbky...@gmail.com wrote:

 On Thu, Oct 16, 2014 at 7:19 AM, Kevin L. Mitchell 
 kevin.mitch...@rackspace.com wrote:

 On Wed, 2014-10-15 at 12:39 -0400, Andrew Laski wrote:
  On 10/15/2014 11:49 AM, Kevin L. Mitchell wrote:
   Now that we have an API working group forming, I'd like to kick off
 some
   discussion over one point I'd really like to see our APIs using (and
   I'll probably drop it in to the repo once that gets fully set up):
 the
   difference between synchronous and asynchronous operations.  Using
 nova
   as an example—right now, if you kick off a long-running operation,
 such
   as a server create or a reboot, you watch the resource itself to
   determine the status of the operation.  What I'd like to propose is
 that
   future APIs use a separate operation resource to track status
   information on the particular operation.  For instance, if we were to
   rebuild the nova API with this idea in mind, booting a new server
 would
   give you a server handle and an operation handle; querying the server
   resource would give you summary information about the state of the
   server (running, not running) and pending operations, while querying
 the
   operation would give you detailed information about the status of the
   operation.  As another example, issuing a reboot would give you the
   operation handle; you'd see the operation in a queue on the server
   resource, but the actual state of the operation itself would be
 listed
   on that operation.  As a side effect, this would allow us (not
 require,
   though) to queue up operations on a resource, and allow us to cancel
 an
   operation that has not yet been started.
  
   Thoughts?
 
  Something like https://review.openstack.org/#/c/86938/ ?
 
  I know that Jay has proposed a similar thing before as well.  I would
  love to get some feedback from others on this as it's something I'm
  going to propose for Nova in Kilo.

 Yep, something very much like that :)  But the idea behind my proposal
 is to make that a codified API guideline, rather than just an addition
 to Nova.


 Perhaps the best way to make this move faster is for developers not from
 Nova
 who are interested to help develop the tasks api spec Andrew pointed to.
 Its been
  on the Nova to-do list for a few cycles now and had quite a bit of
 discussion both
 at mid cycles and summit meetings.

 Once we have a nova spec approved we 

Re: [openstack-dev] [api] API recommendation

2014-10-16 Thread Dean Troyer
On Thu, Oct 16, 2014 at 4:57 AM, Salvatore Orlando sorla...@nicira.com
wrote:

 From an API guideline viewpoint, I understand that
 https://review.openstack.org/#/c/86938/ proposes the introduction of a
 rather simple endpoint to query active tasks and filter them by resource
 uuid or state, for example.


That review/blueprint contains one thing that I want to address in more
detail below along with Sal's comment on persistence...


 While this is hardly questionable, I wonder if it might be worth
 typifying the task, ie: adding a resource_type attribute, and/or allowing
 to retrieve active tasks as a chile resource of an object, eg.: GET
 /servers/server_id/tasks?state=running or if just for running tasks GET
 /servers/server_id/active_tasks


I'd prefer the filter approach, but more importantly, it should be the
_same_ structure as listing resources themselves.

To note: here is another API design detail, specifying resource types in
the URL path:

/server/server/foo

vs

/server/foo

or what we have today, for example, in compute:

/tenant/foo

The proposed approach for the multiple server create case also makes sense
 to me. Other than bulk operations there are indeed cases where a single
 API operation needs to perform multiple tasks. For instance, in Neutron,
 creating a port implies L2 wiring, setting up DHCP info, and securing it on
 the compute node by enforcing anti-spoof rules and security groups. This
 means there will be 3/4 active tasks. For this reason I wonder if it might
 be the case of differentiating between the concept of operation and
 tasks where the former is the activity explicitly initiated by the API
 consumer, and the latter are the activities which need to complete to
 fulfil it. This is where we might leverage the already proposed request_id
 attribute of the task data structure.


I like the ability to track the fan-out, especially if I can get the state
of the entire set of tasks in a single round-trip.  This also makes it
easier to handle backout of failed requests without having to maintain a
lot of client-side state, or make a lot of round-trips.

Finally, a note on persistency. How long a completed task, successfully or
 not should be stored for? Do we want to store them until the resource they
 operated on is deleted?
 I don't think it's a great idea to store them indefinitely in the DB.
 Tying their lifespan to resources is probably a decent idea, but time-based
 cleanup policies might also be considered (e.g.: destroy a task record 24
 hours after its completion)


I can envision an operator/user wanting to be able to pull a log of an
operation/task for not only cloud debugging (x failed to build, when/why?)
but also app-level debugging (concrete use case not ready at deadline).
This would require a minimum of life-of-resource + some-amount-of-time.
The time might also be variable, failed operations might actually need to
stick around longer.

Even as an operator with access to backend logging, pulling these state
transitions out should not be hard, and should be available to the resource
owner (project).

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] API recommendation

2014-10-15 Thread Andrew Laski


On 10/15/2014 11:49 AM, Kevin L. Mitchell wrote:

Now that we have an API working group forming, I'd like to kick off some
discussion over one point I'd really like to see our APIs using (and
I'll probably drop it in to the repo once that gets fully set up): the
difference between synchronous and asynchronous operations.  Using nova
as an example—right now, if you kick off a long-running operation, such
as a server create or a reboot, you watch the resource itself to
determine the status of the operation.  What I'd like to propose is that
future APIs use a separate operation resource to track status
information on the particular operation.  For instance, if we were to
rebuild the nova API with this idea in mind, booting a new server would
give you a server handle and an operation handle; querying the server
resource would give you summary information about the state of the
server (running, not running) and pending operations, while querying the
operation would give you detailed information about the status of the
operation.  As another example, issuing a reboot would give you the
operation handle; you'd see the operation in a queue on the server
resource, but the actual state of the operation itself would be listed
on that operation.  As a side effect, this would allow us (not require,
though) to queue up operations on a resource, and allow us to cancel an
operation that has not yet been started.

Thoughts?


Something like https://review.openstack.org/#/c/86938/ ?

I know that Jay has proposed a similar thing before as well.  I would 
love to get some feedback from others on this as it's something I'm 
going to propose for Nova in Kilo.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] API recommendation

2014-10-15 Thread Michael McCune


- Original Message -
 Thoughts?


I like this idea. From my experience with the Sahara project I think there is 
definite opportunity for this mechanic especially with regards to cluster 
creation and job executions.

regards,
mike

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [api] API recommendation

2014-10-15 Thread Sam Harwell
Hi Kevin,

In an asynchronous environment that may have multiple clients sending commands 
to the same resource in a service, an operation-type resource is a 
fundamental prerequisite to creating client applications which report the 
status of ongoing operations. Without this resource, there is no way to tell a 
user whether an operation they attempted succeeded or failed.

Due to the importance of allowing users to see the results of individual 
operations on resources, I would treat the other features potentially provided 
by this type of resource, such as queuing or canceling operations, separately 
from the fundamental status reporting behavior.

Thank you,
Sam Harwell

-Original Message-
From: Kevin L. Mitchell [mailto:kevin.mitch...@rackspace.com] 
Sent: Wednesday, October 15, 2014 10:49 AM
To: openstack-dev
Subject: [openstack-dev] [api] API recommendation

Now that we have an API working group forming, I'd like to kick off some 
discussion over one point I'd really like to see our APIs using (and I'll 
probably drop it in to the repo once that gets fully set up): the difference 
between synchronous and asynchronous operations.  Using nova as an 
example—right now, if you kick off a long-running operation, such as a server 
create or a reboot, you watch the resource itself to determine the status of 
the operation.  What I'd like to propose is that future APIs use a separate 
operation resource to track status information on the particular operation.  
For instance, if we were to rebuild the nova API with this idea in mind, 
booting a new server would give you a server handle and an operation handle; 
querying the server resource would give you summary information about the state 
of the server (running, not running) and pending operations, while querying the 
operation would give you detailed information about the status of the 
operation.  As another example, issuing a reboot would give you the operation 
handle; you'd see the operation in a queue on the server resource, but the 
actual state of the operation itself would be listed on that operation.  As a 
side effect, this would allow us (not require,
though) to queue up operations on a resource, and allow us to cancel an 
operation that has not yet been started.

Thoughts?
--
Kevin L. Mitchell kevin.mitch...@rackspace.com Rackspace


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev