Re: [Openstack] [Nova-orchestration] Thoughts on Orchestration (was Re: Documentation on Caching)

2012-04-03 Thread Ziad Sawalha
Just confirming what Sandy said; I am playing around with SpiffWorkflow.
I'll post my findings when I'm done on the wiki under the Nova
Orchestration page.

So far I've found some of the documentation lacking and concepts
confusing, which has resulted in a steep learning curve and made it
difficult to integrate into something like RabbitMQ (for long-running
tasks). But the thinking behind it (http://www.workflowpatterns.com/)
seems sound and I will continue to investigate it.

Z

On 3/29/12 5:56 PM, Sriram Subramanian sri...@computenext.com wrote:

Guys,

Sorry for missing the meeting today. Thanks for the detailed summary/
logs. I am cool with the action item : #action sriram to update the
Orchestration session proposal. This is my understanding the logs of
things to be updated in the blueprint:

1) orchestration service provides state management with client side APIs
2) add API design and state storage as topics for the orchestration
session at the Summit
3) add implementation plan as session topic

Please correct me if I missed anything.

Just to bring everyone to same page, here are the new links

Folsom BluePrint: 
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Folsom Session proposal:
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Wiki: http://wiki.openstack.org/NovaOrchestration (I will clean this up
tonight)

Maoy: Sandy's pointers are in this email thread (which n0ano meant to fwd
you)
Mikeyp: Moving the conversation to the main mailing list per your
suggestion

Thanks,
_Sriram

-Original Message-
From: Dugger, Donald D [mailto:donald.d.dug...@intel.com]
Sent: Thursday, March 29, 2012 12:52 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re:
Documentation on Caching)

NP, I'll be on the IRC for whoever wants to talk.  Maybe we can try and
do the sync you want via email, that's always been my favorite way to
communicate (it allows you to focus thoughts and deals with timezones
nicely).

--
Don Dugger
Censeo Toto nos in Kansa esse decisse. - D. Gale
Ph: 303/443-3786


-Original Message-
From: Sriram Subramanian [mailto:sri...@computenext.com]
Sent: Thursday, March 29, 2012 1:45 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Dugger, Donald D; Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re:
Documentation on Caching)

I will most likely be running little late from my 12 - 1 meeting which
doesn't seem to be ending anytime now :(

I haven't gotten a chance to submit a branch yet. Hopefully by this week
end (at least a bare bones)

If you are available for offline sync later this week - I would
appreciate that. Apologies for possibly missing the sync.

Thanks,
-Sriram
 
-Original Message-
From: 
nova-orchestration-bounces+sriram=computenext@lists.launchpad.net
[mailto:nova-orchestration-bounces+sriram=computenext.com@lists.launchpad.
net] On Behalf Of Sriram Subramanian
Sent: Wednesday, March 28, 2012 2:44 PM
To: Sandy Walsh
Cc: nova-orchestrat...@lists.launchpad.net
Subject: Re: [Nova-orchestration] Thoughts on Orchestration (was Re:
Documentation on Caching)

Thanks for the pointers Sandy. I will try to spend some cycles on the
branch per your suggestion; we will also discuss more tomorrow.

Yes, BP is not far off from last summit, and would like to flush out more
for this summit. 

Thanks,
-Sriram

-Original Message-
From: Sandy Walsh [mailto:sandy.wa...@rackspace.com]
Sent: Wednesday, March 28, 2012 11:31 AM
To: Sriram Subramanian
Cc: Michael Pittaro; Dugger, Donald D (donald.d.dug...@intel.com);
nova-orchestrat...@lists.launchpad.net
Subject: Thoughts on Orchestration (was Re: Documentation on Caching)

Ah, gotcha.

I don't think the caching stuff will really affect the Orchestration
layer all that much. Certainly the Cells stuff that comstud is working on
should be considered.

The BP isn't really too far off from what we discussed last summit.
Although I would give more consideration to the stuff Redhat is thinking
about and some of the efforts by HP and IBM with respect to scheduling
(mostly HPC stuff). Unifying and/or understanding those efforts would be
important.

That said, as with all things OpenStack, code speaks louder than words.
The best way to solicit input on an idea is to submit a branch. That's
the approach I'd take now if I had the cycles to put back into Orch. I'd
likely build something on top of Amazon Workflow services (in such a way
as it could be ripped out later) http://aws.amazon.com/swf/ The
replacement could be a new OS Service with SWF as the api template.

What I've been thinking about lately has been how to make a proof of
concept operate with trunk side-by-side without busting the existing
stuff. Tricky. Orchestration touches a lot of stuff. The error handling
is OS could be an issue and unifying the 3 Enum State Machine on Instance
could 

Re: [Openstack] [Nova-orchestration] Thoughts on Orchestration (was Re: Documentation on Caching)

2012-04-03 Thread Sandy Walsh
Can't wait to hear about it Ziad!

Very cool!

-S

From: Ziad Sawalha
Sent: Tuesday, April 03, 2012 6:56 PM
To: Sriram Subramanian; Dugger, Donald D; Sandy Walsh
Cc: nova-orchestrat...@lists.launchpad.net; openstack@lists.launchpad.net
Subject: Re: [Openstack] [Nova-orchestration] Thoughts on Orchestration (was 
Re: Documentation on Caching)

Just confirming what Sandy said; I am playing around with SpiffWorkflow.
I'll post my findings when I'm done on the wiki under the Nova
Orchestration page.

So far I've found some of the documentation lacking and concepts
confusing, which has resulted in a steep learning curve and made it
difficult to integrate into something like RabbitMQ (for long-running
tasks). But the thinking behind it (http://www.workflowpatterns.com/)
seems sound and I will continue to investigate it.

Z

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp


Re: [Openstack] [Nova-orchestration] Thoughts on Orchestration (was Re: Documentation on Caching)

2012-04-03 Thread Yun Mao
Hi Ziad,

Thanks for taking the effort. Do you know which ones out of the 43
workflows patterns are relavant to us? I'm slightly concerned that
SpiffWorkflow might be an overkill and bring unnecessary complexity
into the game. There was a discussion a while ago suggesting that
relatively simple sequential execution pattern:
https://lists.launchpad.net/nova-orchestration/msg00043.html

Thanks,

Yun

On Tue, Apr 3, 2012 at 5:56 PM, Ziad Sawalha ziad.sawa...@rackspace.com wrote:
 Just confirming what Sandy said; I am playing around with SpiffWorkflow.
 I'll post my findings when I'm done on the wiki under the Nova
 Orchestration page.

 So far I've found some of the documentation lacking and concepts
 confusing, which has resulted in a steep learning curve and made it
 difficult to integrate into something like RabbitMQ (for long-running
 tasks). But the thinking behind it (http://www.workflowpatterns.com/)
 seems sound and I will continue to investigate it.

 Z

 On 3/29/12 5:56 PM, Sriram Subramanian sri...@computenext.com wrote:

Guys,

Sorry for missing the meeting today. Thanks for the detailed summary/
logs. I am cool with the action item : #action sriram to update the
Orchestration session proposal. This is my understanding the logs of
things to be updated in the blueprint:

1) orchestration service provides state management with client side APIs
2) add API design and state storage as topics for the orchestration
session at the Summit
3) add implementation plan as session topic

Please correct me if I missed anything.

Just to bring everyone to same page, here are the new links

Folsom BluePrint:
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Folsom Session proposal:
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Wiki: http://wiki.openstack.org/NovaOrchestration (I will clean this up
tonight)

Maoy: Sandy's pointers are in this email thread (which n0ano meant to fwd
you)
Mikeyp: Moving the conversation to the main mailing list per your
suggestion

Thanks,
_Sriram

-Original Message-
From: Dugger, Donald D [mailto:donald.d.dug...@intel.com]
Sent: Thursday, March 29, 2012 12:52 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re:
Documentation on Caching)

NP, I'll be on the IRC for whoever wants to talk.  Maybe we can try and
do the sync you want via email, that's always been my favorite way to
communicate (it allows you to focus thoughts and deals with timezones
nicely).

--
Don Dugger
Censeo Toto nos in Kansa esse decisse. - D. Gale
Ph: 303/443-3786


-Original Message-
From: Sriram Subramanian [mailto:sri...@computenext.com]
Sent: Thursday, March 29, 2012 1:45 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Dugger, Donald D; Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re:
Documentation on Caching)

I will most likely be running little late from my 12 - 1 meeting which
doesn't seem to be ending anytime now :(

I haven't gotten a chance to submit a branch yet. Hopefully by this week
end (at least a bare bones)

If you are available for offline sync later this week - I would
appreciate that. Apologies for possibly missing the sync.

Thanks,
-Sriram

-Original Message-
From:
nova-orchestration-bounces+sriram=computenext@lists.launchpad.net
[mailto:nova-orchestration-bounces+sriram=computenext.com@lists.launchpad.
net] On Behalf Of Sriram Subramanian
Sent: Wednesday, March 28, 2012 2:44 PM
To: Sandy Walsh
Cc: nova-orchestrat...@lists.launchpad.net
Subject: Re: [Nova-orchestration] Thoughts on Orchestration (was Re:
Documentation on Caching)

Thanks for the pointers Sandy. I will try to spend some cycles on the
branch per your suggestion; we will also discuss more tomorrow.

Yes, BP is not far off from last summit, and would like to flush out more
for this summit.

Thanks,
-Sriram

-Original Message-
From: Sandy Walsh [mailto:sandy.wa...@rackspace.com]
Sent: Wednesday, March 28, 2012 11:31 AM
To: Sriram Subramanian
Cc: Michael Pittaro; Dugger, Donald D (donald.d.dug...@intel.com);
nova-orchestrat...@lists.launchpad.net
Subject: Thoughts on Orchestration (was Re: Documentation on Caching)

Ah, gotcha.

I don't think the caching stuff will really affect the Orchestration
layer all that much. Certainly the Cells stuff that comstud is working on
should be considered.

The BP isn't really too far off from what we discussed last summit.
Although I would give more consideration to the stuff Redhat is thinking
about and some of the efforts by HP and IBM with respect to scheduling
(mostly HPC stuff). Unifying and/or understanding those efforts would be
important.

That said, as with all things OpenStack, code speaks louder than words.
The best way to solicit input on an idea is to submit a branch. That's
the approach I'd take now if I had the cycles to put back into Orch. I'd
likely 

Re: [Openstack] [Nova-orchestration] Thoughts on Orchestration (was Re: Documentation on Caching)

2012-03-29 Thread Sriram Subramanian
Guys,

Sorry for missing the meeting today. Thanks for the detailed summary/ logs. I 
am cool with the action item : #action sriram to update the Orchestration 
session proposal. This is my understanding the logs of things to be updated in 
the blueprint:

1) orchestration service provides state management with client side APIs
2) add API design and state storage as topics for the orchestration session at 
the Summit
3) add implementation plan as session topic 

Please correct me if I missed anything.

Just to bring everyone to same page, here are the new links

Folsom BluePrint:  
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Folsom Session proposal:  
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Wiki: http://wiki.openstack.org/NovaOrchestration (I will clean this up tonight)

Maoy: Sandy's pointers are in this email thread (which n0ano meant to fwd you)
Mikeyp: Moving the conversation to the main mailing list per your suggestion

Thanks,
_Sriram

-Original Message-
From: Dugger, Donald D [mailto:donald.d.dug...@intel.com] 
Sent: Thursday, March 29, 2012 12:52 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

NP, I'll be on the IRC for whoever wants to talk.  Maybe we can try and do the 
sync you want via email, that's always been my favorite way to communicate (it 
allows you to focus thoughts and deals with timezones nicely).

--
Don Dugger
Censeo Toto nos in Kansa esse decisse. - D. Gale
Ph: 303/443-3786


-Original Message-
From: Sriram Subramanian [mailto:sri...@computenext.com]
Sent: Thursday, March 29, 2012 1:45 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Dugger, Donald D; Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

I will most likely be running little late from my 12 - 1 meeting which doesn't 
seem to be ending anytime now :(

I haven't gotten a chance to submit a branch yet. Hopefully by this week end 
(at least a bare bones)

If you are available for offline sync later this week - I would appreciate 
that. Apologies for possibly missing the sync.

Thanks,
-Sriram
 
-Original Message-
From: nova-orchestration-bounces+sriram=computenext@lists.launchpad.net 
[mailto:nova-orchestration-bounces+sriram=computenext@lists.launchpad.net] 
On Behalf Of Sriram Subramanian
Sent: Wednesday, March 28, 2012 2:44 PM
To: Sandy Walsh
Cc: nova-orchestrat...@lists.launchpad.net
Subject: Re: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

Thanks for the pointers Sandy. I will try to spend some cycles on the branch 
per your suggestion; we will also discuss more tomorrow. 

Yes, BP is not far off from last summit, and would like to flush out more for 
this summit. 

Thanks,
-Sriram

-Original Message-
From: Sandy Walsh [mailto:sandy.wa...@rackspace.com]
Sent: Wednesday, March 28, 2012 11:31 AM
To: Sriram Subramanian
Cc: Michael Pittaro; Dugger, Donald D (donald.d.dug...@intel.com); 
nova-orchestrat...@lists.launchpad.net
Subject: Thoughts on Orchestration (was Re: Documentation on Caching)

Ah, gotcha.

I don't think the caching stuff will really affect the Orchestration layer all 
that much. Certainly the Cells stuff that comstud is working on should be 
considered.

The BP isn't really too far off from what we discussed last summit.
Although I would give more consideration to the stuff Redhat is thinking about 
and some of the efforts by HP and IBM with respect to scheduling (mostly HPC 
stuff). Unifying and/or understanding those efforts would be important.

That said, as with all things OpenStack, code speaks louder than words.
The best way to solicit input on an idea is to submit a branch. That's the 
approach I'd take now if I had the cycles to put back into Orch. I'd likely 
build something on top of Amazon Workflow services (in such a way as it could 
be ripped out later) http://aws.amazon.com/swf/ The replacement could be a new 
OS Service with SWF as the api template.

What I've been thinking about lately has been how to make a proof of concept 
operate with trunk side-by-side without busting the existing stuff. Tricky. 
Orchestration touches a lot of stuff. The error handling is OS could be an 
issue and unifying the 3 Enum State Machine on Instance could be nasty (Power, 
Virt  Task).

I know Ziad is playing around with SpiffWorkflow so perhaps he might have some 
fresh insights?

Hope it helps! Nice to see an effort still going on here. I think it's an 
important project!

-S


On 03/28/2012 03:08 PM, Sriram Subramanian wrote:
 Hi Sandy,
 
  We wanted to understand the scheduler improvements in Essex so that we can 
 update Orchestration blueprint for Folsom. We also wanted to identify the 
 gaps in the current bp so that we can address them.
 
 Towards the 

Re: [Openstack] [Nova-orchestration] Thoughts on Orchestration (was Re: Documentation on Caching)

2012-03-29 Thread Dugger, Donald D
Sriram-

Great, this matches my understanding of what we agreed to today.  Tnx for 
agreeing to update the BP.

--
Don Dugger
Censeo Toto nos in Kansa esse decisse. - D. Gale
Ph: 303/443-3786


-Original Message-
From: Sriram Subramanian [mailto:sri...@computenext.com] 
Sent: Thursday, March 29, 2012 4:57 PM
To: Dugger, Donald D; Sandy Walsh
Cc: Michael Pittaro (mik...@lahondaresearch.org); 
nova-orchestrat...@lists.launchpad.net; openstack@lists.launchpad.net
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

Guys,

Sorry for missing the meeting today. Thanks for the detailed summary/ logs. I 
am cool with the action item : #action sriram to update the Orchestration 
session proposal. This is my understanding the logs of things to be updated in 
the blueprint:

1) orchestration service provides state management with client side APIs
2) add API design and state storage as topics for the orchestration session at 
the Summit
3) add implementation plan as session topic 

Please correct me if I missed anything.

Just to bring everyone to same page, here are the new links

Folsom BluePrint:  
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Folsom Session proposal:  
https://blueprints.launchpad.net/nova/+spec/nova-orchestration
Wiki: http://wiki.openstack.org/NovaOrchestration (I will clean this up tonight)

Maoy: Sandy's pointers are in this email thread (which n0ano meant to fwd you)
Mikeyp: Moving the conversation to the main mailing list per your suggestion

Thanks,
_Sriram

-Original Message-
From: Dugger, Donald D [mailto:donald.d.dug...@intel.com] 
Sent: Thursday, March 29, 2012 12:52 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

NP, I'll be on the IRC for whoever wants to talk.  Maybe we can try and do the 
sync you want via email, that's always been my favorite way to communicate (it 
allows you to focus thoughts and deals with timezones nicely).

--
Don Dugger
Censeo Toto nos in Kansa esse decisse. - D. Gale
Ph: 303/443-3786


-Original Message-
From: Sriram Subramanian [mailto:sri...@computenext.com]
Sent: Thursday, March 29, 2012 1:45 PM
To: Sriram Subramanian; Sandy Walsh
Cc: Dugger, Donald D; Michael Pittaro (mik...@lahondaresearch.org)
Subject: RE: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

I will most likely be running little late from my 12 - 1 meeting which doesn't 
seem to be ending anytime now :(

I haven't gotten a chance to submit a branch yet. Hopefully by this week end 
(at least a bare bones)

If you are available for offline sync later this week - I would appreciate 
that. Apologies for possibly missing the sync.

Thanks,
-Sriram
 
-Original Message-
From: nova-orchestration-bounces+sriram=computenext@lists.launchpad.net 
[mailto:nova-orchestration-bounces+sriram=computenext@lists.launchpad.net] 
On Behalf Of Sriram Subramanian
Sent: Wednesday, March 28, 2012 2:44 PM
To: Sandy Walsh
Cc: nova-orchestrat...@lists.launchpad.net
Subject: Re: [Nova-orchestration] Thoughts on Orchestration (was Re: 
Documentation on Caching)

Thanks for the pointers Sandy. I will try to spend some cycles on the branch 
per your suggestion; we will also discuss more tomorrow. 

Yes, BP is not far off from last summit, and would like to flush out more for 
this summit. 

Thanks,
-Sriram

-Original Message-
From: Sandy Walsh [mailto:sandy.wa...@rackspace.com]
Sent: Wednesday, March 28, 2012 11:31 AM
To: Sriram Subramanian
Cc: Michael Pittaro; Dugger, Donald D (donald.d.dug...@intel.com); 
nova-orchestrat...@lists.launchpad.net
Subject: Thoughts on Orchestration (was Re: Documentation on Caching)

Ah, gotcha.

I don't think the caching stuff will really affect the Orchestration layer all 
that much. Certainly the Cells stuff that comstud is working on should be 
considered.

The BP isn't really too far off from what we discussed last summit.
Although I would give more consideration to the stuff Redhat is thinking about 
and some of the efforts by HP and IBM with respect to scheduling (mostly HPC 
stuff). Unifying and/or understanding those efforts would be important.

That said, as with all things OpenStack, code speaks louder than words.
The best way to solicit input on an idea is to submit a branch. That's the 
approach I'd take now if I had the cycles to put back into Orch. I'd likely 
build something on top of Amazon Workflow services (in such a way as it could 
be ripped out later) http://aws.amazon.com/swf/ The replacement could be a new 
OS Service with SWF as the api template.

What I've been thinking about lately has been how to make a proof of concept 
operate with trunk side-by-side without busting the existing stuff. Tricky. 
Orchestration touches a lot of stuff. The error handling is OS could be an 
issue and