On 24-Sep-14 00:25, Joshua Harlow wrote:
> I believe heat has its own dependency graph implementation but if that was 
> switched to networkx[1] that library has a bunch of nice read/write 
> capabilities.
> 
> See: https://github.com/networkx/networkx/tree/master/networkx/readwrite
> 
> And one made for sqlalchemy @ https://pypi.python.org/pypi/graph-alchemy/
> 
> Networkx has worked out pretty well for taskflow (and I believe mistral is 
> also using it).
> 
> [1] https://networkx.github.io/
> 
> Something to think about...
> 
> On Sep 23, 2014, at 11:32 AM, Zane Bitter <zbit...@redhat.com> wrote:
> 
>> On 23/09/14 09:44, Anant Patil wrote:
>>> On 23-Sep-14 09:42, Clint Byrum wrote:
>>>> Excerpts from Angus Salkeld's message of 2014-09-22 20:15:43 -0700:
>>>>> On Tue, Sep 23, 2014 at 1:09 AM, Anant Patil <anant.pa...@hp.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> One of the steps in the direction of convergence is to enable Heat
>>>>>> engine to handle concurrent stack operations. The main convergence spec
>>>>>> talks about it. Resource versioning would be needed to handle concurrent
>>>>>> stack operations.
>>>>>>
>>>>>> As of now, while updating a stack, a backup stack is created with a new
>>>>>> ID and only one update runs at a time. If we keep the raw_template
>>>>>> linked to it's previous completed template, i.e. have a back up of
>>>>>> template instead of stack, we avoid having backup of stack.
>>>>>>
>>>>>> Since there won't be a backup stack and only one stack_id to be dealt
>>>>>> with, resources and their versions can be queried for a stack with that
>>>>>> single ID. The idea is to identify resources for a stack by using stack
>>>>>> id and version. Please let me know your thoughts.
>>>>>>
>>>>>
>>>>> Hi Anant,
>>>>>
>>>>> This seems more complex than it needs to be.
>>>>>
>>>>> I could be wrong, but I thought the aim was to simply update the goal 
>>>>> state.
>>>>> The backup stack is just the last working stack. So if you update and 
>>>>> there
>>>>> is already an update you don't need to touch the backup stack.
>>>>>
>>>>> Anyone else that was at the meetup want to fill us in?
>>>>>
>>>>
>>>> The backup stack is a device used to collect items to operate on after
>>>> the current action is complete. It is entirely an implementation detail.
>>>>
>>>> Resources that can be updated in place will have their resource record
>>>> superseded, but retain their physical resource ID.
>>>>
>>>> This is one area where the resource plugin API is particularly sticky,
>>>> as resources are allowed to raise the "replace me" exception if in-place
>>>> updates fail. That is o-k though, at that point we will just comply by
>>>> creating a replacement resource as if we never tried the in-place update.
>>>>
>>>> In order to facilitate this, we must expand the resource data model to
>>>> include a version. Replacement resources will be marked as "current" and
>>>> to-be-removed resources marked for deletion. We can also keep all current
>>>> - 1 resources around to facilitate rollback until the stack reaches a
>>>> "complete" state again. Once that is done, we can remove the backup stack.
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev@lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>> Backup stack is a good way to take care of rollbacks or cleanups after
>>> the stack action is complete. By cleanup I mean the deletion of
>>> resources that are no longer needed after the new update. It works very
>>> well when one engine is processing the stack request and the stacks are
>>> in memory.
>>
>> It's actually a fairly terrible hack (I wrote it ;)
>>
>> It doesn't work very well because in practice during an update there are 
>> dependencies that cross between the real stack and the backup stack (due to 
>> some resources remaining the same or being updated in place, while others 
>> are moved to the backup stack ready for replacement). So in the event of a 
>> failure that we don't completely roll back on the spot, we lose some 
>> dependency information.
>>
>>> As a step towards distributing the stack request processing and making
>>> it fault-tolerant, we need to persist the dependency task graph. The
>>> backup stack can also be persisted along with the new graph, but then
>>> the engine has to traverse both the graphs to proceed with the operation
>>> and later identify the resources to be cleaned-up or rolled back using
>>> the stack id. There would be many resources for the same stack but
>>> different stack ids.
>>
>> Right, yeah this would be a mistake because in reality there is only one 
>> graph, so that's how we need to model it internally.
>>
>>> In contrast, when we store the current dependency task graph(from the
>>> latest request) in DB, and version the resources, we can identify those
>>> resources that need to be rolled-back or cleaned up after the stack
>>> operations is done, by comparing their versions. With versioning of
>>> resources and template, we can avoid creating a deep stack of backup
>>> stacks. The processing of stack operation can happen from multiple
>>> engines, and IMHO, it is simpler when all the engines just see one stack
>>> and versions of resources, instead of seeing many stacks with many
>>> resources for each stack.
>>
>> Bingo.
>>
>> I think all you need to do is record in the resource the particular template 
>> and set of parameters it was tied to (maybe just generate a UUID for each 
>> update... or perhaps a SHA hash of the actual data for better rollbacks?). 
>> Then any resource that isn't part of the latest template should get deleted 
>> during the cleanup phase of the dependency graph traversal.
>>
>> As you mentioned above, we'll also need to store the dependency graph of the 
>> stack in the database somewhere. Right now we generate it afresh from the 
>> template by assuming that each resource name corresponds to one entry in the 
>> DB. Since that will no longer be true, we'll need it to be a graph of 
>> resource IDs that we store.
>>
>> cheers,
>> Zane.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 

Thanks Zane for your thoughts.

I have submitted a spec under convergence spec to implement this and
would like all to review it. I created this spec so that we can submit
patches towards these specs. To identify which parts of convergence are
being addressed by the patches, I have created this subsidiary spec
under Convergence Specification.

- Anant

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to