Zane,

We do follow shardy's suggestion of having worker/observer as eventlet in 
heat-engine. No new process. The timer will be executed under an engine's 
worker.

Question:
1. heat-engine processing resource-action failed (process killed)
2. heat-engine processing timeout for a stack fails (process killed)

In the above mentioned cases, I thought celery tasks would come to our rescue.

Convergence-poc implementation can recover from error and retry if there is a 
notification available.


-Vishnu

-----Original Message-----
From: Zane Bitter [mailto:zbit...@redhat.com] 
Sent: Thursday, November 13, 2014 7:05 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Heat] Using Job Queues for timeout ops

On 13/11/14 06:52, Angus Salkeld wrote:
> On Thu, Nov 13, 2014 at 6:29 PM, Murugan, Visnusaran 
> <visnusaran.muru...@hp.com <mailto:visnusaran.muru...@hp.com>> wrote:
>
>     Hi all,____
>
>     __ __
>
>     Convergence-POC distributes stack operations by sending resource
>     actions over RPC for any heat-engine to execute. Entire stack
>     lifecycle will be controlled by worker/observer notifications. This
>     distributed model has its own advantages and disadvantages.____
>
>     __ __
>
>     Any stack operation has a timeout and a single engine will be
>     responsible for it. If that engine goes down, timeout is lost along
>     with it. So a traditional way is for other engines to recreate
>     timeout from scratch. Also a missed resource action notification
>     will be detected only when stack operation timeout happens. __ __
>
>     __ __
>
>     To overcome this, we will need the following capability:____
>
>     __1.__Resource timeout (can be used for retry)
>
> We will shortly have a worker job, can't we have a job that just 
> sleeps that gets started in parallel with the job that is doing the work?
> It gets to the end of the sleep and runs a check.

What if that worker dies too? There's no guarantee that it'd even be a 
different worker. In fact, there's not even a guarantee that we'd have multiple 
workers.

BTW Steve Hardy's suggestion, which I have more or less come around to, is that 
the engines themselves should be the workers in convergence, to save operators 
deploying two types of processes. (The observers will still be a separate 
process though, in phase 2.)

>     ____
>
>     __2.__Recover from engine failure (loss of stack timeout, resource
>     action notification)____
>
>     __
>
>
> My suggestion above could catch failures as long as it was run in a 
> different process.
>
> -Angus
>
>     __
>
>     __ __
>
>     Suggestion:____
>
>     __1.__Use task queue like celery to host timeouts for both stack and
>     resource.____
>
>     __2.__Poll database for engine failures and restart timers/
>     retrigger resource retry (IMHO: This would be a traditional and
>     weighs heavy)____
>
>     __3.__Migrate heat to use TaskFlow. (Too many code change)____
>
>     __ __
>
>     I am not suggesting we use Task Flow. Using celery will have very
>     minimum code change. (decorate appropriate functions) ____
>
>     __ __
>
>     __ __
>
>     Your thoughts.____
>
>     __ __
>
>     -Vishnu____
>
>     IRC: ckmvishnu____
>
>
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev@lists.openstack.org
>     <mailto:OpenStack-dev@lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to