On Thu, Nov 13, 2014 at 6:29 PM, Murugan, Visnusaran < visnusaran.muru...@hp.com> wrote:
> Hi all, > > > > Convergence-POC distributes stack operations by sending resource actions > over RPC for any heat-engine to execute. Entire stack lifecycle will be > controlled by worker/observer notifications. This distributed model has its > own advantages and disadvantages. > > > > Any stack operation has a timeout and a single engine will be responsible > for it. If that engine goes down, timeout is lost along with it. So a > traditional way is for other engines to recreate timeout from scratch. Also > a missed resource action notification will be detected only when stack > operation timeout happens. > > > > To overcome this, we will need the following capability: > > 1. Resource timeout (can be used for retry) > We will shortly have a worker job, can't we have a job that just sleeps that gets started in parallel with the job that is doing the work? It gets to the end of the sleep and runs a check. > 2. Recover from engine failure (loss of stack timeout, resource > action notification) > > > My suggestion above could catch failures as long as it was run in a different process. -Angus > > > Suggestion: > > 1. Use task queue like celery to host timeouts for both stack and > resource. > > 2. Poll database for engine failures and restart timers/ retrigger > resource retry (IMHO: This would be a traditional and weighs heavy) > > 3. Migrate heat to use TaskFlow. (Too many code change) > > > > I am not suggesting we use Task Flow. Using celery will have very minimum > code change. (decorate appropriate functions) > > > > > > Your thoughts. > > > > -Vishnu > > IRC: ckmvishnu > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev