Hi Jan, On Wed, 2012-05-23 at 09:13 +0200, Jan Provaznik wrote: > Deltacloud is going to implement a stateful app on top of DC API. This > app will do very similar job to what we planned with Cloud State. Some > basic ideas/thoughts about Cloud State are described on wiki: > https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Cloud_State > > So for us it would be much easier to use this deltacloud app instead of > implementing the same thing.
I think this would be a very useful addition to DC, as an optional add-on (meaning you can either use DC as a strict proxy as you do today, or add state tracking, with some additional infrastructure, like a DB) Architecturally, this seems to me like something we'd want to hook into the existing drivers - essentially have a state tracking driver that decorates the actual backend driver (though I have to think this through in a little more detail) A DC+state tracking server would expose only the 'normal' IaaS API as much as possible, with a few additions to control state tracking. For the DC API it's pretty clear how we can expose these additions, for CIMI somewhat clear, and for EC2 it's still very murky ;) But I assume your immediate need is for the 'classic' DC API. > Here are some basic requirements which we would need on Conductor side > from the stateful app: > 1) the app should check state of not only instances but also other > resources - we need especially realms because we check availability status We need to define in detail for each of these resources what needs to be tracked, and what status changes constitute an 'event', but as a general requirement this is fine. I suggest though that we start with something very basic, like tracking only instance state changes, and expand from there gradually. > 2) it should be possible to register hooks for a resource I want to > watch, hook could be just an URL, when the resource is changed data are > POSTed to the registered URL. Do you just want a callback every time an event happens or more detail about what changed (like 'instance went from pending to started') ? > TODO: authentication when posting data to hook url? we use oauth between > other components now For the very first cut, I'd either not do any auth, or do something incredibly cheesy like a token. > TODO: how to handle hook failures (conductor is not accessible and hook > can't be invoked)? For sure, we'd retry for a while ... after that, retry very infrequently (like once a day) + provide an API to retrieve undelivered events. That way, if the conductor failure is transient, everything should catch up fairly quickly. If the conductor failure was longer (some multi-hour maintenance event), conductor can catch up by asking for events explicitly. > TODO: how to handle credentials? will the stateful app keep credentails > permanently for each instance being checked? As much as this worries me from a security standpoint, I don't see another way around this - cloud API's generally don't allow any delegation of auth. There's a couple more TODO's connected to credentials: TODO: how are credentials changes handled (user revokes API Key and generates a new one) ? [not for the first cut] TODO: when are stored credentials purged ? We want to make sure we get rid of them as quickly as possible. > 3) frequency of instance status checking: we probably don't need to have > this configurable, but the checking frequency should be as low as > possible - various actions are hooked to instance state changes (for > example runtime is measured by this). It should be also scalable - if > there are 100 instances for one provider account, we still need check > all of them. Would be nice to use multiple parallel requests to provider > if possible or checking interval might be dynamically increased > according to the actual length of queue). I think combining these should be fairly straightforward; for frequency I imagine we'll build something in based on the anticipated state change: for example, while an instance is pending, we might poll its state pretty frequently, once it's running, we'd back off and poll much less often. What poll frequencies does conductor use today ? David
