On this proposal? No, not yet. Just popped in my mind yesterday. API there is a bit on the wiki.
> On 8 Feb 2017, at 14:31, Jeremiah Lowin <[email protected]> wrote: > > Makes a lot of sense. At the NY meetup there was considerable interest in > using the API (and quite a few hacks around exposing the CLI!) -- is there > more complete documentation anywhere? > > Thanks Bolke > > On Wed, Feb 8, 2017 at 1:36 AM Bolke de Bruin <[email protected]> wrote: > >> Hi All, >> >> Now that we have an API in place. I would like to propose a new state for >> tasks named “WAITING_ON_CALLBACK”. Currently, we have tasks that have a >> kind of polling mechanism (ie. Sensors) that wait for an action to happen >> and check if that action happened by regularly polling a particular >> backend. This will always use a slot from one of the workers and could >> starve an airflow cluster for resources. What if a callback to Airflow >> could happen that task to change its status by calling a callback mechanism >> without taking up a worker slot. A timeout could (should) be associated >> with the required callback so that the task can fail if required. So a bit >> more visual: >> >> >> Task X from DAG Z does some work and sets “WAITING_ON_CALLBACK” -> API >> post to /dags/Z/dag_runs/20170101T00:00:00/tasks/X with payload “set status >> to SUCCESS” >> >> DAG Z happily continues. >> >> Or >> >> Task X from DAG Z sets “WAITING_ON_CALLBACK” with timeout of 300s -> time >> passes -> scheduler sets task to FAILED. >> >> >> Any thoughts? >> >> - Bolke
