Hi Max,
Thank you for this information. I had not come across the TriggerDagRunOperator so thank you for pointing it out - it looks like a useful operator. We are planning to interact with Airflow from a non-python apps so believe the API is necessary. Cheers, Luke ________________________________ From: Maxime Beauchemin <[email protected]> Sent: 05 December 2016 16:28 To: [email protected] Subject: Re: API additions Hi Oliver, Have you looked into: https://airflow.incubator.apache.org/code.html#airflow.operators.TriggerDagRunOperator The idea here is to have a specific high frequency simple DAG with this operator (with a schedule_interval or every 5 minutes or so) that triggers another not-scheduled (schedule_interval=None) DAG when the criteria is met. Your `every_five_minutes` DAG can be reused for anything that follows that pattern. Also note that DagRuns have a payload field that should be easy to access (at least from PythonOperator and templates) in which you can make parameters available for tasks in your DagRun (similar to XCom). It's great to have the API, and vital if the triggering app isn't written in Python, but you can also use the ORM directly `from airflow.models import DagRun`, of course there are pros/cons to that approach. Max On Mon, Dec 5, 2016 at 6:31 AM, Maycock, Luke < [email protected]> wrote: > Hello, > > We've had a requirement for Airflow to be able to start DAG Runs and check > on the state of tasks in a specific DAG Run via API. We'd seen that an > experimental API had been put in place and expanded on this, but there has > since been a significant overhaul, mostly around security and the setup in > this PR: https://github.com/apache/incubator-airflow/pull/1783 > > Before we rework our additions (https://github.com/owlabs/ > incubator-airflow/pull/11, https://github.com/owlabs/ > incubator-airflow/pull/11, https://github.com/owlabs/ > incubator-airflow/pull/14) , we thought it might be worth asking if > anybody had any thoughts about what we've done so far and whether we're > treading on anybody else's toes with these features. > > The endpoints that we've written are: > > - Get details of a task instance > - Create a DAG Run (this is somewhat covered by the trigger_dag endpoint > that has now been added, but we also have a version to allow you to specify > the execution date) > - Add a value to xcom for a task instance (initially required the task to > exist, but we had a use to be able to add xcom for tasks that didn't exist, > so we removed that restriction) > > Is anybody else working on these endpoints or any other significant > changes to this area that we'd do better to wait for before we rework these > for PR? > > Thanks, > Luke Maycock > OLIVER WYMAN > [email protected]<mailto:luke. > [email protected]> > www.oliverwyman.com<http://www.oliverwyman.com/> > > > ________________________________ > This e-mail and any attachments may be confidential or legally privileged. > If you received this message in error or are not the intended recipient, > you should destroy the e-mail message and any attachments or copies, and > you are prohibited from retaining, distributing, disclosing or using any > information contained herein. Please inform us of the erroneous delivery by > return e-mail. Thank you for your cooperation. > ________________________________ This e-mail and any attachments may be confidential or legally privileged. If you received this message in error or are not the intended recipient, you should destroy the e-mail message and any attachments or copies, and you are prohibited from retaining, distributing, disclosing or using any information contained herein. Please inform us of the erroneous delivery by return e-mail. Thank you for your cooperation.
