Here is the link to the AIP for folk's convenience: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-31%3A+Airflow+functional+DAG+API
The proposal and this all looks really good to me :)! I do want to call out to others that it's important we get the interface 95%+ right from the get-go since it could be hard to change later once users start depending on it. On Fri, Feb 21, 2020 at 12:07 PM Gerard Casas Saez <[email protected]> wrote: > Hi everyone, > > Sending a new message to everyone to gather feedback on the AIP-31 about > Airflow functional DAG API. This was initially discussed and proposed in > [DISCUSS] Airflow functional DAGs. After leaving open a small doc to > iterate on the proposal for a couple weeks, I decided to move forward to > formalize it as an AIP document in confluence. I will still be tracking > comments on the doc but would rather move the conversation here. > > AIP - 31 focuses on solving the issue about implicit message passing in > Airflow by extending the DAG/Operator API to include a way to set up a > message passing dependency on the same DAG file. This is complimentary to > task dependency declaration and is intended to be used as another option to > declare dependencies by declaring message dependencies. > > In addition, AIP-31 proposes a way to declare PythonOperators from a > function using decorators. This should help embed custom behavior into DAGs > without needing to create custom operators for everything. > > Changes proposed: > > > • Add __call__ function in BaseOperator: Add a functional interface to > replace class attributes on execution time. > • XComArg class: This object is a reference to an XCom value that has not > been created and will need to be resolved in the future. > • PythonFunctionalOperator and Python Function Operator: Extend > PythonOperator to map op_args and op_kwargs from a decorated interface for > easier set up and add a decorator to create PythonOperators from a function > in an easier way. > > > See AIP document for a DAG example. > > Any help on how to proceed with this will be appreciated (also joined > Slack and posted in. #airflow-creative). My guess is once we agree on what > the API should look like, next step is to do a vote and if successful to > create JIRA issues and GitHub PRs with the mentioned changes. > > Please let me know if there’s any aspect that people feel > strongly opinionated against or aspects that are not clear and I should > work on expanding further. > > Best, > > Gerard Casas Saez > Twitter | Cortex | @casassaez >
