Ah yeah... I totally forgot about that :) (shame on me) ... But it does seem appropriate if I came to the same conclusion again looking from another angle.... :D
J. On Mon, Feb 24, 2020 at 6:25 PM Gerard Casas Saez <[email protected]> wrote: > Agree, I initially pitched the idea on the lineage thread and was > encouraged to pitch it separately. I would love to help figure out how to > align this 2 projects better. > > Bolke - want to set up a call or how should we discuss this better? Would > love to hear feedback on my proposal. > > Gerard Casas Saez > Twitter | Cortex | @casassaez > On Feb 23, 2020, 1:44 AM -0700, Jarek Potiuk <[email protected]>, > wrote: > > I like the idea a lot. Good direction. > > > > I know we have a few people who are better in functional thinking than me > > so I think I would love those people to work it out. Happy to listen to > the > > discussions :) > > > > One thing that struck me however. I am not sure if that falls in the same > > camp, but I have a feeling that there is at least some common part with > the > > proposal from Bolke about the Lineage. > > Those two things (functional DAG API and Lineage) are not directly > > connected but I think the design of both has at least some common part > and > > it would make sense that at least we talk about this and how they play > > together. > > > > WDYT Bolke? > > > > > > J > > > > On Fri, Feb 21, 2020 at 7:13 PM Dan Davydov <[email protected] > > > > wrote: > > > > > Here is the link to the AIP for folk's convenience: > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-31%3A+Airflow+functional+DAG+API > > > > > > The proposal and this all looks really good to me :)! I do want to > call out > > > to others that it's important we get the interface 95%+ right from the > > > get-go since it could be hard to change later once users start > depending on > > > it. > > > > > > On Fri, Feb 21, 2020 at 12:07 PM Gerard Casas Saez > > > <[email protected]> wrote: > > > > > > > Hi everyone, > > > > > > > > Sending a new message to everyone to gather feedback on the AIP-31 > about > > > > Airflow functional DAG API. This was initially discussed and > proposed in > > > > [DISCUSS] Airflow functional DAGs. After leaving open a small doc to > > > > iterate on the proposal for a couple weeks, I decided to move > forward to > > > > formalize it as an AIP document in confluence. I will still be > tracking > > > > comments on the doc but would rather move the conversation here. > > > > > > > > AIP - 31 focuses on solving the issue about implicit message passing > in > > > > Airflow by extending the DAG/Operator API to include a way to set up > a > > > > message passing dependency on the same DAG file. This is > complimentary to > > > > task dependency declaration and is intended to be used as another > option > > > to > > > > declare dependencies by declaring message dependencies. > > > > > > > > In addition, AIP-31 proposes a way to declare PythonOperators from a > > > > function using decorators. This should help embed custom behavior > into > > > DAGs > > > > without needing to create custom operators for everything. > > > > > > > > Changes proposed: > > > > > > > > > > > > • Add __call__ function in BaseOperator: Add a functional interface > to > > > > replace class attributes on execution time. > > > > • XComArg class: This object is a reference to an XCom value that > has not > > > > been created and will need to be resolved in the future. > > > > • PythonFunctionalOperator and Python Function Operator: Extend > > > > PythonOperator to map op_args and op_kwargs from a decorated > interface > > > for > > > > easier set up and add a decorator to create PythonOperators from a > > > function > > > > in an easier way. > > > > > > > > > > > > See AIP document for a DAG example. > > > > > > > > Any help on how to proceed with this will be appreciated (also joined > > > > Slack and posted in. #airflow-creative). My guess is once we agree on > > > what > > > > the API should look like, next step is to do a vote and if > successful to > > > > create JIRA issues and GitHub PRs with the mentioned changes. > > > > > > > > Please let me know if there’s any aspect that people feel > > > > strongly opinionated against or aspects that are not clear and I > should > > > > work on expanding further. > > > > > > > > Best, > > > > > > > > Gerard Casas Saez > > > > Twitter | Cortex | @casassaez > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
