> I always assumed that this was the reason why it's impossible to create dags from API, no one wanted to open this particular can of worms. I think if you need to synchronize these objects, the cleaner way would be to describe them in some sort of a shared config file and let respective dag-processors create them independently of each other.
Just to clarify this one: - creating DAGs via API has been resented mostly because of security reasons - where you would want to submit Python DAG code via API. There is (and it has never been) a problem with opening up submitting "structured" DAGs. This has never been implemented, but if you would like to limit to just modifying or creating resulting DAG structure, that would be possible - for example there is no fundamental problem with generating a DAG from (say) visual representation and submitting a resulting DAG structure without creating a DAG python file (so essentially playing the role of DAG file processor to serialize DAGs). It would have a number of limitations (for example callbacks would not work., timetables would be a challenge etc.), but other than that it's quite possible (and possibly even in the future we might have something like that). Following that - there are no fundamental problems with submitting datasets - because they are not Python code, they are pure "metadata" objects. Still the questions remains how it plays with the DAG-created datasets is an important aspect of the proposal. J. On Tue, Jan 23, 2024 at 2:01 PM Tornike Gurgenidze <togur...@freeuni.edu.ge> wrote: > Maybe I'm missing something, but I can't see how rest endpoints for > datasets could work in practice. afaik, Airflow has some objects that cab > be created by a dag processor (Dags, Datasets) and others that can be > created with API/UI (Connections, Variables), but never both at the same > time. How would update/delete endpoints work if Dataset was initially > created declaratively from a dag file? Would it throw an exception or make > an update that will then be reverted in a little while by a dag-processor > anyway? > > I always assumed that this was the reason why it's impossible to create > dags from API, no one wanted to open this particular can of worms. I think > if you need to synchronize these objects, the cleaner way would be to > describe them in some sort of a shared config file and let respective > dag-processors create them independently of each other. > > On Tue, Jan 23, 2024 at 4:02 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > I am also pretty cool with adding/updating/datasets externally, however I > > know there are some ongoing discussions on how to improve/change datasets > > and bind them together with multiple other features of Airflow - not sure > > what the state of those, but it would be great those effort are > coordinated > > so that we are not pulling stuff in multiple directions. > > > > From what I've heard/overheard noticed about Datasets are those things: > > > > * AIP-60 - > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-60+Standard+URI+representation+for+Airflow+Datasets > > - already almost passed > > * Better coupling of datasets with OpenLineage > > * Partial datasets - allowing to have datasets with data intervals > > * Triggering dags on external dataset changes > > * Objects Storage integration with datasets > > > > All of which sound very promising and are definitely important for > Dataset > > usage. > > > > So I think we really make sure when we are doing anything with datasets, > > the people who think/work on those aspects above have a say in those > > proposals/discussions - it would be a shame if we add something that will > > partially invalidate or make terribly complex to implement some of the > > other things. > > > > I am not saying it's the case here, I am just saying that we should at > > least make sure that people who are currently thinking about these things > > don't come surprised if we merge something that will make their job > harder. > > > > I am a little surprised - knowing the *thinking* happening in dataset > area > > that I am aware of that there are so little comments on that one (even if > > "hey looks cool" - works well for the things I am thinking about) :). > > > > J. > > > > > > > > > > On Tue, Jan 23, 2024 at 3:53 AM Ryan Hatter > > <ryan.hat...@astronomer.io.invalid> wrote: > > > > > I don't think it makes sense to include the create endpoint without > also > > > including dataset update and delete endpoints and updating the Datasets > > > view in the UI to be able to manage externally created Datasets. > > > > > > With that said, I don't think the fact that Datasets are tightly > coupled > > > with DAGs is a good reason not to include additional Dataset endpoints. > > It > > > makes sense to me to be able to interact with Datasets from outside of > > > Airflow. > > > > > > On Sat, Jan 20, 2024 at 6:13 AM Eduardo Nicastro < > edu.nicas...@gmail.com > > > > > > wrote: > > > > > > > Hello all, I have created a Pull Request ( > > > > https://github.com/apache/airflow/pull/36929) to make it possible to > > > > create a dataset through the API as a modest step forward. This PR is > > > open > > > > for your feedback. I'm preparing another PR to build upon the > insights > > > from > > > > https://github.com/apache/airflow/pull/29433. Your thoughts and > > > > contributions are highly encouraged. > > > > > > > > Best Regards, > > > > Eduardo Nicastro > > > > > > > > On Thu, Jan 11, 2024 at 4:30 PM Eduardo Nicastro < > > edu.nicas...@gmail.com > > > > > > > > wrote: > > > > > > > >> Hello all, > > > >> > > > >> I'm reaching out to propose a topic for discussion that has recently > > > >> emerged in our GitHub discussion threads (#36723 > > > >> <https://github.com/apache/airflow/discussions/36723>). It revolves > > > >> around enhancing the management of datasets in a multi-tenant > Airflow > > > >> architecture. > > > >> > > > >> Use case/motivation > > > >> In our multi-instance setup, synchronizing dataset dependencies > across > > > >> instances poses significant challenges. With the advent of dataset > > > >> listeners, a new door has opened for cross-instance dataset > > awareness. I > > > >> propose we explore creating endpoints to export dataset updates to > > make > > > it > > > >> possible to trigger DAGs consuming from a Dataset across tenants. > > > >> > > > >> Context > > > >> Below I will give some context about our current situation and > > solution > > > >> we have in place and propose a new workflow that would be more > > > efficient. > > > >> To be able to implement this new workflow we would need a way to > > export > > > >> Dataset updates as mentioned. > > > >> > > > >> Current Workflow > > > >> In our organization, we're dealing with multiple Airflow tenants, > > let's > > > >> say Tenant 1 and Tenant 2, as examples. To synchronize Dataset A > > across > > > >> these tenants, we currently have a complex setup: > > > >> > > > >> 1. Containers run on a schedule to export metadata to CosmosDB > > (these > > > >> will be replaced by the listener). > > > >> 2. Additional scheduled containers pull data from CosmosDB and > > write > > > >> it to a shared file system, enabling generated DAGS to read it > and > > > mirror a > > > >> dataset across tenants. > > > >> > > > >> > > > >> Proposed Workflow > > > >> Here's a breakdown of our proposed workflow: > > > >> > > > >> 1. Cross-Tenant Dataset Interaction: We have Dags in Tenant 1 > > > >> producing Dataset A. We need a mechanism to trigger all Dags > > > consuming > > > >> Dataset A in Tenant 2. This interaction is crucial for our data > > > pipeline's > > > >> efficiency and synchronicity. > > > >> 2. Dataset Listener Implementation: Our approach involves > > > >> implementing a Dataset listener that programmatically creates > > > Dataset A in > > > >> all tenants where it's not present (like Tenant 2) and export > > Dataset > > > >> updates when they happen. This would trigger an update on all > Dags > > > >> consuming from that Dataset. > > > >> 3. Standardized Dataset Names: We plan to use standardized > dataset > > > >> names, which makes sense since a URI is its identifier and > > > uniqueness is a > > > >> logical requirement. > > > >> > > > >> [image: image.png] > > > >> > > > >> Why This Matters: > > > >> > > > >> - It offers a streamlined, automated way to manage datasets > across > > > >> different Airflow instances. > > > >> - It aligns with a need for efficient, interconnected workflows > in > > a > > > >> multi-tenant environment. > > > >> > > > >> > > > >> I invite the community to discuss: > > > >> > > > >> - Are there alternative methods within Airflow's current > framework > > > >> that could achieve similar goals? > > > >> - Any insights or experiences that could inform our approach? > > > >> > > > >> Your feedback and suggestions are invaluable, and I look forward to > a > > > >> collaborative discussion. > > > >> > > > >> Best Regards, > > > >> Eduardo Nicastro > > > >> > > > > > > > > > >
