TJaniF commented on issue #59294:
URL: https://github.com/apache/airflow/issues/59294#issuecomment-3950618522
Adding to the above list by @cmarteepants:
One-to-many would also be interesting for segment partitions. I.e. Creating
a daily report, with X partitioned Dag runs all running at the same time each
creating the report for a different department. Currently this would be solved
with DTM but has the disadvantage that you can't easily rerun the whole Dag
just for one department (if, for example, the data for R&D would often be
incomplete etc).
From what I understand this and other segment-based partitions would depend
on the ability to "attach" partitions from within upstream Dags, similar to how
you can attach an extra to an AssetEvent right now.
Something like
```python
my_asset_1 = Asset("asset1")
@task(outlets=[my_asset_1])
def t1():
yield Partitions(my_asset_1, ["R&D", "Marketing", "Finance"])
return "hello :)"
```
Also worth thinking through what if the daily report for Marketing needs
data from 2 different locations, e.g. asset1 partitioned with "Marketing" and
asset2 partitioned with marketing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
