jens-scheffler-bosch commented on issue #34206:
URL: https://github.com/apache/airflow/issues/34206#issuecomment-1734392443

   I was thinking about this a bit. I like more "dynamics" in data sets but was 
thinking about the use case.
   
   Let's assume the dynamic comes from an S3 bucket where you want to have the 
dataset being "flexible" based on the object dropped in the bucket. Dynamically 
defining the outlet would give the burden hat you also need to create a 
consuming DAG dynamically or you need to glob a dataset pattern. But this also 
would generate a huge list of datasets over time which might be temporary.
   
   How about using additional "meta data" like information and a rather high 
level dataset? For example the dataset itself just points to the S3 bucket and 
the outlet is triggered to this but can leave some meta data with the event 
about which file (or whatsoever details) are triggering in the bucket. Think 
about a JSON structure describing for example `{ "filename": 
"my_test_record.csv" }`. Such (dict?) kind of meta data could be used as 
`dag_run.conf` for the DAGs triggered by the Dataset. This would use existing 
mecahnisms and no need to dynamically register DAGs for (might be flexible but 
not known at point of authoring) datasets.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to