eladkal commented on issue #57693: URL: https://github.com/apache/airflow/issues/57693#issuecomment-3477751814
> Relationship between logical date and partition date? Unless I am missing something. I don't think these two dates should have direct relationship. I see this in a simplistic (maybe over simplistic) kind of way. Assets partitions can be updated regardless of Airflow DAGs. For example the asset represents some path in S3 that is being update by another system and asset update is invoked by API. In this case there is no logical date binding because the update of the Asset did not come from internal Airflow signal - it was not updated via a DAG. **Description of scenario:** Asset being updated externally as I described above once a day (daily) but DAG that depended on this asset runs hourly. Thus there is no 1 to 1 mapping between the asset and the dag scheduling. Dag scheduling should be based on window. Let me as DAG author to describe what is the window I am OK with about this asset update. since I am not into the details of the AIP I don't know how much of this is covered (if any) but I do hope that we will work towards this capability. It is important to note that not all Assets (data dependencies) are equal in importance. Some might be just enrichments. The concept of window allows you to schedule despite some delays because it's good enough. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
