Shadowsong27 opened a new issue #10194: URL: https://github.com/apache/airflow/issues/10194
**Description** So I have been encountering odd time schedule request recently, say some updates need to be done on every second and third Friday each month, 4pm. Tried to search for solution but could not find a perfect ones, I can think of the following solutions: 1. use a cron and a separate DAG 2. embed trigger in the actual task logic, that is to check whether the code should be executed in that date, otherwise return success (equivalent to skip and mark success) Drawbacks (at least the one i can think of): 1. it creates extra DAG, and when you have a bunch of these requests, it's not really clean. Besides, you need to add equal number of ExternalTaskSensor to preserve inter-DAG dependencies. It also however, will need to provision other dependent resources (such as Kafka consumers) 2. not very generic and reusable apparently, and if you are going for a full containerized solution - each Task is executed in container - you also need to start a container first to reach that simple if-else check, which seems a bit wasteful **Use case / motivation** Basically I have a bunch of extraction scripts that need to be executed at very different and very weird times, very straight forward (?), without the need to replicating provision tasks, too many inter-DAG dependencies Potential solutions I have in mind: - not touching Airflow core (workaround) inherit and extend the Operators i am using - troublesome and needs to repeat for multiple operators - touching Airflow core an extra tuple params `allowed_dates` passed into the operators (BaseOperator), and then inject the date check right before the operator execution, essentially giving all operator the ability to be programmatically skipped in any DAG - might be an overkill ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
