raj-manvar opened a new issue #13788: URL: https://github.com/apache/airflow/issues/13788
<!-- Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions. Don't worry if they're not all applicable; just try to include what you can :-) If you need to include code snippets or logs, please put them in fenced code blocks. If they're super-long, please use the details tag like <details><summary>super-long log</summary> lots of stuff </details> Please delete these comment blocks before submitting the issue. --> **Description** It could be beneficial for Airflow to support Jenkins' "H" cron syntax in Airflow scheduling. The reason for this is to mitigate a stampede of tasks at the top of every hour / some interval, which can currently straining resources based on applications. "H" syntax specifies to run the DAG during a window of time, allowing the scheduler to spread out jobs based on a hash value. For instance, the syntax "H(0-15) * * * *" means to schedule any time in the first 15 minutes, or "H * * * *" would mean to schedule during any minute of the hour. <!-- A short description of your feature --> **Use case / motivation** <!-- What do you want to happen? Rather than telling us how you might implement this solution, try to take a step back and describe what you are trying to achieve. --> Aim is to resolve the stampede of task occuring at some hour of day or at some midnight of day of month. Currently we need to reserve more resources for Airflow to handle peaks of many tasks trying to schedule because of this. H syntax will help with better distribution of load with time and save resources. **Are you willing to submit a PR?** <!--- We accept contributions! --> Yup. from some code digging, it looks like Airflow does the crontab scheduling using some Python library. If the library already supports H syntax, it'd be simpler, but if not I'd need some more guidance / research support **Related Issues** <!-- Is there currently another issue associated with this? --> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
