GitHub user chalapathi-dba added a comment to the discussion: How we know scheduler is overloaded. Not in terms of CPU or RAM. Is there anyway to check from Airflow side
Hi @potiuk A ton thanks for your response. The case is, I have a DAG having 5 tasks. Task execution are completed in 1 or 2 seconds but task pickup between one to other there is delay (Means, after 1st task completed in 2 seconds, second task started after 2 mins or more the pipeline is simple task1>>task2>>task3>>task4>>task5 ) so whole DAG taking 7 mins. **This is happening intermittently.** What I am thinking the scheduler unable to handle the load during the times hence I need to prove to application team saying that there is heaving load on scheduler interms of DAGs runnig or tasking picking. GitHub link: https://github.com/apache/airflow/discussions/53910#discussioncomment-13955645 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
