SalvaChiLlo opened a new issue, #36795: URL: https://github.com/apache/airflow/issues/36795
### Apache Airflow version 2.8.0 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Tasks are taking to long to get executed, workers receive the order to execute a task but once the workers execute the command, it takes several seconds for the actual task to get executed. For example, example_complex dag is taking between a minute and a half and two minutes to complete, but the actual task execution time for each of the tasks is less than a second. I have custom workers that receive tasks from kafka and execute them. I'll append a few logs: ### What you think should happen instead? Tasks take less than a second to get actually executed, but the total amount of time executing the dag is much bigger than expected. ### How to reproduce I just try to execute the example_complex dag although the described behaviour is the same for every other dag. ### Operating System Debian GNU/Linux 12 (bookworm) ### Versions of Apache Airflow Providers pip freeze | grep providers apache-airflow-providers-amazon==8.16.0 apache-airflow-providers-celery==3.5.0 apache-airflow-providers-cncf-kubernetes==7.11.0 apache-airflow-providers-common-io==1.1.0 apache-airflow-providers-common-sql==1.9.0 apache-airflow-providers-docker==3.8.2 apache-airflow-providers-elasticsearch==5.3.0 apache-airflow-providers-ftp==3.7.0 apache-airflow-providers-google==10.12.0 apache-airflow-providers-grpc==3.4.0 apache-airflow-providers-hashicorp==3.6.0 apache-airflow-providers-http==4.8.0 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-microsoft-azure==8.4.0 apache-airflow-providers-mysql==5.5.0 apache-airflow-providers-odbc==4.2.0 apache-airflow-providers-openlineage==1.3.0 apache-airflow-providers-postgres==5.9.0 apache-airflow-providers-redis==3.5.0 apache-airflow-providers-sendgrid==3.4.0 apache-airflow-providers-sftp==4.8.0 apache-airflow-providers-slack==8.5.0 apache-airflow-providers-snowflake==5.2.0 apache-airflow-providers-sqlite==3.6.0 apache-airflow-providers-ssh==3.9.0 ### Deployment Official Apache Airflow Helm Chart ### Deployment details My deployment is pretty similar to the one using docker compose but a big difference is that I have a Kafka executor and workers, for discarding I made several tests to ensure that they are not introducing unexpected delays. ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
