sujith9591 opened a new issue, #39981: URL: https://github.com/apache/airflow/issues/39981
### Apache Airflow version Other Airflow 2 version (please specify below) ### If "Other Airflow 2 version" selected, which one? 2.7.3 ### What happened? Error while running Airflow CLI Commands using BashOperator the below command. airflow tasks run 'DAG_ID' 'TASK_ID' 'RUN_ID' -f -A -i Initially with less number of dags(~500) in the prod environment, the dag(created for running the task by ignoring dependencies) is working as expected. But, when we have deployed more number of dags(~2500) in the prod environment, the same dag which worked fine is failing with the below error. "airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code -9." As per the logs,we get to see that before running the specific task in the given dag by ignoring all its dependencies, it is trying to parse all the dags which are under the dags folder. Based on our investigation, we found the failure is because of the memory issue. The worker is using a lot of memory to process all the dags in the dags folder before performing the actual task. Note: Increase the worker memory has fixed the issue for us. But Going forward, the dag number will also gets increased and incresing the worker memory everytime we face this issue is not an optimal solution. Also, we have found an subdir option, which helps us to specify the File location or directory from which to look for the dag. But as mentioned above, we will encounter the same memory issue in case if there are huge number of dags in the specified folder and the same leads to dags failure. Note: Even running a simple command to list the dags(airflow dags list) using bash operator is also yielding a failure. Env Details: Astro Version: 9.14 Airflow Version: 2.7.3 Cloud: Azure Please refer to the below zip folder for the dag code and error log details [ignore_dependencies_log.txt](https://github.com/user-attachments/files/15516871/ignore_dependencies_log.txt) [dagcode_log_details.zip](https://github.com/user-attachments/files/15516859/dagcode_log_details.zip) ### What you think should happen instead? So, our request is "Instead of parsing all the dags available in the dags folder/Specified subdir folder.it will be better if it process/parse only the dag that we are referring as part of the command" ### How to reproduce Please run the following command with higher number of dags in the dags folder. airflow tasks run 'DAG_ID' 'TASK_ID' 'RUN_ID' -f -A -i ### Operating System Linux ### Versions of Apache Airflow Providers _No response_ ### Deployment Astronomer ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
