t4n1o opened a new issue #19192: URL: https://github.com/apache/airflow/issues/19192
### Apache Airflow version 2.1.4 ### Operating System Linux / Ubuntu Server ### Versions of Apache Airflow Providers apache-airflow-providers-ftp==2.0.1 apache-airflow-providers-http==2.0.1 apache-airflow-providers-imap==2.0.1 apache-airflow-providers-postgres==2.3.0 apache-airflow-providers-sqlite==2.0.1 ### Deployment Virtualenv installation ### Deployment details Airflow v2.1.4 Postgres 13 LocalExecutor Installed with Virtualenv / ansible - https://github.com/idealista/airflow-role ### What happened  I run a single BashOperator (for a long running task, we have to download data for 8+ hours initially to get in sync with the data source, then download more each day in small increments). We're only using 3% CPU and 2 GB of memory (out of 64 GB) but the scheduler is unable to run any other simple task at the same time. Currently only the long task is running, everything else is queued, even thought we have more resources:  ### What you expected to happen I expect my long running BashOperator task to run, but for airflow to have the resources to run other tasks without hanging like this. ### How to reproduce I run a command with bashoperator (I use it because I have python, C, and rust programs being scheduled by airflow). bash_command='umask 002 && cd /opt/my_code/ && /opt/my_code/venv/bin/python -m path.to.my.python.namespace', Configuration: ``` airflow_executor: LocalExecutor airflow_database_conn: 'postgresql+psycopg2://airflow:airflow_pass@localhost:5432/airflow' # "sqlite:///{{ airflow_app_home }}/airflow.db" airflow_database_engine_encoding: utf-8 airflow_database_engine_collation_for_ids: airflow_database_pool_enabled: True airflow_database_pool_size: 3 airflow_database_max_overflow: 10 airflow_database_pool_recycle: 2000 airflow_database_pool_pre_ping: True airflow_database_schema: airflow_database_connect_args: airflow_parallelism: 10 airflow_dag_concurrency: 7 airflow_dags_are_paused_at_creation: True airflow_max_active_runs_per_dag: 16 airflow_load_examples: False airflow_load_default_connections: False airflow_plugins_folder: "{{ airflow_app_home }}/plugins" # [operators] airflow_operator_default_owner: airflow airflow_operator_default_cpus: 1 airflow_operator_default_ram: 512 airflow_operator_default_disk: 512 airflow_operator_default_gpus: 0 airflow_default_queue: default airflow_allow_illegal_arguments: False ``` ### Anything else This occurs every time consistently, also on 2.1.2 The other tasks have this state:  ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
