bertrand-buffat opened a new issue, #26394:
URL: https://github.com/apache/airflow/issues/26394
### Apache Airflow version
2.3.4
### What happened
Tasks are no longer scheduled since upgrading to airflow.2.3.4, with logs:
```
{base_executor.py:211} INFO - task TaskInstanceKey(dag_id='sys_liveness',
task_id='liveness', run_id='scheduled__2022-09-14T12:15:00+00:00',
try_number=1, map_index=-1) is still running
{base_executor.py:215} ERROR - could not queue task
TaskInstanceKey(dag_id='sys_liveness', task_id='liveness',
run_id='scheduled__2022-09-14T12:15:00+00:00', try_number=1, map_index=-1)
(still running after 4 attempts)
```
Even with only one dag running in a simple deployment, and enough slot in
the default pool.
### What you think should happen instead
Tasks should be executed.
### How to reproduce
_No response_
### Operating System
Debian GNU/Linux
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon = "4.1.0"
apache-airflow-providers-cncf-kubernetes = "4.0.2"
apache-airflow-providers-http = "2.1.2"
apache-airflow-providers-mysql = "2.2.3"
apache-airflow-providers-postgres = "4.1.0"
apache-airflow-providers-ssh = "2.4.4"
apache-airflow-providers-sqlite = "2.1.3"
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
`kubernetes executor`
```
[core]
dags_folder = /data-foundation-airflow/
hostname_callable = socket.getfqdn
default_timezone = utc
executor = KubernetesExecutor
parallelism = 512
max_active_tasks_per_dag = 128
dags_are_paused_at_creation = False
max_active_runs_per_dag = 1
load_examples = False
plugins_folder = /home/airflow/plugins
execute_tasks_new_python_interpreter = False
fernet_key = 46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho=
donot_pickle = False
dagbag_import_timeout = 30.0
dagbag_import_error_tracebacks = True
dagbag_import_error_traceback_depth = 2
dag_file_processor_timeout = 300
task_runner = StandardTaskRunner
default_impersonation =
security =
unit_test_mode = False
enable_xcom_pickling = False
killed_task_cleanup_time = 60
dag_run_conf_overrides_params = True
dag_discovery_safe_mode = True
dag_ignore_file_syntax = regexp
default_task_retries = 0
default_task_weight_rule = downstream
default_task_execution_timeout =
min_serialized_dag_update_interval = 30
compress_serialized_dags = False
min_serialized_dag_fetch_interval = 30
max_num_rendered_ti_fields_per_task = 30
check_slas = True
xcom_backend = airflow.models.xcom.BaseXCom
lazy_load_plugins = True
lazy_discover_providers = True
hide_sensitive_var_conn_fields = True
sensitive_var_conn_names =
default_pool_task_slot_count = 128
max_map_length = 1024
daemon_umask = 0o077
secure_mode = False
dag_concurrency = 256
non_pooled_task_slot_count = 512
sql_alchemy_conn =
postgresql+psycopg2://airflow:lc9c2w74uhyqi...@airflow-postgres-instance-1-cluster.ctyrh80ang3y.eu-central-1.rds.amazonaws.com:5432/airflow?options=-csearch_path%3Dairflow
sql_alchemy_schema = airflow
[logging]
base_log_folder = /home/airflow/logs
remote_logging = True
remote_log_conn_id =
google_key_path =
remote_base_log_folder =
s3://doctolib-data-foundation-staging/airflow/staging/logs/
encrypt_s3_logs = True
logging_level = INFO
celery_logging_level =
fab_logging_level = WARNING
logging_config_class = log_config.LOGGING_CONFIG
colored_console_log = True
colored_log_format = [%(blue)s%(asctime)s%(reset)s]
{%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s
- %(log_color)s%(message)s%(reset)s
colored_formatter_class =
airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s -
%(message)s
simple_log_format = %(asctime)s %(levelname)s - %(message)s
log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware
task_log_prefix_template =
log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id
}}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index
}}/{% endif %}attempt={{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location =
/home/airflow/logs/dag_processor_manager/dag_processor_manager.log
task_log_reader = task
extra_logger_names =
worker_log_server_port = 8793
[api]
enable_experimental_api = False
auth_backends =
airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session
maximum_page_limit = 100
fallback_page_limit = 100
google_oauth2_audience =
google_key_path =
access_control_allow_headers =
access_control_allow_methods =
access_control_allow_origins =
auth_backend = airflow.api.auth.backend.basic_auth
[webserver]
base_url = https://airflow-staging.doctolibdata.com
default_ui_timezone = UTC
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert = /home/airflow/ssl/cert.pem
web_server_ssl_key = /home/airflow/ssl/key.pem
session_backend = database
web_server_master_timeout = 120
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 30
reload_on_plugin_change = False
secret_key = kCjfcp+BFuDIGas1zi3NkQ==
workers = 4
worker_class = sync
access_logfile = -
error_logfile = -
access_logformat =
expose_config = True
expose_hostname = True
expose_stacktrace = True
dag_default_view = grid
dag_orientation = LR
log_fetch_timeout_sec = 5
log_fetch_delay_sec = 2
log_auto_tailing_offset = 30
log_animation_speed = 1000
hide_paused_dags_by_default = False
page_size = 100
navbar_color = #F1C40F
default_dag_run_display_number = 25
enable_proxy_fix = False
proxy_fix_x_for = 1
proxy_fix_x_proto = 1
proxy_fix_x_host = 1
proxy_fix_x_port = 1
proxy_fix_x_prefix = 1
cookie_secure = False
cookie_samesite = Lax
default_wrap = False
x_frame_enabled = True
show_recent_stats_for_completed_runs = True
update_fab_perms = True
session_lifetime_minutes = 43200
instance_name_has_markup = True
auto_refresh_interval = 3
warn_deployment_exposure = True
audit_view_excluded_events =
gantt,landing_times,tries,duration,calendar,graph,grid,tree,tree_data
filter_by_owner = True
authenticate = True
auth_backend = airflow.contrib.auth.backends.google_auth
rbac = True
[smtp]
smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
smtp_port = 587
smtp_mail_from = [email protected]
smtp_timeout = 30
smtp_retry_limit = 5
smtp_user = [email protected]
smtp_password = svjfxmprcyxsytdu
[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
num_runs = -1
scheduler_idle_sleep_time = 1
min_file_process_interval = 400
deactivate_stale_dags_interval = 60
dag_dir_list_interval = 300
print_stats_interval = 30
pool_metrics_interval = 5.0
scheduler_health_check_threshold = 30
orphaned_tasks_check_interval = 300.0
child_process_log_directory = /home/airflow/logs/scheduler
scheduler_zombie_task_threshold = 300
zombie_detection_interval = 10.0
catchup_by_default = True
ignore_first_depends_on_past_by_default = True
max_tis_per_query = 512
use_row_level_locking = True
max_dagruns_to_create_per_loop = 10
max_dagruns_per_loop_to_schedule = 20
schedule_after_task_execution = True
parsing_processes = 4
file_parsing_sort_mode = modified_time
standalone_dag_processor = False
max_callbacks_per_loop = 20
use_job_schedule = True
allow_trigger_in_future = False
dependency_detector =
airflow.serialization.serialized_objects.DependencyDetector
trigger_timeout_check_interval = 15
max_threads = 5
```
### Anything else
When we restart the scheduler, any queued tasks are executed. But the next
task after isn't and is stucked in queued again.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
