[
https://issues.apache.org/jira/browse/AIRFLOW-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17288246#comment-17288246
]
Abdul Wahab commented on AIRFLOW-5858:
--------------------------------------
I am using 1.10.9 and this issue happened to us yesterday again and resulted in
breaking a lot of important thins in Prod. Can someone please provide an update
if it got fixed in any of the Airflow 1.*.* versions? if yes, i would like to
then upgrade.
> Airflow celery worker missing heartbeat
> ---------------------------------------
>
> Key: AIRFLOW-5858
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5858
> Project: Apache Airflow
> Issue Type: Bug
> Components: celery
> Affects Versions: 1.10.2
> Reporter: Nikhil SInghal
> Priority: Major
>
> Our Airflow setup uses Celery Executors and Redis as a broker. We are facing
> a issue of missing heartbeat from celery workers. Once this heartbeat is
> missed the worker stops taking any new task. It also appears as offline in
> the Celery flower UI. Manual restart of workers fixes this problem.
> Wanted to know if this is a known issue or being faced by other users in the
> community
> These are the logs from failure.
> [2019-11-06 02:30:36,368: INFO/MainProcess] missed heartbeat from
> celery@dp-airflow-worker-6cb4b596f8-nzdrt
> worker: Warm shutdown (MainProcess)
> -------------- celery@dp-airflow-worker-6cb4b596f8-4qkfg v4.1.1 (latentcall)
> ---- **** -----
> --- * *** * -- Linux-4.9.0-9-amd64-x86_64-with-debian-10.1 2019-11-05 15:24:34
> -- * - **** ---
> - ** ---------- [config]
> - ** ---------- .> app: airflow.executors.celery_executor:0x7f2a01250cf8
> - ** ---------- .> transport:
> redis://:**@redis-11313.internal.c3160.ap-southeast-1-mz.ec2.cloud.rlrcp.com:11313//
> - ** ---------- .> results:
> postgresql://airflow:**@airflowdbprod.ckvce9fjaook.ap-southeast-1.rds.amazonaws.com:5432/airflowdb
> - *** --- * --- .> concurrency: 64 (prefork)
> -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this
> worker)
> --- ***** -----
> -------------- [queues]
> .> default exchange=default(direct) key=default
> [tasks]
> . airflow.executors.celery_executor.execute_command
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
