Hi Airflow users, We are currently running Airflow v2.1.2 and keep getting the following reoccurring error.
"Error while health checking kube watcher process. Process died for unknown reasons" No DAGs are currently being set to run. Stack trace of error is as follows: ############################################################################# Show 91 collapsed crumbs 07:14:24 query SELECT dag.dag_id AS dag_dag_id, dag.root_dag_id AS dag_root_dag_id, dag.is_paused AS dag_is_paused, dag.is_subdag AS dag_is_subdag, dag.is_active AS dag_is_active, dag.last_parsed_time AS dag_last_parsed_time, dag.last_pickled AS dag_last_pickled, dag.last_expired AS dag_last_expired, dag.scheduler_lock AS dag_scheduler_lock, dag.pickle_id AS dag_pickle_id, dag.fileloc AS dag_fileloc, dag.owners AS dag_owners, dag.description AS dag_description, dag.default_view AS dag_default_view, dag.schedule_interva... 07:14:24 query: SELECT dag_run.dag_id AS dag_run_dag_id, dag_run.execution_date AS dag_run_execution_date FROM dag_run WHERE 1 != 1 07:14:24 query: SELECT dag_run.dag_id AS dag_run_dag_id, count(%(count_2)s) AS count_1 FROM dag_run WHERE 1 != 1 AND dag_run.state = %(state_1)s AND dag_run.external_trigger IS false GROUP BY dag_run.dag_id 07:14:24 query: SELECT dag_run.state AS dag_run_state, dag_run.id AS dag_run_id, dag_run.dag_id AS dag_run_dag_id, dag_run.execution_date AS dag_run_execution_date, dag_run.start_date AS dag_run_start_date, dag_run.end_date AS dag_run_end_date, dag_run.run_id AS dag_run_run_id, dag_run.creating_job_id AS dag_run_creating_job_id, dag_run.external_trigger AS dag_run_external_trigger, dag_run.run_type AS dag_run_run_type, dag_run.conf AS dag_run_conf, dag_run.last_scheduling_decision AS dag_run_last_scheduling_decision, da... 07:14:24 query: SELECT task_instance.dag_id AS task_instance_dag_id, task_instance.execution_date AS task_instance_execution_date FROM task_instance WHERE 1 != 1 AND task_instance.state NOT IN (%(state_1)s, %(state_2)s, %(state_3)s, %(state_4)s, %(state_5)s) GROUP BY task_instance.dag_id, task_instance.execution_date 07:14:24 query : SELECT dag_run.state AS dag_run_state, dag_run.id AS dag_run_id, dag_run.dag_id AS dag_run_dag_id, dag_run.execution_date AS dag_run_execution_date, dag_run.start_date AS dag_run_start_date, dag_run.end_date AS dag_run_end_date, dag_run.run_id AS dag_run_run_id, dag_run.creating_job_id AS dag_run_creating_job_id, dag_run.external_trigger AS dag_run_external_trigger, dag_run.run_type AS dag_run_run_type, dag_run.conf AS dag_run_conf, dag_run.last_scheduling_decision AS dag_run_last_scheduling_decision, da... 07:14:24 query: SELECT slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots FROM slot_pool FOR UPDATE NOWAIT 07:14:24 query: SELECT task_instance.pool AS task_instance_pool, task_instance.state AS task_instance_state, sum(task_instance.pool_slots) AS sum_1 FROM task_instance WHERE task_instance.state IN (%(state_1)s, %(state_2)s) GROUP BY task_instance.pool, task_instance.state 07:14:24 query: SELECT task_instance.try_number AS task_instance_try_number, task_instance.task_id AS task_instance_task_id, task_instance.dag_id AS task_instance_dag_id, task_instance.execution_date AS task_instance_execution_date, task_instance.start_date AS task_instance_start_date, task_instance.end_date AS task_instance_end_date, task_instance.duration AS task_instance_duration, task_instance.state AS task_instance_state, task_instance.max_tries AS task_instance_max_tries, task_instance.hostname AS task_instance_ho... 07:14:24 message: Error while health checking kube watcher process. Process died for unknown reasons ################################################################## Anyone have any experience with this error? Regards, Eric
