jonathonbattista opened a new issue #14040:
URL: https://github.com/apache/airflow/issues/14040


   **Apache Airflow version**: 1.10.14
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl 
version`): v1.18.12-gke.1205
   
   **Environment**: GKE using KubernetesExecutor with MySQL backend.
   
   **What happened**:  Pods are not spawning because Scheduler is out of sync 
with the DB. 
   
   The webserver is reporting queued tasks but the scheduler has no idea about 
them.
   
   A restart of the Scheduler pod temporarily fixes the problem.
   
   Our `default_pool` has `128` slots but we set`core-parallelism` to `120`.
   
   Webserver:
   
![image](https://user-images.githubusercontent.com/42357520/106660653-21492780-6555-11eb-9b55-29eeaa9ef6ab.png)
   
   Scheduler:
   ```
   airflow-scheduler [2021-02-02 20:21:28,025] {base_executor.py:122} DEBUG - 3 
running task instances                                                          
                                                                                
              
   airflow-scheduler [2021-02-02 20:21:28,025] {base_executor.py:123} DEBUG - 0 
in queue                                                                        
                                                                                
              
   airflow-scheduler [2021-02-02 20:21:28,025] {base_executor.py:124} DEBUG - 
117 open slots
   ```
   **What you expected to happen**:
   Webserver and Scheduler should be in sync.
   
   <!-- What do you think went wrong? -->
   
   **How to reproduce it**:
   Deploy 1.10.14 on K8s with KubernetesExectuor and MySQL backend.
   
   
   **Anything else we need to know**:
   
   <!--
   
   How often does this problem occur? Once? Every time etc?
   
   Any relevant logs to include? Put them here in side a detail tag:
   <details><summary>x.log</summary> lots of stuff </details>
   
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to