patsevanton commented on issue #15340:
URL: https://github.com/apache/airflow/issues/15340#issuecomment-818472351
**kubectl describe -n xxxxx pod airflow-scheduler-658d5d4454-r2sgl**
```
Name: airflow-scheduler-658d5d4454-r2sgl
Namespace: xxxxx
Priority: 0
Node: ubuntu1804/192.168.22.7
Start Time: Tue, 13 Apr 2021 05:54:59 +0000
Labels: component=scheduler
pod-template-hash=658d5d4454
release=airflow
tier=airflow
Annotations: checksum/airflow-config:
d84f720b402097e58a879efc896869845ec8bae56455470bf241221b2a016f19
checksum/extra-configmaps:
2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
checksum/extra-secrets:
bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
checksum/metadata-secret:
a954626eab69d09b0c9bfd44128c793948c18d943d9e97431903985654b350c5
checksum/pgbouncer-config-secret:
da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
checksum/result-backend-secret:
af25d110685219c9219e6a4f9b268566118a4b732de33192387a111d1f241c89
cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status: Pending
IP: 10.1.78.6
IPs:
IP: 10.1.78.6
Controlled By: ReplicaSet/airflow-scheduler-658d5d4454
Init Containers:
wait-for-airflow-migrations:
Container ID:
containerd://ac2a25e781647e59aa341e5e308ebbef60408d69b1a2f6b5f2d83df808718ec2
Image: apache/airflow:2.0.0
Image ID:
docker.io/apache/airflow@sha256:e973fef20d3be5b6ea328d2707ac87b90f680382790d1eb027bd7766699b2409
Port: <none>
Host Port: <none>
Args:
python
-c
import airflow
import logging
import os
import time
from alembic.config import Config
from alembic.runtime.migration import MigrationContext
from alembic.script import ScriptDirectory
from airflow import settings
package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
directory = os.path.join(package_dir, 'migrations')
config = Config(os.path.join(package_dir, 'alembic.ini'))
config.set_main_option('script_location', directory)
config.set_main_option('sqlalchemy.url',
settings.SQL_ALCHEMY_CONN.replace('%', '%%'))
script_ = ScriptDirectory.from_config(config)
timeout=60
with settings.engine.connect() as connection:
context = MigrationContext.configure(connection)
ticker = 0
while True:
source_heads = set(script_.get_heads())
db_heads = set(context.get_current_heads())
if source_heads == db_heads:
break
if ticker >= timeout:
raise TimeoutError("There are still unapplied migrations
after {} seconds.".format(ticker))
ticker += 1
time.sleep(1)
logging.info('Waiting for migrations... %s second(s)', ticker)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 13 Apr 2021 06:15:15 +0000
Finished: Tue, 13 Apr 2021 06:16:24 +0000
Ready: False
Restart Count: 7
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in
secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in
secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in
secret 'airflow-airflow-metadata'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from
airflow-scheduler-token-q6zfr (ro)
Containers:
scheduler:
Container ID:
Image: apache/airflow:2.0.0
Image ID:
Port: <none>
Host Port: <none>
Args:
bash
-c
exec airflow scheduler
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Liveness: exec [python -Wignore -c import os
os.environ['AIRFLOW__CORE__LOGGING_LEVEL'] = 'ERROR'
os.environ['AIRFLOW__LOGGING__LOGGING_LEVEL'] = 'ERROR'
from airflow.jobs.scheduler_job import SchedulerJob
from airflow.utils.db import create_session
from airflow.utils.net import get_hostname
import sys
with create_session() as session:
job =
session.query(SchedulerJob).filter_by(hostname=get_hostname()).order_by(
SchedulerJob.latest_heartbeat.desc()).limit(1).first()
sys.exit(0 if job.is_alive() else 1)
] delay=10s timeout=5s period=30s #success=1 #failure=10
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in
secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in
secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in
secret 'airflow-airflow-metadata'> Optional: false
Mounts:
/opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
/opt/airflow/logs from logs (rw)
/opt/airflow/pod_templates/pod_template_file.yaml from config
(ro,path="pod_template_file.yaml")
/var/run/secrets/kubernetes.io/serviceaccount from
airflow-scheduler-token-q6zfr (ro)
scheduler-gc:
Container ID:
Image: apache/airflow:2.0.0
Image ID:
Port: <none>
Host Port: <none>
Args:
bash
/clean-logs
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/opt/airflow/logs from logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from
airflow-scheduler-token-q6zfr (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-airflow-config
Optional: false
logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
airflow-scheduler-token-q6zfr:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-scheduler-token-q6zfr
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25m default-scheduler Successfully
assigned xxxxx/airflow-scheduler-658d5d4454-r2sgl to ubuntu1804
Normal Pulling 24m kubelet Pulling image
"apache/airflow:2.0.0"
Normal Pulled 24m kubelet Successfully
pulled image "apache/airflow:2.0.0"
Normal Created 17m (x5 over 24m) kubelet Created
container wait-for-airflow-migrations
Normal Started 17m (x5 over 24m) kubelet Started
container wait-for-airflow-migrations
Normal Pulled 17m (x4 over 22m) kubelet Container
image "apache/airflow:2.0.0" already present on machine
Warning BackOff 4m58s (x50 over 21m) kubelet Back-off
restarting failed container
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
