andormarkus opened a new issue #16010:
URL: https://github.com/apache/airflow/issues/16010
**Apache Airflow version**: 2.0.2
**Helm chart version**: 1.0.0
**Kubernetes version**: 1.20
**What happened**:
I have tried to enable HTTPS connection, however when I enable it liveness
and readiness probes are failing
```bash
▶ kubectl -n airflow describe pod/airflow-webserver-564fcf9c5c-jqtt4
Name: airflow-webserver-564fcf9c5c-jqtt4
Namespace: airflow
Priority: 0
Node: ip-10-10-13-31.eu-central-1.compute.internal/10.10.13.31
Start Time: Sun, 23 May 2021 11:37:08 +0200
Labels: component=webserver
pod-template-hash=564fcf9c5c
release=airflow
tier=airflow
Annotations: checksum/airflow-config:
4e5f26f48f157218dcaf8eddac95e912aa7a47a4eb75b04697bb3701fd550340
checksum/extra-configmaps:
2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
checksum/extra-secrets:
bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
checksum/metadata-secret:
765446a9def21895960ebc0df295399c35d3467cdd84be6ef8bc65c18ca0f7e5
checksum/pgbouncer-config-secret:
da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
checksum/webserver-config:
4a2281a4e3ed0cc5e89f07aba3c1bb314ea51c17cb5d2b41e9b045054a6b5c72
kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.13.56
IPs:
IP: 10.10.13.56
Controlled By: ReplicaSet/airflow-webserver-564fcf9c5c
Init Containers:
wait-for-airflow-migrations:
Container ID:
docker://da6e1a36ac496982bc6d9058e1d94791cf99bf7d70dbbe10be2f2bbe0446eae8
Image: apache/airflow:2.0.2
Image ID:
docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
Port: <none>
Host Port: <none>
Args:
python
-c
import airflow
import logging
import os
import time
from alembic.config import Config
from alembic.runtime.migration import MigrationContext
from alembic.script import ScriptDirectory
from airflow import settings
package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
directory = os.path.join(package_dir, 'migrations')
config = Config(os.path.join(package_dir, 'alembic.ini'))
config.set_main_option('script_location', directory)
config.set_main_option('sqlalchemy.url',
settings.SQL_ALCHEMY_CONN.replace('%', '%%'))
script_ = ScriptDirectory.from_config(config)
timeout=60
with settings.engine.connect() as connection:
context = MigrationContext.configure(connection)
ticker = 0
while True:
source_heads = set(script_.get_heads())
db_heads = set(context.get_current_heads())
if source_heads == db_heads:
break
if ticker >= timeout:
raise TimeoutError("There are still unapplied migrations
after {} seconds.".format(ticker))
ticker += 1
time.sleep(1)
logging.info('Waiting for migrations... %s second(s)', ticker)
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 23 May 2021 11:37:11 +0200
Finished: Sun, 23 May 2021 11:37:24 +0200
Ready: True
Restart Count: 0
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in
secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in
secret 'airflow-postgres-password'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in
secret 'airflow-postgres-password'> Optional: false
AWS_DEFAULT_REGION: eu-central-1
AWS_REGION: eu-central-1
AWS_ROLE_ARN:
arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
AWS_WEB_IDENTITY_TOKEN_FILE:
/var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token
(ro)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb
(ro)
Containers:
webserver:
Container ID:
docker://2b8ee8a99ebd088b545685b3d1bc290adb0a7675e4935322a9549a152339a249
Image: apache/airflow:2.0.2
Image ID:
docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
Port: 8080/TCP
Host Port: 0/TCP
Args:
bash
-c
exec airflow webserver
State: Running
Started: Sun, 23 May 2021 11:39:48 +0200
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 23 May 2021 11:37:26 +0200
Finished: Sun, 23 May 2021 11:39:48 +0200
Ready: False
Restart Count: 1
Liveness: http-get http://:8080/airflow/health delay=15s
timeout=30s period=5s #success=1 #failure=20
Readiness: http-get http://:8080/airflow/health delay=15s
timeout=30s period=5s #success=1 #failure=20
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in
secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in
secret 'airflow-postgres-password'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in
secret 'airflow-postgres-password'> Optional: false
AWS_DEFAULT_REGION: eu-central-1
AWS_REGION: eu-central-1
AWS_ROLE_ARN:
arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
AWS_WEB_IDENTITY_TOKEN_FILE:
/var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
/opt/airflow/dags from dags (rw)
/opt/airflow/logs from logs (rw)
/opt/airflow/secrets/ssl from airflow-ssl (ro)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token
(ro)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb
(ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 86400
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-airflow-config
Optional: false
dags:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: airflow-dags
ReadOnly: false
logs:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: airflow-logs
ReadOnly: false
airflow-ssl:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-ssl
Optional: false
airflow-token-9cqzb:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-token-9cqzb
Optional: false
QoS Class: BestEffort
Node-Selectors: namespace=airflow
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m14s default-scheduler Successfully
assigned airflow/airflow-webserver-564fcf9c5c-jqtt4 to
ip-10-10-13-31.eu-central-1.compute.internal
Normal Pulled 4m11s kubelet Container
image "apache/airflow:2.0.2" already present on machine
Normal Created 4m11s kubelet Created
container wait-for-airflow-migrations
Normal Started 4m11s kubelet Started
container wait-for-airflow-migrations
Normal Pulled 3m57s kubelet Container
image "apache/airflow:2.0.2" already present on machine
Normal Created 3m57s kubelet Created
container webserver
Normal Started 3m56s kubelet Started
container webserver
Warning Unhealthy 3m37s kubelet Liveness
probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp
10.10.13.56:8080: connect: connection refused
Warning Unhealthy 3m36s (x2 over 3m41s) kubelet Readiness
probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp
10.10.13.56:8080: connect: connection refused
Warning Unhealthy 3m2s kubelet Liveness
probe failed: Get "http://10.10.13.56:8080/airflow/health": context deadline
exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 3m1s kubelet Readiness
probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:37846->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 3m1s kubelet Liveness
probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:38016->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 3m1s kubelet Readiness
probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:38046->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 3m1s kubelet Readiness
probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:38048->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 2m57s kubelet Liveness
probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:38056->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 2m56s kubelet Readiness
probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:38060->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 2m32s (x9 over 2m52s) kubelet (combined
from similar events): Liveness probe failed: Get
"http://10.10.13.56:8080/airflow/health": read tcp
10.10.13.31:38192->10.10.13.56:8080: read: connection reset by peer
```
**How to reproduce it**:
Helm chart myconf.yaml looks like this
```yaml
executor: KubernetesExecutor
fernetKey: "XXXXXXXXXX"
webserver:
base_url: "https://k8s.foobaa.com/airflow";
web_server_ssl_cert: "/opt/airflow/secrets/ssl/tls.crt"
web_server_ssl_key: "/opt/airflow/secrets/ssl/tls.key"
webserver:
service:
type: NodePort
extraVolumeMounts:
- name: airflow-ssl
mountPath: /opt/airflow/secrets/ssl
readOnly: true
extraVolumes:
- name: airflow-ssl
secret:
secretName: airflow-ssl
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
