Ciateon opened a new issue, #30077: URL: https://github.com/apache/airflow/issues/30077
### Apache Airflow version Other Airflow 2 version (please specify below) ### What happened Hi all, As mentioned by Airflow Community and various forum what I have heard that it's much better to use the official docker Image rather than using the custom docker image. So we started using the docker image which is build by Airflow community rather than writing our own image as its much more optimized and have much greater security than our own image. So currently I'm trying to deploy Airflow 2.2.3 as its most compatible with our system. I used to following image from Docker hub *`airflow:2.2.3-python3.9`* . But we are unable to deploy Airflow using the following image. We have three deployment config as of now for the deployment of Airflow * Airflow Webserver * Airflow Database[Postgres] * Airflow Redis[Message broker] Message Broker and Database are up and running but whenever we are trying to deploy the **airflow webserver** we are not able to launch get the Airflow up and running. We have stored the Airflow Docker Image(2.2.3) in our ImageStream of our platform aka Openshift. Strange part we are able to get airflow Up and running with the docker build from source, but not from the official image of Airflow:2.2.3  ### What you think should happen instead Ideally we should get the airflow log mentioning airflow is up and running, but in our case the pod for the Airflow-webserver is dying within few second. ### How to reproduce We are using Openshift V 4.6 as a platform for the deployment of Airflow. ### Operating System RHEL ### Versions of Apache Airflow Providers _No response_ ### Deployment Other ### Deployment details The deployment Config that we have used for the deployment of Airflow is as follows: *airflow-webserver.yaml* ``` apiVersion: apps.openshift.io/v1 kind: DeploymentConfig metadata: name: airflow-webserver namespace: CUSTOM_NAMESPACE labels: app: airflow spec: strategy: type: Rolling triggers: - type: ConfigChange - type: ImageChange imageChangeParams: automatic: true containerNames: - airflow-webserver from: kind: ImageStreamTag namespace: CUSTOM_NAMESPACE name: "airflow:2.2.3-python3.9" replicas: 1 revisionHistoryLimit: 10 paused: false selector: app: airflow deploymentconfig: airflow-webserver template: metadata: labels: name: airflow-webserver app: airflow deploymentconfig: airflow-webserver spec: volumes: - name: airflow-dags persistentVolumeClaim: claimName: airflow-dags - name: airflow-logs persistentVolumeClaim: claimName: airflow-logs - name: airflow-redis persistentVolumeClaim: claimName: airflow-redis containers: - name: airflow-webserver image: image-registry.openshift-image-registry.svc:5000/CUSTOM_NAMESPACE/airflow resources: limits: memory: 6Gi env: - name : AIRFLOW__CORE__SQL_ALCHEMY_CONN value: postgresql+psycopg2://CUSTOM_NAME:CUSTOM_PASS@airflow-db/airflow - name: AIRFLOW__CORE__EXECUTOR value: CeleryExecutor - name : AIRFLOW__CELERY__BROKER_URL value: redis://:CUSTOM_PASS@CUSTOM_REDIS_SVC:6379/0 - name: AIRFLOW__CELERY__RESULT_BACKEND value: db+postgresql://CUSTOM_NAME:CUSTOM_NAME@airflow-db/airflow - name: AIRFLOW__CORE__FERNET_KEY valueFrom: secretKeyRef: name: airflow-helm key: airflow-fernet-key - name: AIRFLOW__WEBSERVER__SECRET_KEY valueFrom: secretKeyRef: name: airflow-helm key: airflow-secret-key - name: AIRFLOW_LOAD_EXAMPLES value: "no" - name: AIRFLOW_WEBSERVER_PORT_NUMBER value: "8080" - name: AIRFLOW_USERNAME value: user - name: AIRFLOW_PASSWORD value: user - name: AIRFLOW__WEBSERVER__BASE_URL value: "http://airflow-image-custom_url.router.com"; - name: AIRFLOW_LDAP_ENABLE value: "no" - name: AIRFLOW_UID value: "50000" ports: - containerPort: 8080 protocol: TCP volumeMounts: - name: airflow-dags mountPath: /opt/airflow/dags - name: airflow-logs mountPath: /opt/airflow/logs - name: airflow-redis mountPath: /var/lib/redis/data ``` Deployment Config for the Metadatabase is as follows: ``` apiVersion: apps.openshift.io/v1 kind: DeploymentConfig metadata: name: airflow-db namespace: CUSTOM_NAMESPACE labels: app: airflow spec: strategy: type: Rolling triggers: - type: ConfigChange replicas: 1 revisionHistoryLimit: 10 paused: false selector: app: airflow deploymentconfig: airflow-db template: metadata: labels: name: airflow-db app: airflow deploymentconfig: airflow-db spec: volumes: - name: airflow-db persistentVolumeClaim: claimName: airflow-db containers: - name: airflow-db image: image-registry.openshift-image-registry.svc:5000/openshift/postgresql-13-rhel8 resources: limits: memory: 512Mi env: - name: POSTGRESQL_USER value: CUSTOM_NAME - name: POSTGRESQL_PASSWORD value: CUSTOM_PASS - name: POSTGRESQL_ROOT_PASSWORD value : CUSTOM_PASS - name: POSTGRESQL_DATABASE value: CUSTOM_NAME ports: - containerPort: 5432 protocol: TCP volumeMounts: - name: airflow-db mountPath: /var/lib/postgresql/data securityContext: capabilities: {} privileged: false livenessProbe: exec: command: - pg_isready - -h - localhost - -U - postgres initialDelaySeconds: 30 timeoutSeconds: 5 readinessProbe: exec: command: - pg_isready - -h - localhost - -U - postgres initialDelaySeconds: 5 timeoutSeconds: 1 ``` Deployment Config for Redis is as follows: ``` apiVersion: apps.openshift.io/v1 kind: DeploymentConfig metadata: name: air-redis namespace: CUSTOM_NAMESPACE label: app: airflow spec: strategy: type: Rolling triggers: - type: ConfigChange replicas: 1 revisionHistoryLimit: 10 paused: false selector: app: airflow deploymentconfig: air-redis template: metadata: labels: name: air-redis app: airflow deploymentconfig: air-redis spec: volumes: - name: airflow-redis persistentVolumeClaim: claimName: airflow-redis containers: - name: air-redis image: image-registry.openshift-image-registry.svc:5000/openshift/redis-rhel-8 resources: limits: memory: 512Mi env: - name: REDIS_PASSWORD value: CUSTOM_PASSWORD ports: - containerPort: 6379 protocol: TCP volumeMounts: - name: airflow-redis mountPath: /var/lib/redis/data readinessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 5 timeoutSeconds: 1 ``` All the password are taken from the *secrets* but for the sake of simplicity i have made here hard-coded value. ### Anything else Error Log while Deploying Airflow Webserver is as follows: *airflow-webserver pod is crashing within few seconds* <img width="801" alt="airflow-meta-image2" src="https://user-images.githubusercontent.com/35499473/224782401-c3de61cf-a734-4fe2-bb80-341777d61d5c.PNG";> Corresponding error logs : ``` usage: airflow [-h] GROUP_OR_COMMAND ... positional arguments: GROUP_OR_COMMAND Groups: celery Celery components config View configuration connections Manage connections dags Manage DAGs db Database operations jobs Manage jobs kubernetes Tools to help run the KubernetesExecutor pools Manage pools providers Display providers roles Manage roles tasks Manage tasks users Manage users variables Manage variables Commands: cheat-sheet Display cheat sheet info Show information about current Airflow and environment kerberos Start a kerberos ticket renewer plugins Dump information about loaded plugins rotate-fernet-key Rotate encrypted connection credentials and variables scheduler Start a scheduler instance standalone Run an all-in-one copy of Airflow sync-perm Update permissions for existing roles and optionally DAGs triggerer Start a triggerer instance version Show the version webserver Start a Airflow webserver instance optional arguments: -h, --help show this help message and exit airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above. ``` No able to understand if there what is actual issue when I'm using the official Image from Airflow community. I was able to deploy using the custom image which I wrote but not from the official image. Do I need to change anything in the Deployment Config or there is some other which i'm not able to figure it out. Thanks in Advance. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/35499473/224782401-c3de61cf-a734-4fe2-bb80-341777d61d5c.PNG"](https://user-images.githubusercontent.com/35499473/224782401-c3de61cf-a734-4fe2-bb80-341777d61d5c.PNG"){kind=link}