potiuk commented on issue #28941: URL: https://github.com/apache/airflow/issues/28941#issuecomment-1383257889
Airflow's docker compose is not fool-proof. Redis is an in-memory data store and killing it and restarting will cause those problems. The Docker-compose of ours is not "official" way of running a production-ready deployment. Quite contrary - we are very explicit that this is quick-start only and you should make it much more robust if you want to make it production ready (and what you are describing is part of production-ready deployment): In the docker-compose: > # WARNING: This configuration is for local development. Do not use it in a production deployment. and in the [docs](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#running-airflow-in-docker): > This procedure can be useful for learning and exploration. However, adapting it for use in real-world situations can be complicated. Making changes to this procedure will require specialized expertise in Docker & Docker Compose, and the Airflow community may not be able to help you. > For that reason, we recommend using Kubernetes with the [Official Airflow Community Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) when you are ready to run Airflow in production. The docker-compose of ours uses redis in it's very-basic form, which is without persistence - all data is kept in memory. If you need persistence in order ot survive restarts, you need to configure and modify the docker-compose: Redis has https://redis.io/docs/management/persistence/ But it's up to you to make it robust and production-ready and resilient to any kind of failures, I am afraid. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
