GitHub user mvngne closed a discussion: Save logs to S3 (ceph)
Hello, everyone. I'm trying to deploy Apache Airflow with the Helm chart to our Kubernetes cluster. After some initial difficulties, I have it running now. However, there is one remaining issue: Logging. Currently, without any special settings, the logs aren't persistent. Since the pods are deleted by default after execution, I can't access the logs afterwards. We would like to save the logs to S3 (Ceph), but I can't seem to get that to work. There are some parts in the documentation on how to configure AWS S3, which is similar but slightly different, that I tried to use, but it just won't work. My goal is to use the deployment to add all dependencies so that the logs from all DAGs, which are synced from Git, are automatically saved in S3 and can be viewed later. Is there someone who has configured logging to S3 in Airflow 3+ with Helm who can give me hints on how to configure it? GitHub link: https://github.com/apache/airflow/discussions/58889 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
