dennisbakhuis opened a new issue #16777: URL: https://github.com/apache/airflow/issues/16777
I feel a bit like an idiot but I cannot seem to get the logging working on persistent volumes when using the KubernetesExecutor and the freshly released official Helm chart (version 1.0.0). After creating a simple [PV](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pv.yaml) and [PVC](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pvc.yaml) manually, I changed the following on the bottom of `values.yaml` file: ``` logs: persistence: # Enable persistent volume for storing logs enabled: true # Volume size for logs size: 100Gi # If using a custom storageClass, pass name here storageClassName: ## the name of an existing PVC to use existingClaim: airflow-logs ``` This process is partly described in the [official Helm documentation](https://airflow.apache.org/docs/helm-chart/stable/manage-logs.html). Still, the airflow-scheduler pod crashes due to permission errors as i cannot write in the mounted logs folder: [logs in this pastebin](https://pastebin.com/ek3iH4EZ). When the persistent logging is turned off, all is working, except for task logging as these are deleted when the worker-pod is deleted. A solution might be to have an init container to set the folder permissions but I guess that is not yet implemented in the current chart (or I do not know how to do it). Any help towards a solution would be greatly appreciated! **Apache Airflow version**: 2.1.0 ** Helm chart version**: 1.0.0 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.20.7 **Environment**: Microsoft AKS - **Cloud provider or hardware configuration**: Microsoft AKS - **OS** (e.g. from /etc/os-release): - **Kernel** (e.g. `uname -a`): - **Install tools**: - **Others**: **What happened**: I added a persistent volume (PV) and a persistent volume claim (PVC) manually to my AKS cluster. Next, I set the persistent logging settings on the bottom, i.e. logs.persistence.enabled=true; logs.persistence.existingClaim=airflow-logs The exact [PV](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pv.yaml) and [PVC](https://github.com/marclamberti/webinar-airflow-chart/blob/master/pvc.yaml) settings very basic so I would not expect anything from that. After updating the deployment using `helm upgrade` the scheduler pod crashes. The full error log is [here](https://pastebin.com/ek3iH4EZ). **What you expected to happen**: I expect Airflow and the worker pods to write all logs to the persistent volume, mounted at `/opt/airflow/logs` Inspecting the logs it looks like the mkdir is failing due to permission issues which makes the whole pod crash. **How to reproduce it**: create the PV and PVC, turn on logs.persistence.enable and set logs.persistence.existingClaim=<PVC-name> **Anything else we need to know**: I have seen a similar problem with the unofficial chart. The solution was to include a extraInitContainer that 'fixes' the permissions of the logs folder. I have tried that but helm complained that such a setting does not exist. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
