GitHub user carlosjourdan added a comment to the discussion: Large number of
Transactions for the logs when using WASB
We were also facing this problem using the helm chart. Turns out that the
`xxx-log-groomer` containers in each pod (worker, triggerer, scheduler) was
calling the `/clean-logs` script, which runs a `find` command in the
`$AIRFLOW_HOME/logs` folder. This being actually a persistent volume with
`azurefile-csi` storage class, a lot of transactions were sent to the storage
account.
We solved it by disabling logging persistence at the helm chart level, and
enabling remote logging in the airflow configuration level. We also changed all
the remaining storage classes to `managed-csi`.
The relevant parts of the helm template look like this:
```yaml
config:
logging:
remote_logging: "True"
remote_log_conn_id: "wasb" #in our case we created the connection using the
AzureKeyVaultBackend integration
remote_base_log_folder:
"wasb://logs@${STORAGE_ACCOUNT_NAME}.blob.core.windows.net"
azure_remote_logging:
remote_wasb_log_container: "logs"
logs:
persistence:
enabled: false
```
GitHub link:
https://github.com/apache/airflow/discussions/34973#discussioncomment-12706367
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
