GitHub user pawgajda-drs added a comment to the discussion: Azure blob remote
logging using the official helm chart
If anyone's stumbles upon this issue when trying to make it work, here are my
notes.
1. First of all, when running Helm Chart version 1.18.0, you're on Airflow
3.0.2. Override your images to 3.0.6, otherwise it just won't work at all:
```
images:
airflow:
repository: apache/airflow
tag: "3.0.6"
pod_template:
repository: apache/airflow
tag: "3.0.6"
```
2. Configure WASB/Azure Blob Storage Conenction via Airflow UI or via
Environment Variable.
More information here:
https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables
If you want to configure it via Environment Variable, I recommend this format:
```
AIRFLOW_CONN_WASB_LOGGING='{"conn_type": "wasb", "host":
"__STORAGE_ACCOUNT_HOST__", "login": "__STORAGE_ACCOUNT_NAME__", "password":
"__STORAGE_ACCOUNT_SHARED_KEY__"}'
```
bare minimum version:
```
AIRFLOW_CONN_WASB_LOGGING='{"conn_type": "wasb", "login":
"__STORAGE_ACCOUNT_NAME__", "password": "__STORAGE_ACCOUNT_SHARED_KEY__"}'
```
I recommend setting it up as `Secret` then pass it to Helm Chart like this:
```
extraEnvFrom: |
- secretRef:
name: airflow-remote-logging-secret
```
Finally, the configuration itself:
```
env:
# Enable Remote Logging
- name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
value: "True"
# Delete Local Logs when Remote Logging is Enabled
- name: "AIRFLOW__LOGGING__DELETE_LOCAL_LOGS"
value: "True"
# Directory inside Blob Container
# needs to start with 'wasb' or 'wasb/' or 'wasb://'
- name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
value: "wasb/logs"
# Blob Container Name
- name: "AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER"
value: "airflow-logs"
# Auth for Blob Storage
# The naming convention is AIRFLOW_CONN_{CONN_ID} (lowercase value here)
- name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID"
value: "wasb_logging"
```
A few notes:
Regarding `AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER` value, documentation here:
https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/logging/index.html
says to use
```
wasb://path/to/logs
```
format, but for version 3.0.6 do `wasb/path/to/logs`, otherwise your path will
look like `wasb:/noname/path/to/logs`.
Regarding `AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID` set it to lowercase name of
your variable or the name of connection you created via UI, that's it.
Thanks to everyone who participated in this discussion and other discussion
related to this topic:
https://github.com/apache/airflow/issues/54192
https://github.com/apache/airflow/issues/54413
GitHub link:
https://github.com/apache/airflow/discussions/37461#discussioncomment-14508496
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
