GitHub user ahipp13 created a discussion: Remote Logging to Azure Blob Storage Still Broke in Airflow 3.0.4
### Apache Airflow Provider(s) microsoft-azure ### Versions of Apache Airflow Providers apache-airflow @ [file:///home/build/dist/apache_airflow-3.0.4-py3-none-any.whl](file://home/build/dist/apache_airflow-3.0.4-py3-none-any.whl) apache-airflow-core==3.0.4 apache-airflow-providers-amazon @ [file:///home/build/providers/amazon/dist/apache_airflow_providers_amazon-9.7.0-py3-none-any.whl](file://home/build/providers/amazon/dist/apache_airflow_providers_amazon-9.7.0-py3-none-any.whl) apache-airflow-providers-apache-kafka==1.10.2 apache-airflow-providers-celery @ [file:///home/build/providers/celery/dist/apache_airflow_providers_celery-3.10.6-py3-none-any.whl](file://home/build/providers/celery/dist/apache_airflow_providers_celery-3.10.6-py3-none-any.whl) apache-airflow-providers-cncf-kubernetes @ [file:///home/build/providers/cncf/kubernetes/dist/apache_airflow_providers_cncf_kubernetes-10.4.2-py3-none-any.whl](file://home/build/providers/cncf/kubernetes/dist/apache_airflow_providers_cncf_kubernetes-10.4.2-py3-none-any.whl) apache-airflow-providers-common-compat==1.7.3 apache-airflow-providers-common-io==1.6.2 apache-airflow-providers-common-sql==1.27.5 apache-airflow-providers-docker @ [file:///home/build/providers/docker/dist/apache_airflow_providers_docker-4.3.1-py3-none-any.whl](file://home/build/providers/docker/dist/apache_airflow_providers_docker-4.3.1-py3-none-any.whl) apache-airflow-providers-elasticsearch @ [file:///home/build/providers/elasticsearch/dist/apache_airflow_providers_elasticsearch-6.2.2-py3-none-any.whl](file://home/build/providers/elasticsearch/dist/apache_airflow_providers_elasticsearch-6.2.2-py3-none-any.whl) apache-airflow-providers-fab @ [file:///home/build/providers/fab/dist/apache_airflow_providers_fab-2.2.0-py3-none-any.whl](file://home/build/providers/fab/dist/apache_airflow_providers_fab-2.2.0-py3-none-any.whl) apache-airflow-providers-ftp @ [file:///home/build/providers/ftp/dist/apache_airflow_providers_ftp-3.12.3-py3-none-any.whl](file://home/build/providers/ftp/dist/apache_airflow_providers_ftp-3.12.3-py3-none-any.whl) apache-airflow-providers-google @ [file:///home/build/providers/google/dist/apache_airflow_providers_google-15.1.0-py3-none-any.whl](file://home/build/providers/google/dist/apache_airflow_providers_google-15.1.0-py3-none-any.whl) apache-airflow-providers-grpc @ [file:///home/build/providers/grpc/dist/apache_airflow_providers_grpc-3.7.3-py3-none-any.whl](file://home/build/providers/grpc/dist/apache_airflow_providers_grpc-3.7.3-py3-none-any.whl) apache-airflow-providers-hashicorp @ [file:///home/build/providers/hashicorp/dist/apache_airflow_providers_hashicorp-4.1.1-py3-none-any.whl](file://home/build/providers/hashicorp/dist/apache_airflow_providers_hashicorp-4.1.1-py3-none-any.whl) apache-airflow-providers-http @ [file:///home/build/providers/http/dist/apache_airflow_providers_http-5.2.2-py3-none-any.whl](file://home/build/providers/http/dist/apache_airflow_providers_http-5.2.2-py3-none-any.whl) apache-airflow-providers-imap @ [file:///home/build/providers/imap/dist/apache_airflow_providers_imap-3.8.3-py3-none-any.whl](file://home/build/providers/imap/dist/apache_airflow_providers_imap-3.8.3-py3-none-any.whl) apache-airflow-providers-microsoft-azure @ [file:///home/build/providers/microsoft/azure/dist/apache_airflow_providers_microsoft_azure-12.3.1-py3-none-any.whl](file://home/build/providers/microsoft/azure/dist/apache_airflow_providers_microsoft_azure-12.3.1-py3-none-any.whl) apache-airflow-providers-mysql @ [file:///home/build/providers/mysql/dist/apache_airflow_providers_mysql-6.2.1-py3-none-any.whl](file://home/build/providers/mysql/dist/apache_airflow_providers_mysql-6.2.1-py3-none-any.whl) apache-airflow-providers-odbc @ [file:///home/build/providers/odbc/dist/apache_airflow_providers_odbc-4.9.2-py3-none-any.whl](file://home/build/providers/odbc/dist/apache_airflow_providers_odbc-4.9.2-py3-none-any.whl) apache-airflow-providers-openlineage @ [file:///home/build/providers/openlineage/dist/apache_airflow_providers_openlineage-2.1.3-py3-none-any.whl](file://home/build/providers/openlineage/dist/apache_airflow_providers_openlineage-2.1.3-py3-none-any.whl) apache-airflow-providers-postgres @ [file:///home/build/providers/postgres/dist/apache_airflow_providers_postgres-6.1.3-py3-none-any.whl](file://home/build/providers/postgres/dist/apache_airflow_providers_postgres-6.1.3-py3-none-any.whl) apache-airflow-providers-redis @ [file:///home/build/providers/redis/dist/apache_airflow_providers_redis-4.0.2-py3-none-any.whl](file://home/build/providers/redis/dist/apache_airflow_providers_redis-4.0.2-py3-none-any.whl) apache-airflow-providers-sendgrid @ [file:///home/build/providers/sendgrid/dist/apache_airflow_providers_sendgrid-4.0.1-py3-none-any.whl](file://home/build/providers/sendgrid/dist/apache_airflow_providers_sendgrid-4.0.1-py3-none-any.whl) apache-airflow-providers-sftp @ [file:///home/build/providers/sftp/dist/apache_airflow_providers_sftp-5.2.0-py3-none-any.whl](file://home/build/providers/sftp/dist/apache_airflow_providers_sftp-5.2.0-py3-none-any.whl) apache-airflow-providers-slack @ [file:///home/build/providers/slack/dist/apache_airflow_providers_slack-9.0.4-py3-none-any.whl](file://home/build/providers/slack/dist/apache_airflow_providers_slack-9.0.4-py3-none-any.whl) apache-airflow-providers-smtp==2.2.0 apache-airflow-providers-snowflake==6.5.2 apache-airflow-providers-sqlite @ [file:///home/build/providers/sqlite/dist/apache_airflow_providers_sqlite-4.0.2-py3-none-any.whl](file://home/build/providers/sqlite/dist/apache_airflow_providers_sqlite-4.0.2-py3-none-any.whl) apache-airflow-providers-ssh @ [file:///home/build/providers/ssh/dist/apache_airflow_providers_ssh-4.0.1-py3-none-any.whl](file://home/build/providers/ssh/dist/apache_airflow_providers_ssh-4.0.1-py3-none-any.whl) apache-airflow-providers-standard==1.5.0 apache-airflow-task-sdk==1.0.4 google-cloud-orchestration-airflow==1.17.5 ### Apache Airflow version 3.0.4 ### Operating System bash-5.3$ cat /etc/os-release ID=chainguard NAME="Chainguard" PRETTY_NAME="Chainguard" VERSION_ID="20230214" HOME_URL="https://chainguard.dev/"; ### Deployment Official Apache Airflow Helm Chart ### Deployment details Deploying helm chart on to Kubernetes version v1.32.4-gke.200. For remote logging I have these environment variables set: ``` # Remote Logging - name: "AIRFLOW__LOGGING__REMOTE_LOGGING" value: "True" - name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER" value: "wasb://airflow-logs@ACCOUNT_NAME.blob.core.windows.net/wasb-airflow-logs" - name: "AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER" value: "airflow-logs" - name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID" value: "edw_az_conn" - name: "AIRFLOW__LOGGING__DELETE_LOCAL_LOGS" value: "True" ``` And I have this connection created: <img width="867" height="779" alt="Image" src="https://github.com/user-attachments/assets/4bbda1a1-5445-46a0-bb42-84294480c1f2"; /> <img width="858" height="758" alt="Image" src="https://github.com/user-attachments/assets/526a3a6b-d3c0-480d-9d15-689738634dd2"; /> ### What happened Whenever I run one of my DAGs, it will complete successfully, but then when it tries to write the logs it will give an error that says "The conn_id `edw_az_conn` isn't defined" when it clearly is as you can see from the screenshots above. Full log below ``` {"event":"Could not write logs to wasb-airflow-logs/dag_id=assettracking_sand_meili_namespaces_dag/run_id=manual__2025-08-12T13:54:52.960501+00:00/task_id=process_namespa │ │ ces_sand/attempt=1.log","level":"error","logger":"airflow.providers.microsoft.azure.log.wasb_task_handler.WasbRemoteLogIO","timestamp":"2025-08-12T13:55:53.482145Z","exce │ │ ption":[{"exc_type":"AirflowNotFoundException","exc_value":"The conn_id `edw_az_conn` isn't defined","exc_notes":[],"syntax_error":null,"is_cause":false,"frames":[{"filen │ │ ame":"/opt/airflow/lib/python3.12/site-packages/airflow/providers/microsoft/azure/log/wasb_task_handler.py","lineno":170,"name":"write"},{"filename":"/opt/airflow/lib/pyt │ │ hon3.12/site-packages/airflow/providers/microsoft/azure/hooks/wasb.py","lineno":359,"name":"load_string"},{"filename":"/opt/airflow/lib/python3.12/site-packages/airflow/p │ │ roviders/microsoft/azure/hooks/wasb.py","lineno":416,"name":"upload"},{"filename":"/opt/airflow/lib/python3.12/site-packages/airflow/providers/microsoft/azure/hooks/wasb. │ │ py","lineno":231,"name":"_get_blob_client"},{"filename":"/usr/lib/python3.12/functools.py","lineno":998,"name":"__get__"},{"filename":"/opt/airflow/lib/python3.12/site-pa │ │ ckages/airflow/providers/microsoft/azure/hooks/wasb.py","lineno":156,"name":"blob_service_client"},{"filename":"/opt/airflow/lib/python3.12/site-packages/airflow/provider │ │ s/microsoft/azure/hooks/wasb.py","lineno":160,"name":"get_conn"},{"filename":"/opt/airflow/lib/python3.12/site-packages/airflow/hooks/base.py","lineno":64,"name":"get_con │ │ nection"},{"filename":"/opt/airflow/lib/python3.12/site-packages/airflow/models/connection.py","lineno":512,"name":"get_connection_from_secrets"}],"is_group":false,"excep │ │ tions":[]}]} ``` ### What you think should happen instead It should log to the blob storage like normal. I have this same configuration running in Airflow 2.11 ### How to reproduce Would need to set up Azure remote logging with a storage account and use the airflow provider. ### Anything else I brought up this issue here: https://github.com/apache/airflow/issues/54192 I was told upgrading to 3.0.4 would fix it, but it is still occurring to me. Another note, I know it is not the connection that is broken, because I can read logs in the blob storage before I upgraded the environment... ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) GitHub link: https://github.com/apache/airflow/discussions/54489 ---- This is an automatically sent email for commits@airflow.apache.org. To unsubscribe, please send an email to: commits-unsubscr...@airflow.apache.org
