girip11 opened a new issue, #71768: URL: https://github.com/apache/airflow/issues/71768
### Under which category would you file this issue? Providers ### Apache Airflow version 3.2.2 ### What happened and how to reproduce it? I have got an Airflow self hosted deployment on Azure Kubernetes cluster with remote logging enabled to azure blob storage. I was migrating my existing deployment from airflow version 2.8.4 to 3.2.2. My setup involves `remote_logging` enabled with logs copied to azure blob storage. I use a custom `logging_config_class`. This was all working on airflow version 2.8.4. I followed what was mentioned in the documentation of version [3.2.2 advanced logging configuration](https://airflow.apache.org/docs/apache-airflow/3.2.2/administration-and-deployment/logging-monitoring/advanced-logging-configuration.html). After migrating to v3.2.2 I was getting this warning in the scheduler logs > warning ] remote_log_handler_unavailable [airflow.logging.remote] loc=log.py:236 note='Remote log handler could not be loaded; logs will be available locally only.' And the logs were not copied to my azure blob storage container. Upon going through the source code for the remote logging to work, I found that we need to set the following global variables in the python module. ``` REMOTE_TASK_LOG: RemoteLogIO | None = MyRemoteLogIO() DEFAULT_REMOTE_CONN_ID: str | None = "my_remote_conn" ``` This wasn't captured in the airflow 3.2.2 documentation and after fixing this, the scheduler warning related to remote logging disappeared and the remote logs were written to azure blob storage. ### What you think should happen instead? The documentation for the advanced logging that uses custom logging_config_class along with remote logging enabled scenario is not captured correctly. The documentation for airflow version 3.3.1 is updated with this information, but the documentation for airflow version 3.2.2 should be updated with this information. ### Operating System NAME="Debian GNU/Linux", VERSION_ID="12", VERSION_CODENAME=bookworm ### Deployment Official Apache Airflow Helm Chart ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers ``` apache-airflow-providers-amazon 9.29.0 apache-airflow-providers-apache-livy 4.5.6 apache-airflow-providers-cncf-kubernetes 10.17.1 apache-airflow-providers-common-compat 1.15.0 apache-airflow-providers-common-io 1.7.2 apache-airflow-providers-common-sql 2.0.0 apache-airflow-providers-fab 3.6.4 apache-airflow-providers-http 6.0.2 apache-airflow-providers-microsoft-azure 13.3.0 apache-airflow-providers-postgres 6.7.0 apache-airflow-providers-smtp 3.0.1 apache-airflow-providers-ssh 5.0.2 apache-airflow-providers-standard 1.13.1 ``` ### Official Helm Chart version 1.22.0 (latest released) ### Kubernetes Version 1.35.3 ### Helm Chart configuration Airflow logging related YAML configuration ```YAML logging: remote_logging: "True" remote_log_conn_id: "azblob_logging" remote_base_log_folder: "wasb://wasb-airflow-logs-aks" base_log_folder: "/opt/airflow/logs" logging_level: "INFO" fab_logging_level: "WARN" colored_console_log: "True" colored_formatter_class: "airflow.utils.log.colored_log.CustomTTYColoredFormatter" simple_log_format: "%%(asctime)s %%(levelname)s - %%(message)s" dag_processor_child_process_log_directory: "/opt/airflow/logs/dag_processor" task_log_reader: "task" encrypt_s3_logs: "False" delete_local_logs: "False" logging_config_class: "config.log_config.LOGGING_CONFIG" azure_remote_logging: remote_wasb_log_container: "airflow-logs" ``` ### Docker Image customizations Not Applicable ### Anything else? This was my original logging configuration ```python from copy import deepcopy from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG) # Prevent task log records from propagating to the root console handler. # The default propagate=True means every record is handled twice (FileTaskHandler # + root console), which the Airflow UI merges into duplicate log lines. LOGGING_CONFIG["loggers"]["airflow.task"]["propagate"] = False ``` ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
