Adaverse commented on PR #32779:
URL: https://github.com/apache/airflow/pull/32779#issuecomment-1657838038
> I know we still have provider-specific code in "logging" section, but with
the recent move of configuration to providers, we should rather aim to
eventually move all of those to provider-specific configuration.
So in order to remove provider specific config from loading from core
Airflow, below are the steps I would follow -
1. Introduce a Airflow config variable (ex - AIRFLOW__LOGGING__REMOTE_LOGGER
?) that can take values like s3, cloudwatch, gcs, wasb etc.. that will help us
eliminate the if else we have which can be dynamically fetched from the
provider yaml from `name` field.
2. Next, we can dynamically create the task handler dictionary based on the
input from provider.yaml from the specific provider
for example for Azure provider.yaml-->
```
logging:
- airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler
- name: wasb
- formatter: airflow
- base_log_folder: AIRFLOW__LOGGING__BASE_LOG_FOLDER
- wasb_container: AIRFLOW_LOGGING__REMOTE_WASB_LOG_CONTAINER
```
So we would compare the value in `name` with the value provided in
AIRFLOW__LOGGING__REMOTE_LOGGER and then load the corresponding dict and update
the `DEFAULT_LOGGING_CONFIG`
If we can dynamically update the config
[doc](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging)
to only contain the possible values of the supported remote loggers, then i
guess we are good to go.
Let me know if above sounds fine or any changes you suggest?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
