AutomationDev85 opened a new issue, #58770:
URL: https://github.com/apache/airflow/issues/58770
### Apache Airflow version
3.1.3
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
We encountered a breaking change when upgrading from Airflow 2.11.0 to 3.1.3
related to remote_task_handler_kwargs in airflow_local_settings.py.
Issue:
In Airflow 2.x, remote_task_handler_kwargs configured the FileTaskHandler
(or its subclasses like WasbTaskHandler). We successfully used this to set
max_bytes for log file size limiting:
remote_task_handler_kwargs = {"max_bytes": 5000000, "backup_count": 5}
After upgrading to Airflow 3.x, this configuration raises:
TypeError: WasbRemoteLogIO.__init__() got an unexpected keyword argument
'max_bytes'
### What you think should happen instead?
Root cause:
PR https://github.com/apache/airflow/pull/48491 introduced separate
RemoteLogIO classes for remote storage operations. The current implementation
in airflow_local_settings.py now passes remote_task_handler_kwargs to
WasbRemoteLogIO.__init__(), which only accepts base_log_folder, remote_base,
delete_local_copy, and wasb_container.
Handler-level parameters like max_bytes and backup_count should go to
FileTaskHandler, but are instead being consumed by the I/O layer, which doesn't
support them.
Misleading naming:
The variable name remote_task_handler_kwargs suggests these parameters
configure the handler, but the current code path applies them to the remote I/O
abstraction.
Question:
How should max_bytes and similar FileTaskHandler parameters be configured in
Airflow 3.x with the new RemoteLogIO architecture? Should there be separate
config keys for I/O-layer vs handler-layer parameters, or should the current
implementation be fixed to properly separate them?
Expected behavior:
Parameters like max_bytes, backup_count and delay should configure the
FileTaskHandler instance, while WasbRemoteLogIO receives only its own relevant
parameters.
### How to reproduce
Configure remote logging e.g. using WasbFileTaskHandler to upload logs to a
blob stroage and add:
[logging]
remote_logging = True
remote_base_log_folder =
wasb-<container>://<account>.blob.core.windows.net/<path>
remote_task_handler_kwargs = {"max_bytes": 5000000, "backup_count": 5}
and start application. And check for TypeError during init of the logging
system:
TypeError: WasbRemoteLogIO.__init__() got an unexpected keyword argument
'max_bytes'
### Operating System
Debian GNU/Linux 12
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
