jason810496 opened a new issue, #70270:
URL: https://github.com/apache/airflow/issues/70270
Part of #70265 (related: #67056).
## Why
#67056 decoupled remote logging from the hardcoded branches in
`airflow_local_settings.py`: core and
the Task SDK now resolve the handler via `ProvidersManager` dispatch on the
`[logging] remote_base_log_folder` URL scheme, instantiating the provider
class through a no-arg
`from_config()` classmethod. This issue migrates the `hdfs` scheme.
## What
- [ ] Add `HdfsRemoteLogIO.from_config()` in
`providers/apache/hdfs/src/airflow/providers/apache/hdfs/log/hdfs_task_handler.py`,
mirroring
the legacy branch:
https://github.com/apache/airflow/blob/104ad12e190db7197fa303d98a3ab68879eafd33/airflow-core/src/airflow/config_templates/airflow_local_settings.py#L287-L302
— including the `[logging] remote_task_handler_kwargs` IO-kwargs merge
and `expanduser` on
`base_log_folder`, so behavior is unchanged for existing configs.
- [ ] Register the `hdfs` scheme under a `remote-logging:` section in
`providers/apache/hdfs/provider.yaml` and mirror it in
`providers/apache/hdfs/src/airflow/providers/apache/hdfs/get_provider_info.py`.
- [ ] Add tests mirroring `TestS3RemoteLogIOFromConfig` in
`providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`.
- [ ] Verify end to end with a real system test: set up the actual backend
yourself (a real
service or a local equivalent), run a task with remote logging
enabled, and confirm logs are
uploaded and read back through the new dispatch path. Include the
setup and verification
results in the PR description.
## Notes
The legacy branch passes `urlsplit(remote_base_log_folder).path` as
`remote_base` — the
`hdfs://host` part is stripped and only the path is kept. Mirror this in
`from_config`.
## Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If
`from_config` raises on a bad
config, the shared factory falls back to the legacy path, so this is not a
breaking change.
The legacy branch derives a default connection id from
`WebHDFSHook.default_conn_name`; handle the
connection id the same way the merged amazon PRs do (the hook reads
`[logging] remote_log_conn_id` itself).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
