jason810496 opened a new issue, #70265:
URL: https://github.com/apache/airflow/issues/70265
- related: #67056
## Why
#67056 decoupled remote logging from the hardcoded if/elif chain in
`airflow-core/src/airflow/config_templates/airflow_local_settings.py`. Core
and the Task SDK now
share one resolver
(`airflow_shared.logging.factory.resolve_remote_task_log`) that dispatches on
the
URL scheme of `[logging] remote_base_log_folder` via `ProvidersManager` and
instantiates the
provider class through a no-arg `from_config()` classmethod.
The legacy per-scheme chain in `airflow_local_settings.py` is transitional:
it stays as a fallback
until every remote logging backend has been migrated to the new mechanism.
This issue tracks that
migration, one sub-issue per backend.
## The pattern
Each migration is a small, self-contained PR with three parts (see the
merged amazon examples:
s3 — #69817, cloudwatch — #69816):
1. **Add `from_config()`** — a no-arg classmethod on the provider's
`RemoteLogIO` class that mirrors
the corresponding legacy branch in `airflow_local_settings.py`, including
the
`[logging] remote_task_handler_kwargs` IO-kwargs merge (where the legacy
branch applies it) and
`expanduser` on `base_log_folder`, so behavior is unchanged for existing
configs.
2. **Register the scheme** — add a `remote-logging:` section to the
provider's `provider.yaml` and
mirror it in `get_provider_info.py`:
```yaml
remote-logging:
- classpath:
airflow.providers.amazon.aws.log.s3_task_handler.S3RemoteLogIO
scheme: s3
```
3. **Tests** — mirror `TestS3RemoteLogIOFromConfig` in
`providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`:
`from_config` field mapping,
IO-kwargs filtering, scheme registration, and end-to-end
`resolve_remote_task_log` dispatch.
4. **Real system test** — set up the actual backend yourself (a real service
or a local
equivalent), run a task with remote logging enabled, and verify logs are
uploaded and read back
through the new dispatch path. Include the setup and verification results
in the PR description.
If `from_config()` raises on a bad config, the shared factory falls back to
the legacy path — so
these migrations are non-breaking by construction.
## Backends
- [x] `s3` (amazon) — #69817
- [x] `cloudwatch` (amazon) — #69816
- [ ] `gs` (google) — #SUB_GS
- [ ] `stackdriver` (google) — #SUB_STACKDRIVER
- [ ] `wasb` (microsoft.azure) — #SUB_WASB
- [ ] `oss` (alibaba) — #SUB_OSS
- [ ] `hdfs` (apache.hdfs) — #SUB_HDFS
- [ ] `elasticsearch` — #SUB_ES
- [ ] `opensearch` — #SUB_OS
## Follow-up once backends are migrated
- [ ] Once all provider migrations are merged **and released**, add a
deprecation warning to the
legacy branches in `airflow_local_settings.py`, gated on the provider
versions that ship
`from_config` (step 2 of the plan in #67056).
- [ ] Remove the hardcoded remote logging config from core in a future minor
or major release,
whichever the community prefers.
## How to contribute
Comment on a sub-issue to claim it — one PR per scheme, following the merged
amazon examples.
The `elasticsearch` and `opensearch` sub-issues include a small design
decision (those backends are
currently selected by host config rather than URL scheme); design feedback
is welcome there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
