jason810496 opened a new issue, #70272:
URL: https://github.com/apache/airflow/issues/70272
Part of #70265 (related: #67056).
## Why
#67056 decoupled remote logging from the hardcoded branches in
`airflow_local_settings.py`: core and
the Task SDK now resolve the handler via `ProvidersManager` dispatch on the
`[logging] remote_base_log_folder` URL scheme, instantiating the provider
class through a no-arg
`from_config()` classmethod. This issue migrates the OpenSearch backend.
## Design note (feedback welcome)
Unlike the object-storage backends, OpenSearch is currently selected by
`[opensearch] host` being
set — `remote_base_log_folder` typically has no scheme for these
deployments, so pure scheme
dispatch cannot reach it. The proposal (same as the sibling Elasticsearch
issue #70271):
- Register an `opensearch` scheme and document setting
`[logging] remote_base_log_folder = opensearch://` as the forward-looking
configuration.
- Keep the legacy host-based selection working via the transitional fallback
in
`airflow_local_settings.py` until the chain is removed (tracked by the
meta issue), so existing
configs are unaffected.
## What
- [ ] Add `OpensearchRemoteLogIO.from_config()` in
`providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py`,
mirroring the
legacy branch:
https://github.com/apache/airflow/blob/104ad12e190db7197fa303d98a3ab68879eafd33/airflow-core/src/airflow/config_templates/airflow_local_settings.py#L349-L379
— reading the `[opensearch]` options the branch reads (`host`, `port`,
`username`,
`password`, `target_index`, `write_stdout`, `write_to_os`,
`json_format`, `host_field`,
`offset_field`, and `log_id_template` with its
`{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}` fallback) plus
`[logging] base_log_folder` (with `expanduser`) and `[logging]
delete_local_logs`, so
behavior is unchanged for existing configs. Note the legacy branch
does *not* merge
`remote_task_handler_kwargs` IO-kwargs for this backend — mirror that
too.
- [ ] Register the `opensearch` scheme under a `remote-logging:` section in
`providers/opensearch/provider.yaml` and mirror it in
`providers/opensearch/src/airflow/providers/opensearch/get_provider_info.py`.
- [ ] Document the `remote_base_log_folder = opensearch://` form in the
provider's logging docs.
- [ ] Add tests mirroring `TestS3RemoteLogIOFromConfig` in
`providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`.
- [ ] Verify end to end with a real system test: set up the actual backend
yourself (a real
OpenSearch cluster or a local one, e.g. via Docker), run a task with
remote logging enabled,
and confirm logs are written and read back through the new dispatch
path. Include the setup
and verification results in the PR description.
## Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If
`from_config` raises on a bad
config, the shared factory falls back to the legacy path, so this is not a
breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
