jason810496 opened a new issue, #70271:
URL: https://github.com/apache/airflow/issues/70271
Part of #70265 (related: #67056).
## Why
#67056 decoupled remote logging from the hardcoded branches in
`airflow_local_settings.py`: core and
the Task SDK now resolve the handler via `ProvidersManager` dispatch on the
`[logging] remote_base_log_folder` URL scheme, instantiating the provider
class through a no-arg
`from_config()` classmethod. This issue migrates the Elasticsearch backend.
## Design note (feedback welcome)
Unlike the object-storage backends, Elasticsearch is currently selected by
`[elasticsearch] host`
being set — `remote_base_log_folder` typically has no scheme for these
deployments, so pure scheme
dispatch cannot reach it. The proposal:
- Register an `elasticsearch` scheme and document setting
`[logging] remote_base_log_folder = elasticsearch://` as the
forward-looking configuration.
- Keep the legacy host-based selection working via the transitional fallback
in
`airflow_local_settings.py` until the chain is removed (tracked by the
meta issue), so existing
configs are unaffected.
## What
- [ ] Add `ElasticsearchRemoteLogIO.from_config()` in
`providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py`,
mirroring the legacy branch:
https://github.com/apache/airflow/blob/104ad12e190db7197fa303d98a3ab68879eafd33/airflow-core/src/airflow/config_templates/airflow_local_settings.py#L304-L347
— reading the `[elasticsearch]` options the branch reads (`host`,
`target_index`,
`write_stdout`, `write_to_es`, `json_format`, `log_id_template`,
`host_field`,
`offset_field`) plus `[logging] base_log_folder` (with `expanduser`)
and
`[logging] delete_local_logs`, so behavior is unchanged for existing
configs. Note the legacy
branch does *not* merge `remote_task_handler_kwargs` IO-kwargs for
this backend — mirror that
too.
- [ ] Register the `elasticsearch` scheme under a `remote-logging:` section
in
`providers/elasticsearch/provider.yaml` and mirror it in
`providers/elasticsearch/src/airflow/providers/elasticsearch/get_provider_info.py`.
- [ ] Document the `remote_base_log_folder = elasticsearch://` form in the
provider's logging docs.
- [ ] Add tests mirroring `TestS3RemoteLogIOFromConfig` in
`providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py`.
- [ ] Verify end to end with a real system test: set up the actual backend
yourself (a real
Elasticsearch cluster or a local one, e.g. via Docker), run a task
with remote logging
enabled, and confirm logs are written and read back through the new
dispatch path. Include
the setup and verification results in the PR description.
## Notes
The legacy branch also swaps the `task` handler class inside
`DEFAULT_LOGGING_CONFIG`; that dict is
kept only for import compatibility (see the comment at its definition in
`airflow_local_settings.py`), so `from_config` should not need to replicate
it — but please verify
task-log reads work end to end with the new dispatch.
## Reference
Merged examples to follow: #69817 (s3), #69816 (cloudwatch). If
`from_config` raises on a bad
config, the shared factory falls back to the legacy path, so this is not a
breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
