seanghaeli opened a new issue, #77056:
URL: https://github.com/apache/airflow/issues/77056
These remote log handlers find logs by task instance, which an asset watcher
doesn't have. `ElasticsearchRemoteLogIO` and `OpensearchRemoteLogIO` return
early from both `upload()` and `read()` when `ti` is `None`, so a watcher's log
files (#76527) are never stored. `CloudLoggingRemoteLogIO` does send each
watcher line as it's written, labelled with its `log_path`, but its `read()`
returns nothing without a `ti`, so the lines can't be read back.
With #76527 and #76560 (draft), `GET /api/v2/triggers/{trigger_id}/logs` on
these handlers shows only a watcher's current file, read live from the
triggerer running it, and none of its history.
Suggestion: in the elasticsearch, opensearch and google providers, store a
watcher's lines under its trigger id when there's no task instance (every
watcher line carries `trigger_id`), and have `read()` find them by trigger id
when it's asked for a trigger log path without a `ti`. One PR per provider is
welcome.
Done when: for each of the three handlers, a watcher's stored lines can be
read back through `GET /api/v2/triggers/{trigger_id}/logs` after it has left
its triggerer, with tests.
This needs #76527 and #76560 to merge first. I'd recommend building on top
of #76527 and #76560 and rebasing once they're merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
