devgonvarun commented on issue #53926:
URL: https://github.com/apache/airflow/issues/53926#issuecomment-3149749676
Hi @eladkal and @Owen-CH-Leung did you get a chance to look into this?
Since the OpensearchTaskHandler doesn't yet support write_to_es like the
ElasticsearchTaskHandler, I have to use a log shipper (e.g., Fluentd) to send
task logs to OpenSearch. I extract the worker pod logs (kubernetes executor in
my case) and construct a log_id from the Kubernetes pod labels. This log_id is
used by the OpensearchTaskHandler to find hits in OpenSearch and fetch the
correct corresponding logs for display in the airflow UI. Presently the log_id
uses logical_date but the Kubernetes pod labels don't include the logical_date
anymore, hence this issue. Fortunately, both run_id and map_index are available
in the kubernetes pod labels, so if the log_id in OpensearchTaskHandler works
as expected using the default log_id_template
{dag_id}-{task_id}-{run_id}-{map_index}-{try_number} then everything should
work fine. Once the log_id is correctly constructed to match the format
expected by the OpensearchTaskHandler, the log retrieval should work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
