killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1627823172
Hey @getaaron , I have implemented the k-way-merge and it works fine. For
now, I have specified the limit as 50 lines per call and it keeps calling till
the end_of_log is reached. The metadata that gets sent back-and-forth looks
something like this
```json
{
"metadata": {
"end_of_log": false,
"executor_logs": {
"log_pos": 0
},
"immediate_tail": true,
"local_logs": {
"/root/airflow/logs/dag_id=log_generator_dag/run_id=manual__2023-07-09T19:52:37.154177+00:00/task_id=generate_logs_5mb/attempt=1.log":
{
"done": false,
"lines": 200,
"need": 50,
"prev_position": 14269
}
},
"remote_logs": {
"log_pos": 0
},
"served_logs": {
"log_pos": 0
}
}
}
```
However, I just realized that it is used in
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_log.
Should we be creating a separate flow for auto streaming logs then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
