killua1zoldyck commented on issue #31105:
URL: https://github.com/apache/airflow/issues/31105#issuecomment-1627826299
Hey @getaaron , I have implemented the k-way-merge and it works fine. For
now, I have specified the limit as 50 lines per call and it keeps calling till
the end_of_log is reached. The metadata that gets sent back-and-forth looks
something like this
json
{
"metadata": {
"end_of_log": false,
"executor_logs": {
"log_pos": 0
},
"immediate_tail": true,
"local_logs": {
"/root/airflow/logs/dag_id=log_generator_dag/run_id=manual__2023-07-09T19:52:37.154177+00:00/task_id=generate_logs_5mb/attempt=1.log":
{
"done": false,
"lines": 200,
"need": 50,
"prev_position": 14269
}
},
"remote_logs": {
"log_pos": 0
},
"served_logs": {
"log_pos": 0
}
}
}
Where previously it looked something like this
```json
{
"metadata": {
"end_of_log": false,
"log_pos: 100,
}
}
I thought this would only affect the Airflow UI piece that calls the
/get_logs_with_metadata. However, I just realized that it is used in
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_log.
Should we be creating a separate flow for auto streaming logs from the UI and
have the same for the rest api?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
