keon94 opened a new issue, #2693: URL: https://github.com/apache/incubator-devlake/issues/2693
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-devlake/issues?q=is%3Aissue) and found no similar feature requirement. ### Description With #2578 it is assumed that all the logs exist on the main devlake service. If we have Temporal running, then we are unable to get logs from the worker instances. We need to be able to fetch those logs as well, and put them all in the same archive. ### Use case Have the download logs API return the logs from all workers that participated in the pipeline execution. Possible solutions: 1. Have the main Devlake service track the workers involved, and upon their completion, query them for their logs. This becomes challenging if we try to build resiliency against a worker dying. 2. Have the workers periodically publish their logs back to the main service (use a message queue on the main service to listen for these events and append to the log files) - I prefer this one, and it's also safe against dying workers. ### Related issues #2578 ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
