rickchengx opened a new issue, #13017: URL: https://github.com/apache/dolphinscheduler/issues/13017
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement. ### Description # Why remote logging? * Avoid task log loss after worker is torn down * Easier to obtain logs and troubleshoot after logs are aggregated in remote storage * Enhanced cloud-native support for DS # Feature Design ## Connect to different remote targets DS can support a variety of common remote storage, and has strong scalability to support other types of remote storage * S3 * OSS * ElasticSearch * Azure Blob Storage * Google Cloud Storage * ... ## When to write logs to remote storage Like airflow, DS writes the task logs to remote storage after the task completes (success or fail). ## How to read logs Since the task log is stored in both the worker's local and remote storage, when the `api-server` needs to read the log of a certain task instance, it needs to determine the reading strategy. Airflow first tries to read the logs stored remotely, and if it fails, reads the local logs. But I prefer to try to read the local log first, and then read the remote log if the local log file does not exist. We could discuss this further. ## Log retention strategy For example, the maximum capacity of remote storage can be set, and old logs can be deleted by rolling. # Sub-tasks WIP # Ref * [Logging and Monitoring architecture in Airflow](https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-architecture.html) * [Remote logging in Airflow](https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/logging.html) Any comments or suggestions are welcome. ### Use case Discussed above. ### Related issues * #8543 * This issue is a sub-task of #12436 ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
