rickchengx commented on issue #13017: URL: https://github.com/apache/dolphinscheduler/issues/13017#issuecomment-1341891161
Here are some discussions on the weekly meeting **Q1: In the k8s environment, users can choose to mount persistent volumes (E.g., [OSS](https://help.aliyun.com/document_detail/130911.html)) to synchronize task logs to remote storage.** R1: This is indeed a way to synchronize logs to remote storage, and only need to mount persistent volumes (PV). But there are still some shortcomings as below: * **Efficiency**: Since the PV is connected to the remote storage, the speed of log writing will be reduced, which will further affect the task execution of the worker. On the contrary, uploading the task log to the remote storage asynchronously through the remote logging mechanism will not affect the execution of the task. * **Generality**: PV is not suitable for some remote storage, such as elasticsearch. And also it is not applicable to DS deployed in non-k8s environment. **Q2: Users can configure whether to use remote storage for task logs** R2: Yes, users can decide whether to enable log remote storage through configuration, and specify the corresponding configuration of remote logging. **Q3: The master-server also has task logs, which need to be uploaded to remote storage in a unified manner.** R3: Yes, users can set the master's task log related remote storage configuration in Master's configuration. **Q4: Is it possible to set the task log retention policy through the configuration supported by the remote storage itself?** R4: This is a good idea and it can simplify the design of remote logging, I'll look into it. Thanks again for all the suggestions at the weekly meeting, please correct me if I'm wrong. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
