ruanwenjun commented on issue #15938: URL: https://github.com/apache/dolphinscheduler/issues/15938#issuecomment-2082789170
I am very -1 to this DSIP. In standalone mode, users can directly use the local file system. In cluster mode, users can use distribution file system or some shared file system like NFS, the distribution file system is very commonly, this is not an expensive technology. Back to the implementation, do you understand the complexity of implementing a file system? The only thing I can think of as an implemenration is we broadcast the file to all workers, but this means the cluster will be hard to scaled. In additional, DS is not must rely on a distribution file system, this is different with spark/flink which is stronge rely on a file system to store the checkpoint data, these systems still don't have plan to implement a file system. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
