danny0405 commented on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-838074201
> @danny0405 if your concern is memory usage alone and not the time taken to spin up the timeline server for each write task, then worth exploring if you want to use `EMBEDDED_KV_STORE` or `SPILLABLE_DISK` FileSystemViewStorageType instead. That will reduce memory pressure. > > If you truly want the server to run with each TaskManager, I think we can do what you are trying to do here, but lets add a flag, that controls the reuse behavior and turn it on only for Flink path? I do think this will also involve testing of the incremental sync part more rigorously. Without the incremental sync, I think we end up calling AbstractTableFileSystemView#runSync(), which will reset and reinit file listings anyway, by refreshing the timeline (causes more listings to .hoodie/ each call to timeline server to check if local view is behind, see RequestHandler#syncIfLocalViewBehind) > > Hope that helps! Thanks so much for the feedback, add the config option is reasonable for me, adding the timeline service to `JobManager` is not a good choice because in Flink, it is hard to do RPC between `TaskManager` and `JobManager`, the logic in `JobManager` should be lightweight and clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
