danny0405 edited a comment on pull request #2899: URL: https://github.com/apache/hudi/pull/2899#issuecomment-837659380
> > each write task write out these records buffer (grouping by file group id) using a write client there. > > Understood. So each task manager is, sending RPC calls to the timeline server running locally? This is very different from Spark. What's the life cycle of the `write client` in each JVM process? i.e how often do we re-initialize it? (it can't be per record, so wondering) > > I get why you want a singleton now. Let's work together to figure something out. The life cycle of the flink `write client` is: it is living until the job ends, we start a write client for each task and we use that client always. In my thoughts, if i request the embedded timeline server with a new instant time, the server should sync itself first, it is more easy for server to keep the freshness than client(i.e each client should figure out how and when to re-initialize the fs view server). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
