hudi-bot opened a new issue, #14996: URL: https://github.com/apache/hudi/issues/14996
Recently, call `HoodieTableMetaClient.reloadActiveTimeline` many times in one operation, and this will reload the timeline fully. Perhaps, to support to reload in Incremental mode will increase the performance. ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-3232 - Type: Improvement --- ## Comments 23/Jan/22 19:40;shivnarayan;this definitely looks beneficial. may I know how you plan to get this done. Do you plan to add some filtering in below method in HoodieTableMetaClient. {code:java} public List<HoodieInstant> scanHoodieInstantsFromFileSystem(Path timelinePath, Set<String> includedExtensions, boolean applyLayoutVersionFilters) throws IOException { {code} something like, HTMC (HoodieTableMetaClient) will maintain last loaded instant or mod time of last loaded instant file. So, every time when reload is invoked, we just filter for files modified after the last known mod time. Or you had something else in mind ? ;;; --- 25/Jan/22 07:38;[email protected];[~shivnarayan] i plan to do something in HoodieActiveTimeline. all the methods in HoodieActiveTimeline will judge whether need to load the latest instant. If yes, load incrementally and execute the following codes. Also, all the methods that will create a new instant will force to load after the operation. WDYT?;;; -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
