danny0405 commented on PR #7627: URL: https://github.com/apache/hudi/pull/7627#issuecomment-1552739597
> We can implement this by adding a extra flag in some way or just simply comparing the file modified time of the last archived log. Also, to accelerate filtering out instants with greater state transient time in archive timeline we can also compare the modification time of the archived log. You are right, we need some facilities to speed up the query for transition time -> completion time mappings, in Hudi release 1.0, we are planning to remove the archival timeline and there is only one timeline which is always active, so in my opition, fixing the hive sync by using the completion time on the active timeline can solve most of the multi-writer use cases, if we fond any instants are archived, we should fallback to a full sync. And we already support such fallback in PR: https://github.com/apache/hudi/pull/8388 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
