stream2000 commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1833238919
For 1.0.0 and later hudi version which supports efficient completion time queries on the timeline(#9565), we can get partition's `lastModifiedTime` by scanning the timeline and get the last write commit for the partition. Also for efficiency, we can store the partitions' last modified time and current completion time in the replace commit metadata. The next time we need to calculate the partitions' last modified time, we can build incrementally from the replace commit metadata of the last ttl management. @danny0405 Added new `lastModifiedTime` calculation method for 1.0.0 and later hudi version. We plan to implement the file listing based `lastModifiedTime` at first and implement the timeline-based `lastModifiedTime` calculation in a separate PR. This will help users with earlier hudi versions easy to pick the function to their code base. I have addressed all comments according to online/offline discussions. If there is no other concern, we can move on this~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
