danny0405 commented on code in PR #9291: URL: https://github.com/apache/hudi/pull/9291#discussion_r1275646230
########## website/docs/timeline.md: ########## @@ -39,4 +44,36 @@ organization reflects the actual time or `event time`, the data was intended for When there is late arriving data (data intended for 9:00 arriving >1 hr late at 10:20), we can see the upsert producing new data into even older time buckets/folders. With the help of the timeline, an incremental query attempting to get all new data that was committed successfully since 10:00 hours, is able to very efficiently consume -only the changed files without say scanning all the time buckets > 07:00. \ No newline at end of file +only the changed files without say scanning all the time buckets > 07:00. + +### Active and Archived timeline +Hudi divides the entire timeline into active and archived timeline. As the name suggests active timeline is consulted all +the time to serve metadata on valid data files and to ensure reads on the timeline does not incur unnecessary latencies +as timeline grows, the active timeline needs to be bounded on the metadata (timeline instants) it can serve. To ensure this, +after certain thresholds the archival kicks in to move older timeline events to the archived timeline. In general, archival +timeline is never contacted for regular operations of the table and is merely used for book-keeping and debugging purposes. Review Comment: We are rebuilding the archived timeline stuff, it could served for more frequent queries by the writer/readers. Should be fine we put it like this in current branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
