[
https://issues.apache.org/jira/browse/IOTDB-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17211594#comment-17211594
]
Houliang Qi commented on IOTDB-918:
-----------------------------------
Good idea, this will reduce the time taken by the snapshot and speed up the
snapshot task.
I think we can add the commit log index to the tsFileResource when close one
TsFile, and when a follower request a snapshot, the leader needs to collects
the TsFiles that need to be sent to the follower, with the commit log index in
the TsFile resource, it's easy to filter the TsFile that do not need to be
collected.
BTW, according to my understanding, the commit log index can only indicate that
the data in the TsFile will not exceed the commit log index, but it cannot
guarantee that the data smaller than the commit log index must exist in the
TsFile. Is that the truth?
> [Distributed] Bind TsFiles with log index to accelerate snapshots
> -----------------------------------------------------------------
>
> Key: IOTDB-918
> URL: https://issues.apache.org/jira/browse/IOTDB-918
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/Cluster
> Reporter: Tian Jiang
> Priority: Major
> Labels: snapshot
>
> During snapshot catch-up, the leader will collect information of all TsFiles
> as RemoteTsFileResources, and send them to the followers. The follower will
> check if it has the file and decide whether to pull it from the leader.
> To be smarter, the leader may be able to know that a TsFile of a certain
> version only contains data precedes to a certain log index, and the leader
> can avoid sending files that would be surely on the receiver side.
> For example, when a TsFile `3.tsfile` is closed and the current committed log
> index is 100, the leader knows for sure that the file will not contain data
> corresponding to logs whose indexes are more than 100. When a follower whose
> committed log index exceeds 100 applies for a snapshot, the leader can skip
> `3.tsfile` since the follower must have it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
