[
https://issues.apache.org/jira/browse/RATIS-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691685#comment-17691685
]
Xinyu Tan commented on RATIS-1787:
----------------------------------
[~szetszwo] Thank you for your continued support of the IoTDB community. What
are your thoughts on this issue? If our requirements are not common, can we
work out a configuration so that more performance-oriented projects don't have
to store these md5 files.
> Don't generate md5 file for each file when a file is received during
> InstallSnapshot to reduce the pressure on the file system
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: RATIS-1787
> URL: https://issues.apache.org/jira/browse/RATIS-1787
> Project: Ratis
> Issue Type: Improvement
> Reporter: Xinyu Tan
> Priority: Major
>
> The IoTDB community has experienced a number of performance issues with MD5
> calculations when using the Snapshot feature.
> Originally, MD5 calculation is to prevent errors in file transmission.
> However, in the previous implementation, when we generate a Snapshot, we need
> to load all files to calculate the snapshot, even though we do not need to
> send snapshot at this time, which causes a lot of resource preemption in the
> background. [~William Song] in RATIS-1597 has combined client-side MD5
> calculations with streaming, thus avoiding a lot of background IO and
> computing tasks. However, at the Snapshot receiver, one MD5 file is still
> currently stored for each file. At present, the snapshot file level of IoTDB
> may be tens of thousands. These tens of thousands of small md5 files cause a
> lot of pressure on the file system. Is it possible that we do not store an
> md5 file for each file when receiving?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
