Xinyu Tan created RATIS-1787:
--------------------------------
Summary: Don't generate md5 file for each file when a file is
received during InstallSnapshot to reduce the pressure on the file system
Key: RATIS-1787
URL: https://issues.apache.org/jira/browse/RATIS-1787
Project: Ratis
Issue Type: Improvement
Reporter: Xinyu Tan
The IoTDB community has experienced a number of performance issues with MD5
calculations when using the Snapshot feature.
Originally, MD5 calculation is to prevent errors in file transmission. However,
in the previous implementation, when we generate a Snapshot, we need to load
all files to calculate the snapshot, even though we do not need to send
snapshot at this time, which causes a lot of resource preemption in the
background. [~William Song] in RATIS-1597 has combined client-side MD5
calculations with streaming, thus avoiding a lot of background IO and computing
tasks. However, at the Snapshot receiver, one MD5 file is still currently
stored for each file. At present, the snapshot file level of IoTDB may be tens
of thousands. These tens of thousands of small md5 files cause a lot of
pressure on the file system. Is it possible that we do not store an md5 file
for each file when receiving?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
