Marton Elek created HDDS-4459:
---------------------------------
Summary: Improve closed container replication
Key: HDDS-4459
URL: https://issues.apache.org/jira/browse/HDDS-4459
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Reporter: Marton Elek
Assignee: Marton Elek
Closed are immutable therefore the replication is different from Ratis.
Closed containers are downloaded from other datanodes over a specific GRPC
channel.
While the first implementation works as intended it has many spaces for
improvements.
1. the container replica is compressed with gzip. A more efficient compression
codec like zstd might be better (reported Wei-Chiu)
2. Current downloader uses only one thread (reported Wei-Chiu)
3. Temporary space is configurable, but by default uses /tmp. This means we
are bottlenecked by the root disk, even though we have 48 disks on each DN.
There should be an option to store the temp file under the destination disk
instead. (reported Wei-Chiu)
4. Buffer size is hard coded (1MB) and may not be the ideal
5. There are no metrics which clearly shows the performance of the replication
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
