[
https://issues.apache.org/jira/browse/HDDS-4459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Elek updated HDDS-4459:
------------------------------
Description:
Closed are immutable therefore the replication is different from Ratis.
Closed containers are downloaded from other datanodes over a specific GRPC
channel.
While the first implementation works as intended it has many spaces for
improvements.
1. the container replica is compressed with gzip. A more efficient compression
codec like zstd might be better (reported Wei-Chiu)
2. Current downloader uses only one thread (reported Wei-Chiu)
3. Temporary space is configurable, but by default uses /tmp. This means we
are bottlenecked by the root disk, even though we have 48 disks on each DN.
There should be an option to store the temp file under the destination disk
instead. (reported Wei-Chiu)
4. Buffer size is hard coded (1MB) and may not be the ideal
5. There are no metrics which clearly shows the performance of the replication
6. Separate client and server2server RPC listeners
was:
Closed are immutable therefore the replication is different from Ratis.
Closed containers are downloaded from other datanodes over a specific GRPC
channel.
While the first implementation works as intended it has many spaces for
improvements.
1. the container replica is compressed with gzip. A more efficient compression
codec like zstd might be better (reported Wei-Chiu)
2. Current downloader uses only one thread (reported Wei-Chiu)
3. Temporary space is configurable, but by default uses /tmp. This means we
are bottlenecked by the root disk, even though we have 48 disks on each DN.
There should be an option to store the temp file under the destination disk
instead. (reported Wei-Chiu)
4. Buffer size is hard coded (1MB) and may not be the ideal
5. There are no metrics which clearly shows the performance of the replication
> Improve closed container replication
> ------------------------------------
>
> Key: HDDS-4459
> URL: https://issues.apache.org/jira/browse/HDDS-4459
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Marton Elek
> Assignee: Marton Elek
> Priority: Major
>
> Closed are immutable therefore the replication is different from Ratis.
> Closed containers are downloaded from other datanodes over a specific GRPC
> channel.
> While the first implementation works as intended it has many spaces for
> improvements.
> 1. the container replica is compressed with gzip. A more efficient
> compression codec like zstd might be better (reported Wei-Chiu)
> 2. Current downloader uses only one thread (reported Wei-Chiu)
> 3. Temporary space is configurable, but by default uses /tmp. This means we
> are bottlenecked by the root disk, even though we have 48 disks on each DN.
> There should be an option to store the temp file under the destination disk
> instead. (reported Wei-Chiu)
> 4. Buffer size is hard coded (1MB) and may not be the ideal
> 5. There are no metrics which clearly shows the performance of the
> replication
> 6. Separate client and server2server RPC listeners
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
