[jira] [Commented] (HDFS-10612) Optimize mechanism when block report size exceed the limit of PB message
[ https://issues.apache.org/jira/browse/HDFS-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376229#comment-15376229 ] Yuanbo Liu commented on HDFS-10612: --- Block report size is difficult to calculate, for the block number and block report size are not in linear relationship if datanode use blockbuffer. And it will bring performance loss to calculate report size when we write block. The least we can do is to add warn log, make block report size as a metric and add this metric to datanode web ui. > Optimize mechanism when block report size exceed the limit of PB message > > > Key: HDFS-10612 > URL: https://issues.apache.org/jira/browse/HDFS-10612 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yuanbo Liu > > Community has made block report size configurable in HDFS-10312. But there is > still a risk for Hadoop. If block report size exceeds PB message size, the > cluster may be in a danger situation -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10612) Optimize mechanism when block report size exceed the limit of PB message
[
https://issues.apache.org/jira/browse/HDFS-10612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372410#comment-15372410
]
Yuanbo Liu commented on HDFS-10612:
---
Here are my proposals:
1. Stop writing blocks when block report size almost reach the limit of PB
message size
2. Add warning log of block report size
We can modify the implements of {{VolumeChoosingPolicy}}, and stop datanode
choosing the volume whose block report size almost reach the limit of PB
message size.
> Optimize mechanism when block report size exceed the limit of PB message
>
>
> Key: HDFS-10612
> URL: https://issues.apache.org/jira/browse/HDFS-10612
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>
> Community has made block report size configurable in HDFS-10312. But there is
> still a risk for Hadoop. If block report size exceeds PB message size, the
> cluster may be in a danger situation
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
