yangzhanling created HDFS-16416:
-----------------------------------
Summary: Namenode was unavailable when high concurrency
Key: HDFS-16416
URL: https://issues.apache.org/jira/browse/HDFS-16416
Project: Hadoop HDFS
Issue Type: Improvement
Components: namanode
Affects Versions: 2.7.7
Environment:
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager#outputBufferCapacity
private int outputBufferCapacity = 512 * 1024;
so we can update sample:
//first add key and value in DFSConfigKeys
//org.apache.hadoop.hdfs.DFSConfigKeys
//hdfs-site.xml
public static final String DFS_NAMENODE_BUFFER_KEY =
"dfs.namenode.buffer.key";
public static final int DFS_NAMENODE_BUFFER_VALUE_DEFAULT = 512 * 1024;
//second,add param from configurature
//org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
this.outputBufferCapacity = conf.getInt(
DFSConfigKeys.DFS_NAMENODE_BUFFER_KEY,
DFSConfigKeys.DFS_NAMENODE_BUFFER_VALUE_DEFAULT
);
//third ,add config in hdfs-site.xml
<property>
<name>dfs.namenode.buffer.key"</name>
<value>1024</value>
</property>
Reporter: yangzhanling
Namenode was unavailable when high concurrency when meta data wrote to
DoubleBuffer,and the currentBuff was full (512k) ,the other cursyncBuffer was
happend sync to dist or journalnode,so we request to namenode was unavailable.
Because the currentBuffer size is 512k,it is smaller in some bigger request
cluster.but it is not updated by configuring with *hdfs-site.xml*.it is
hardcoding in Java source code and unreasonable.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
