- Fair vs Non-Fair locking for the HDFS Namenode. IIUC this seems to be a code change rather than a jvm setting tunable, but I am wondering if others have experience with different locking mechanisms in production for HDFS.
It looks like talking about java.util.concurrent.locks.ReentrantReadWriteLock used by NameNode. * https://issues.apache.org/jira/browse/HDFS-5239 * https://github.com/apache/hadoop/blob/rel/release-3.2.2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java#L121-L124 "fair" is the default value in HDFS. I guess it is used in many clusters. On 2021/06/11 15:52, Luca Toscano wrote:
Hi everybody, https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr is a nice blog post reading. There are some interesting follow ups in my opinion: - Fair vs Non-Fair locking for the HDFS Namenode. IIUC this seems to be a code change rather than a jvm setting tunable, but I am wondering if others have experience with different locking mechanisms in production for HDFS. - Observer HDFS Namenode. IIUC this was introduced in Hadoop 2.10, it would be nice if we could offer it via puppet for the docker provisioner (if we don't already do it, I didn't find it). Having a separate Namenode to handle read requests could be interesting for busy clusters. Has anybody already deployed it? Thanks in advance, Luca
