Summary: webhdfs of federated namenode does not  work properly
                 Key: HDFS-13138
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: webhdfs
    Affects Versions: 3.0.0, 2.7.1
            Reporter: KWON BYUNGCHANG

my cluster has multiple namenodes using HDFS Federation.

webhdfs that is not defaultFS does not work properly.

when I uploaded to non defaultFS namenode  using webhdfs.

uploaded file was founded at defaultFS namenode.


I think root cause is that

  clientNamenodeAddress of non defaultFS namenode is always fs.defaultFS.


   * Set the namenode address that will be used by clients to access this
   * namenode or name service. This needs to be called before the config
   * is overriden.
  public void setClientNamenodeAddress(Configuration conf) {
    String nnAddr = conf.get(FS_DEFAULT_NAME_KEY);
    if (nnAddr == null) {
      // default fs is not set.
      clientNamenodeAddress = null;
    }"{} is {}", FS_DEFAULT_NAME_KEY, nnAddr);
    URI nnUri = URI.create(nnAddr);

    String nnHost = nnUri.getHost();
    if (nnHost == null) {
      clientNamenodeAddress = null;

    if (DFSUtilClient.getNameServiceIds(conf).contains(nnHost)) {
      // host name is logical
      clientNamenodeAddress = nnHost;
    } else if (nnUri.getPort() > 0) {
      // physical address with a valid port
      clientNamenodeAddress = nnUri.getAuthority();
    } else {
      // the port is missing or 0. Figure out real bind address later.
      clientNamenodeAddress = null;
    }"Clients are to use {} to access"
        + " this namenode/service.", clientNamenodeAddress );


so webhdfs is redirected to datanode having wrong namenoderpcaddress parameter

finally file was located namenode of fs,defaultFS


workaround is

  configure fs.defaultFS of each namenode to its own nameservice.  


  hdfs://ns1  has fs.defaultFS=hdfs://ns1

  hdfs://ns2  has fs.defaultFS=hdfs://ns2






This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to