ayushtkn commented on a change in pull request #2053: URL: https://github.com/apache/hadoop/pull/2053#discussion_r435860568
########## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md ########## @@ -693,4 +693,42 @@ Usage: `hdfs debug recoverLease -path <path> [-retries <num-retries>]` | [`-path` *path*] | HDFS path for which to recover the lease. | | [`-retries` *num-retries*] | Number of times the client will retry calling recoverLease. The default number of retries is 1. | -Recover the lease on the specified path. The path must reside on an HDFS filesystem. The default number of retries is 1. +Recover the lease on the specified path. The path must reside on an HDFS file system. The default number of retries is 1. + +dfsadmin with ViewFsOverloadScheme +---------------------------------- + +Usage: `hdfs dfsadmin -fs <child fs mount link URI> <dfsadmin command options>` + +| COMMAND\_OPTION | Description | +|:---- |:---- | +| `-fs` *child fs mount link URI* | Its a logical mount link path to child file system in ViewFS world. This uri typically formed as src mount link prefixed with fs.defaultFS. Please note, this is not an actual child file system uri, instead its a logical mount link uri pointing to actual child file system| + +Example command usage: + `hdfs dfsadmin -fs hdfs://nn1 -safemode enter` + +In ViewFsOverloadScheme, we may have multiple child file systems as mount point mappings as shown in [ViewFsOverloadScheme Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic parameter supported by dfsadmin. When users want to execute commands on one of the child file system, they need to pass that file system mount mapping link uri to -fs option. Let's take an example mount link configuration and dfsadmin command below. + +Mount link: + +```xml +<property> + <name>fs.defaultFS</name> + <value>hdfs://MyCluster1</value> +</property> + +<property> + <name>fs.viewfs.mounttable.MyCluster1./user</name> + <value>hdfs://MyCluster2/user</value> + <!-- mount table name : MyCluster1 + mount link mapping: hdfs://MyCluster1/user --> hdfs://MyCluster2/user + mount link path: /user + mount link uri: hdfs://MyCluster1/user + mount target uri for /user: hdfs://MyCluster2/user --> +</property> +``` + +If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option (`-fs hdfs://MyCluster1/user`) +Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve the passed (`-fs hdfs://MyCluster1/user`) to target fs (`hdfs://MyCluster2/user`). +This was users can get the access all hdfs child file systems in ViewFsOverloadScheme. Review comment: This was -> This "way" ? get the access all -> get the access "to" all ########## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md ########## @@ -0,0 +1,163 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +View File System Overload Scheme Guide +====================================== + +<!-- MACRO{toc|fromDepth=0|toDepth=3} --> + +Introduction +------------ + +The View File System Overload Scheme introduced to solve two key challenges with the View File System(ViewFS). The first problem is, to use ViewFS, users need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second problem is that users need to copy the mount-table configurations to all the client nodes. +The ViewFileSystemOverloadScheme is addressing these challenges. + +View File System Overload Scheme +-------------------------------- + +### Details + +The View File System Overload Scheme is an extension to the View File System. This will allow users to continue to use their existing fs.defaultFS configured scheme or any new scheme name instead of using scheme `viewfs`. Mount link configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html). If a user wants to continue use the same fs.defaultFS and wants to have more mount points, then mount link configurations should have the current fs.defaultFS authority name as mount table name. Example if fs.defaultFS is `hdfs://mycluster`, then the mount link configuration key name should be like in the following format `fs.viewfs.mounttable.*mycluster*.<mountLinkPath>`. We will discuss more example configurations in following sections. + +Another important improvement with the ViewFileSystemOverloadScheme is, administrators need not copy the `mount-table.xml` configuration file to 1000s of client nodes. Instead they can keep the mount-table configuration file in a Hadoop compatible file system. So, keeping the configuration file in a central place makes administrators life easier as they can update mount-table in single place. + +### Enabling View File System Overload Scheme + +To use this class, the following configurations needed to be added in core-site.xml file. + +```xml +<property> + <name>fs.<scheme>.impl</name> + <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> +</property> +``` +Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS. For example if fs.defaultFS was configured with `hdfs://mycluster`, then the above configuration would be like below: + +```xml +<property> + <name>fs.hdfs.impl</name> + <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> +</property> +``` + +### Example Configurations + +**Example 1:** + +If users want some of their existing cluster (`hdfs://mycluster`) data to mount with hdfs(`hdfs://mycluster`) and other object store clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following example configurations can show how to add mount links. + + +```xml +<property> + <name>fs.viewfs.mounttable.Cluster./user</name> + <value>hdfs://mycluster/user</value> +</property> + +<property> + <name>fs.viewfs.mounttable.Cluster./data</name> + <value>o3fs://bucket1.volume1/data</value> +</property> + +<property> + <name>fs.viewfs.mounttable.Cluster./backup</name> + <value>s3a://bucket1/backup/</value> +</property> +``` + +Let's consider the following operations to understand to where these operations will be delegated based on mount links. Review comment: "to understand to where" -> "to understand where" ########## File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md ########## @@ -0,0 +1,163 @@ +<!--- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +View File System Overload Scheme Guide +====================================== + +<!-- MACRO{toc|fromDepth=0|toDepth=3} --> + +Introduction +------------ + +The View File System Overload Scheme introduced to solve two key challenges with the View File System(ViewFS). The first problem is, to use ViewFS, users need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second problem is that users need to copy the mount-table configurations to all the client nodes. +The ViewFileSystemOverloadScheme is addressing these challenges. + +View File System Overload Scheme +-------------------------------- + +### Details + +The View File System Overload Scheme is an extension to the View File System. This will allow users to continue to use their existing fs.defaultFS configured scheme or any new scheme name instead of using scheme `viewfs`. Mount link configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html). If a user wants to continue use the same fs.defaultFS and wants to have more mount points, then mount link configurations should have the current fs.defaultFS authority name as mount table name. Example if fs.defaultFS is `hdfs://mycluster`, then the mount link configuration key name should be like in the following format `fs.viewfs.mounttable.*mycluster*.<mountLinkPath>`. We will discuss more example configurations in following sections. + +Another important improvement with the ViewFileSystemOverloadScheme is, administrators need not copy the `mount-table.xml` configuration file to 1000s of client nodes. Instead they can keep the mount-table configuration file in a Hadoop compatible file system. So, keeping the configuration file in a central place makes administrators life easier as they can update mount-table in single place. + +### Enabling View File System Overload Scheme + +To use this class, the following configurations needed to be added in core-site.xml file. + +```xml +<property> + <name>fs.<scheme>.impl</name> + <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> +</property> +``` +Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS. For example if fs.defaultFS was configured with `hdfs://mycluster`, then the above configuration would be like below: + +```xml +<property> + <name>fs.hdfs.impl</name> + <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value> +</property> +``` + +### Example Configurations + +**Example 1:** + +If users want some of their existing cluster (`hdfs://mycluster`) data to mount with hdfs(`hdfs://mycluster`) and other object store clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following example configurations can show how to add mount links. + + +```xml +<property> + <name>fs.viewfs.mounttable.Cluster./user</name> + <value>hdfs://mycluster/user</value> +</property> + +<property> + <name>fs.viewfs.mounttable.Cluster./data</name> + <value>o3fs://bucket1.volume1/data</value> +</property> + +<property> + <name>fs.viewfs.mounttable.Cluster./backup</name> + <value>s3a://bucket1/backup/</value> +</property> +``` + +Let's consider the following operations to understand to where these operations will be delegated based on mount links. + + *Op1:* Create a file with the the path `hdfs://mycluster/user/fileA`, then physically this file will be created at `hdfs://mycluster/user/fileA`. This delegation happened based on the first configuration parameter in above configurations. Here `/user` mapped to `hdfs://mycluster/user/`. + + *Op2:* Create a file the the path `hdfs://mycluster/data/datafile`, then this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This delegation happened based on second configurations parameter in above configurations. Here `/data` was mapped with `o3fs://bucket1.volume1.omhost/data/`. + + *Op3:* Create a file with the the path `hdfs://Cluster/backup/data.zip`, then physically this file will be created at `s3a://bucket1/backup/data.zip`. This delegation happened based on the third configuration parameter in above configurations. Here `/backup` was mapped to `s3a://bucket1/backup/`. + + +**Example 2:** + +If users want some of their existing cluster (`s3a://bucketA/`) data to mount with other hdfs cluster(`hdfs://Cluster`) and object store clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucketA/`), the following example configurations can show how to add mount links. + + +```xml +<property> + <name>fs.viewfs.mounttable.bucketA./user</name> + <value>hdfs://Cluster/user</value> +</property> + +<property> + <name>fs.viewfs.mounttable.bucketA./data</name> + <value>o3fs://bucket1.volume1.omhost/data</value> +</property> + +<property> + <name>fs.viewfs.mounttable.bucketA./salesDB</name> + <value>s3a://bucketA/salesDB/</value> +</property> +``` +Let's consider the following operations to understand to where these operations will be delegated based on mount links. + + *Op1:* Create a file with the the path `s3a://bucketA/user/fileA`, then this file will be created physically at `hdfs://Cluster/user/fileA`. This delegation happened based on the first configuration parameter in above configurations. Here `/user` mapped to `hdfs://Cluster/user`. + + *Op2:* Create a file the the path `s3a://bucketA/data/datafile`, then this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This delegation happened based on second configurations parameter in above configurations. Here `/data` was mapped with `o3fs://bucket1.volume1.omhost/data/`. + + *Op3:* Create a file with the the path `s3a://bucketA/salesDB/dbfile`, then physically this file will be created at `s3a://bucketA/salesDB/dbfile`. This delegation happened based on the third configuration parameter in above configurations. Here `/salesDB` was mapped to `s3a://bucket1/salesDB`. + +Note: In above examples we used create operation only, but the same mechanism applies to any other file system APIs here. + +The following picture shows how the different schemes can be used in ViewFileSystemOverloadScheme compared to the ViewFileSystem. + +<img src="./images/ViewFSOverloadScheme.png" width="1050" height="550"/> + +### Central Mount Table Configurations + +To enabled central mount table configuration, we need to configure `fs.viewfs.mounttable.path` in `core-site.xml` with the value as the Hadoop compatible file system directory/file path, where the `mount-table-<versionNumber>.xml` file copied. Here versionNumber is an integer number and need to increase the version number and upload new file in same directory. Review comment: "To enabled" -> "To enable" ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org