ayushtkn commented on a change in pull request #2053:
URL: https://github.com/apache/hadoop/pull/2053#discussion_r435860568
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
##########
@@ -693,4 +693,42 @@ Usage: `hdfs debug recoverLease -path <path> [-retries
<num-retries>]`
| [`-path` *path*] | HDFS path for which to recover the lease. |
| [`-retries` *num-retries*] | Number of times the client will retry calling
recoverLease. The default number of retries is 1. |
-Recover the lease on the specified path. The path must reside on an HDFS
filesystem. The default number of retries is 1.
+Recover the lease on the specified path. The path must reside on an HDFS file
system. The default number of retries is 1.
+
+dfsadmin with ViewFsOverloadScheme
+----------------------------------
+
+Usage: `hdfs dfsadmin -fs <child fs mount link URI> <dfsadmin command options>`
+
+| COMMAND\_OPTION | Description |
+|:---- |:---- |
+| `-fs` *child fs mount link URI* | Its a logical mount link path to child
file system in ViewFS world. This uri typically formed as src mount link
prefixed with fs.defaultFS. Please note, this is not an actual child file
system uri, instead its a logical mount link uri pointing to actual child file
system|
+
+Example command usage:
+ `hdfs dfsadmin -fs hdfs://nn1 -safemode enter`
+
+In ViewFsOverloadScheme, we may have multiple child file systems as mount
point mappings as shown in [ViewFsOverloadScheme
Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic
parameter supported by dfsadmin. When users want to execute commands on one of
the child file system, they need to pass that file system mount mapping link
uri to -fs option. Let's take an example mount link configuration and dfsadmin
command below.
+
+Mount link:
+
+```xml
+<property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://MyCluster1</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.MyCluster1./user</name>
+ <value>hdfs://MyCluster2/user</value>
+ <!-- mount table name : MyCluster1
+ mount link mapping: hdfs://MyCluster1/user --> hdfs://MyCluster2/user
+ mount link path: /user
+ mount link uri: hdfs://MyCluster1/user
+ mount target uri for /user: hdfs://MyCluster2/user -->
+</property>
+```
+
+If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option
(`-fs hdfs://MyCluster1/user`)
+Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve
the passed (`-fs hdfs://MyCluster1/user`) to target fs
(`hdfs://MyCluster2/user`).
+This was users can get the access all hdfs child file systems in
ViewFsOverloadScheme.
Review comment:
This was -> This "way" ?
get the access all -> get the access "to" all
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
##########
@@ -0,0 +1,163 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+View File System Overload Scheme Guide
+======================================
+
+<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
+
+Introduction
+------------
+
+The View File System Overload Scheme introduced to solve two key challenges
with the View File System(ViewFS). The first problem is, to use ViewFS, users
need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second
problem is that users need to copy the mount-table configurations to all the
client nodes.
+The ViewFileSystemOverloadScheme is addressing these challenges.
+
+View File System Overload Scheme
+--------------------------------
+
+### Details
+
+The View File System Overload Scheme is an extension to the View File System.
This will allow users to continue to use their existing fs.defaultFS configured
scheme or any new scheme name instead of using scheme `viewfs`. Mount link
configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html).
If a user wants to continue use the same fs.defaultFS and wants to have more
mount points, then mount link configurations should have the current
fs.defaultFS authority name as mount table name. Example if fs.defaultFS is
`hdfs://mycluster`, then the mount link configuration key name should be like
in the following format `fs.viewfs.mounttable.*mycluster*.<mountLinkPath>`. We
will discuss more example configurations in following sections.
+
+Another important improvement with the ViewFileSystemOverloadScheme is,
administrators need not copy the `mount-table.xml` configuration file to 1000s
of client nodes. Instead they can keep the mount-table configuration file in a
Hadoop compatible file system. So, keeping the configuration file in a central
place makes administrators life easier as they can update mount-table in single
place.
+
+### Enabling View File System Overload Scheme
+
+To use this class, the following configurations needed to be added in
core-site.xml file.
+
+```xml
+<property>
+ <name>fs.<scheme>.impl</name>
+ <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS.
For example if fs.defaultFS was configured with `hdfs://mycluster`, then the
above configuration would be like below:
+
+```xml
+<property>
+ <name>fs.hdfs.impl</name>
+ <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+
+### Example Configurations
+
+**Example 1:**
+
+If users want some of their existing cluster (`hdfs://mycluster`) data to
mount with hdfs(`hdfs://mycluster`) and other object store
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.Cluster./user</name>
+ <value>hdfs://mycluster/user</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.Cluster./data</name>
+ <value>o3fs://bucket1.volume1/data</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.Cluster./backup</name>
+ <value>s3a://bucket1/backup/</value>
+</property>
+```
+
+Let's consider the following operations to understand to where these
operations will be delegated based on mount links.
Review comment:
"to understand to where" -> "to understand where"
##########
File path:
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
##########
@@ -0,0 +1,163 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+View File System Overload Scheme Guide
+======================================
+
+<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
+
+Introduction
+------------
+
+The View File System Overload Scheme introduced to solve two key challenges
with the View File System(ViewFS). The first problem is, to use ViewFS, users
need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second
problem is that users need to copy the mount-table configurations to all the
client nodes.
+The ViewFileSystemOverloadScheme is addressing these challenges.
+
+View File System Overload Scheme
+--------------------------------
+
+### Details
+
+The View File System Overload Scheme is an extension to the View File System.
This will allow users to continue to use their existing fs.defaultFS configured
scheme or any new scheme name instead of using scheme `viewfs`. Mount link
configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html).
If a user wants to continue use the same fs.defaultFS and wants to have more
mount points, then mount link configurations should have the current
fs.defaultFS authority name as mount table name. Example if fs.defaultFS is
`hdfs://mycluster`, then the mount link configuration key name should be like
in the following format `fs.viewfs.mounttable.*mycluster*.<mountLinkPath>`. We
will discuss more example configurations in following sections.
+
+Another important improvement with the ViewFileSystemOverloadScheme is,
administrators need not copy the `mount-table.xml` configuration file to 1000s
of client nodes. Instead they can keep the mount-table configuration file in a
Hadoop compatible file system. So, keeping the configuration file in a central
place makes administrators life easier as they can update mount-table in single
place.
+
+### Enabling View File System Overload Scheme
+
+To use this class, the following configurations needed to be added in
core-site.xml file.
+
+```xml
+<property>
+ <name>fs.<scheme>.impl</name>
+ <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS.
For example if fs.defaultFS was configured with `hdfs://mycluster`, then the
above configuration would be like below:
+
+```xml
+<property>
+ <name>fs.hdfs.impl</name>
+ <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+
+### Example Configurations
+
+**Example 1:**
+
+If users want some of their existing cluster (`hdfs://mycluster`) data to
mount with hdfs(`hdfs://mycluster`) and other object store
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.Cluster./user</name>
+ <value>hdfs://mycluster/user</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.Cluster./data</name>
+ <value>o3fs://bucket1.volume1/data</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.Cluster./backup</name>
+ <value>s3a://bucket1/backup/</value>
+</property>
+```
+
+Let's consider the following operations to understand to where these
operations will be delegated based on mount links.
+
+ *Op1:* Create a file with the the path `hdfs://mycluster/user/fileA`, then
physically this file will be created at `hdfs://mycluster/user/fileA`. This
delegation happened based on the first configuration parameter in above
configurations. Here `/user` mapped to `hdfs://mycluster/user/`.
+
+ *Op2:* Create a file the the path `hdfs://mycluster/data/datafile`, then
this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`.
This delegation happened based on second configurations parameter in above
configurations. Here `/data` was mapped with
`o3fs://bucket1.volume1.omhost/data/`.
+
+ *Op3:* Create a file with the the path `hdfs://Cluster/backup/data.zip`,
then physically this file will be created at `s3a://bucket1/backup/data.zip`.
This delegation happened based on the third configuration parameter in above
configurations. Here `/backup` was mapped to `s3a://bucket1/backup/`.
+
+
+**Example 2:**
+
+If users want some of their existing cluster (`s3a://bucketA/`) data to mount
with other hdfs cluster(`hdfs://Cluster`) and object store
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucketA/`), the following
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+ <name>fs.viewfs.mounttable.bucketA./user</name>
+ <value>hdfs://Cluster/user</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.bucketA./data</name>
+ <value>o3fs://bucket1.volume1.omhost/data</value>
+</property>
+
+<property>
+ <name>fs.viewfs.mounttable.bucketA./salesDB</name>
+ <value>s3a://bucketA/salesDB/</value>
+</property>
+```
+Let's consider the following operations to understand to where these
operations will be delegated based on mount links.
+
+ *Op1:* Create a file with the the path `s3a://bucketA/user/fileA`, then this
file will be created physically at `hdfs://Cluster/user/fileA`. This delegation
happened based on the first configuration parameter in above configurations.
Here `/user` mapped to `hdfs://Cluster/user`.
+
+ *Op2:* Create a file the the path `s3a://bucketA/data/datafile`, then this
file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This
delegation happened based on second configurations parameter in above
configurations. Here `/data` was mapped with
`o3fs://bucket1.volume1.omhost/data/`.
+
+ *Op3:* Create a file with the the path `s3a://bucketA/salesDB/dbfile`, then
physically this file will be created at `s3a://bucketA/salesDB/dbfile`. This
delegation happened based on the third configuration parameter in above
configurations. Here `/salesDB` was mapped to `s3a://bucket1/salesDB`.
+
+Note: In above examples we used create operation only, but the same mechanism
applies to any other file system APIs here.
+
+The following picture shows how the different schemes can be used in
ViewFileSystemOverloadScheme compared to the ViewFileSystem.
+
+<img src="./images/ViewFSOverloadScheme.png" width="1050" height="550"/>
+
+### Central Mount Table Configurations
+
+To enabled central mount table configuration, we need to configure
`fs.viewfs.mounttable.path` in `core-site.xml` with the value as the Hadoop
compatible file system directory/file path, where the
`mount-table-<versionNumber>.xml` file copied. Here versionNumber is an integer
number and need to increase the version number and upload new file in same
directory.
Review comment:
"To enabled" -> "To enable"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
