This is an automated email from the ASF dual-hosted git repository.

umamahesh pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new e7cead1  HDFS-15330. Document the ViewFSOverloadScheme details in 
ViewFS guide. Contributed by Uma Maheswara Rao G.
e7cead1 is described below

commit e7cead114314412d461ed1c4b28ba33726c65b9b
Author: Uma Maheswara Rao G <umamah...@apache.org>
AuthorDate: Fri Jun 5 10:58:21 2020 -0700

    HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. 
Contributed by Uma Maheswara Rao G.
    
    (cherry picked from commit 76fa0222f0d2e2d92b4a1eedba8b3e38002e8c23)
    (cherry picked from commit 418580446b65be3a0674762e76fc2cb9a1e5629a)
---
 .../hadoop-hdfs/src/site/markdown/HDFSCommands.md  |  40 ++++-
 .../hadoop-hdfs/src/site/markdown/ViewFs.md        |   6 +
 .../src/site/markdown/ViewFsOverloadScheme.md      | 163 +++++++++++++++++++++
 .../site/resources/images/ViewFSOverloadScheme.png | Bin 0 -> 190004 bytes
 hadoop-project/src/site/site.xml                   |   1 +
 5 files changed, 209 insertions(+), 1 deletion(-)

diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
index 306884c..32e1a7b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
@@ -672,4 +672,42 @@ Usage: `hdfs debug recoverLease -path <path> [-retries 
<num-retries>]`
 | [`-path` *path*] | HDFS path for which to recover the lease. |
 | [`-retries` *num-retries*] | Number of times the client will retry calling 
recoverLease. The default number of retries is 1. |
 
-Recover the lease on the specified path. The path must reside on an HDFS 
filesystem. The default number of retries is 1.
+Recover the lease on the specified path. The path must reside on an HDFS file 
system. The default number of retries is 1.
+
+dfsadmin with ViewFsOverloadScheme
+----------------------------------
+
+Usage: `hdfs dfsadmin -fs <child fs mount link URI> <dfsadmin command options>`
+
+| COMMAND\_OPTION | Description |
+|:---- |:---- |
+| `-fs` *child fs mount link URI* | Its a logical mount link path to child 
file system in ViewFS world. This uri typically formed as src mount link 
prefixed with fs.defaultFS. Please note, this is not an actual child file 
system uri, instead its a logical mount link uri pointing to actual child file 
system|
+
+Example command usage:
+   `hdfs dfsadmin -fs hdfs://nn1 -safemode enter`
+
+In ViewFsOverloadScheme, we may have multiple child file systems as mount 
point mappings as shown in [ViewFsOverloadScheme 
Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic 
parameter supported by dfsadmin. When users want to execute commands on one of 
the child file system, they need to pass that file system mount mapping link 
uri to -fs option. Let's take an example mount link configuration and dfsadmin 
command below.
+
+Mount link:
+
+```xml
+<property>
+  <name>fs.defaultFS</name>
+  <value>hdfs://MyCluster1</value>
+</property>
+
+<property>
+  <name>fs.viewfs.mounttable.MyCluster1./user</name>
+  <value>hdfs://MyCluster2/user</value>
+  <!-- mount table name : MyCluster1
+       mount link mapping: hdfs://MyCluster1/user --> hdfs://MyCluster2/user
+       mount link path: /user
+       mount link uri: hdfs://MyCluster1/user
+       mount target uri for /user: hdfs://MyCluster2/user -->
+</property>
+```
+
+If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option 
(`-fs hdfs://MyCluster1/user`)
+Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve 
the passed (`-fs hdfs://MyCluster1/user`) to target fs 
(`hdfs://MyCluster2/user`).
+This way users can get the access to all hdfs child file systems in 
ViewFsOverloadScheme.
+If there is no `-fs` option provided, then it will try to connect to the 
configured fs.defaultFS cluster if a cluster running with the fs.defaultFS uri.
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
index f851ef6..52ad49c 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
@@ -361,6 +361,12 @@ resume its work, it's a good idea to provision some sort 
of cron job to purge su
 
     Delegation tokens for the cluster to which you are submitting the job 
(including all mounted volumes for that cluster’s mount table), and for input 
and output paths to your map-reduce job (including all volumes mounted via 
mount tables for the specified input and output paths) are all handled 
automatically. In addition, there is a way to add additional delegation tokens 
to the base cluster configuration for special circumstances.
 
+Don't want to change scheme or difficult to copy mount-table configurations to 
all clients?
+-------------------------------------------------------------------------------------------
+
+Please refer to the [View File System Overload Scheme 
Guide](./ViewFsOverloadScheme.html)
+
+
 Appendix: A Mount Table Configuration Example
 ---------------------------------------------
 
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
new file mode 100644
index 0000000..5fd8633
--- /dev/null
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFsOverloadScheme.md
@@ -0,0 +1,163 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+View File System Overload Scheme Guide
+======================================
+
+<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
+
+Introduction
+------------
+
+The View File System Overload Scheme introduced to solve two key challenges 
with the View File System(ViewFS). The first problem is, to use ViewFS, users 
need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second 
problem is that users need to copy the mount-table configurations to all the 
client nodes.
+The ViewFileSystemOverloadScheme is addressing these challenges.
+
+View File System Overload Scheme
+--------------------------------
+
+### Details
+
+The View File System Overload Scheme is an extension to the View File System. 
This will allow users to continue to use their existing fs.defaultFS configured 
scheme or any new scheme name instead of using scheme `viewfs`. Mount link 
configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html). 
If a user wants to continue use the same fs.defaultFS and wants to have more 
mount points, then mount link configurations should have the current 
fs.defaultFS authority name as mo [...]
+
+Another important improvement with the ViewFileSystemOverloadScheme is, 
administrators need not copy the `mount-table.xml` configuration file to 1000s 
of client nodes. Instead they can keep the mount-table configuration file in a 
Hadoop compatible file system. So, keeping the configuration file in a central 
place makes administrators life easier as they can update mount-table in single 
place.
+
+### Enabling View File System Overload Scheme
+
+To use this class, the following configurations needed to be added in 
core-site.xml file.
+
+```xml
+<property>
+  <name>fs.<scheme>.impl</name>
+  <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS. 
For example if fs.defaultFS was configured with `hdfs://mycluster`, then the 
above configuration would be like below:
+
+```xml
+<property>
+  <name>fs.hdfs.impl</name>
+  <value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
+</property>
+```
+
+### Example Configurations
+
+**Example 1:**
+
+If users want some of their existing cluster (`hdfs://mycluster`) data to 
mount with hdfs(`hdfs://mycluster`) and other object store 
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following 
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+  <name>fs.viewfs.mounttable.Cluster./user</name>
+  <value>hdfs://mycluster/user</value>
+</property>
+
+<property>
+  <name>fs.viewfs.mounttable.Cluster./data</name>
+  <value>o3fs://bucket1.volume1/data</value>
+</property>
+
+<property>
+  <name>fs.viewfs.mounttable.Cluster./backup</name>
+  <value>s3a://bucket1/backup/</value>
+</property>
+```
+
+Let's consider the following operations to understand where these operations 
will be delegated based on mount links.
+
+  *Op1:* Create a file with the the path `hdfs://mycluster/user/fileA`, then 
physically this file will be created at `hdfs://mycluster/user/fileA`. This 
delegation happened based on the first configuration parameter in above 
configurations. Here `/user` mapped to `hdfs://mycluster/user/`.
+
+  *Op2:* Create a file the the path `hdfs://mycluster/data/datafile`, then 
this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. 
This delegation happened based on second configurations parameter in above 
configurations. Here `/data` was mapped with 
`o3fs://bucket1.volume1.omhost/data/`.
+
+  *Op3:* Create a file with the the path `hdfs://Cluster/backup/data.zip`, 
then  physically this file will be created at `s3a://bucket1/backup/data.zip`. 
This delegation happened based on the third configuration parameter in above 
configurations. Here `/backup` was mapped to `s3a://bucket1/backup/`.
+
+
+**Example 2:**
+
+If users want some of their existing cluster (`s3a://bucketA/`) data to mount 
with other hdfs cluster(`hdfs://Cluster`) and object store 
clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucketA/`), the following 
example configurations can show how to add mount links.
+
+
+```xml
+<property>
+  <name>fs.viewfs.mounttable.bucketA./user</name>
+  <value>hdfs://Cluster/user</value>
+</property>
+
+<property>
+  <name>fs.viewfs.mounttable.bucketA./data</name>
+  <value>o3fs://bucket1.volume1.omhost/data</value>
+</property>
+
+<property>
+  <name>fs.viewfs.mounttable.bucketA./salesDB</name>
+  <value>s3a://bucketA/salesDB/</value>
+</property>
+```
+Let's consider the following operations to understand to where these 
operations will be delegated based on mount links.
+
+ *Op1:* Create a file with the the path `s3a://bucketA/user/fileA`, then this 
file will be created physically at `hdfs://Cluster/user/fileA`. This delegation 
happened based on the first configuration parameter in above configurations. 
Here `/user` mapped to `hdfs://Cluster/user`.
+
+  *Op2:* Create a file the the path `s3a://bucketA/data/datafile`, then this 
file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This 
delegation happened based on second configurations parameter in above 
configurations. Here `/data` was mapped with 
`o3fs://bucket1.volume1.omhost/data/`.
+
+  *Op3:* Create a file with the the path `s3a://bucketA/salesDB/dbfile`, then  
physically this file will be created at `s3a://bucketA/salesDB/dbfile`. This 
delegation happened based on the third configuration parameter in above 
configurations. Here `/salesDB` was mapped to `s3a://bucket1/salesDB`.
+
+Note: In above examples we used create operation only, but the same mechanism 
applies to any other file system APIs here.
+
+The following picture shows how the different schemes can be used in 
ViewFileSystemOverloadScheme compared to the ViewFileSystem.
+
+<img src="./images/ViewFSOverloadScheme.png" width="1050" height="550"/>
+
+### Central Mount Table Configurations
+
+To enable central mount table configuration, we need to configure 
`fs.viewfs.mounttable.path` in `core-site.xml` with the value as the Hadoop 
compatible file system directory/file path, where the 
`mount-table-<versionNumber>.xml` file copied. Here versionNumber is an integer 
number and need to increase the version number and upload new file in same 
directory.
+
+The ViewFileSystemOverloadScheme always loads the highest version number 
`mount-table-<versionNumber>.xml`. Please don't replace the file with same 
name. Always increment the version number to take new file picked by newly 
initializing clients. Why we don't recommend to replace the files is that, some 
client might have already opened the connections to old mount-table files 
already and in middle of loading configuration files, and replacing files can 
make them fail.
+
+```xml
+<property>
+  <name>fs.viewfs.mounttable.path</name>
+  <value>hdfs://Cluster/config/mount-table-dir</value>
+</property>
+```
+ If you are sure, you will never do updates to mount-table file, you can also 
configure file path directly like below. If you configure file path, it will 
not check any highest version number loading. Whatever file configured it will 
be loaded. However file name format should be same.
+
+```xml
+<property>
+  <name>fs.viewfs.mounttable.path</name>
+  
<value>hdfs://Cluster/config/mount-table-dir/mount-table-<versionNumber>.xml</value>
+</property>
+```
+Note: we recommend not to configure mount-links in `core-site.xml` if you 
configure above valid path. Otherwise both mount links will be mixed and can 
lead to a confused behavior.
+
+If you copy the `mount-table-<versionNumber>.xml`, you may consider having big 
replication factor depending on your cluster size. So, that file will be 
available locally to majority of clients as applications(MR/YARN/HBASE..etc) 
use locality on HDFS when reading `mount-table-<versionNumber>.xml`.
+
+DFSAdmin commands with View File System Overload Scheme
+-------------------------------------------------------
+
+Please refer to the [HDFSCommands 
Guide](./HDFSCommands.html#dfsadmin_with_ViewFsOverloadScheme)
+
+Appendix: A Mount Table Configuration with XInclude
+---------------------------------------------------
+
+If users have a HTTP server in trusted network and don't need authentication 
mechanism to it, you can also place your mount-table.xml file in that server 
and configure
+ XInclude xml tag with `mount-table.xml` file.
+
+```xml
+<configuration xmlns:xi="http://www.w3.org/2001/XInclude";>
+  <xi:include href="http://myserver/mountTable/mountTable.xml"; />
+</configuration>
+```
+
+The Apache Hadoop configuration has the capability to read the http urls from 
XInclude and load into configurations. If you choose this option, please don't 
configure mount-table configuration items in `core-site.xml` or at 
`fs.viewfs.mounttable.path`. Please note, Hadoop configuration XInclude does 
not use SPNego authentication when opening url. So, this will not work if http 
server where you placed `mount-table.xml` needs authentication.
diff --git 
a/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png
 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png
new file mode 100644
index 0000000..b5029c1
Binary files /dev/null and 
b/hadoop-hdfs-project/hadoop-hdfs/src/site/resources/images/ViewFSOverloadScheme.png
 differ
diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml
index 36f2e0d..ba08ef9 100644
--- a/hadoop-project/src/site/site.xml
+++ b/hadoop-project/src/site/site.xml
@@ -83,6 +83,7 @@
       <item name="Observer NameNode" 
href="hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html"/>
       <item name="Federation" 
href="hadoop-project-dist/hadoop-hdfs/Federation.html"/>
       <item name="ViewFs" href="hadoop-project-dist/hadoop-hdfs/ViewFs.html"/>
+      <item name="ViewFsOverloadScheme" 
href="hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html"/>
       <item name="Snapshots" 
href="hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html"/>
       <item name="Edits Viewer" 
href="hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html"/>
       <item name="Image Viewer" 
href="hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html"/>


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

Reply via email to