[ https://issues.apache.org/jira/browse/HDFS-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631318#comment-15631318 ]
Manoj Govindassamy commented on HDFS-11058: ------------------------------------------- # {{ViewFsMountPoint}} : Yes, this replaces _ViewFileSystem#MountPoint_. This is more than a refactor as the new definition is slightly different from the older one. So far there has not been any real outside user for _ViewFileSystem#MountPoint_. As part of HDFS-5684, I added annotations in MountPoint and extended the test case to print MountPoints. {{DfUsage}} for ViewFileSystem is the first real outside user for MountPoint, and I thought this would be right time for defining a proper {{ViewFsMountPoint}}. IMHO, {{ViewFsMountPoint}} should be abstracted and expose only the needed attributes -- the MountedOn path and its target FileSystem. The FileSystem could be a _hdfs://_ or it could be a one for _MergeFs_, but I don't see a need for exposing all the NameServices, at least for now. # {{ViewFsUtil}} : This is the new helper routine for ViewFileSystem. Already added the annotation _InterfaceAudience.Public, InterfaceStability.Evolving_ to it. Utility functions inside this class are introduced only for {{DfUsage}}. Is it still worth to separate this out to a new patch when there are no other callers ? # Responsive and non-responsive filesystems: Yes, this is a new behavior for {{DF}}, but this is only in the context of ViewFileSystem. I was contemplating on _Availability_ vs _Consistency_ for the command and inclined towards the Availability. When any one of the NameNode or backing filesystem is not reachable, then DF command will error out normally and none of the information would be printed. Whereas by skipping the unreachable ones, at least the reachable FileSystems are printed out. I have seen the unix {{df}} command getting stuck at times when NFS servers are not reachable. But, I am totally ok to remove this extra feature and error out when any of the backing NameServices are not reachable. May I will propse this as a separate patch and not part of this issue. > Implement 'hadoop fs -df' command for ViewFileSystem > ------------------------------------------------------- > > Key: HDFS-11058 > URL: https://issues.apache.org/jira/browse/HDFS-11058 > Project: Hadoop HDFS > Issue Type: Task > Affects Versions: 3.0.0-alpha1 > Reporter: Manoj Govindassamy > Assignee: Manoj Govindassamy > Labels: viewfs > Attachments: HDFS-11058.01.patch > > > Df command doesn't seem to work well with ViewFileSystem. It always reports > used data as 0. Here is the client mount table configuration I am using > against a federated clusters of 2 NameNodes and 2 DataNoes. > {code} > 1 <?xml version="1.0" ?> > 2 <configuration> > 3 <property> > 4 <name>fs.defaultFS</name> > 5 <value>viewfs://ClusterX/</value> > 6 </property> > .. > 11 <property> > 12 <name>fs.default.name</name> > 13 <value>viewfs://ClusterX/</value> > 14 </property> > .. > 23 <property> > 24 <name>fs.viewfs.mounttable.ClusterX.link./nn0</name> > 25 <value>hdfs://127.0.0.1:50001/</value> > 26 </property> > 27 <property> > 28 <name>fs.viewfs.mounttable.ClusterX.link./nn1</name> > 29 <value>hdfs://127.0.0.1:51001/</value> > 30 </property> > 31 <property> > 32 <name>fs.viewfs.mounttable.ClusterX.link./nn2</name> > 33 <value>hdfs://127.0.0.1:52001/nn2</value> > 34 </property> > 35 <property> > 36 <name>fs.viewfs.mounttable.ClusterX.link./nn3</name> > 37 <value>hdfs://127.0.0.1:52001/nn3</value> > 38 </property> > 39 <property> > 40 <name>fs.viewfs.mounttable.ClusterY.linkMergeSlash</name> > 41 <value>hdfs://127.0.0.1:50001/</value> > 42 </property> > 43 </configuration> > {code} > {{Df}} command always reports Size/Available as 8.0E and the usage as 0 for > any federated cluster. > {noformat} > # hadoop fs -fs viewfs://ClusterX/ -df / > Filesystem Size Used Available Use% > viewfs://ClusterX/ 9223372036854775807 0 9223372036854775807 0% > # hadoop fs -fs viewfs://ClusterX/ -df -h / > Filesystem Size Used Available Use% > viewfs://ClusterX/ 8.0 E 0 8.0 E 0% > # hadoop fs -fs viewfs://ClusterY/ -df -h / > Filesystem Size Used Available Use% > viewfs://ClusterY/ 8.0 E 0 8.0 E 0% > {noformat} > Whereas {{Du}} command seems to work as expected even with ViewFileSystem. > {noformat} > # hadoop fs -fs viewfs://ClusterY/ -du -h / > 10.6 K 31.8 K /build.log.16y > 0 0 /user > # hadoop fs -fs viewfs://ClusterX/ -du -h / > 10.6 K 31.8 K /nn0 > 0 0 /nn1 > 20.2 K 35.8 K /nn3 > 40.6 K 34.3 K /nn4 > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org