[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429081#comment-16429081 ] Xiao Chen commented on HDFS-11968: -- Pushed this to branch-3.0, thanks for the work here > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: 3.1.0, 3.0.3 > > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch, HDFS-11968.011.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190135#comment-16190135 ] Hudson commented on HDFS-11968: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13014 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13014/]) HDFS-11968. ViewFS: StoragePolicies commands fail with HDFS federation. (arp: rev b91305119b434d23b99ae7e755aea6639f48b6ab) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestStoragePolicyCommands.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/InodeTree.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/StoragePolicyAdmin.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestWebHDFSStoragePolicyCommands.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestViewFSStoragePolicyCommands.java > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: 3.1.0 > > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch, HDFS-11968.011.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189504#comment-16189504 ] Surendra Singh Lilhore commented on HDFS-11968: --- +1 LGTM. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch, HDFS-11968.011.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189288#comment-16189288 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 52s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 12s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 30s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 3s{color} | {color:orange} root: The patch generated 3 new + 27 unchanged - 3 fixed = 30 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 21s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 12s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 36s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}202m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestLeaseRecoveryStriped | | | hadoop.hdfs.qjournal.server.TestJournalNodeSync | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890087/HDFS-11968.011.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f5a37e49cc99 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d4d2fd1 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189221#comment-16189221 ] Arpit Agarwal commented on HDFS-11968: -- +1 for the v11 patch. Thanks [~msingh]. [~surendrasingh]/[~manojg], are you okay with the latest patch? I will hold off committing for now in you have additional comments. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch, HDFS-11968.011.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189189#comment-16189189 ] Mukul Kumar Singh commented on HDFS-11968: -- Thanks for the review [~arpitagarwal], I have fixed the review comments in the latest patch. I have also raised HDFS-12581 to fix the issue with getAllStoragePolicies. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch, HDFS-11968.011.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188892#comment-16188892 ] Arpit Agarwal commented on HDFS-11968: -- Thanks [~msingh]. Your patch looks good to me. We can fix the behavior introduced by HDFS-13272. Two minor comments, but I am +1 basically. # The {{instanceof}} check is unnecessary. You can just print the policy object if it is non-null. {code} + if ((policy != null) && (policy instanceof BlockStoragePolicy)) { System.out.println("\t" + policy); {code} # The following error should go to stderr (via System.err.println) {code} System.out.println(getName() + " is not supported for filesystem " + fs.getScheme() + " on path " + path); {code} > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16187767#comment-16187767 ] Mukul Kumar Singh commented on HDFS-11968: -- The support for StoragePolicy was added to ViewFileSystem with HADOOP-13272. The current patch makes use of this api to find out the underlying storage policies. Here is the output for 2 namenode setup {code} [hdfs@y126 ~]$ /opt/hadoop/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies 17/10/02 08:39:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Block Storage Policies: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]} BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]} BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]} BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]} {code} > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177689#comment-16177689 ] Mukul Kumar Singh commented on HDFS-11968: -- Hi [~arpitagarwal], I ran the following commands on a viewfs,hdfs and webhdfs based instances and following are the list of commands and their output. ViewFS based uri {code} *core-site.xml* fs.default.name viewfs://cluster1 true fs.viewfs.mounttable.cluster1.link./home hdfs://127.0.0.1:8020/dir1 fs.viewfs.mounttable.cluster1.link./tmp hdfs://127.0.0.1:8020/dir2 *list storage policies* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies Block Storage Policies: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]} BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]} BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]} BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]} *set storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -setStoragePolicy -path /home -policy ALL_SSD Set storage policy ALL_SSD on /home *get storage policies after setting a policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -getStoragePolicy -path /home The storage policy of /home: BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK] *unset storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -unsetStoragePolicy -path /home Unset storage policy from /home {code} Web HDFS based uri {code} *core site* fs.default.name webhdfs://127.0.0.1:9870/ *list storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies Block Storage Policies: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]} BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]} BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]} BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]} *set storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -setStoragePolicy -path /dir2 -policy WARM Set storage policy WARM on /dir2 *get storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -getStoragePolicy -path /dir2 The storage policy of /dir2: BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]} {code} HDFS based uri {code} *core site* fs.default.name hdfs://127.0.0.1:8020/ *list storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies Block Storage Policies: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE], replicationFallbacks=[DISK, ARCHIVE]} BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD, DISK]} BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]} BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK], replicationFallbacks=[DISK]} *set storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -setStoragePolicy -path /dir1 -policy COLD Set storage policy COLD on /dir1 *get storage policy* [root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -getStoragePolicy -path /dir1 The storage policy of /dir1: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177588#comment-16177588 ] Surendra Singh Lilhore commented on HDFS-11968: --- Thanks [~arpitagarwal]. If the given path is mounted with the *hdfs* filesystem then no change in behavior, otherwise command will fail and give message like {color:red}"getStoragePolicies is not supported for filesystem viewfs on path ".{color} > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177264#comment-16177264 ] Arpit Agarwal commented on HDFS-11968: -- Hi [~msingh], [~surendrasingh], What is the expected behavior now when running the getStoragePolicies command against a viewfs path? > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176264#comment-16176264 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 58s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 0s{color} | {color:orange} root: The patch generated 3 new + 27 unchanged - 3 fixed = 30 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 4s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 16s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}166m 17s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | | | hadoop.hdfs.qjournal.server.TestJournalNodeSync | | | hadoop.hdfs.server.namenode.TestReencryption | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888457/HDFS-11968.010.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9d19ab1c894d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / c71d137 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/21301/artifact/patchprocess/diff-checkstyle-root.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21301/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/21301/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176130#comment-16176130 ] Surendra Singh Lilhore commented on HDFS-11968: --- +1 LGTM, pending jenkins. [~arpitagarwal], [~manojg] do you want to take a look at latest patch? > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch, HDFS-11968.010.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176022#comment-16176022 ] Surendra Singh Lilhore commented on HDFS-11968: --- bq. We can decide to either have different error messages for ViewFileSystem or this can be fixed by modifying the exception message in InodeTree.java:resolve Its better to handle in {{InodeTree.java:resolve}}. bq. In order to write unit test with WebHDFSFileSystem and ViewFileSystem, fields conf, cluster and fs have been made protected which is resulting in the checkstyle warning. okay > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175968#comment-16175968 ] Mukul Kumar Singh commented on HDFS-11968: -- Thanks for the review [~surendrasingh] 1) No need to catch FileNotFoundException explicitly in UnsetStoragePolicyCommand and SetStoragePolicyCommand. In GetStoragePolicyCommand handling FileNotFoundException because of compatibility issue. Message give by DistributedFileSystem in FileNotFoundException is different from the message in GetStoragePolicyCommand. I don't want to change it. bq. The exception handling in UnsetStoragePolicyCommand/SetStoragePolicyCommand is needed to make sure that the error message returned as part of ViewFileSystem is the same as the one returned by HDFS. If FileNotFoundException is not handled then the error message will be {{FileNotFoundException: /fooz}} where as for HDFS a FileNotFoundException results in error mesage {{File/Directory does not exist: /fooz }}. We can decide to either have different error messages for ViewFileSystem or this can be fixed by modifying the exception message in {{InodeTree.java:resolve}} {code} failedAt.append('/').append(path[j]); } throw (new FileNotFoundException(failedAt.toString())); } {code} or by handling the FileNotFoundException in the UnsetStoragePolicyCommand/SetStoragePolicyCommand handler. Please let me know of you opinion :) 2) Please handle the checkstyle warnings. bq. In order to write unit test with WebHDFSFileSystem and ViewFileSystem, fields conf, cluster and fs have been made protected which is resulting in the checkstyle warning. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175936#comment-16175936 ] Surendra Singh Lilhore commented on HDFS-11968: --- Thanks [~msingh] for patch.. A few comments: # No need to catch {{FileNotFoundException}} explicitly in {{UnsetStoragePolicyCommand}} and {{SetStoragePolicyCommand}}. In {{GetStoragePolicyCommand}} handling {{FileNotFoundException}} because of compatibility issue. Message give by DistributedFileSystem in {{FileNotFoundException}} is different from the message in {{GetStoragePolicyCommand}}. I don't want to change it. {code} HdfsFileStatus status = dfs.getClient().getFileInfo( Path.getPathWithoutSchemeAndAuthority(p).toString()); if (status == null) { System.err.println("File/Directory does not exist: " + path); return 2; } {code} # Please handle the checkstyle warnings. Otherwise LGTM.. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch, > HDFS-11968.009.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174760#comment-16174760 ] Hadoop QA commented on HDFS-11968: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 7s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 5 unchanged - 3 fixed = 8 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 48s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}126m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888258/HDFS-11968.009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 6b06d72c8b58 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 50849ec | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/21278/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/21278/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/21278/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments:
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174565#comment-16174565 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 5 unchanged - 3 fixed = 8 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 1s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}106m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs | | | Redundant nullcheck of policy, which is known to be non-null in org.apache.hadoop.hdfs.tools.StoragePolicyAdmin$ListStoragePoliciesCommand.run(Configuration, List) Redundant null check at StoragePolicyAdmin.java:is known to be non-null in org.apache.hadoop.hdfs.tools.StoragePolicyAdmin$ListStoragePoliciesCommand.run(Configuration, List) Redundant null check at StoragePolicyAdmin.java:[line 113] | | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888240/HDFS-11968.008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 74968f765a03 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8b33663 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/21276/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/21276/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html | | unit |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174470#comment-16174470 ] Mukul Kumar Singh commented on HDFS-11968: -- Hi [~surendrasingh], Thanks for the review. I have applied all of you review comments in the patch v8. Thanks for your the review. Please have a look again. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173700#comment-16173700 ] Surendra Singh Lilhore commented on HDFS-11968: --- Please handle some more comments 1. No need of adding new methods in {{BlockStoragePolicySpi}}. Remove these methods. {code} /** + byte getId(); + + byte getRealId(); {code} 2. Please don't change any code which is not related to this patch, like {code} - AdminHelper.printUsage(false, "storagepolicies", COMMANDS); + AdminHelper + .printUsage(false, "storagepolicies", COMMANDS); {code} 3. {{InodeTree.java}} changes is not required. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173678#comment-16173678 ] Surendra Singh Lilhore commented on HDFS-11968: --- Thanks [~msingh] patch. {{getStoragePolicy}} command is depend on {{HdfsFileStatus}} and {{BlockStoragePolicy}} to get the policy Id, which is specific to the HDFS. Lets not change any public interface({{BlockStoragePolicySpi}}) to get the policy ID. Try this code for {{getStoragePolicy}} command. {code} try { FileStatus status; try { status = fs.getFileStatus(new Path(path)); } catch (FileNotFoundException e) { System.err.println("File/Directory does not exist: " + path); return 2; } if (status instanceof HdfsFileStatus) { byte storagePolicyId = ((HdfsFileStatus) status).getStoragePolicy(); if (storagePolicyId == HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) { System.out.println("The storage policy of " + path + " is unspecified"); return 0; } Collection policies = fs .getAllStoragePolicies(); for (BlockStoragePolicySpi policy : policies) { if (policy instanceof BlockStoragePolicy) { if (((BlockStoragePolicy) policy).getId() == storagePolicyId) { System.out.println("The storage policy of " + path + ":\n" + policy); return 0; } } } } System.out.println(getName() + " is not supported for filesystem " + fs.getScheme() + " on path " + path); return 2; } catch (Exception e) { System.err.println(AdminHelper.prettifyException(e)); return 2; } {code} > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171565#comment-16171565 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 15m 28s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 37s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 12m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 40s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 18s{color} | {color:orange} root: The patch generated 3 new + 188 unchanged - 4 fixed = 191 total (was 192) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 33s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 45s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 35s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}200m 4s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestPread | | | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | | | hadoop.hdfs.TestBalancerBandwidth | | | hadoop.hdfs.server.namenode.TestDecommissioningStatus | | | hadoop.hdfs.TestParallelShortCircuitReadNoChecksum | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestLeaseRecoveryStriped | | | hadoop.hdfs.TestParallelUnixDomainRead | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | | Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile | \\ \\ || Subsystem || Report/Notes || | Docker |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171294#comment-16171294 ] Mukul Kumar Singh commented on HDFS-11968: -- Thanks for the review [~surendrasingh], 1) ViewFileSystem#resolveStoragePolicyPath() why this method is required ?. fsState.resolve already throwing FileNotFoundException. Only thing is proper exception message is missing in fsState.resolve. bq. Fixed by modifying the message in fsState.resolve. 2) What is realId in BlockStoragePolicySpi? bq. As you have already noted, this was added to get the local policy id. With ViewFilesystem we cannot use the HdfsFileStatus. Hence I thought of adding another field to fetch the local policy id of the command. This also avoid two rpc calls one for a) getting the filestatus and b) for getting the list of all the storage policies for a namenode. I have updated the patch for 1) Please have a look and let me know of your comments. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch, HDFS-11968.007.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171202#comment-16171202 ] Surendra Singh Lilhore commented on HDFS-11968: --- Thanks [~msingh] for patch. I have some question for your patch. 1. {{ViewFileSystem#resolveStoragePolicyPath()}} why this method is required ?. {{fsState.resolve}} already throwing {{FileNotFoundException}}. Only thing is proper exception message is missing in {{fsState.resolve}}. 2. What is realId in BlockStoragePolicySpi? {code} + /** + * Return the real id of the storage policy. + * + * @return the real id of the storage policy. + */ + byte getRealId(); {code} If it is local policy id for a file/directory then no need to add here. {{BlockStoragePolicySpi}} define one policy and its information. I think you are trying to get the file/directory local policy ID for {{StoragePolicyAdmin#GetStoragePolicyCommand}} command, which is currently {{HdfsFileStatus#getStoragePolicy()}} giving. One way to get it is {{HDFS-11298}}, but need to think we can add policyid in FileStatus or not. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169529#comment-16169529 ] Mukul Kumar Singh commented on HDFS-11968: -- Thanks for the awesome suggestion [~surendrasingh], I have address the review comment in the latest patch. I have also added 2 tests as well, one for Viewfs and another for WebHdfs. We already have one test for HDFS. This will help in extending storage policy admin commands to all the supported filesystem. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch, HDFS-11968.005.patch, > HDFS-11968.006.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169393#comment-16169393 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 8s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 16s{color} | {color:orange} root: The patch generated 5 new + 217 unchanged - 3 fixed = 222 total (was 220) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 3s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 33s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 40s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 39s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}200m 48s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | | | hadoop.hdfs.server.namenode.TestReencryptionWithKMS | | | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | | | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | | | hadoop.hdfs.TestLeaseRecoveryStriped | | | hadoop.hdfs.server.namenode.TestNamenodeRetryCache | | Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12887552/HDFS-11968.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16155227#comment-16155227 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 41s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 15s{color} | {color:orange} root: The patch generated 8 new + 187 unchanged - 2 fixed = 195 total (was 189) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 10s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 17s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}187m 0s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.fs.shell.TestCopyFromLocal | | | hadoop.metrics2.sink.TestFileSink | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | | | hadoop.hdfs.TestClientProtocolForPipelineRecovery | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | | | hadoop.hdfs.TestEncryptedTransfer | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 | | | hadoop.hdfs.TestHdfsAdmin | | | hadoop.hdfs.TestApplyingStoragePolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | | | hadoop.hdfs.TestLeaseRecoveryStriped | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144821#comment-16144821 ] Surendra Singh Lilhore commented on HDFS-11968: --- Hi [~msingh], Thanks for the patch. I have one suggestion. HADOOP-13272 implemented storage policy APIs in ViewFs. I think you can remove DFS check and directly send the call to give filesystem. If in give filesystem this APIs are not implemented then command will get UnsupportedOperationException. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch, HDFS-11968.004.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143632#comment-16143632 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 14 unchanged - 0 fixed = 19 total (was 14) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 17s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}129m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | | | hadoop.hdfs.TestClientProtocolForPipelineRecovery | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | | | hadoop.hdfs.TestFileAppendRestart | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration | | | hadoop.hdfs.TestDFSInputStream | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | | | hadoop.hdfs.TestLeaseRecoveryStriped | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.TestFileCreationEmpty | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 | | Timed out junit tests | org.apache.hadoop.hdfs.TestWriteReadStripedFile | | | org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding | \\ \\ || Subsystem || Report/Notes || | Docker |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142102#comment-16142102 ] Arpit Agarwal commented on HDFS-11968: -- Thanks for reporting this issue [~msingh]. A few comments: # I don't think listing policies from all namespaces in getStoragePolicy is the right behavior. Policies with the same name may have different meanings in different clusters. It's okay to print an error message for viewfs and maybe list all available DistributedFileSystems so the administrator can query each. # Can you please add javadocs to getResolvedPath to summarize the behavior? Also perhaps it should be renamed to getResolvedDfsPath. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, > HDFS-11968.003.patch > > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138312#comment-16138312 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 54s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 14 unchanged - 0 fixed = 17 total (was 14) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 57s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 91m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12883320/HDFS-11968.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 85ba2539c450 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4249172 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20822/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20822/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20822/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20822/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > ViewFS: StoragePolicies commands fail with HDFS federation >
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138239#comment-16138239 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 14 unchanged - 0 fixed = 20 total (was 14) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 30s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}112m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12883299/HDFS-11968.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 0831602413a0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4249172 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20819/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20819/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20819/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20819/console | |
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136647#comment-16136647 ] Hadoop QA commented on HDFS-11968: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 12 new + 14 unchanged - 0 fixed = 26 total (was 14) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}103m 45s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11968 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12883086/HDFS-11968.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e53bd04faafe 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d5ff57a | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20801/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20801/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key:
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047113#comment-16047113 ] Manoj Govindassamy commented on HDFS-11968: --- [~msingh], As you pointed out, the storage policy command {{StoragePolicyAdmin}} assumes the path is always DFS and bails out immediately on seeing non-dfs paths. Also, ViewFs only has partial support for storage policy commands. It supports get and set of storage policies by invoking the command on the target filesystem based on the path. But, I don't see it supporting the listing of all storage policies. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046472#comment-16046472 ] Mukul Kumar Singh commented on HDFS-11968: -- Hi [~brahmareddy], Thanks for pointing me to HDFS-11177. As you have pointed out, this bug is support storage policies command with HDFS federation. This bug will add support to resolve Viewfs based paths to HDFS ones and then apply storage policy commands on them. > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.7.1 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046441#comment-16046441 ] Brahma Reddy Battula commented on HDFS-11968: - [~msingh] thanks for reporting. which version..? is it failing even after HDFS-11177..? > ViewFS: StoragePolicies commands fail with HDFS federation > -- > > Key: HDFS-11968 > URL: https://issues.apache.org/jira/browse/HDFS-11968 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > > hdfs storagepolicies command fails with HDFS federation. > For storage policies commands, a given user path should be resolved to a HDFS > path and > storage policy command should be applied onto the resolved HDFS path. > {code} > static DistributedFileSystem getDFS(Configuration conf) > throws IOException { > FileSystem fs = FileSystem.get(conf); > if (!(fs instanceof DistributedFileSystem)) { > throw new IllegalArgumentException("FileSystem " + fs.getUri() + > " is not an HDFS file system"); > } > return (DistributedFileSystem)fs; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org