[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937785#comment-15937785 ] Hudson commented on HADOOP-9631: SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11443 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11443/]) HADOOP-9631. ViewFs should use underlying FileSystem's server side (zhz: rev 59d69257a888347f0fb9c51bb000afc986b64f98) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/local/RawLocalFs.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFs.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/fs/Hdfs.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFs.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFsBaseTest.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ftp/FtpFs.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFs.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestAfsCheckPath.java > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu >Assignee: Erik Krogen > Labels: BB2015-05-TBR > Fix For: 2.9.0, 2.7.4, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-9631.005.patch, HADOOP-9631.006.patch, > HADOOP-9631.007.patch, HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937605#comment-15937605 ] Hadoop QA commented on HADOOP-9631: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 42s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 46s{color} | {color:green} root generated 0 new + 777 unchanged - 1 fixed = 777 total (was 778) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}109m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-9631 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12860051/HADOOP-9631.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux f73fff65cb25 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f462e1f | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/11887/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11887/testReport/ | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11887/console | | Powered by | Apache Yetus
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937454#comment-15937454 ] Zhe Zhang commented on HADOOP-9631: --- Thanks [~xkrogen] for the work! v6 patch LGTM, with a couple of minor comments: # {{ViewFs#getServerDefaults}} has unnecessary exceptions in signature # Can we enhance the test to cover the case where {{ViewFs#getServerDefaults(f)}} where f is an internal dir? > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu >Assignee: Erik Krogen > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.005.patch, HADOOP-9631.006.patch, > HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937448#comment-15937448 ] Hadoop QA commented on HADOOP-9631: --- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 57s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 35s{color} | {color:green} root generated 0 new + 777 unchanged - 1 fixed = 777 total (was 778) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 25s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 14s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}108m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-9631 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12860026/HADOOP-9631.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 641114bbca38 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f462e1f | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/11884/testReport/ | | modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client U: . | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/11884/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > ViewFs should use underlying FileSystem's server side defaults >
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936893#comment-15936893 ] Hadoop QA commented on HADOOP-9631: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 54s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 54s{color} | {color:red} root generated 1 new + 777 unchanged - 1 fixed = 778 total (was 778) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 5s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}111m 27s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | | | hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs | | | hadoop.fs.viewfs.TestViewFsLocalFs | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | HADOOP-9631 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12859959/HADOOP-9631.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 534a3124848e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f462e1f | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/11879/artifact/patchprocess/diff-compile-javac-root.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/11879/artifact/patchprocess/whitespace-eol.txt | | unit |
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362107#comment-15362107 ] Yuming Wang commented on HADOOP-9631: - [~lohit], Can you please help me upload the [latest patch|https://drive.google.com/open?id=0BxL8Kzbd2N--QW5VaU9wX1o4Z0k], I'm not authorized. > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360192#comment-15360192 ] Hadoop QA commented on HADOOP-9631: --- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color} | {color:red} HADOOP-9631 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12588630/HADOOP-9631.trunk.4.patch | | JIRA Issue | HADOOP-9631 | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9920/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360190#comment-15360190 ] Yuming Wang commented on HADOOP-9631: - The test failures still exist? > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524720#comment-14524720 ] Hadoop QA commented on HADOOP-9631: --- \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | patch | 0m 0s | The patch command could not apply the patch during dryrun. | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12588630/HADOOP-9631.trunk.4.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / f1a152c | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6316/console | This message was automatically generated. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13790714#comment-13790714 ] Sanjay Radia commented on HADOOP-9631: -- Here are two early comments (haven't finished reviewing the whole patch). * viewfs#getServerDefaults(path) can be simplified. See how open or list are implemented and it take advantage of the internal class InternalDirOfViewOf. Something like this should work: {code} viewfs#getServerDefaults(f) { InodeTree.ResolveResultAbstractFileSystem res = fsState.resolve(getUriPath(f), true); return res.targetFileSystem.getServerDefailts(res.remainingPath); } InternalDirOfViewFs#getServerDefaults() { return LocalConfigKeys.getServerDefaults(); } InternalDirOfViewFs#getServerDefaults(f) { checkPathIsSlash(f); return LocalConfigKeys.getServerDefaults(); } {code} * FIleSystem#getServerDefaults(f) is incorrect due to getDefaultReplication(). It should use getDefaultReplciation(f). Hence move the code for FIleSystem#getServerDefaults() to FIleSystem#getServerDefaults(f), changing the getDefaultReplication to pass the pathname f. Have FIleSystem#getServerDefaults() call FIleSystem#getServerDefaults(/); ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770111#comment-13770111 ] Lohit Vijayarenu commented on HADOOP-9631: -- [~cnauroth] Can you please help review latest patch ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13688247#comment-13688247 ] Hadoop QA commented on HADOOP-9631: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588630/HADOOP-9631.trunk.4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2675//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2675//console This message is automatically generated. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687530#comment-13687530 ] Hadoop QA commented on HADOOP-9631: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12588487/HADOOP-9631.trunk.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2670//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2670//console This message is automatically generated. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13686409#comment-13686409 ] Lohit Vijayarenu commented on HADOOP-9631: -- Taking a look at how ViewFs mountable is constructed, it looks there is no straightforward way to get TargetFileSystem given mount point or path. I started with changing getServerDefaults to accept Path. Now, create call would pass a Path which does not exists. So, we will have to resolve the Path to mountpoint. I see that ViewFileSystem::resolve can fetch the mountpoint, but not an easy way to fetch underlying FileSystem. '/' seem to always resolve to InteralViewOfDir which does not have targetFileSytem. Any suggestions of easy way to fetch this? ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683523#comment-13683523 ] Lohit Vijayarenu commented on HADOOP-9631: -- Can anyone please review this. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683623#comment-13683623 ] Chris Nauroth commented on HADOOP-9631: --- Hi Lohit, I'm looking at version 2 of the patch, which always resolves to the home directory to get server defaults. This might appear to be working for the specific use case you need (YARN NodeManager log aggregation), but I don't think it would be generally correct in all cases. For example, imagine a viewfs configured with mount points mapping /user to local file system and /data to an HDFS cluster. Then, creating a file in /data would still get the undesired replication value. That example is a bit contrived, but we can also consider the more realistic example from the {{ViewFs}} javadocs: {code} * For example one could have a mount table that provides links such as * ul * li /user - hdfs://nnContainingUserDir/user * li /project/foo - hdfs://nnProject1/projects/foo * li /project/bar - hdfs://nnProject2/projects/bar * li /tmp - hdfs://nnTmp/privateTmpForUserXXX * /ul {code} Now, assume the client is creating file /project/foo/myFile. The operation would pick up server defaults associated with /user (configured on nnContainingUserDir) instead of the desired /project/foo (configured on nnProject1). The server defaults may differ between nnContainingUserDir and nnProject1. It seems like we need a patch that is sensitive to the actual {{Path}} accessed in the client operation, similar to what Daryn mentioned earlier. Also, I don't think the new unit test exercises this code, because {{TestHDFSFileContextMainOperations}} doesn't configure a {{ViewFs}}. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683631#comment-13683631 ] Lohit Vijayarenu commented on HADOOP-9631: -- [~cnauroth] Thanks for the review. I see the problem where viewfs might be mapped to different filesystems. Let me revisit my initial patch which was trying to use Path to resolve to FileSystem mountpoint and update JIRA. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680988#comment-13680988 ] Hadoop QA commented on HADOOP-9631: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587367/HADOOP-9631.trunk.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2639//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2639//console This message is automatically generated. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13680695#comment-13680695 ] Hadoop QA commented on HADOOP-9631: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12587250/HADOOP-9631.trunk.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.viewfs.TestViewFsWithAuthorityLocalFs org.apache.hadoop.fs.viewfs.TestViewFsLocalFs org.apache.hadoop.fs.viewfs.TestViewFsHdfs org.apache.hadoop.fs.viewfs.TestViewFsAtHdfsRoot {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2632//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2632//console This message is automatically generated. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira