[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-9631: -- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 2.8.1 2.7.4 2.9.0 Status: Resolved (was: Patch Available) Committed to above mentioned branches. Thanks Lohit and Erik for the contribution! > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu >Assignee: Erik Krogen > Labels: BB2015-05-TBR > Fix For: 2.9.0, 2.7.4, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-9631.005.patch, HADOOP-9631.006.patch, > HADOOP-9631.007.patch, HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-9631: -- Hadoop Flags: Reviewed Target Version/s: 2.9.0, 2.7.4, 2.8.1, 3.0.0-alpha3 Thanks [~xkrogen] for the update! +1 on v7 patch. I just committed to trunk, working on backports. I think this should go into branch-2, branch-2.8, and branch-2.7. > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu >Assignee: Erik Krogen > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.005.patch, HADOOP-9631.006.patch, > HADOOP-9631.007.patch, HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-9631: Attachment: HADOOP-9631.007.patch Zhe's comments addressed in v007 patch > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu >Assignee: Erik Krogen > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.005.patch, HADOOP-9631.006.patch, > HADOOP-9631.007.patch, HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, > HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[
https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HADOOP-9631:
Attachment: HADOOP-9631.006.patch
Forgot a try/catch... My mistake.
One note - to maintain backwards compatibility, if you call
{{ViewFs#getServerDefaults()}} (with no argument) or call it on one of the
internal directories, you just get the {{LocalConfigKeys.getServerDefaults()}}
(maintaining existing behavior). This is in contrast to {{ViewFileSystem}}'s
behavior in which either of these calls results in a
{{NotInMountpointException}}.
> ViewFs should use underlying FileSystem's server side defaults
> --
>
> Key: HADOOP-9631
> URL: https://issues.apache.org/jira/browse/HADOOP-9631
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, viewfs
>Affects Versions: 2.0.4-alpha
>Reporter: Lohit Vijayarenu
>Assignee: Erik Krogen
> Labels: BB2015-05-TBR
> Attachments: HADOOP-9631.005.patch, HADOOP-9631.006.patch,
> HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch,
> HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java
>
>
> On a cluster with ViewFS as default FileSystem, creating files using
> FileContext will always result with replication factor of 1, instead of
> underlying filesystem default (like HDFS)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated HADOOP-9631: Attachment: HADOOP-9631.005.patch Resuming this work; attaching a v005 patch incorporating [~sanjay.radia]'s suggestions. Added a unit test to verify. > ViewFs should use underlying FileSystem's server side defaults > -- > > Key: HADOOP-9631 > URL: https://issues.apache.org/jira/browse/HADOOP-9631 > Project: Hadoop Common > Issue Type: Bug > Components: fs, viewfs >Affects Versions: 2.0.4-alpha >Reporter: Lohit Vijayarenu >Assignee: Erik Krogen > Labels: BB2015-05-TBR > Attachments: HADOOP-9631.005.patch, HADOOP-9631.trunk.1.patch, > HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, > HADOOP-9631.trunk.4.patch, TestFileContext.java > > > On a cluster with ViewFS as default FileSystem, creating files using > FileContext will always result with replication factor of 1, instead of > underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-9631: - Labels: BB2015-05-TBR (was: ) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Labels: BB2015-05-TBR Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Open (was: Patch Available) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Patch Available (was: Open) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Attachment: HADOOP-9631.trunk.4.patch Patch to fix javadoc warning. org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks failure does not look to be related to this patch. I see this failure for other patches too. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, HADOOP-9631.trunk.4.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Attachment: HADOOP-9631.trunk.3.patch Another try at this patch. Problem I was facing was passing down getServerDefaults() call from InternalViewOfDir class. So, in ViewFs::getServerDeaults() did a match of mountpoints and if there exists a matching mountpoint return the filesystem, else return LocalConfig as earlier. This was needed so that we can handle case of FileNotFound, invalid dir/file creation and such. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Patch Available (was: Open) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, HADOOP-9631.trunk.3.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Open (was: Patch Available) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Attachment: HADOOP-9631.trunk.1.patch Attached is patch which deprecates getServerDefaults() and adds getServerDefaults(Path). Tested this by deploying on one of our YARN cluster and could see App logs getting created with replication factor of 3 instead of 1. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Patch Available (was: Open) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Open (was: Patch Available) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Status: Patch Available (was: Open) ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[ https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu updated HADOOP-9631: - Attachment: HADOOP-9631.trunk.2.patch While trying to see why testcases failed, realized that there is easy way to this. In the end each filesystem has ServerDefaults, so in viewfs we just had to pick right filesystem and pass it down. Made a change to use homedirectory path and chose underlying filesystem. Attaching new patch with this change. Now all viewfs tests as well as new test works. ViewFs should use underlying FileSystem's server side defaults -- Key: HADOOP-9631 URL: https://issues.apache.org/jira/browse/HADOOP-9631 Project: Hadoop Common Issue Type: Bug Components: fs, viewfs Affects Versions: 2.0.4-alpha Reporter: Lohit Vijayarenu Attachments: HADOOP-9631.trunk.1.patch, HADOOP-9631.trunk.2.patch, TestFileContext.java On a cluster with ViewFS as default FileSystem, creating files using FileContext will always result with replication factor of 1, instead of underlying filesystem default (like HDFS) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9631) ViewFs should use underlying FileSystem's server side defaults
[
https://issues.apache.org/jira/browse/HADOOP-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lohit Vijayarenu updated HADOOP-9631:
-
Attachment: TestFileContext.java
We see a case where NodeManager Log Aggregator service uses FileContext to move
container log from local filesystem to HDFS. If default filesystem on HDFS
cluster is set to viewfs:/// then FileContext would internally end up using
ViewFs. While doing so, any files created via FileContext would be created with
replication factor of 1. This is because ViewFs getServerDefaults seems to
return local fileystem defaults
{noformat}
@Override
public FsServerDefaults getServerDefaults() throws IOException {
return LocalConfigKeys.getServerDefaults();
}
{noformat}
This would become problem for anyone using new FileContext API running on top
of a federated namespace.
Note that this is not problem with FileSystem APIs. Attached is a test program
which creates file using both APIs on HDFS and doing ls on that file shows both
have different replication factor. File created using FileContext has
replication factor of 1, while file created using FileSystem has replication
factor of 3.
ViewFs should use underlying FileSystem's server side defaults
--
Key: HADOOP-9631
URL: https://issues.apache.org/jira/browse/HADOOP-9631
Project: Hadoop Common
Issue Type: Bug
Components: fs, viewfs
Affects Versions: 2.0.4-alpha
Reporter: Lohit Vijayarenu
Attachments: TestFileContext.java
On a cluster with ViewFS as default FileSystem, creating files using
FileContext will always result with replication factor of 1, instead of
underlying filesystem default (like HDFS)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
