[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17251164#comment-17251164
]
Steve Loughran commented on HADOOP-15691:
-
Sorry, missed that I'd pulled this back as part of some of the dir marker work.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.2.3
>
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> This complements the {{StreamCapabilities}} interface by allowing
> applications
> to probe for a specific path on a specific instance of a {{FileSystem}}
> or {{FileContext}} offering a specific feature.
> This is intended to allow applications to determine
> * Whether a method is implemented before calling it and dealing with
> any subsequent UnsupportedOperationException.
> * Whether a specific feature is believed to be available in the remote store.
> As well as a common set of capabilities defined in CommonPathCapabilities,
> file systems are free to add their own capabilities, prefixed with
> fs. + schema + .
>
> The plan is to identify and document more capabilities -and for file systems
> which add new features, for a declaration of the availability of the feature
> to
> always be available.
> The interface may be offered by other classes too; there is no restriction
> here.
> Note
> * The remote store is not expected to be checked for the feature;
> It is more a check of client API and the client's configuration/knowledge
> of the state of the remote system.
> * Permissions are not checked.
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938507#comment-16938507
]
Steve Loughran commented on HADOOP-15691:
-
Apart from its breadth, I'd like to backport this to 3.2.x; though I'd do the
new open() call first.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> This complements the {{StreamCapabilities}} interface by allowing
> applications
> to probe for a specific path on a specific instance of a {{FileSystem}}
> or {{FileContext}} offering a specific feature.
> This is intended to allow applications to determine
> * Whether a method is implemented before calling it and dealing with
> any subsequent UnsupportedOperationException.
> * Whether a specific feature is believed to be available in the remote store.
> As well as a common set of capabilities defined in CommonPathCapabilities,
> file systems are free to add their own capabilities, prefixed with
> fs. + schema + .
>
> The plan is to identify and document more capabilities -and for file systems
> which add new features, for a declaration of the availability of the feature
> to
> always be available.
> The interface may be offered by other classes too; there is no restriction
> here.
> Note
> * The remote store is not expected to be checked for the feature;
> It is more a check of client API and the client's configuration/knowledge
> of the state of the remote system.
> * Permissions are not checked.
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937719#comment-16937719
]
Adam Antal commented on HADOOP-15691:
-
Thanks for working on this [~ste...@apache.org]!
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> This complements the {{StreamCapabilities}} interface by allowing
> applications
> to probe for a specific path on a specific instance of a {{FileSystem}}
> or {{FileContext}} offering a specific feature.
> This is intended to allow applications to determine
> * Whether a method is implemented before calling it and dealing with
> any subsequent UnsupportedOperationException.
> * Whether a specific feature is believed to be available in the remote store.
> As well as a common set of capabilities defined in CommonPathCapabilities,
> file systems are free to add their own capabilities, prefixed with
> fs. + schema + .
>
> The plan is to identify and document more capabilities -and for file systems
> which add new features, for a declaration of the availability of the feature
> to
> always be available.
> The interface may be offered by other classes too; there is no restriction
> here.
> Note
> * The remote store is not expected to be checked for the feature;
> It is more a check of client API and the client's configuration/knowledge
> of the state of the remote system.
> * Permissions are not checked.
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937651#comment-16937651 ] Hudson commented on HADOOP-15691: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17384 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17384/]) HADOOP-15691 Add PathCapabilities to FileSystem and FileContext. (stevel: rev e346e3638c595a512cd582739ff51fb64c3b4950) * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystem.java * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonPathCapabilities.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/index.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/http/AbstractHttpFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFs.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractConcatTest.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/DelegateToFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractAppendTest.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileContext.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ChRootedFileSystem.java * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/FsLinkResolution.java * (edit) hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/ContractTestUtils.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitConstants.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java * (add) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/pathcapabilities.md * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/PathCapabilitiesSupport.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/select/SelectConstants.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/DfsPathCapabilities.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ABlockOutputStream.java * (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/PathCapabilities.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/select/SelectTool.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/filesystem.md * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/HttpFSFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/select/ITestS3Select.java > Add PathCapabilities to FS and FC to complement StreamCapabilities > -- > > Key: HADOOP-15691 > URL: https://issues.apache.org/jira/browse/HADOOP-15691 > Project: Hadoop Common > Issue Type: New Feature >
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934199#comment-16934199
]
Aaron Fabbri commented on HADOOP-15691:
---
+1 on latest pull request pending clean precommit result. This has been long
overdue, thanks for the contribution and thanks for reviewing [~adam.antal]
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of a filesystem through both the
> FileSystem and FileContext APIs
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882108#comment-16882108
]
Adam Antal commented on HADOOP-15691:
-
Hi [~ste...@apache.org], is there any update on this?
Would be happy to see this committed for YARN-9607 and YARN-9525.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of a filesystem through both the
> FileSystem and FileContext APIs
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868449#comment-16868449
]
Adam Antal commented on HADOOP-15691:
-
Reviewed and had some questions commented on the PR.
Great job so far [~ste...@apache.org].
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of a filesystem through both the
> FileSystem and FileContext APIs
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868329#comment-16868329
]
Hadoop QA commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-15691 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15691 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954452/HADOOP-15691-004.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16335/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch, HADOOP-15691-004.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of a filesystem through both the
> FileSystem and FileContext APIs
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739714#comment-16739714
]
Hadoop QA commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
3s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 37s{color} | {color:orange} root: The patch generated 8 new + 520 unchanged
- 1 fixed = 528 total (was 521) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 22s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
47s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
24s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
32s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
57s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.ssl.TestSSLFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707821#comment-16707821
]
Hadoop QA commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HADOOP-15691 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-15691 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12940787/HADOOP-15691-003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15600/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-14707: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623885#comment-16623885
]
Hadoop QA commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 30s{color} | {color:orange} root: The patch generated 4 new + 525 unchanged
- 0 fixed = 529 total (was 525) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 12 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
58s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
40s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
58s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623684#comment-16623684
]
Steve Loughran commented on HADOOP-15691:
-
Patch 003
* revise based on feedback
* catch up with trunk (pathhandles)
* add option for an FS connector to say "We know we are read only" (http, har)
* add checksum capability, wire up so those filesystems with it (DFS, S3A when
its enabled)
* symlink capability is based on {{symlinksEnabled}} flag in FileSystem class.
* expanded docs
* ~0 tests
I now realise that a bool "true/false" isn't quite right, as it doesn't cover
the "we think we'd support it but you'd have to try". Symlinks being a key
example. There's DFS: we know we have it, other stores "we know we don't", but
localFS is either "known disabled" or "try it and see"
An enum seems the outcome here, something like
{code}
known-available: its a known feature and known to work in this instance (ex:
symlinks on hdfs)
known-unavailable: we know of it, we know we don't support it (ex: append on
s3a)
known-maybe-available: we know of it, but to see if it works, you will have to
try the operation. (symlink on local)
unknown: we don't know of this feature.
{code}
the base FS would say "unknown feature" for ~everything; implementations would
list what they know about and their interpretation, so that everything with
rename would add "fs:rename"; posix-ish filesytems with atomic rename,
"fs-rename-atomic"; the object stores would mark the latter as
"known-unavailable", etc.
We'd also need to qualify everything with "depending on your permissions". An
FS may know of append, but it might be a read-only mount. I've already added to
filesystem.md the fact that the state may change depending on external events.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch,
> HADOOP-15691-003.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623450#comment-16623450
]
Steve Loughran commented on HADOOP-15691:
-
bq. Should we document a well defined capabilities? For example, we can say in
Hadoop 3.3, we define the following list of path capabilities (like
fs:snapshots) and file system implementations should support probing of these
capabilities. Right now these capabilities exist in the code, not in user docs.
bq. Let's leave to the javadocs, so that its more likely to be in sync with the
code. We can add a ref to it though in the FS docs.
bq. Instead of defining the capabilities in StreamCapabilities.java, why not
create a PathCapabilities class for these capability definitions?
I'd just added them alongside the existing ones, but yes, it makes sense. Will
do this.
bq. Are these capabilities well defined somewhere? I am not a file system guy
so I am not sure if people interpret these capabilities differently.
Not yet, we can be specific in the javadocs
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622516#comment-16622516
]
Wei-Chiu Chuang commented on HADOOP-15691:
--
Thanks Steve for the work. I skimmed through the patch and I think it makes
perfect sense. It'll also make some Hadoop internal implementation much cleaner.
Just a few comments
# Should we document a well defined capabilities? For example, we can say in
Hadoop 3.3, we define the following list of path capabilities (like
fs:snapshots) and file system implementations should support probing of these
capabilities. Right now these capabilities exist in the code, not in user docs.
# Instead of defining the capabilities in StreamCapabilities.java, why not
create a PathCapabilities class for these capability definitions?
# Are these capabilities well defined somewhere? I am not a file system guy so
I am not sure if people interpret these capabilities differently.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596377#comment-16596377
]
Steve Loughran commented on HADOOP-15691:
-
BTW, I've realised that I could expose this not just through a new API, but
simply as a set of config options which you can query through
FileSystem.getConf().getBoolean("fs.capabilities.fs.atomic-rename"). FS
instances would add their list of capabilities in init(), so existing code can
see if a feature is there without needing to call the API.
This'd make it easier for apps like HBase, Tez & Spark to be able to probe for
specific features, e.g. whether they could skip the use of a temp file +
rename. It would complicate subclassed filesystems which changed the
capabilites though, especially those which removed them, and I'd also need to
add some property declaring that the capabilities had been registered, e.g.
"fs.capabilities.registered"
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch, HADOOP-15691-002.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16594178#comment-16594178
]
genericqa commented on HADOOP-15691:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 20s{color} | {color:orange} root: The patch generated 2 new + 454 unchanged
- 0 fixed = 456 total (was 454) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
58s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 2s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
54s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
40s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-azure-datalake in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593934#comment-16593934
]
Steve Loughran commented on HADOOP-15691:
-
Should be of interest to: [~sanjay.radia], [~tmarquardt], [~fabbri], [~ehiggs]
[~thomasdemoor], [~rdblue], [~arpitagarwal] [~jzhuge] [~sseth]
The goal here initially is just to let callers see if things like concat/append
work before trying, but I really want to line things up for being able to ask
if a store can do a direct write to a dest path without it being visible until
closed (so you don't need to write to a temp path & rename for that
one-operation commit), when to know that rename() doesn't do what posix
expects,
I know this breaks the idea that all filesystems should work just like HDFS,
but that is true today: we just need to be able to determine which APIs *and
semantics* are offered by specific FS instances.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities
[
https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593926#comment-16593926
]
Steve Loughran commented on HADOOP-15691:
-
Patch 001: initial implementation. This defines a new interface,
PathCapabiliities, implemented by FileSystem, AbstractFileSystem and
FileContext, to probe for features. It has one method
{code}
boolean hasPathCapability(Path path, String capability) throws IOException;
{code}
* There's a core list of features added in StreamCapabilities (right place?);
for the main optional FS operations & features: permissions, xattrs, storage
policies, snapshots, append, truncate, concat, and wired up for the all built
in filesystems
* No tests; they can go into contract tests (e.g the append test can verify
that having the append capability -> append works, etc)
* Base implementation returns false for everything but the "fs:symlinks" probe,
which is forwarded to {{supportsSymlinks()}}
To do in separate JIRA;
* Object store operations/semantics, where I want to be able to declare when a
store does a PUT at the end of a write (e.g: no rename needed), and to declare
when renames are slow/non-atomic for dirs. We can use that in the mrv2
committers to know when to warn, perhaps.
* Multipart upload availablity
h3. Semantics of return value
w.r/t this patch, the spec says "return true iff you know the feature is
available". This means "false" can mean any of
* The capability is unknown.
* The capability is known, but not available on this instance.
* The capability is known but the connector does not know if it is supported by
this specific instance.
We could think about having an enum of all four states, so that you can
distinguish them, but I fear it gets complicated with forwarded filesystems.
h3. FSLinkResolver
new class {{org.apache.hadoop.fs.FSLinkResolver}} for implementing FileContext
operations as closures, here
{code}
FsLinkResolution.resolve(this,
fixRelativePart(path),
(fs, p) -> fs.hasPathCapability(p, capability));
{code}
It should be straightforward to move all FileContext operations to this & make
the file a lot leaner.
> Add PathCapabilities to FS and FC to complement StreamCapabilities
> --
>
> Key: HADOOP-15691
> URL: https://issues.apache.org/jira/browse/HADOOP-15691
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15691-001.patch
>
>
> Add a {{PathCapabilities}} interface to both FileSystem and FileContext to
> declare the capabilities under the path of an FS
> This is needed for
> * HADOOP-15407: declare that a dest FS supports permissions
> * object stores to declare that they offer PUT-in-place alongside
> (slow-rename)
> * Anything else where the implementation semantics of an FS is so different
> caller apps would benefit from probing for the underlying semantics
> I know, we want all filesystem to work *exactly* the same. But it doesn't
> hold, especially for object stores —and to efficiently use them, callers need
> to be able to ask for specific features.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
