[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[
https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749011#comment-15749011
]
Steve Loughran commented on HADOOP-13863:
-
* Needs some docs, ideally a section in
{{hadoop-tools/hadoop-azure/src/site/markdown/index.md}}
* default values _MAY_ go into
{{hadoop-common-project/hadoop-common/src/main/resources/core-default.xml}}
* I'd like to have all exceptions subclass {{AzureException}} unless there's a
particularly compelling reason why not
I ran all the tests for this locally, they did complete but it did take a long
time. I should check to see if there's any different when run without this
patch.
{code}
Results :
Tests run: 646, Failures: 0, Errors: 0, Skipped: 8
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 29:23 min
[INFO] Finished at: 2016-12-14T17:58:13+00:00
{code}
> Hadoop - Azure: Add a new SAS key mode for WASB.
>
>
> Key: HADOOP-13863
> URL: https://issues.apache.org/jira/browse/HADOOP-13863
> Project: Hadoop Common
> Issue Type: Improvement
> Components: azure, fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, WASB-SAS
> Key Mode-Design Proposal.pdf
>
>
> Current implementation of WASB, only supports Azure storage keys and SAS key
> being provided via org.apache.hadoop.conf.Configuration, which results in
> these secrets residing in the same address space as the WASB process and
> providing complete access to the Azure storage account and its containers.
> Added to the fact that WASB does not inherently support ACL's, WASB is its
> current implementation cannot be securely used for environments like secure
> hadoop cluster. This JIRA is created to add a new mode in WASB, which
> operates on Azure Storage SAS keys, which can provide fine grained timed
> access to containers and blobs, providing a segway into supporting WASB for
> secure hadoop cluster.
> More details about the issue and the proposal are provided in the design
> proposal document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[
https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748885#comment-15748885
]
Steve Loughran commented on HADOOP-13863:
-
Any reason why {{SASKeyGenerationException extends Exception}} and {{
SASKeyModeNonSupportedException extends RuntimeException}}?
Generally we like everything to extend IOException, as that tends to get
relayed around the most
> Hadoop - Azure: Add a new SAS key mode for WASB.
>
>
> Key: HADOOP-13863
> URL: https://issues.apache.org/jira/browse/HADOOP-13863
> Project: Hadoop Common
> Issue Type: Improvement
> Components: azure, fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, WASB-SAS
> Key Mode-Design Proposal.pdf
>
>
> Current implementation of WASB, only supports Azure storage keys and SAS key
> being provided via org.apache.hadoop.conf.Configuration, which results in
> these secrets residing in the same address space as the WASB process and
> providing complete access to the Azure storage account and its containers.
> Added to the fact that WASB does not inherently support ACL's, WASB is its
> current implementation cannot be securely used for environments like secure
> hadoop cluster. This JIRA is created to add a new mode in WASB, which
> operates on Azure Storage SAS keys, which can provide fine grained timed
> access to containers and blobs, providing a segway into supporting WASB for
> secure hadoop cluster.
> More details about the issue and the proposal are provided in the design
> proposal document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743543#comment-15743543 ] Mingliang Liu commented on HADOOP-13863: Good, thanks [~dchickabasapa] for updating the patch. I'll review this code this week (or before the holiday) if no reviews from others. > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, WASB-SAS > Key Mode-Design Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[
https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733604#comment-15733604
]
Mingliang Liu commented on HADOOP-13863:
{code}
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
[ERROR] COMPILATION ERROR :
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java:[978,53]
cannot find symbol
symbol: class MockStorageInterface
location: class org.apache.hadoop.fs.azure.AzureNativeFileSystemStore
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-azure: Compilation failure
[ERROR]
/Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java:[978,53]
cannot find symbol
[ERROR] symbol: class MockStorageInterface
[ERROR] location: class org.apache.hadoop.fs.azure.AzureNativeFileSystemStore
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hadoop-azure
{code}
> Hadoop - Azure: Add a new SAS key mode for WASB.
>
>
> Key: HADOOP-13863
> URL: https://issues.apache.org/jira/browse/HADOOP-13863
> Project: Hadoop Common
> Issue Type: Improvement
> Components: azure, fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13863.001.patch, WASB-SAS Key Mode-Design
> Proposal.pdf
>
>
> Current implementation of WASB, only supports Azure storage keys and SAS key
> being provided via org.apache.hadoop.conf.Configuration, which results in
> these secrets residing in the same address space as the WASB process and
> providing complete access to the Azure storage account and its containers.
> Added to the fact that WASB does not inherently support ACL's, WASB is its
> current implementation cannot be securely used for environments like secure
> hadoop cluster. This JIRA is created to add a new mode in WASB, which
> operates on Azure Storage SAS keys, which can provide fine grained timed
> access to containers and blobs, providing a segway into supporting WASB for
> secure hadoop cluster.
> More details about the issue and the proposal are provided in the design
> proposal document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[
https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733601#comment-15733601
]
Mingliang Liu commented on HADOOP-13863:
This patch does not apply.
{code}
978 if (!(this.storageInteractionLayer instanceof
MockStorageInterface) && useSasKeyMode) {
979 connectToAzureStorageInSASKeyMode(accountName, containerName,
sessionUri);
980 return;
981 }
{code}
Probing the implementation details is not ideal, not to mention it's a mocked
class in tests.
> Hadoop - Azure: Add a new SAS key mode for WASB.
>
>
> Key: HADOOP-13863
> URL: https://issues.apache.org/jira/browse/HADOOP-13863
> Project: Hadoop Common
> Issue Type: Improvement
> Components: azure, fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13863.001.patch, WASB-SAS Key Mode-Design
> Proposal.pdf
>
>
> Current implementation of WASB, only supports Azure storage keys and SAS key
> being provided via org.apache.hadoop.conf.Configuration, which results in
> these secrets residing in the same address space as the WASB process and
> providing complete access to the Azure storage account and its containers.
> Added to the fact that WASB does not inherently support ACL's, WASB is its
> current implementation cannot be securely used for environments like secure
> hadoop cluster. This JIRA is created to add a new mode in WASB, which
> operates on Azure Storage SAS keys, which can provide fine grained timed
> access to containers and blobs, providing a segway into supporting WASB for
> secure hadoop cluster.
> More details about the issue and the proposal are provided in the design
> proposal document.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15721980#comment-15721980 ] Steve Loughran commented on HADOOP-13863: - + [~lmccay] > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, WASB-SAS Key Mode-Design > Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720581#comment-15720581 ] Dushyanth commented on HADOOP-13863: Thanks [~ste...@apache.org] for the review of the draft. To answer your questions: 1) https://docs.microsoft.com/en-us/azure/storage/storage-dotnet-shared-access-signature-part-1 2) The layer is going to be a layer with auth support, as well as logic to support renewel of the SAS keys. Hence it made more sense to have a separate implementation rather than polluting the current StorageInterfaceImpl implementation. We expect the feature to be added iteratively, the first iteration is expected to be introduction of the SAS key mode with just copy-paste and get an initial read on the approach, and then the plan is to add the renewel process. 3) Yes, the local SAS key mode will allow to desk the SAS key mode of execution on anyone's desktop. The HttpClient mode, will have to be tested is specific setups. > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: WASB-SAS Key Mode-Design Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15719783#comment-15719783 ] Steve Loughran commented on HADOOP-13863: - # any link for more info on SAS keys? # this new storage interface: is it going to be a think layer of extra auth, or a copy & paste of the existing code? Because C is something to absolutely avoid...if it can't be done without some changes for better extensibility in the NativeAzureStorage classes, I'd prefer adding those extension points. # testing. The local one works with today's keys, so can be run on anyone's desktop? If so, that's essential. It does mean that the other codepath, the httpclient stuff, is going to have to be tested in the release process by volunteers with access to that feature S3 has a similar problem there and its IAM credential auth on EC2 machines. We trust AWS to test their HTTP client, but even there, in HADOOP-13727, we had to do some patching to deal with how IAM/AWS throttled clients —the kind of problem which we didn't see during our integration tests, even running in-EC2. It'd be good to make sure that the Azure client avoids the same problem > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: WASB-SAS Key Mode-Design Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
