[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dushyanth updated HADOOP-13930:
---
Attachment: HADOOP-13930.002.patch
Adding re-based patch and addressing some of Steve's comment. Will address test
related comments in the next patch.
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15771586#comment-15771586 ] Dushyanth commented on HADOOP-13863: I have created HADOOP-13938 to fix the javadoc errors. > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > HADOOP-13863.003.patch, HADOOP-13863.004.patch, HADOOP-13863.005.patch, > HADOOP-13863.006.patch, HADOOP-13863.007.patch, Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13938) Address javadoc errors in Azure WASB.
Dushyanth created HADOOP-13938: -- Summary: Address javadoc errors in Azure WASB. Key: HADOOP-13938 URL: https://issues.apache.org/jira/browse/HADOOP-13938 Project: Hadoop Common Issue Type: Bug Components: azure, fs/azure Affects Versions: 2.8.0 Reporter: Dushyanth As observed in HADOOP-13863, there are few javadocs error that are thrown while building WASB in QA builds. This JIRA is created to track the fix for these javadoc errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: HADOOP-13863.007.patch [~liuml07] Thanks a lot for the review. I have addressed comments in Patch v7 > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > HADOOP-13863.003.patch, HADOOP-13863.004.patch, HADOOP-13863.005.patch, > HADOOP-13863.006.patch, HADOOP-13863.007.patch, Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dushyanth updated HADOOP-13930:
---
Attachment: HADOOP-13930.001.patch
Attaching first patch. This patch is using WasbRemoteCallHelper class that is
yet to be committed via HADOOP-13863.
The patch has been tested against Live Azure storage tests and also newly added
tests for the feature.
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: azure, fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Dushyanth
> Attachments: HADOOP-13930.001.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13930) Azure: Add Authorization support to WASB
Dushyanth created HADOOP-13930:
--
Summary: Azure: Add Authorization support to WASB
Key: HADOOP-13930
URL: https://issues.apache.org/jira/browse/HADOOP-13930
Project: Hadoop Common
Issue Type: Improvement
Components: azure, fs/azure
Affects Versions: 2.8.0
Reporter: Dushyanth
Assignee: Dushyanth
As highlighted in HADOOP-13863, current implementation of WASB does not support
authorization to any File System operations. This jira is created to add
authorization support for WASB. The current approach is to enforce
authorization via an external REST service (One approach could be to use
component like Ranger to enforce authorization). The support for authorization
would be hiding behind a configuration flag : "fs.azure.enable.authorization"
and the remote service is expected to be provided via config :
"fs.azure.remote.auth.service.url".
The remote service is expected to provide support for the following REST call:
{URL}/CHECK_AUTHORIZATION```
An example request:
{URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type=_token=
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: HADOOP-13863.005.patch Attaching Patch v5 with fixes for stylecheck, findbugs and xml encoding errors. > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > HADOOP-13863.003.patch, HADOOP-13863.004.patch, HADOOP-13863.005.patch, > Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: HADOOP-13863.004.patch Thanks [~ste...@apache.org] for the review. I have addressed your comments in Patch v4. Also the lines have been reduced 80 chars per line. > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > HADOOP-13863.003.patch, HADOOP-13863.004.patch, Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15750027#comment-15750027 ] Dushyanth edited comment on HADOOP-13863 at 12/15/16 1:23 AM: -- Thanks [~ste...@apache.org] for the review. 1) I have added documentation to index.md file for the change. 2) These are azure specific configs name as "fs.zure.*" so not sure if it is a good idean to add them to hadoop-common. 3) I have modified the newly added Exceptions to extend AzureException The new patch also has support for the RemoteSASKeyGenerationImpl. was (Author: dchickabasapa): Thanks [~ste...@apache.org] for the review. 1) I have added documentation to index.md file for the change. 2) These are azure specific configs name as "fs.zure.*" so not sure if it is a good idean to add them to hadoop-common. 3) I have modified the newly added Exceptions to extend AzureException > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > HADOOP-13863.003.patch, Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: HADOOP-13863.003.patch Thanks [~ste...@apache.org] for the review. 1) I have added documentation to index.md file for the change. 2) These are azure specific configs name as "fs.zure.*" so not sure if it is a good idean to add them to hadoop-common. 3) I have modified the newly added Exceptions to extend AzureException > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > HADOOP-13863.003.patch, Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: Proposal-Document.pdf Modified the proposal document with the Remote service requirements > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, > Proposal-Document.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: (was: WASB-SAS Key Mode-Design Proposal.pdf) > Azure: Add a new SAS key mode for WASB. > --- > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: HADOOP-13863.002.patch I apologize, I was not seeing this issue after the initial build was successful. Was able to repro the problem with a clean build. I have fixed the issue, and removed the implementation checks. The check was put in place to not affect Mock tests, but have disabled those tests for SAS mode in the new patch. > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, HADOOP-13863.002.patch, WASB-SAS > Key Mode-Design Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13675: --- Attachment: HADOOP-13675.004.patch Fixed checkstyle warnings from the previous patch. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch, > HADOOP-13675.003.patch, HADOOP-13675.004.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: HADOOP-13863.001.patch First iteration of the support to add SAS key mode for WASB. This patch has following limitations: 1) Supports only LocalSASKey mode. 2) No support for renewal of SAS keys Testing: This patch has been tested for both Non-SAS key mode and SAS-key mode for live azure storage tests. > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-13863.001.patch, WASB-SAS Key Mode-Design > Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15720581#comment-15720581 ] Dushyanth commented on HADOOP-13863: Thanks [~ste...@apache.org] for the review of the draft. To answer your questions: 1) https://docs.microsoft.com/en-us/azure/storage/storage-dotnet-shared-access-signature-part-1 2) The layer is going to be a layer with auth support, as well as logic to support renewel of the SAS keys. Hence it made more sense to have a separate implementation rather than polluting the current StorageInterfaceImpl implementation. We expect the feature to be added iteratively, the first iteration is expected to be introduction of the SAS key mode with just copy-paste and get an initial read on the approach, and then the plan is to add the renewel process. 3) Yes, the local SAS key mode will allow to desk the SAS key mode of execution on anyone's desktop. The HttpClient mode, will have to be tested is specific setups. > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: WASB-SAS Key Mode-Design Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13675: --- Attachment: HADOOP-13675.003.patch Thanks [~liuml07] for the review. I have addressed your comments for the previous patch and attached Patch 3. Testing: The patch contains new test to verify the changes made. Also changes have been tested against live tests for the both Block Blobs and Page Blobs. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch, > HADOOP-13675.003.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Attachment: WASB-SAS Key Mode-Design Proposal.pdf Attaching the first draft of the design proposal to support SAS Key mode in WASB. > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: WASB-SAS Key Mode-Design Proposal.pdf > > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13863: --- Assignee: Dushyanth > Hadoop - Azure: Add a new SAS key mode for WASB. > > > Key: HADOOP-13863 > URL: https://issues.apache.org/jira/browse/HADOOP-13863 > Project: Hadoop Common > Issue Type: Improvement > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > > Current implementation of WASB, only supports Azure storage keys and SAS key > being provided via org.apache.hadoop.conf.Configuration, which results in > these secrets residing in the same address space as the WASB process and > providing complete access to the Azure storage account and its containers. > Added to the fact that WASB does not inherently support ACL's, WASB is its > current implementation cannot be securely used for environments like secure > hadoop cluster. This JIRA is created to add a new mode in WASB, which > operates on Azure Storage SAS keys, which can provide fine grained timed > access to containers and blobs, providing a segway into supporting WASB for > secure hadoop cluster. > More details about the issue and the proposal are provided in the design > proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13863) Hadoop - Azure: Add a new SAS key mode for WASB.
Dushyanth created HADOOP-13863: -- Summary: Hadoop - Azure: Add a new SAS key mode for WASB. Key: HADOOP-13863 URL: https://issues.apache.org/jira/browse/HADOOP-13863 Project: Hadoop Common Issue Type: Improvement Components: azure, fs/azure Affects Versions: 2.8.0 Reporter: Dushyanth Current implementation of WASB, only supports Azure storage keys and SAS key being provided via org.apache.hadoop.conf.Configuration, which results in these secrets residing in the same address space as the WASB process and providing complete access to the Azure storage account and its containers. Added to the fact that WASB does not inherently support ACL's, WASB is its current implementation cannot be securely used for environments like secure hadoop cluster. This JIRA is created to add a new mode in WASB, which operates on Azure Storage SAS keys, which can provide fine grained timed access to containers and blobs, providing a segway into supporting WASB for secure hadoop cluster. More details about the issue and the proposal are provided in the design proposal document. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707688#comment-15707688 ] Dushyanth edited comment on HADOOP-13675 at 11/30/16 6:17 AM: -- [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I have attached Hadoop-13675.002.patch that Jameel had worked on, kindly review the patch. was (Author: dchickabasapa): [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I have attached Hadoop-13675.002.patch that Jameel had worked on. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707688#comment-15707688 ] Dushyanth edited comment on HADOOP-13675 at 11/30/16 6:17 AM: -- [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I have attached Hadoop-13675.002.patch that Jameel had worked on. was (Author: dchickabasapa): [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I am attaching Patch-2 that Jameel had worked on. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13675: --- Attachment: HADOOP-13675.002.patch > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707688#comment-15707688 ] Dushyanth commented on HADOOP-13675: [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I am attaching Patch-2 that Jameel had worked on. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth reassigned HADOOP-13675: -- Assignee: Dushyanth (was: Jameel Naina Mohamed) > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13675: --- Attachment: HADOOP-13675.001.patch Adding first iteration of the patch to fix the return type handling for deletes. Testing: The patch contains new test to verify the changes made. Also changes have been tested against FileSystemContractLive tests for the both Block Blobs and Page Blobs. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13675) Bug in return value for delete() calls in WASB
Dushyanth created HADOOP-13675: -- Summary: Bug in return value for delete() calls in WASB Key: HADOOP-13675 URL: https://issues.apache.org/jira/browse/HADOOP-13675 Project: Hadoop Common Issue Type: Bug Components: azure, fs/azure Affects Versions: 2.8.0 Reporter: Dushyanth Fix For: 2.9.0 Current implementation of WASB does not correctly handle multiple threads/clients calling delete on the same file. The expected behavior in such scenarios is only one of the thread should delete the file and return true, while all other threads should receive false. However in the current implementation even though only one thread deletes the file, multiple clients incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13475) Adding Append Blob support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-13475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422050#comment-15422050 ] Dushyanth commented on HADOOP-13475: [~Raulmsm] Thank you very much for the change. I see that there are lot of diffs in the patch that are getting introduced because of probably different formatter being used. Kindly clean those up. Inside retrieveAppendStream(..) method probably it is a good idea to move the block blob handling into a else statement following the same pattern as other similar methods in the class. Considering AppendBlobs unlike BlockBlobs would be writing data onto the actual blobs should the output stream be supporting flush() api i.e should it implement syncable? I dont see any implementation for the interface : CloudBlockAppendWrapper nor is the interface extended in the patch. Am I missing something? > Adding Append Blob support for WASB > --- > > Key: HADOOP-13475 > URL: https://issues.apache.org/jira/browse/HADOOP-13475 > Project: Hadoop Common > Issue Type: New Feature > Components: azure >Affects Versions: 2.7.1 >Reporter: Raul da Silva Martins >Assignee: Raul da Silva Martins >Priority: Critical > Attachments: 0001-Added-Support-for-Azure-AppendBlobs.patch > > > Currently the WASB implementation of the HDFS interface does not support the > utilization of Azure AppendBlobs underneath. As owners of a large scale > service who intend to start writing to Append blobs, we need this support in > order to be able to keep using our HDI capabilities. > This JIRA is added to implement Azure AppendBlob support to WASB. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13354) Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients
[ https://issues.apache.org/jira/browse/HADOOP-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383163#comment-15383163 ] Dushyanth commented on HADOOP-13354: +1 to HADOOP-13354.004.patch > Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft > Azure Storage Clients > --- > > Key: HADOOP-13354 > URL: https://issues.apache.org/jira/browse/HADOOP-13354 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg > Attachments: HADOOP-13354.001.patch, HADOOP-13354.002.patch, > HADOOP-13354.003.patch, HADOOP-13354.004.patch, Test-Results-With-4.2.0-fixes > > > Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft > Azure Storage Clients. > We are currently using version 2.2.0 of the SDK. > Version 4.2.0 brings some breaking changes. > Need to fix code to resolve all these breaking changes and certify that > everything works properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13181) WASB append support: getPos incorrect
[
https://issues.apache.org/jira/browse/HADOOP-13181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383079#comment-15383079
]
Dushyanth commented on HADOOP-13181:
+1 to change.
> WASB append support: getPos incorrect
> -
>
> Key: HADOOP-13181
> URL: https://issues.apache.org/jira/browse/HADOOP-13181
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure
>Affects Versions: 2.8.0
>Reporter: Bogdan Raducanu
> Attachments: HADOOP-13181.001.patch, append.java
>
>
> See attached code.
> Cause:
> In NativeAzureFileSystem.java: the append method returns
> {code}
> new FSDataOutputStream(appendStream, statistics)
> {code}
> Instead, it should probably return
> {code}
> new FSDataOutputStream(appendStream, statistics, meta.getLength())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12717) NPE when trying to rename a directory in Windows Azure Storage FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-12717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15178763#comment-15178763 ] Dushyanth commented on HADOOP-12717: +1 on the Patch. > NPE when trying to rename a directory in Windows Azure Storage FileSystem > - > > Key: HADOOP-12717 > URL: https://issues.apache.org/jira/browse/HADOOP-12717 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Robert Yokota >Assignee: Robert Yokota > Attachments: HADOOP-12717.001.patch, diff.txt > > > Encountered an NPE when trying to use the HBase utility ExportSnapshot with > Azure as the target. > It turns out verifyAndConvertToStandardFormat is returning null when > determining the hbaseRoot, and this is being added to the atomicRenameDirs > set. > java.lang.NullPointerException > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.isKeyForDirectorySet(AzureNativeFileSystemStore.java:1059) > at > org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.isAtomicRenameKey(AzureNativeFileSystemStore.java:1053) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2098) > at > org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1996) > at > org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:944) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > com.yammer.calmie.snapshot.AbstractSnapshotUtil.exportSnapshot(AbstractSnapshotUtil.java:210) > at > com.yammer.calmie.snapshot.AbstractSnapshotUtil.run(AbstractSnapshotUtil.java:79) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at > com.yammer.calmie.snapshot.SnapshotAzureBlobUtil.main(SnapshotAzureBlobUtil.java:85) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12535) Run FileSystem contract tests with hadoop-azure.
[
https://issues.apache.org/jira/browse/HADOOP-12535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160130#comment-15160130
]
Dushyanth commented on HADOOP-12535:
[~madhuch-ms] This is a known deviation for append support in WASB. Append
expects the application to enforce single modifier semantics for a file. Unlike
HDFS where we have lease or lock on the path, WASB does not have the same
semantics and we expect the applications to enforce that the file is not
modified (rename or delete) during append.
I think we can skip this test for Append at this time and re-enable it when we
bring append support to have the same semantics as HDFS.
cc [~cnauroth]
> Run FileSystem contract tests with hadoop-azure.
>
>
> Key: HADOOP-12535
> URL: https://issues.apache.org/jira/browse/HADOOP-12535
> Project: Hadoop Common
> Issue Type: Bug
> Components: azure, test
>Reporter: Chris Nauroth
>Assignee: madhumita chakraborty
> Attachments: HADOOP-12535.001.patch, HADOOP-12535.002.patch,
> HADOOP-12535.003.patch
>
>
> This issue proposes to implement the Hadoop {{FileSystem}} contract tests for
> hadoop-azure/WASB. The contract tests define the expected semantics of the
> {{FileSystem}}, so running these for hadoop-azure is likely to catch
> potential problems and improve overall quality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.005.patch [~cnauroth] Thanks for the review and feedback. I have addressed all the feedback in v005. The append support is now guarded by "fs.azure.enable.append.support" configuration flag. Testing : Testing with tests added for Append API and contract live tests for block blobs and page blobs. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635-004.patch, > HADOOP-12635.001.patch, HADOOP-12635.002.patch, HADOOP-12635.003.patch, > HADOOP-12635.005.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.006.patch Fixing Findbugs, Stylecheck and whitspace errors. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635-004.patch, > HADOOP-12635.001.patch, HADOOP-12635.002.patch, HADOOP-12635.003.patch, > HADOOP-12635.005.patch, HADOOP-12635.006.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.007.patch [~cnauroth] Thanks for the feedback. I have addressed the comment in v007. Testing: I have added new test to verify the behavior for Append when the configuration flag is set to false plust testing done for v006 > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635-004.patch, > HADOOP-12635.001.patch, HADOOP-12635.002.patch, HADOOP-12635.003.patch, > HADOOP-12635.005.patch, HADOOP-12635.006.patch, HADOOP-12635.007.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635-004.patch Fixing whitespace, author and Findbug errors from v003. Testing same as v003. [~cnauroth] the newly added tests for Append are getting executed in QA run. These are only live tests, is there a way to mark these tests as live-tests and not be run as part of QA run. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635-004.patch, > HADOOP-12635.001.patch, HADOOP-12635.002.patch, HADOOP-12635.003.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092969#comment-15092969 ] Dushyanth commented on HADOOP-12635: [~cnauroth] Sorry I missed your second CR comment. Taking a look at it now. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch, > HADOOP-12635.002.patch, HADOOP-12635.003.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.003.patch [~cnauroth] The conditions for the update of blob metadata in Append depends on the following conditions: 1) If append lease metadata is not part of the Blob. In this case this is the first client to Append so we update the metadata. 2) If append lease metadata is present and timeout has occurred. In this case irrespective of what the value of the append lease is we update the metadata. 3) If append lease metadata is present and is equal to testCondition value (passed as parameter) and timeout has not occurred, we update the metadata. 4) If append lease metadata is present and is not equal to testCondition value (passed as parameter) and timeout has not occurred, we do not update metadata and return false. The updatePossible flag enforces these conditions. But you are correct in that it is difficult to understand and thus maintain. I have made the change to follow the check you have suggested and added comments to specify the condition explicitly. Adding Patch v003 . Testing same as Patch v002. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch, > HADOOP-12635.002.patch, HADOOP-12635.003.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093049#comment-15093049 ] Dushyanth commented on HADOOP-12635: [~cnauroth] When I said that the create() codeflow doesn't have to change, I meant that the logic doesn't have to change however some methods inside AzureNativeFileSystemStore class like storeMetadataAttribute() and rename() do need to change to have a lease on the blob before updating the metadata. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch, > HADOOP-12635.002.patch, HADOOP-12635.003.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15092989#comment-15092989 ] Dushyanth commented on HADOOP-12635: [~cnauroth] Thanks for that catch. But I think in WASB we can solve this problem completely in Append and not touch create codepath by using the fact that WASB for Block Blobs writes to a temporary blob during create. During this write, the metadata of the blob would contain a link back to the temporary blob where the actual writes are happening. So in append() we can check the metadata of the blob to see if there is a temporary link and fail if there exists one. Do you see any usecase that I am missing that would fail with this approach? > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch, > HADOOP-12635.002.patch, HADOOP-12635.003.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.002.patch [~cnauroth] Thank you very much for the code review. I have attached a new patch based on the review comments. 1) You are right, there is a race condition. I have modified the updateBlobAppendMetadata to be a test and set operation. In this way, the creation of Append stream would become atomic. So now if two clients try to open the Append stream, one of them would be successful, the other would fail as the metadata would be updated holding a lease on the blob. 2) Created an initialize() method for creation of thread pool and append lease renewer thread other public API has a check to see if the append stream is initialized. 3) This was a mistake, I had to set the lease renewel period to 10 secs and the timeout to 30 secs. Keeping it to 1 sec for Azure storage I think would create bottleneck issues for the blob, so I have kept the lease renewel to 10 secs, let me know if you feel otherwise. 4) Fix to 4 should address it. 6) Fixed it. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch, > HADOOP-12635.002.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Attachment: HADOOP-12551.004.patch [~cnauroth] I apologise, I mis-interpreted the comment. I have changed all the multi-threaded tests to be conistent in v004 of the patch. Testing is same as v003 > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch, HADOOP-12551.002.patch, > HADOOP-12551.003.patch, HADOOP-12551.004.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Attachment: HADOOP-12551.002.patch [~cnauroth] Thanks a lot for the review. Attaching patch with the Multi-threaded test fix you suggested. While submitting the patch I built it using checkstyle flag, but didn't find any new style check warnings. So I am kicking of a QA build, to see if I am missing anything. Testing: The patch contains new tests to verify the changes made. Also changes have been tested against FileSystemContractLive tests for the both Block Blobs and Page Blobs. > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch, HADOOP-12551.002.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Status: Patch Available (was: Open) > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch, HADOOP-12551.002.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090289#comment-15090289 ] Dushyanth commented on HADOOP-12551: [~cnauroth] I see that there are two findbugs warnings in the QA run. I have commented the code explaining that we would not be hitting the NullReferenceException in the code path that the warnings are raised. I am leaving this as a No-op unless you feel it needs to be fixed. > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch, HADOOP-12551.002.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090354#comment-15090354 ] Dushyanth commented on HADOOP-12551: Testing done for Patch v003 - The newly added tests were executed along with Contract Live tests for Block and Page blobs. > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch, HADOOP-12551.002.patch, > HADOOP-12551.003.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Attachment: HADOOP-12551.003.patch [~cnauroth] This actually is an issue with the way the try-catch. We should not be swallowing the storage exception in scenarios where its a StorageException and not FileNotFoundException. There were other places where I had used this paradigm, I have fixed those try-catch blocks. > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch, HADOOP-12551.002.patch, > HADOOP-12551.003.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: HADOOP-12635.001.patch Attaching first iteration for Append support in WASB. Testing: The patch contains new tests to verify the append support. Also changes have been tested against FileSystemContractLive tests for the both Block Blobs and Page Blobs. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement > Components: azure >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx, HADOOP-12635.001.patch > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Summary: Introduce FileNotFoundException for WASB FileSystem API (was: Introduce FileNotFoundException for open and getFileStatus API's in WASB) > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for WASB FileSystem API
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Attachment: HADOOP-12551.001.patch Attaching first iteration for the JIRA. The patch contains modification to the handling of calls made to the Azure Storage layer wrapped in a try catch block to handle BlobNotFound exception. The patch handles open(), rename(), delete(), listStatus(), setOwner(), setPermission() APIs. Testing: The patch contains new tests to verify the changes made. Also changes have been tested against FileSystemContractLive tests for the both Block Blobs and Page Blobs. > Introduce FileNotFoundException for WASB FileSystem API > --- > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12551.001.patch > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15073367#comment-15073367 ] Dushyanth commented on HADOOP-12608: [~xyao] Thanks for the review. Testing: The patch contains new tests to verify the changes made. Also changes have been tested against FileSystemContractLive tests for the both Block Blobs and Page Blobs. > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: HADOOP-12608.001.patch, HADOOP-12608.002.patch, > HADOOP-12608.003.patch, HADOOP-12608.004.patch, HADOOP-12608.005.patch > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for open and getFileStatus API's in WASB
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Description: HADOOP-12533 introduced FileNotFoundException to the read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior. This jira also re-examines other Azure storage store calls to check for BlobNotFoundException in setPermission(), setOwner, rename(), delete(), open(), listStatus() APIs. was: HADOOP-12533 introduced FileNotFoundException to the read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior. This jira also fixes the store calls in > Introduce FileNotFoundException for open and getFileStatus API's in WASB > > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also re-examines other Azure storage store calls to check for > BlobNotFoundException in setPermission(), setOwner, rename(), delete(), > open(), listStatus() APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for open and getFileStatus API's in WASB
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Target Version/s: (was: 2.8.0) > Introduce FileNotFoundException for open and getFileStatus API's in WASB > > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also fixes the store calls in -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for open and getFileStatus API's in WASB
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Description: HADOOP-12533 introduced FileNotFoundException to the read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior. This jira also fixes the store calls in was:HADOOP-12533 introduced FileNotFoundException to the read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior. > Introduce FileNotFoundException for open and getFileStatus API's in WASB > > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. > This jira also fixes the store calls in -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12608: --- Attachment: HADOOP-12608.004.patch Fixing the issue with creating patch and attaching the patch. > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12608.001.patch, HADOOP-12608.002.patch, > HADOOP-12608.003.patch, HADOOP-12608.004.patch > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12608: --- Attachment: HADOOP-12608.005.patch Fixed the white-space tab issue. > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12608.001.patch, HADOOP-12608.002.patch, > HADOOP-12608.003.patch, HADOOP-12608.004.patch, HADOOP-12608.005.patch > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12608: --- Attachment: HADOOP-12608.003.patch Synced to the latest and created the patch. > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12608.001.patch, HADOOP-12608.002.patch, > HADOOP-12608.003.patch > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12608: --- Attachment: HADOOP-12608.002.patch [~ste...@apache.org] Thanks Steve for the review. 1) I would like to keep this way, because it gives an indication the error is with respect azure storage keys not being provided and it is azure specific. 2) I have added a test that simulates and verifies the error message. 3) I have tested as before with the live azure storage tests. I would start mentioning that going forward with new submissions. > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12608.001.patch, HADOOP-12608.002.patch > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth reassigned HADOOP-12635: -- Assignee: Dushyanth > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Dushyanth >Assignee: Dushyanth > Attachments: Append API.docx > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12635) Adding Append API support for WASB
Dushyanth created HADOOP-12635: -- Summary: Adding Append API support for WASB Key: HADOOP-12635 URL: https://issues.apache.org/jira/browse/HADOOP-12635 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.7.1 Reporter: Dushyanth Attachments: Append API.docx Currently the WASB implementation of the HDFS interface does not support Append API. This JIRA is added to design and implement the Append API support to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12635) Adding Append API support for WASB
[ https://issues.apache.org/jira/browse/HADOOP-12635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12635: --- Attachment: Append API.docx Attaching is the first draft of the specification for Append. > Adding Append API support for WASB > -- > > Key: HADOOP-12635 > URL: https://issues.apache.org/jira/browse/HADOOP-12635 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.1 >Reporter: Dushyanth > Attachments: Append API.docx > > > Currently the WASB implementation of the HDFS interface does not support > Append API. This JIRA is added to design and implement the Append API support > to WASB. The intended support for Append would only support a single writer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
Dushyanth created HADOOP-12608: -- Summary: Fix error message in WASB in connecting through Anonymous Credential codepath Key: HADOOP-12608 URL: https://issues.apache.org/jira/browse/HADOOP-12608 Project: Hadoop Common Issue Type: Bug Components: tools Affects Versions: 2.8.0 Reporter: Dushyanth Fix For: 2.8.0 Users of WASB have raised complaints over the error message returned back from WASB when they are trying to connect to Azure storage with anonymous credentials. Current implementation returns the correct message when we encounter a Storage exception, however for scenarios like querying to check if a container exists does not throw a StorageException but returns false when URI is directly specified (Anonymous access) the error message returned does not clearly state that credentials for storage account is not provided. This JIRA tracks the fix the error message to return what is returned when a storage exception is hit and also correct spelling mistakes in the error message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth reassigned HADOOP-12608: -- Assignee: Dushyanth > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12608) Fix error message in WASB in connecting through Anonymous Credential codepath
[ https://issues.apache.org/jira/browse/HADOOP-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12608: --- Attachment: HADOOP-12608.001.patch Adding patch for the first iteration. > Fix error message in WASB in connecting through Anonymous Credential codepath > - > > Key: HADOOP-12608 > URL: https://issues.apache.org/jira/browse/HADOOP-12608 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12608.001.patch > > > Users of WASB have raised complaints over the error message returned back > from WASB when they are trying to connect to Azure storage with anonymous > credentials. Current implementation returns the correct message when we > encounter a Storage exception, however for scenarios like querying to check > if a container exists does not throw a StorageException but returns false > when URI is directly specified (Anonymous access) the error message returned > does not clearly state that credentials for storage account is not provided. > This JIRA tracks the fix the error message to return what is returned when a > storage exception is hit and also correct spelling mistakes in the error > message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for open and getFileStatus API's in WASB
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Assignee: Dushyanth > Introduce FileNotFoundException for open and getFileStatus API's in WASB > > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > > HADOOP-12533 introduced FileNotFoundException to be introduced in read and > seek API for WASB. The open and getFileStatus api currently throws > FileNotFoundException correctly when the file does not exists when the API is > called but does not throw the same exception if there is another > thread/process deletes the file during its execution. This Jira fixes that > behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12551) Introduce FileNotFoundException for open and getFileStatus API's in WASB
Dushyanth created HADOOP-12551: -- Summary: Introduce FileNotFoundException for open and getFileStatus API's in WASB Key: HADOOP-12551 URL: https://issues.apache.org/jira/browse/HADOOP-12551 Project: Hadoop Common Issue Type: Bug Components: tools Affects Versions: 2.8.0 Reporter: Dushyanth Fix For: 2.8.0 HADOOP-12533 introduced FileNotFoundException to be introduced in read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12551) Introduce FileNotFoundException for open and getFileStatus API's in WASB
[ https://issues.apache.org/jira/browse/HADOOP-12551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12551: --- Description: HADOOP-12533 introduced FileNotFoundException to the read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior. (was: HADOOP-12533 introduced FileNotFoundException to be introduced in read and seek API for WASB. The open and getFileStatus api currently throws FileNotFoundException correctly when the file does not exists when the API is called but does not throw the same exception if there is another thread/process deletes the file during its execution. This Jira fixes that behavior.) > Introduce FileNotFoundException for open and getFileStatus API's in WASB > > > Key: HADOOP-12551 > URL: https://issues.apache.org/jira/browse/HADOOP-12551 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > > HADOOP-12533 introduced FileNotFoundException to the read and seek API for > WASB. The open and getFileStatus api currently throws FileNotFoundException > correctly when the file does not exists when the API is called but does not > throw the same exception if there is another thread/process deletes the file > during its execution. This Jira fixes that behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985539#comment-14985539 ] Dushyanth commented on HADOOP-12533: [~cnauroth] I see that Hadoop QA run failed in mvninstall subsystem. The failure seems to be in hadoop-common. The only change between the previous patch and this one was making close() synchronized. Which I doubt would have any effect on mvninstall. Is this a legitimate issue or a transient failure? [~Steve Loughran] This is hitting a bit of deadline for a release, so it would be great if we can get a review from you. Sorry for the inconvenience. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch, HADOOP-12533.002.patch, > HADOOP-12533.003.patch, HADOOP-12533.004.patch, HADOOP-12533.005.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Attachment: HADOOP-12533.004.patch Addressed Steve's Comments. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch, HADOOP-12533.002.patch, > HADOOP-12533.003.patch, HADOOP-12533.004.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Attachment: HADOOP-12533.005.patch Addressed [~cnauroth] comments. [~ste...@apache.org] can you please take a look. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch, HADOOP-12533.002.patch, > HADOOP-12533.003.patch, HADOOP-12533.004.patch, HADOOP-12533.005.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Attachment: HADOOP-12533.003.patch [~ste...@apache.org] [~cnauroth] Thanks a lot for the CR. Steve I addressed both comments on the previous iteration. Chris I fixed the style check warnings. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch, HADOOP-12533.002.patch, > HADOOP-12533.003.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Priority: Blocker (was: Major) > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14983374#comment-14983374 ] Dushyanth commented on HADOOP-12533: [~ste...@apache.org] Thanks steve. For 2. Since the files are deleted and the correctness of the test depends on getting the exception, I didn't close the streams. Thanks for the suggestion. I think adding the AbstractFSContract suite of tests can be done as improvement. I would let Chris comment on this. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth >Priority: Blocker > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Priority: Major (was: Critical) > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Priority: Critical (was: Blocker) > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth >Priority: Critical > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Attachment: HADOOP-12533.002.patch Thanks [~cnauroth] for the quick CR. 1) I have made the change to use error codes from the Azure Storage SDK classes. However the Java Azure storage SDK, does not handle error codes consistently. In some places the error code returned back from the SDK is from [~StorageErrorCode] enum class (https://github.com/Azure/azure-storage-java/blob/9248873511c14f112fcae5f6402447a7fab76162/microsoft-azure-storage/src/com/microsoft/azure/storage/StorageErrorCode.java) as its the case here in [~StorageException.java] (https://github.com/Azure/azure-storage-java/blob/9248873511c14f112fcae5f6402447a7fab76162/microsoft-azure-storage/src/com/microsoft/azure/storage/StorageException.java#L128) and in some cases it is from the StorageErrorCodeStrings class. So I have refactored the code to handle both possible cases. 2) I have added exception message for FileNotFoundException. 3) I have added teardown method to close the inputstream after each test. 4) I will check with my manager to see if I need to pick up the adding of AbstractFSContract suite to WASB. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch, HADOOP-12533.002.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
Dushyanth created HADOOP-12533: -- Summary: Introduce FileNotFoundException in WASB for read and seek API Key: HADOOP-12533 URL: https://issues.apache.org/jira/browse/HADOOP-12533 Project: Hadoop Common Issue Type: Bug Components: tools Affects Versions: 2.8.0 Reporter: Dushyanth Fix For: 2.8.0 Currently WASB throws a IOException in read and seek API for both Block and Page blobs for scenarios where the back blobs do not exists. This creates problems for applications like HBase which expect a FileNotFoundException in these scenarios. The fix for the problem is to check if the exceptions from Azure storage is because for Blob not found and throw FileNotFound exception if that is the case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Attachment: HADOOP-12533.001.patch Added first iteration for the fix. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the back blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth reassigned HADOOP-12533: -- Assignee: Dushyanth > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the back blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12533) Introduce FileNotFoundException in WASB for read and seek API
[ https://issues.apache.org/jira/browse/HADOOP-12533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12533: --- Description: Currently WASB throws a IOException in read and seek API for both Block and Page blobs for scenarios where the backing blobs do not exists. This creates problems for applications like HBase which expect a FileNotFoundException in these scenarios. The fix for the problem is to check if the exceptions from Azure storage is because for Blob not found and throw FileNotFound exception if that is the case. was: Currently WASB throws a IOException in read and seek API for both Block and Page blobs for scenarios where the back blobs do not exists. This creates problems for applications like HBase which expect a FileNotFoundException in these scenarios. The fix for the problem is to check if the exceptions from Azure storage is because for Blob not found and throw FileNotFound exception if that is the case. > Introduce FileNotFoundException in WASB for read and seek API > - > > Key: HADOOP-12533 > URL: https://issues.apache.org/jira/browse/HADOOP-12533 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: HADOOP-12533.001.patch > > > Currently WASB throws a IOException in read and seek API for both Block and > Page blobs for scenarios where the backing blobs do not exists. This creates > problems for applications like HBase which expect a FileNotFoundException in > these scenarios. > The fix for the problem is to check if the exceptions from Azure storage is > because for Blob not found and throw FileNotFound exception if that is the > case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: HADOOP-12350.007.patch Attaching with the tab character style check warning fix. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch, > HADOOP-12350.004.patch, HADOOP-12350.005.patch, HADOOP-12350.006.patch, > HADOOP-12350.007.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: HADOOP-12350.006.patch Addressing the stylecheck warnings from the previous iteration. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch, > HADOOP-12350.004.patch, HADOOP-12350.005.patch, HADOOP-12350.006.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14942762#comment-14942762 ] Dushyanth commented on HADOOP-12350: [~cnauroth] Sorry for the additional iteration needed here. The line numbers didnt match so I probably missed the warnings. I have addressed the comments and added a new iteration. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch, > HADOOP-12350.004.patch, HADOOP-12350.005.patch, HADOOP-12350.006.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: HADOOP-12350.005.patch Added a new patch that has the fix for the checkstyle and whitespace warnings. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch, > HADOOP-12350.004.patch, HADOOP-12350.005.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: HADOOP-12350.004.patch Adding the patch after rebasing to the latest on Trunk > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch, > HADOOP-12350.004.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch [~cnauroth] Thanks for the review. I have added a new iteration addressing your comments in the previous review. Please take a look. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Improve-WASB-Logging-Itr3.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743888#comment-14743888 ] Dushyanth commented on HADOOP-12350: [~cnauroth] Thanks for the input chris. I have added a patch that moves the logging framework to Slf4j. As discussed in the JIRA I have added a private cleanup method inside NativeAzureFileSystem class. To address the comment on AzureNativeFileSystemStore, yes the log statement is specifically for fatal exception that can't be retried again. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch Adding patch that changes to slf4j logging framework. > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch, > 0001-HADOOP-12350-Moving-from-commons.logging-to-slf4j-lo.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737857#comment-14737857 ] Dushyanth commented on HADOOP-12350: [~cnauroth] Thanks for the review Chris. I see that inside NativeAzureFileSystem class there are calls to IOUtils.cleanup(LOG, out). However IOUtils only exposes cleanup API that take Commons Logging objects and not Slf4j objects. considering that Hadoop community is moving towards use of Slf4j, would the IOUtils be exposing an Api that that takes Slf4j? If not, then I looked at the code for cleanup, it basically closes I looked at the code for cleanup it basically closes the Closeable objects and logs exception message on it. Do you see issues in having this implemented on the NativeAzureFileSystem class itself and move away from using IOUtils? > WASB Logging: Improve WASB Logging around deletes, reads and writes > --- > > Key: HADOOP-12350 > URL: https://issues.apache.org/jira/browse/HADOOP-12350 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.8.0 > > Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch > > > Logging around the WASB component is very limited and it is disabled by > default. This improvement is created to add logging around Reads, Writes and > Deletes when Azure Storage Exception to capture the blobs that hit the > exception. This information is useful while communicating with the Azure > storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
Dushyanth created HADOOP-12350: -- Summary: WASB Logging: Improve WASB Logging around deletes, reads and writes Key: HADOOP-12350 URL: https://issues.apache.org/jira/browse/HADOOP-12350 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Fix For: 2.8.0 Logging around the WASB component is very limited and it is disabled by default. This improvement is created to add logging around Reads, Writes and Deletes when Azure Storage Exception to capture the blobs that hit the exception. This information is useful while communicating with the Azure storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Assignee: Dushyanth WASB Logging: Improve WASB Logging around deletes, reads and writes --- Key: HADOOP-12350 URL: https://issues.apache.org/jira/browse/HADOOP-12350 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Fix For: 2.8.0 Logging around the WASB component is very limited and it is disabled by default. This improvement is created to add logging around Reads, Writes and Deletes when Azure Storage Exception to capture the blobs that hit the exception. This information is useful while communicating with the Azure storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12350) WASB Logging: Improve WASB Logging around deletes, reads and writes
[ https://issues.apache.org/jira/browse/HADOOP-12350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-12350: --- Attachment: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch Added first iteration. Patch contains: 1) Logging statement around Read, Write and Deletes. 2) Removed un-necessary imports WASB Logging: Improve WASB Logging around deletes, reads and writes --- Key: HADOOP-12350 URL: https://issues.apache.org/jira/browse/HADOOP-12350 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Fix For: 2.8.0 Attachments: 0001-HADOOP-12350-Added-WASB-Logging-Statement.patch Logging around the WASB component is very limited and it is disabled by default. This improvement is created to add logging around Reads, Writes and Deletes when Azure Storage Exception to capture the blobs that hit the exception. This information is useful while communicating with the Azure storage team for debugging purposes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Status: In Progress (was: Patch Available) Addressed Code review feedback and attached new patch with fixes. Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Attachment: (was: HADOOP-11960.002.patch) Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Attachment: HADOOP-11960.002.patch [~cnauroth] Thanks very much for the review. I have address your comments and attached a new patch with the fixes. HADOOP-11960.002.patch Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch, HADOOP-11960.002.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Status: Patch Available (was: In Progress) Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch, HADOOP-11960.002.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Attachment: HADOOP-11960.002.patch Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch, HADOOP-11960.002.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Status: Patch Available (was: In Progress) Adding first review patch to enable Azure Storage client-side logging. The core-site config fs.azure.storage.client.logging acts as a knob to control logging. The default behavior is set to False disabling client side logging. Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11960) Enable Azure-Storage Client Side logging.
[ https://issues.apache.org/jira/browse/HADOOP-11960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-11960: --- Attachment: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch Patch Attached for review. Enable Azure-Storage Client Side logging. - Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth Assignee: Dushyanth Attachments: 0001-HADOOP-945-Enabling-WASB-Azure-Storage-Client-side-l.patch AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-11960) Enable Azure-Storage Client Side logging.
Dushyanth created HADOOP-11960: -- Summary: Enable Azure-Storage Client Side logging. Key: HADOOP-11960 URL: https://issues.apache.org/jira/browse/HADOOP-11960 Project: Hadoop Common Issue Type: Improvement Components: tools Reporter: Dushyanth AzureStorageExceptions currently are logged as part of the WAB code which often is not too informative. AzureStorage SDK supports client side logging that can be enabled that logs relevant information w.r.t request made from the Storage client. This JIRA is created to enable Azure Storage Client Side logging at the Job submission level. User should be able to configure Client Side logging on a Per Job bases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
