[jira] [Commented] (HADOOP-14913) Sticky bit implementation for rename() operation in Azure WASB
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201391#comment-16201391 ] Varada Hemeswari commented on HADOOP-14913: --- [~ste...@apache.org], Thanks for the review and commit. Appreciate your endeavour for code cleanup. > Sticky bit implementation for rename() operation in Azure WASB > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Fix For: 2.9.0, 3.1.0 > > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch, > HADOOP-14913-003.patch, HADOOP-14913-004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[
https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199818#comment-16199818
]
Varada Hemeswari commented on HADOOP-14913:
---
Also I have enabled the tests to run by default without skipping for the
following test classes with the [^HADOOP-14913-003.patch] so the anyone running
the command
{code}
mvn -T 1c -Dparallel-tests -DtestsThreadCount=8 clean verify
{code}
do not have to enable any secure flag to run these tests.
So these will not skip now - TestNativeAzureFileSystemAuthorization.java,
ITestNativeAzureFSAuthWithBlobSpecificKeys.java,
ITestNativeAzureFSAuthorizationCaching.java,
> Sticky bit implementation for Rename operation in Azure fs
> --
>
> Key: HADOOP-14913
> URL: https://issues.apache.org/jira/browse/HADOOP-14913
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure
>Reporter: Varada Hemeswari
>Assignee: Varada Hemeswari
> Labels: azure, fs, secure
> Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch,
> HADOOP-14913-003.patch
>
>
> When authorization is enabled in WASB filesystem, there is a need for
> stickybit in cases where multiple users can create files under a shared
> directory. This additional check for sticky bit is reqired since any user can
> delete/rename another user's file because the parent has WRITE permission for
> all users.
> The purpose of this jira is to implement sticky bit equivalent for 'rename'
> call when authorization is enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14913: -- Status: Open (was: Patch Available) > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch, > HADOOP-14913-003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14913: -- Status: Patch Available (was: Open) > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch, > HADOOP-14913-003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[
https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198739#comment-16198739
]
Varada Hemeswari commented on HADOOP-14913:
---
[~ste...@apache.org], Thanks for the review.
I have addressed your comments of moving the decision of exceptions thrown to
rename instead of stickybit check.
However, for some of the other comments regarding FNFE and returning 'false' in
Azure Filesystem, the contract says,
{code}
fs.contract.rename-returns-false-if-source-missing
true
{code}
So accordingly, instead of throwing FNFE, I am returning false. This is same
behaviour as Auth-not-enabled case too in the rename implementation.
I agree that the exception shouldnt be swallowed for apps to know the reason
for failure, but this has nothing to do with sticky bit checks. I am simply
retaining existing semantics.
Similarly 'rename' functionality in Auth-not-enabled case is fully tested by
contract test for Rename.
However, TestNativeAzureFileSystemAuthorization concentrates only on auth
behaviour for all fs calls. Since the expectation is all the implementation for
fs call is same except for additional auth calls and stickybit checks being
made, protected by azureAuthorization flag(so that auth-not-enabled cases are
undisturbed at any point).
Considering this risk is not much higher. I agree about the 'delete' risk since
we introduced partial delete which is huge change.
I tried to add all possible test cases related to changes made in this patch
for sticky bit and moving auth check calls in Rename. I have also added tests
for src not existing use case since stickybit check has to retain this
behaviour.
If your concern is auth enabling is not testing whole paths of all fs calls,
may be we should consider having contract test runs with auth enabled cases.
However it must be a seperate endeavour unrelated to this change('Rename')
alone.
> Sticky bit implementation for Rename operation in Azure fs
> --
>
> Key: HADOOP-14913
> URL: https://issues.apache.org/jira/browse/HADOOP-14913
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, fs/azure
>Reporter: Varada Hemeswari
>Assignee: Varada Hemeswari
> Labels: azure, fs, secure
> Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch,
> HADOOP-14913-003.patch
>
>
> When authorization is enabled in WASB filesystem, there is a need for
> stickybit in cases where multiple users can create files under a shared
> directory. This additional check for sticky bit is reqired since any user can
> delete/rename another user's file because the parent has WRITE permission for
> all users.
> The purpose of this jira is to implement sticky bit equivalent for 'rename'
> call when authorization is enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14913: -- Attachment: HADOOP-14913-003.patch > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch, > HADOOP-14913-003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196485#comment-16196485 ] Varada Hemeswari commented on HADOOP-14913: --- [~ste...@apache.org], Can you please review the patch attached? > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14913: -- Attachment: HADOOP-14193.002.patch Submitted Patch-002 after fixing tests that will have additional checks introduced due to https://issues.apache.org/jira/browse/HADOOP-14845 > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch, HADOOP-14193.002.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192451#comment-16192451 ] Varada Hemeswari commented on HADOOP-14913: --- [~ste...@apache.org], Can you please review the patch attached? > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
[ https://issues.apache.org/jira/browse/HADOOP-14913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14913: -- Attachment: HADOOP-14193.001.patch Tested hadoop-azure module against Azure-South India end point in both secure and un-secure mode. > Sticky bit implementation for Rename operation in Azure fs > -- > > Key: HADOOP-14913 > URL: https://issues.apache.org/jira/browse/HADOOP-14913 > Project: Hadoop Common > Issue Type: New Feature > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure > Attachments: HADOOP-14193.001.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete/rename another user's file because the parent has WRITE permission for > all users. > The purpose of this jira is to implement sticky bit equivalent for 'rename' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14913) Sticky bit implementation for Rename operation in Azure fs
Varada Hemeswari created HADOOP-14913: - Summary: Sticky bit implementation for Rename operation in Azure fs Key: HADOOP-14913 URL: https://issues.apache.org/jira/browse/HADOOP-14913 Project: Hadoop Common Issue Type: New Feature Components: fs, fs/azure Reporter: Varada Hemeswari Assignee: Varada Hemeswari When authorization is enabled in WASB filesystem, there is a need for stickybit in cases where multiple users can create files under a shared directory. This additional check for sticky bit is reqired since any user can delete/rename another user's file because the parent has WRITE permission for all users. The purpose of this jira is to implement sticky bit equivalent for 'rename' call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Patch Available (was: Open) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.007.patch Corrected spaces in MockWasbAuthorizerImpl.java > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Open (was: Patch Available) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: (was: HADOOP-14768.007.patch) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[
https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184226#comment-16184226
]
Varada Hemeswari commented on HADOOP-14768:
---
Thanks [~ste...@apache.org] for the review.
I have addressed most of you comments in [^HADOOP-14768.007.patch]. Except the
following two
1. I did not change the exceptions thrown to PathIsNotEmptyDirectoryException
or org.apache.hadoop.fs.PathIsNotDirectoryException(in both the paths)
2. L2484: use Collection.addAll() to add list - I am not sure of the value add
here. Leaving as it is hoping its not a blocker for commit.
{quote}L2088. Issue: if a recursive delete failed, should that update the last
modified time of the parent dir? Or is it as the actual dir being modified is
itself not deleted, you don't have to. If that's true, given the dir being
deleted is still there, does its modtime need to be changed?{quote}
- Following the convention of auth disabled flow, if the delete is partial,
that is delete fails to completely clean up all files(not just the dir on which
delete is requested), we skip modifying the parent's last updated time of any
file/folder though some of them have successfully deleted..
I have run the hadoop-azure tests again in auth enabled and disabled flows
against azure storage account in South India.
> Honoring sticky bit during Deletion when authorization is enabled in WASB
> -
>
> Key: HADOOP-14768
> URL: https://issues.apache.org/jira/browse/HADOOP-14768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Varada Hemeswari
>Assignee: Varada Hemeswari
> Labels: fs, secure, wasb
> Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch,
> HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch,
> HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch,
> HADOOP-14768.007.patch
>
>
> When authorization is enabled in WASB filesystem, there is a need for
> stickybit in cases where multiple users can create files under a shared
> directory. This additional check for sticky bit is reqired since any user can
> delete another user's file because the parent has WRITE permission for all
> users.
> The purpose of this jira is to implement sticky bit equivalent for 'delete'
> call when authorization is enabled.
> Note : Sticky bit implementation for 'Rename' operation is not done as part
> of this JIRA
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.007.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch, > HADOOP-14768.007.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182360#comment-16182360 ] Varada Hemeswari commented on HADOOP-14768: --- I ran all the tests against azure storage account in South India in both secure and unsecure modes. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180157#comment-16180157 ] Varada Hemeswari commented on HADOOP-14768: --- Thanks [~tmarquardt] for the review. [~ste...@apache.org], Can we make some progress on this with priority? > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178910#comment-16178910 ] Varada Hemeswari edited comment on HADOOP-14768 at 9/25/17 12:03 PM: - Addressed comments from [~tmarquardt]. Separated delete paths for authorization enabled and disabled flows in [^HADOOP-14768.006.patch] was (Author: vahemesw): addressed comments from [~tmarquardt]. Seperated delete paths for authorization enabled and disabled flows. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.006.patch addressed comments from [~tmarquardt]. Seperated delete paths for authorization enabled and disabled flows. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch, HADOOP-14768.006.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176862#comment-16176862 ] Varada Hemeswari commented on HADOOP-14768: --- [~tmarquardt] The code when auth is not enabled in patch 5 is completely the same as it used to be previously with the exception that now the code handles delete case when delete is issued for '/'(root path). Previously it used to throw null ponter exception. Those are the changes you see apart from that there is clear branching when we are getting the contents to delete the file. Maintaining seperate paths from the beginning would be risky since changes done to one may not be done in another. And it would be lot of duplicate code too. Also I have tested majority of the delete scenarios in both auth enabled and disabled cases too. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[
https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176725#comment-16176725
]
Varada Hemeswari commented on HADOOP-14768:
---
[~ste...@apache.org]Intrestingly it doesnt fail when I use this command to run
the tests
{code}
mvn -Dtest=ITestWasbUriAndConfiguration#testCredsFromCredentialProvider test
{code}
However it fails if I use
{code}
mvn -T 1C -Dparallel-tests -DtestsThreadCount=8 clean verify
{code}
The parellelization seems to be causing some issue. I am not quite familiar
with debugging it. Can you take it from here?
Also can you please take a look at the patch. I am blocked on this to work on
sticky bit for rename.
Thanks.
> Honoring sticky bit during Deletion when authorization is enabled in WASB
> -
>
> Key: HADOOP-14768
> URL: https://issues.apache.org/jira/browse/HADOOP-14768
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Varada Hemeswari
>Assignee: Varada Hemeswari
> Labels: fs, secure, wasb
> Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch,
> HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch,
> HADOOP-14768.004.patch, HADOOP-14768.005.patch
>
>
> When authorization is enabled in WASB filesystem, there is a need for
> stickybit in cases where multiple users can create files under a shared
> directory. This additional check for sticky bit is reqired since any user can
> delete another user's file because the parent has WRITE permission for all
> users.
> The purpose of this jira is to implement sticky bit equivalent for 'delete'
> call when authorization is enabled.
> Note : Sticky bit implementation for 'Rename' operation is not done as part
> of this JIRA
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176491#comment-16176491 ] Varada Hemeswari commented on HADOOP-14768: --- [~tmarquardt], i have submitted [^HADOOP-14768.005.patch] with changes such that the new code path takes effect only when authorization is enabled . Please do review. Also I have confirmed that the test failure *ITestWasbUriAndConfiguration.testCredsFromCredentialProvider* is not related to my patch since it was failing even without my changes. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Patch Available (was: Open) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Open (was: Patch Available) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.005.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: (was: HADOOP-14768.005.patch) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.005.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch, HADOOP-14768.005.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175142#comment-16175142 ] Varada Hemeswari edited comment on HADOOP-14768 at 9/21/17 5:26 PM: [~tmarquardt], I agree to the risk of performance and functionality. We are already considering sticky bit only if authorization is enabled. So I think adding another flag is unnecesary. Please note that the changes not only add sticky bit but also change semantics of delete when authorization is enabled.( introducing partial delete whereas previously failure of single auth check used to halt entire delete).These required the changes you pointed out, that may actually cause performance to degrade. So seperate flag for stickybit may not be that useful. I will also be working on rename as soon as this patch is committed making this the base for stickybit. I can make changes such that if authorization is not enabled, delete will continue along the previous legacy path or else the new changes will take effect. Let me know if this works. was (Author: vahemesw): [~tmarq], I agree to the risk of performance and functionality. We are already considering sticky bit only if authorization is enabled. So I think adding another flag is unnecesary. Please note that the changes not only add sticky bit but also change semantics of delete when authorization is enabled.( introducing partial delete whereas previously failure of single auth check used to halt entire delete).These required the changes you pointed out, that may actually cause performance to regress. So seperate flag for stickybit may not be that useful. I can make changes such that if authorization is not enabled, delete will continue along the previous legacy path or else the new changes will take effect. Let me know if this works. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175142#comment-16175142 ] Varada Hemeswari commented on HADOOP-14768: --- [~tmarq], I agree to the risk of performance and functionality. We are already considering sticky bit only if authorization is enabled. So I think adding another flag is unnecesary. Please note that the changes not only add sticky bit but also change semantics of delete when authorization is enabled.( introducing partial delete whereas previously failure of single auth check used to halt entire delete).These required the changes you pointed out, that may actually cause performance to regress. So seperate flag for stickybit may not be that useful. I can make changes such that if authorization is not enabled, delete will continue along the previous legacy path or else the new changes will take effect. Let me know if this works. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174321#comment-16174321 ] Varada Hemeswari commented on HADOOP-14768: --- Thanks [~ste...@apache.org]. [~tmarquardt], [~ste...@apache.org], pinging again for review. Running on tight deadlines here :) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Open (was: Patch Available) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Patch Available (was: Open) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.004.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch, > HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172839#comment-16172839 ] Varada Hemeswari commented on HADOOP-14768: --- Thanks [~tmarquardt] for the review comments. *L1870* - Editing the description of this Jira saying this change is specific to Delete only. I have a seperate WI to deal with the Rename operation to honour stickybit. *L2208* - The partialListing contents are directly added to ArrayList item. there is no way I would know the size of contents forehand. *L2117* - We will need some heuristics to decide the optimal size. Will retain dynamic expansion of hashmap for now since the patch is already huge. Let me know if you think this can block the commit. [~ste...@apache.org], The submit patch does not seem to be running the yetus/jenkins build. Can you please take a look at this with priority. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Description: When authorization is enabled in WASB filesystem, there is a need for stickybit in cases where multiple users can create files under a shared directory. This additional check for sticky bit is reqired since any user can delete another user's file because the parent has WRITE permission for all users. The purpose of this jira is to implement sticky bit equivalent for 'delete' call when authorization is enabled. Note : Sticky bit implementation for 'Rename' operation is not done as part of this JIRA was: When authorization is enabled in WASB filesystem, there is a need for stickybit in cases where multiple users can create files under a shared directory. This additional check for sticky bit is reqired since any user can delete another user's file because the parent has WRITE permission for all users. The purpose of this jira is to implement sticky bit equivalent for 'delete' call when authorization is enabled. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. > Note : Sticky bit implementation for 'Rename' operation is not done as part > of this JIRA -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: (was: HADOOP-14768.004.patch) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.004.patch Re-attaching patch since yetus build was not triggered. > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.004.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: (was: HADOOP-14768.004.patch) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.004.patch Did a pull of integration tests commit. I see a test failing in non-secure mode after the pull. Not sure how it is related to my changes. testCredsFromCredentialProvider(org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration) Time elapsed: 0.221 sec <<< FAILURE! java.lang.AssertionError: AccountKey incorrect. expected: but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.apache.hadoop.fs.azure.ITestWasbUriAndConfiguration.testCredsFromCredentialProvider(ITestWasbUriAndConfiguration.java:347) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch, HADOOP-14768.003.patch, HADOOP-14768.004.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.003.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: (was: HADOOP-14768.003.patch) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167444#comment-16167444 ] Varada Hemeswari commented on HADOOP-14768: --- All the tests have passed in hadoop-azure in both secure and unsecure mode. Tested against storage account in South India > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.003.patch Fixed extra whitespaces and tabs > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch, > HADOOP-14768.003.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Status: Patch Available (was: Open) > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.002.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch, HADOOP-14768.002.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146593#comment-16146593 ] Varada Hemeswari commented on HADOOP-14768: --- [~ste...@apache.org], Can you please review the patch attached? > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
[ https://issues.apache.org/jira/browse/HADOOP-14768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14768: -- Attachment: HADOOP-14768.001.patch > Honoring sticky bit during Deletion when authorization is enabled in WASB > - > > Key: HADOOP-14768 > URL: https://issues.apache.org/jira/browse/HADOOP-14768 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: fs, secure, wasb > Attachments: HADOOP-14768.001.patch > > > When authorization is enabled in WASB filesystem, there is a need for > stickybit in cases where multiple users can create files under a shared > directory. This additional check for sticky bit is reqired since any user can > delete another user's file because the parent has WRITE permission for all > users. > The purpose of this jira is to implement sticky bit equivalent for 'delete' > call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14768) Honoring sticky bit during Deletion when authorization is enabled in WASB
Varada Hemeswari created HADOOP-14768: - Summary: Honoring sticky bit during Deletion when authorization is enabled in WASB Key: HADOOP-14768 URL: https://issues.apache.org/jira/browse/HADOOP-14768 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: Varada Hemeswari Assignee: Varada Hemeswari When authorization is enabled in WASB filesystem, there is a need for stickybit in cases where multiple users can create files under a shared directory. This additional check for sticky bit is reqired since any user can delete another user's file because the parent has WRITE permission for all users. The purpose of this jira is to implement sticky bit equivalent for 'delete' call when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14581: -- Attachment: HADOOP-14581.4.patch > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581-003.patch, HADOOP-14581.1.patch, > HADOOP-14581.2.patch, HADOOP-14581.4.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[
https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083508#comment-16083508
]
Varada Hemeswari commented on HADOOP-14581:
---
[~steve_l],I have addressed your documentation and check style comments in
patch - 4.
Here is the snippet of tests run against Azure South India endpoint.
{code}
[INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-azure ---
[INFO] Surefire report directory:
E:\2\hadoop\hadoop-tools\hadoop-azure\target\surefire-reports
---
T E S T S
---
---
T E S T S
---
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 11.617 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.339 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.531 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 88.856 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.887 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.743 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.879 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.626 sec - in
org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.695 sec -
in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 65.745 sec - in
org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.671 sec - in
org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
Running
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.579 sec - in
org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.266 sec - in
org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.682 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.501 sec - in
org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.525 sec - in
org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
Running org.apache.hadoop.fs.azure.TestBlobDataValidation
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.12 sec - in
org.apache.hadoop.fs.azure.TestBlobDataValidation
Running org.apache.hadoop.fs.azure.TestBlobMetadata
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.099 sec - in
org.apache.hadoop.fs.azure.TestBlobMetadata
Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.546 sec - in
org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
Running org.apache.hadoop.fs.azure.TestContainerChecks
Tests run: 4,
[jira] [Updated] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14581: -- Status: Patch Available (was: Open) > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581-003.patch, HADOOP-14581.1.patch, > HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079214#comment-16079214 ] Varada Hemeswari commented on HADOOP-14581: --- Hi Steve/Ming Liang, Can you please review the latest patch at your earliest possible? Pinging again since we are running tight on deadlines Thanks and regards, Hema > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075937#comment-16075937 ] Varada Hemeswari commented on HADOOP-14581: --- [~steve_l], [~liuml07] Can you please take a look at the recent patch? > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari reassigned HADOOP-14581: - Assignee: Steve Loughran (was: Varada Hemeswari) > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Steve Loughran > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14581: -- Attachment: (was: HADOOP-14581.2.patch) > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14581: -- Attachment: HADOOP-14581.2.patch > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch, > HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068528#comment-16068528 ] Varada Hemeswari commented on HADOOP-14581: --- I think here you could actually use getStrings() to get the list, and treat an empty list as the same as an entry "*": all. Why use that method? Existing regression tests. *--> I am using * to allow all users and "" to not allow anyone. So they could not be treated as same. There are no existing tests that regressed due to this.* needs policy for: * in the string: maybe fail fast for illegal setup? *--> taken care of, in patch2* needs policy for "". Right now it probably fails. Should it be a skip. *--> The failure is intentional. We expect the property to be setup as '*' default value or with list of users. In case it is setup as "", it translates to no one is allowed to chown.* TestNativeAzureFileSystemAuthorization try a string like "user1,user2 , user3 ,,user4 " to see what happens. I'd expect the leading/trailing spaces stripped, empty element skipped. *--> Patch2 tests includes this list* also try " user1, *" to verify that it gets rejected. *-->added a test for this in patch 2* I tested the hadoop-azure module changes on 'vahemeswregion.blob.core.windows.net' storage account Thanks. > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14581: -- Attachment: HADOOP-14581.2.patch > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch, HADOOP-14581.2.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
[ https://issues.apache.org/jira/browse/HADOOP-14581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14581: -- Attachment: HADOOP-14581.1.patch > Restrict setOwner to list of user when security is enabled in wasb > -- > > Key: HADOOP-14581 > URL: https://issues.apache.org/jira/browse/HADOOP-14581 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14581.1.patch > > > Currently in azure FS, setOwner api is exposed to all the users accessing the > file system. > When Authorization is enabled, access to some files/folders is given to > particular users based on whether the user is the owner of the file. > So setOwner has to be restricted to limited set of users to prevent users > from exploiting owner based authorization of files and folders. > Introducing a new config called fs.azure.chown.allowed.userlist which is a > comma seperated list of users who are allowed to perform chown operation when > authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14581) Restrict setOwner to list of user when security is enabled in wasb
Varada Hemeswari created HADOOP-14581: - Summary: Restrict setOwner to list of user when security is enabled in wasb Key: HADOOP-14581 URL: https://issues.apache.org/jira/browse/HADOOP-14581 Project: Hadoop Common Issue Type: Bug Components: fs/azure Affects Versions: 3.0.0-alpha3 Reporter: Varada Hemeswari Assignee: Varada Hemeswari Currently in azure FS, setOwner api is exposed to all the users accessing the file system. When Authorization is enabled, access to some files/folders is given to particular users based on whether the user is the owner of the file. So setOwner has to be restricted to limited set of users to prevent users from exploiting owner based authorization of files and folders. Introducing a new config called fs.azure.chown.allowed.userlist which is a comma seperated list of users who are allowed to perform chown operation when authorization is enabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14442) Owner support for ranger-wasb integration
[ https://issues.apache.org/jira/browse/HADOOP-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14442: -- Attachment: HADOOP-14442.1.patch > Owner support for ranger-wasb integration > - > > Key: HADOOP-14442 > URL: https://issues.apache.org/jira/browse/HADOOP-14442 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: filesystem, secure, wasb > Attachments: HADOOP-14442.1.patch, HADOOP-14442.patch > > > For the ranger-wasb integration, we need owner information from the metadata > information of the files/folders to be passed along to the ranger authorizer. > This patch contains the changes related to retrieving the owner from metadata > and making it available for ranger plugin that is integrated with wasb. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14442) Owner support for ranger-wasb integration
[ https://issues.apache.org/jira/browse/HADOOP-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14442: -- Target Version/s: 3.0.0-alpha2, 3.0.0-alpha1, 2.7.3 (was: 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2) Description: For the ranger-wasb integration, we need owner information from the metadata information of the files/folders to be passed along to the ranger authorizer. This patch contains the changes related to retrieving the owner from metadata and making it available for ranger plugin that is integrated with wasb. > Owner support for ranger-wasb integration > - > > Key: HADOOP-14442 > URL: https://issues.apache.org/jira/browse/HADOOP-14442 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: filesystem, secure, wasb > Attachments: HADOOP-14442.patch > > > For the ranger-wasb integration, we need owner information from the metadata > information of the files/folders to be passed along to the ranger authorizer. > This patch contains the changes related to retrieving the owner from metadata > and making it available for ranger plugin that is integrated with wasb. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14442) Owner support for ranger-wasb integration
[ https://issues.apache.org/jira/browse/HADOOP-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16024766#comment-16024766 ] Varada Hemeswari commented on HADOOP-14442: --- [~liuml07], Can you please review the patch attached. > Owner support for ranger-wasb integration > - > > Key: HADOOP-14442 > URL: https://issues.apache.org/jira/browse/HADOOP-14442 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: filesystem, secure, wasb > Attachments: HADOOP-14442.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14442) Owner support for ranger-wasb integration
[ https://issues.apache.org/jira/browse/HADOOP-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14442: -- Attachment: HADOOP-14442.patch > Owner support for ranger-wasb integration > - > > Key: HADOOP-14442 > URL: https://issues.apache.org/jira/browse/HADOOP-14442 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/azure >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: filesystem, secure, wasb > Attachments: HADOOP-14442.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14442) Owner support for ranger-wasb integration
Varada Hemeswari created HADOOP-14442: - Summary: Owner support for ranger-wasb integration Key: HADOOP-14442 URL: https://issues.apache.org/jira/browse/HADOOP-14442 Project: Hadoop Common Issue Type: Improvement Components: fs, fs/azure Reporter: Varada Hemeswari Assignee: Varada Hemeswari -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14203) performAuthCheck fails with wasbs scheme
[
https://issues.apache.org/jira/browse/HADOOP-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varada Hemeswari updated HADOOP-14203:
--
Labels: azure fs secure; wasb (was: azure fs wasb)
> performAuthCheck fails with wasbs scheme
>
>
> Key: HADOOP-14203
> URL: https://issues.apache.org/jira/browse/HADOOP-14203
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Varada Hemeswari
>Assignee: Sivaguru Sankaridurg
>Priority: Critical
> Labels: azure, fs, secure;, wasb
>
> Accessing Azure file system with 'wasbs' scheme fails on enabling wasb
> authorization.
> Stack trace :
> {code}
> adminuser1@hn0-f6adaa:/etc/hadoop/conf$ yarn jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
> wordcount "/examplefile" "/output"
> 17/03/20 07:58:48 INFO client.AHSProxy: Connecting to Application History
> server at hn0-f6adaa.team2testdomain.onmicrosoft.com/10.45.0.190:10200
> 17/03/20 07:58:48 INFO security.TokenCache: Got dt for
> wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net;
> Kind: WASB delegation, Service: 10.45.0.190:50911, Ident: (owner=adminuser1,
> renewer=yarn, realUser=, issueDate=1489996728687, maxDate=1490601528687,
> sequenceNumber=15, masterKeyId=11)
> org.apache.hadoop.fs.azure.WasbAuthorizationException: getFileStatus
> operation for Path :
> wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net/output
> not allowed
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.performAuthCheck(NativeAzureFileSystem.java:1425)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2058)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
> In the above fs.defaultFS is set to
> "wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net"
> If fs.defaultFS is changed to
> "wasb://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net", the
> job runs fine
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14203) performAuthCheck fails with wasbs scheme
[
https://issues.apache.org/jira/browse/HADOOP-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varada Hemeswari updated HADOOP-14203:
--
Affects Version/s: (was: 2.6.5)
2.7.3
> performAuthCheck fails with wasbs scheme
>
>
> Key: HADOOP-14203
> URL: https://issues.apache.org/jira/browse/HADOOP-14203
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.3
>Reporter: Varada Hemeswari
>Assignee: Sivaguru Sankaridurg
>Priority: Critical
> Labels: azure, fs, secure;, wasb
>
> Accessing Azure file system with 'wasbs' scheme fails on enabling wasb
> authorization.
> Stack trace :
> {code}
> adminuser1@hn0-f6adaa:/etc/hadoop/conf$ yarn jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
> wordcount "/examplefile" "/output"
> 17/03/20 07:58:48 INFO client.AHSProxy: Connecting to Application History
> server at hn0-f6adaa.team2testdomain.onmicrosoft.com/10.45.0.190:10200
> 17/03/20 07:58:48 INFO security.TokenCache: Got dt for
> wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net;
> Kind: WASB delegation, Service: 10.45.0.190:50911, Ident: (owner=adminuser1,
> renewer=yarn, realUser=, issueDate=1489996728687, maxDate=1490601528687,
> sequenceNumber=15, masterKeyId=11)
> org.apache.hadoop.fs.azure.WasbAuthorizationException: getFileStatus
> operation for Path :
> wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net/output
> not allowed
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.performAuthCheck(NativeAzureFileSystem.java:1425)
> at
> org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2058)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> {code}
> In the above fs.defaultFS is set to
> "wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net"
> If fs.defaultFS is changed to
> "wasb://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net", the
> job runs fine
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14203) performAuthCheck fails with wasbs scheme
Varada Hemeswari created HADOOP-14203:
-
Summary: performAuthCheck fails with wasbs scheme
Key: HADOOP-14203
URL: https://issues.apache.org/jira/browse/HADOOP-14203
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.6.5
Reporter: Varada Hemeswari
Assignee: Sivaguru Sankaridurg
Priority: Critical
Accessing Azure file system with 'wasbs' scheme fails on enabling wasb
authorization.
Stack trace :
{code}
adminuser1@hn0-f6adaa:/etc/hadoop/conf$ yarn jar
/usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar
wordcount "/examplefile" "/output"
17/03/20 07:58:48 INFO client.AHSProxy: Connecting to Application History
server at hn0-f6adaa.team2testdomain.onmicrosoft.com/10.45.0.190:10200
17/03/20 07:58:48 INFO security.TokenCache: Got dt for
wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net; Kind:
WASB delegation, Service: 10.45.0.190:50911, Ident: (owner=adminuser1,
renewer=yarn, realUser=, issueDate=1489996728687, maxDate=1490601528687,
sequenceNumber=15, masterKeyId=11)
org.apache.hadoop.fs.azure.WasbAuthorizationException: getFileStatus operation
for Path :
wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net/output
not allowed
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.performAuthCheck(NativeAzureFileSystem.java:1425)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2058)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1447)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
{code}
In the above fs.defaultFS is set to
"wasbs://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net"
If fs.defaultFS is changed to
"wasb://vahemesw-2v6-201703200...@storagewuteam02.blob.core.windows.net", the
job runs fine
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881948#comment-15881948 ] Varada Hemeswari commented on HADOOP-14091: --- [~liuml07] Can you please backport this patch to HDP 2.6 Fenton branch? > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: SECURE, WASB > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14091.001.patch, HADOOP-14091.002.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari reopened HADOOP-14091: --- > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: SECURE, WASB > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14091.001.patch, HADOOP-14091.002.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880334#comment-15880334 ] Varada Hemeswari commented on HADOOP-14091: --- The target Hadoop versions would be 2.7 and higher > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: SECURE, WASB > Attachments: HADOOP-14091.001.patch, HADOOP-14091.002.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14091: -- Attachment: HADOOP-14091.002.patch HADOOP-14091.001.patch file has some format issue. Attaching a proper patch file. > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: SECURE, WASB > Attachments: HADOOP-14091.001.patch, HADOOP-14091.002.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varada Hemeswari updated HADOOP-14091: -- Attachment: HADOOP-14091.001.patch diff for abstract fs implementation for wasbs scheme. > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari > Labels: SECURE, WASB > Attachments: HADOOP-14091.001.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874276#comment-15874276 ] Varada Hemeswari commented on HADOOP-14091: --- I am working on this. Please assign this back to me. > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari > Labels: SECURE, WASB > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
Varada Hemeswari created HADOOP-14091: - Summary: AbstractFileSystem implementaion for 'wasbs' scheme Key: HADOOP-14091 URL: https://issues.apache.org/jira/browse/HADOOP-14091 Project: Hadoop Common Issue Type: Task Components: fs/azure Environment: humboldt Reporter: Varada Hemeswari Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem implementation for 'wasb' scheme. This task refers to providing AbstractFileSystem implementation for 'wasbs' scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
