[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845.004.patch
> Azure wasb: getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Fix For: 2.9.0
>
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845.004.patch,
> HADOOP-14845-branch-2-001.patch.txt, HADOOP-14845-branch-2-002.patch,
> HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189299#comment-16189299
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
[~asuresh], [~steve_l]. I'll try fwd porting, fix the tests and submit another
patch for trunk.
> Azure wasb: getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Fix For: 2.9.0
>
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183766#comment-16183766
]
Sivaguru Sankaridurg edited comment on HADOOP-14845 at 9/28/17 1:04 PM:
[~steve_l]. branch2-patch.003 passed yetus automated build and test.
I tested the code locally against Azure West US.
Could you please review and commit the changes?
was (Author: sisan...@microsoft.com):
[~steve_l]. branch2-patch.003 passed yetus automated build and test.
Could you please review and commit the changes?
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183766#comment-16183766
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
[~steve_l]. branch2-patch.003 passed yetus automated build and test.
Could you please review and commit the changes?
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845-branch-2-003.patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182210#comment-16182210
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
Hi [~steve_l].
{quote}I'm afraid the HADOOP-14553 test changes have broken this going into
trunk right now. Sorry: you'll appreciate the changes once you get used to
them; testing_azure.md covers the changes. For now, how about we get the patch
into branch-2 and then move that forward into trunk?{quote}
Okay.. I'll read through testing.md and try to figure out any changes that I
need to make.
Sounds like a plan. I'll add another patch that addresses your comments and we
can commit to branch-2 and then into trunk (for now).
{quote}NativeAzureFileSystem - If these two methods aren't to be
overridden/used directly by any (mock) subclass, mark them as private for
now{quote}
These are used by {{NativeAzureFileSystem.FolderRenamePending}} class - which
is why these are marked as _protected_.
All other comments have been addressed in branch-2-003.patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, security
>Affects Versions: 2.8.0, 2.7.4
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg reassigned HADOOP-14845:
-
Assignee: Steve Loughran (was: Sivaguru Sankaridurg)
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Steve Loughran
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16169934#comment-16169934
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
[~steve_l]. Both patches (trunk and branch-2) passed yetus build and tests.
Could you please review and commit the changes ?
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845-branch-2-002.patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt,
> HADOOP-14845-branch-2-002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845-branch-2-001.patch.txt
Attaching branch-2 patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Status: Open (was: Patch Available)
Canceling inorder to submit branch-2 patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Status: Patch Available (was: Open)
Submitting branch-2 patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167558#comment-16167558
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
[~steve_l]. Thank you for reverting the change that caused the regression.
I synced to latest, merged in my changes and tested locally. All the tests pass
now.
I have attached 003.patch (which is exactly the same as 002.patch), just to
re-trigger the yetus build.
I expect the build and tests to pass.
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845.003.patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch,
> HADOOP-14845.003.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166379#comment-16166379
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
[~steve_l]. Please also advise about how to proceed with this JIRA.
Should we figure out and fix the regression before proceeding with this JIRA
or should I submit a branch-2 patch and we can check-in into trunk *and*
branch-2 on the basis of the branch-2 build result ?
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166375#comment-16166375
]
Sivaguru Sankaridurg edited comment on HADOOP-14845 at 9/14/17 2:36 PM:
Thanks for confirming [~steve_l].
I suspected HADOOP-14520 tooand I ran tests locally on my box by branching
at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see any failures.
{quote}PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
commit 13eda5000304099d1145631f9be13ce8a00b600d
Author: Steve Loughran
Date: Thu Sep 7 18:35:03 2017 +0100
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked{quote}
was (Author: sisan...@microsoft.com):
Thanks for confirming [~steve_l].
I tried suspected HADOOP-14520 tooand I ran tests locally on my box by
branching at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see
any failures.
{quote}PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
commit 13eda5000304099d1145631f9be13ce8a00b600d
Author: Steve Loughran
Date: Thu Sep 7 18:35:03 2017 +0100
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked{quote}
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166375#comment-16166375
]
Sivaguru Sankaridurg edited comment on HADOOP-14845 at 9/14/17 2:35 PM:
Thanks for confirming [~steve_l].
I tried suspected HADOOP-14520 tooand I ran tests locally on my box by
branching at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see
any failures.
{quote}PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
commit 13eda5000304099d1145631f9be13ce8a00b600d
Author: Steve Loughran
Date: Thu Sep 7 18:35:03 2017 +0100
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked{quote}
was (Author: sisan...@microsoft.com):
Thanks for confirming [~steve_l].
I tried suspected HADOOP-14520 tooand I ran tests locally on my box by
branching at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see
any failures.
{{ PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
commit 13eda5000304099d1145631f9be13ce8a00b600d
Author: Steve Loughran
Date: Thu Sep 7 18:35:03 2017 +0100
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked}}
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166375#comment-16166375
]
Sivaguru Sankaridurg edited comment on HADOOP-14845 at 9/14/17 2:34 PM:
Thanks for confirming [~steve_l].
I tried suspected HADOOP-14520 tooand I ran tests locally on my box by
branching at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see
any failures.
{{ PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
commit 13eda5000304099d1145631f9be13ce8a00b600d
Author: Steve Loughran
Date: Thu Sep 7 18:35:03 2017 +0100
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec -
in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked}}
was (Author: sisan...@microsoft.com):
Thanks for confirming [~steve_l].
I tried suspected HADOOP-14520 tooand I ran tests locally on my box by
branching at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see
any failures.
bq. PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
bq. commit 13eda5000304099d1145631f9be13ce8a00b600d
bq. Author: Steve Loughran
bq. Date: Thu Sep 7 18:35:03 2017 +0100
bq.
bq. HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
bq. Contributed by Georgi Chalakov
bq. Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
bq. Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166375#comment-16166375
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
Thanks for confirming [~steve_l].
I tried suspected HADOOP-14520 tooand I ran tests locally on my box by
branching at {{13eda5000304099d1145631f9be13ce8a00b600d}} ... and I dont see
any failures.
bq. PS E:\g22\hadoop\hadoop-tools\hadoop-azure> git log
bq. commit 13eda5000304099d1145631f9be13ce8a00b600d
bq. Author: Steve Loughran
bq. Date: Thu Sep 7 18:35:03 2017 +0100
bq.
bq. HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
bq. Contributed by Georgi Chalakov
bq. Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
bq. Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.965 sec
- in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166118#comment-16166118
]
Sivaguru Sankaridurg edited comment on HADOOP-14845 at 9/14/17 11:43 AM:
-
[~steve_l], I am seeing these [unit-test errors |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13274/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt]
even without my changes, when I sync to latest on trunk.
It looks like there has been a regression .. can you please confirm ?
was (Author: sisan...@microsoft.com):
[~steve_l], I am seeing these [unit-test errors |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13274/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt]
even without my changes, when I sync to HEAD on trunk.
It looks like there has been a regression .. can you please confirm ?
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166118#comment-16166118
]
Sivaguru Sankaridurg commented on HADOOP-14845:
---
[~steve_l], I am seeing these [unit-test errors |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13274/artifact/patchprocess/patch-unit-hadoop-tools_hadoop-azure.txt]
even without my changes, when I sync to HEAD on trunk.
It looks like there has been a regression .. can you please confirm ?
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845.002.patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Status: Patch Available (was: Open)
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) azure getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14845:
--
Attachment: HADOOP-14845.001.patch
> azure getFileStatus not making any auth checks
> --
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14845.001.patch
>
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14845) getFileStatus not making any auth checks
[
https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg reassigned HADOOP-14845:
-
Assignee: Sivaguru Sankaridurg
> getFileStatus not making any auth checks
>
>
> Key: HADOOP-14845
> URL: https://issues.apache.org/jira/browse/HADOOP-14845
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
>
> The HDFS spec requires only traverse checks for any file accessed via
> getFileStatus ... and since WASB does not support traverse checks, removing
> this call effectively removed all protections for the getFileStatus call. The
> reasoning at that time was that doing a performAuthCheck was the wrong thing
> to do, since it was going against the specand that the correct fix to the
> getFileStatus issue was to implement traverse checks rather than go against
> the spec by calling performAuthCheck. The side-effects of such a change were
> not fully clear at that time, but the thinking was that it was safer to
> remain true to the spec, as far as possible.
> The reasoning remains correct even today. But in view of the security hole
> introduced by this change (that anyone can load up any other user's data in
> hive), and keeping in mind that WASB does not intend to implement traverse
> checks, we propose a compromise.
> We propose (re)introducing a read-access check to getFileStatus(), that would
> check the existing ancestor for read-access whenever invoked. Although not
> perfect (in that it is a departure from the spec), we believe that it is a
> good compromise between having no checks at all; and implementing full-blown
> traverse checks.
> For scenarios that deal with intermediate folders like mkdirs, the call would
> check for read access against an existing ancestor (when invoked from shell)
> for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
> exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
> }}. This can be thought of, as being a close-enough substitute for the
> traverse checks that hdfs does.
> For other scenarios that don't deal with non-existent intermediate folders –
> like read, delete etc, the check will happen against the parent. Once again,
> we can think of the read-check against the parent as a substitute for the
> traverse check, which can be customized for various users with ranger
> policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14845) getFileStatus not making any auth checks
Sivaguru Sankaridurg created HADOOP-14845:
-
Summary: getFileStatus not making any auth checks
Key: HADOOP-14845
URL: https://issues.apache.org/jira/browse/HADOOP-14845
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Reporter: Sivaguru Sankaridurg
The HDFS spec requires only traverse checks for any file accessed via
getFileStatus ... and since WASB does not support traverse checks, removing
this call effectively removed all protections for the getFileStatus call. The
reasoning at that time was that doing a performAuthCheck was the wrong thing to
do, since it was going against the specand that the correct fix to the
getFileStatus issue was to implement traverse checks rather than go against the
spec by calling performAuthCheck. The side-effects of such a change were not
fully clear at that time, but the thinking was that it was safer to remain true
to the spec, as far as possible.
The reasoning remains correct even today. But in view of the security hole
introduced by this change (that anyone can load up any other user's data in
hive), and keeping in mind that WASB does not intend to implement traverse
checks, we propose a compromise.
We propose (re)introducing a read-access check to getFileStatus(), that would
check the existing ancestor for read-access whenever invoked. Although not
perfect (in that it is a departure from the spec), we believe that it is a good
compromise between having no checks at all; and implementing full-blown
traverse checks.
For scenarios that deal with intermediate folders like mkdirs, the call would
check for read access against an existing ancestor (when invoked from shell)
for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/"
exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar"
}}. This can be thought of, as being a close-enough substitute for the traverse
checks that hdfs does.
For other scenarios that don't deal with non-existent intermediate folders –
like read, delete etc, the check will happen against the parent. Once again, we
can think of the read-check against the parent as a substitute for the traverse
check, which can be customized for various users with ranger policies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Status: Patch Available (was: Open)
Submitting branch-2 patch
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch,
> HADOOP-14820-006.patch, HADOOP-14820-007.patch,
> HADOOP-14820-branch-2-001.patch.txt
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Attachment: HADOOP-14820-branch-2-001.patch.txt
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch,
> HADOOP-14820-006.patch, HADOOP-14820-007.patch,
> HADOOP-14820-branch-2-001.patch.txt
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Status: Open (was: Patch Available)
Canceling patch in order to submit branch-2 patch
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch,
> HADOOP-14820-006.patch, HADOOP-14820-007.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150349#comment-16150349
]
Sivaguru Sankaridurg commented on HADOOP-14820:
---
patch 006 :- {{testMkdirsWithExistingHierarchyCheckPositive2}} does not seem
right. childPath3 has already been created.
Later when childPath3 is created again, it won't go up two levels .. it will be
a no-op.
The modifications to the test, to do what you intended is below:
{code:java}
@Test
public void testMkdirsWithExistingHierarchyCheckPositive2() throws Throwable {
Path testPath = new Path("/testMkdirsWithExistingHierarchyCheckPositive2");
Path childPath1 = new Path(testPath, "1");
Path childPath2 = new Path(childPath1, "2");
Path childPath3 = new Path(childPath2, "3");
authorizer.addAuthRule("/",
WasbAuthorizationOperations.WRITE.toString(), true);
authorizer.addAuthRule(childPath1.toString(),
WasbAuthorizationOperations.WRITE.toString(), true);
fs.updateWasbAuthorizer(authorizer);
try {
fs.mkdirs(childPath1);
ContractTestUtils.assertIsDirectory(fs, childPath1);
// Path already exists => no-op.
fs.mkdirs(testPath);
ContractTestUtils.assertIsDirectory(fs, testPath);
// Path already exists => no-op.
fs.mkdirs(childPath1);
ContractTestUtils.assertIsDirectory(fs, childPath1);
// Check permissions against existing ancestor childPath1
fs.mkdirs(childPath3);
ContractTestUtils.assertIsDirectory(fs, childPath3);
}
finally {
allowRecursiveDelete(fs, testPath.toString());
fs.delete(testPath, true);
}
}
{code}
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch,
> HADOOP-14820-006.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148988#comment-16148988
]
Sivaguru Sankaridurg commented on HADOOP-14820:
---
I added another test to cover directories partway up the tree with 005.patch.
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Attachment: HADOOP-14820.005.patch
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch, HADOOP-14820.005.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Wasb mkdirs security checks inconsistent with HDFS
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148921#comment-16148921
]
Sivaguru Sankaridurg commented on HADOOP-14820:
---
Fixed the checkstyle issue with the latest patch.
The new test creates {{/testMkdirsWithExistingHierarchyCheckPositive}} and then
removes permission on the root folder; and tries to create
{{/testMkdirsWithExistingHierarchyCheckPositive}} again. The test fails without
the patch. This test is expected to work, because the second create is a no-op
(w.r.t authorization.)
> Wasb mkdirs security checks inconsistent with HDFS
> --
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Affects Versions: 2.8.1
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Attachment: HADOOP-14820.004.patch
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch, HADOOP-14820.004.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Attachment: HADOOP-14820.003.patch
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch,
> HADOOP-14820.003.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:53 AM:
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {{getAncestor}} should take
care of this. This is the correct fix. The call to {{performAuthCheck}} on
L1714 is required.
*L2426* - This has been fixed. Please see changes to {{getAncestor}} in
003.patch. The naming is consistent with the definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
bq. Ancestor: The last existing component of the requested path. For example,
for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar exists.
The ancestor path is /foo if /foo exists but /foo/bar does not exist.
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {{getAncestor}} should take
care of this. This is the correct fix. The call to {{performAuthCheck}} on
L1714 is required.
*L2426* - This has been fixed. Please see changes to {{getAncestor}} in
003.patch. The naming is consistent with the definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:52 AM:
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {{getAncestor}} should take
care of this. This is the correct fix. The call to {{performAuthCheck}} on
L1714 is required.
*L2426* - This has been fixed. Please see changes to {{getAncestor}} in
003.patch. The naming is consistent with the definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {{getAncestor}} should take
care of this. This is the correct fix. The call to
{noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:52 AM:
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {{getAncestor}} should take
care of this. This is the correct fix. The call to
{noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {panel}getAncestor{panel}
should take care of this. This is the correct fix. The call to
{noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:51 AM:
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to {panel}getAncestor{panel}
should take care of this. This is the correct fix. The call to
{noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to
{code:java}getAncestor{code} should take care of this. This is the correct fix.
The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:50 AM:
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to
{code:java}getAncestor{code} should take care of this. This is the correct fix.
The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to
{noformat}getAncestor(){noformat} should take care of this. This is the correct
fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:49 AM:
*NativeAzureFileSystem.java*
*L1714* - Thanks for pointing this out. The fix to
{noformat}getAncestor(){noformat} should take care of this. This is the correct
fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java*
*L1714 - * Thanks for pointing this out. The fix to
{noformat}getAncestor(){noformat} should take care of this. This is the correct
fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg edited comment on HADOOP-14820 at 8/31/17 5:49 AM:
*NativeAzureFileSystem.java*
*L1714 - * Thanks for pointing this out. The fix to
{noformat}getAncestor(){noformat} should take care of this. This is the correct
fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
was (Author: sisan...@microsoft.com):
*NativeAzureFileSystem.java
L1714 - * Thanks for pointing this out. The fix to
{noformat}getAncestor(){noformat} should take care of this. This is the correct
fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148486#comment-16148486
]
Sivaguru Sankaridurg commented on HADOOP-14820:
---
*NativeAzureFileSystem.java
L1714 - * Thanks for pointing this out. The fix to
{noformat}getAncestor(){noformat} should take care of this. This is the correct
fix. The call to {noformat}performAuthCheck{noformat} on L1714 is required.
*L2426* - This has been fixed. Please see changes to
{noformat}getAncestor{noformat} in 003.patch. The naming is consistent with the
definition mentioned at
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Permission_Checks].
{noformat} Ancestor: The last existing component of the requested path. For
example, for the path /foo/bar/baz, the ancestor path is /foo/bar if /foo/bar
exists. The ancestor path is /foo if /foo exists but /foo/bar does not
exist.{noformat}
*L2459* - Same as above.
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Attachment: HADOOP-14820.002.patch
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch, HADOOP-14820.002.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Attachment: HADOOP-14820.001.patch
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
[
https://issues.apache.org/jira/browse/HADOOP-14820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-14820:
--
Status: Patch Available (was: Open)
> Fix for HDFS semantics parity for mkdirs -p
>
>
> Key: HADOOP-14820
> URL: https://issues.apache.org/jira/browse/HADOOP-14820
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14820.001.patch
>
>
> No authorization checks should be made when a user tries to create (mkdirs
> -p) an existing folder hierarchy.
> For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
> following operations, the results should be as shown below.
> {noformat}
> hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
> hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
> hdiuser@hn0-0d2f67:~$ ls -l
> dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
> hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
> mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
> The first three mkdirs succeed, because the ancestor is already present. The
> fourth one fails because of a permission check against the (shorter) ancestor
> (as compared to the path being created).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14820) Fix for HDFS semantics parity for mkdirs -p
Sivaguru Sankaridurg created HADOOP-14820:
-
Summary: Fix for HDFS semantics parity for mkdirs -p
Key: HADOOP-14820
URL: https://issues.apache.org/jira/browse/HADOOP-14820
Project: Hadoop Common
Issue Type: Bug
Components: fs/azure
Reporter: Sivaguru Sankaridurg
Assignee: Sivaguru Sankaridurg
No authorization checks should be made when a user tries to create (mkdirs -p)
an existing folder hierarchy.
For example, if we start with _/home/hdiuser/prefix_ pre-created, and do the
following operations, the results should be as shown below.
{noformat}
hdiuser@hn0-0d2f67:~$ sudo chown root:root prefix
hdiuser@hn0-0d2f67:~$ sudo chmod 555 prefix
hdiuser@hn0-0d2f67:~$ ls -l
dr-xr-xr-x 3 rootroot 4096 Aug 29 08:25 prefix
hdiuser@hn0-0d2f67:~$ mkdir -p /home
hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser
hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix
hdiuser@hn0-0d2f67:~$ mkdir -p /home/hdiuser/prefix/1
mkdir: cannot create directory â/home/hdiuser/prefix/1â: Permission denied
The first three mkdirs succeed, because the ancestor is already present. The
fourth one fails because of a permission check against the (shorter) ancestor
(as compared to the path being created).
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Status: Patch Available (was: Open) Attached a branch-2 patch. Submitting it. > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch, > HADOOP-14802.003.patch, HADOOP-14802-branch-2-001.patch.txt > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Attachment: HADOOP-14802-branch-2-001.patch.txt > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch, > HADOOP-14802.003.patch, HADOOP-14802-branch-2-001.patch.txt > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Status: Open (was: Patch Available) Cancelling current patch in order to submit a branch-2 patch > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch, > HADOOP-14802.003.patch, HADOOP-14802-branch-2-001.patch.txt > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143592#comment-16143592 ] Sivaguru Sankaridurg commented on HADOOP-14802: --- The test failure in hadoop-common-project does not seem to be related to my changes. > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch, > HADOOP-14802.003.patch > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14802) Add support for using container saskeys for all accesses
[
https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143473#comment-16143473
]
Sivaguru Sankaridurg edited comment on HADOOP-14802 at 8/28/17 7:27 AM:
# I have added the property
{noformat}fs.azure.saskey.usecontainersaskeyforallaccess{noformat} declaration
to core-default.xml.
# The shared sasKey generator makes a request to a service called "delegation
token service" to fetch a sas-key. This is an expensive call. This code change
makes available another option, where the user contacts Delegation token
service only once, to fetch the container sas-key. All other blob-specific keys
re-use the container sas-key and don't make that call to DTS. This provides a
performance boost.
# I tested this manually.
was (Author: sisan...@microsoft.com):
# I have added the property
{code}fs.azure.saskey.usecontainersaskeyforallaccess{code} declaration to
core-default.xml.
# The shared sasKey generator makes a request to a service called "delegation
token service" to fetch a sas-key. This is an expensive call. This code change
makes available another option, where the user contacts Delegation token
service only once, to fetch the container sas-key. All other blob-specific keys
re-use the container sas-key and don't make that call to DTS. This provides a
performance boost.
> Add support for using container saskeys for all accesses
>
>
> Key: HADOOP-14802
> URL: https://issues.apache.org/jira/browse/HADOOP-14802
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch,
> HADOOP-14802.003.patch
>
>
> This JIRA tracks adding support for using container saskey for all storage
> access.
> Instead of using saskeys that are specific to each blob, it is possible to
> re-use the container saskey for all blob accesses.
> This provides a performance improvement over using blob-specific saskeys
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Attachment: HADOOP-14802.003.patch > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch, > HADOOP-14802.003.patch > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14802) Add support for using container saskeys for all accesses
[
https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143473#comment-16143473
]
Sivaguru Sankaridurg commented on HADOOP-14802:
---
# I have added the property
{code}fs.azure.saskey.usecontainersaskeyforallaccess{code} declaration to
core-default.xml.
# The shared sasKey generator makes a request to a service called "delegation
token service" to fetch a sas-key. This is an expensive call. This code change
makes available another option, where the user contacts Delegation token
service only once, to fetch the container sas-key. All other blob-specific keys
re-use the container sas-key and don't make that call to DTS. This provides a
performance boost.
> Add support for using container saskeys for all accesses
>
>
> Key: HADOOP-14802
> URL: https://issues.apache.org/jira/browse/HADOOP-14802
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Sivaguru Sankaridurg
>Assignee: Sivaguru Sankaridurg
> Labels: azure, fs, secure, wasb
> Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch
>
>
> This JIRA tracks adding support for using container saskey for all storage
> access.
> Instead of using saskeys that are specific to each blob, it is possible to
> re-use the container saskey for all blob accesses.
> This provides a performance improvement over using blob-specific saskeys
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140361#comment-16140361 ] Sivaguru Sankaridurg commented on HADOOP-14802: --- Hi Steve, I ran the all azure tests against azure west-us. There were no new tests because no new functionality was introduced. The code only changed how we got a reference to a CloudBlob -- the functionality after getting the reference, remained the same. In any case, I added some tests that flip the flags for using blob-specific sas-keys in the second patch. I also submitted patch.002 to yetus. Thanks Siva > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Status: Patch Available (was: Open) > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Attachment: HADOOP-14802.002.patch > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch, HADOOP-14802.002.patch > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14802) Add support for using container saskeys for all accesses
[ https://issues.apache.org/jira/browse/HADOOP-14802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14802: -- Attachment: HADOOP-14802.001.patch > Add support for using container saskeys for all accesses > > > Key: HADOOP-14802 > URL: https://issues.apache.org/jira/browse/HADOOP-14802 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14802.001.patch > > > This JIRA tracks adding support for using container saskey for all storage > access. > Instead of using saskeys that are specific to each blob, it is possible to > re-use the container saskey for all blob accesses. > This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14802) Add support for using container saskeys for all accesses
Sivaguru Sankaridurg created HADOOP-14802: - Summary: Add support for using container saskeys for all accesses Key: HADOOP-14802 URL: https://issues.apache.org/jira/browse/HADOOP-14802 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: Sivaguru Sankaridurg Assignee: Sivaguru Sankaridurg This JIRA tracks adding support for using container saskey for all storage access. Instead of using saskeys that are specific to each blob, it is possible to re-use the container saskey for all blob accesses. This provides a performance improvement over using blob-specific saskeys -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14642: -- Attachment: HADOOP-14642.005.patch > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Jitendra Nath Pandey > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch, HADOOP-14642.005.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg reassigned HADOOP-14642: - Assignee: Jitendra Nath Pandey (was: Sivaguru Sankaridurg) > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Jitendra Nath Pandey > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14642: -- Attachment: HADOOP-14642.004.patch Fixed checkstyle and javadoc comments with 004.patch > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch, HADOOP-14642.004.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091498#comment-16091498 ] Sivaguru Sankaridurg commented on HADOOP-14642: --- The three tests listed as failed here [https://builds.apache.org/job/PreCommit-HADOOP-Build/12808/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt] are not related to my changes. The tests seem to have failed for other earlier builds too. > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14642: -- Attachment: HADOOP-14642.003.patch Rebased > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch, > HADOOP-14642.003.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14642: -- Attachment: HADOOP-14642.002.patch > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch, HADOOP-14642.002.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14642: -- Attachment: HADOOP-14642.001.patch > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14642.001.patch > > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14642) wasb: add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg reassigned HADOOP-14642: - Assignee: Sivaguru Sankaridurg > wasb: add support for caching Authorization and SASKeys > --- > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14642) Add support for caching Authorization and SASKeys
[ https://issues.apache.org/jira/browse/HADOOP-14642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14642: -- Labels: azure fs secure wasb (was: ) > Add support for caching Authorization and SASKeys > - > > Key: HADOOP-14642 > URL: https://issues.apache.org/jira/browse/HADOOP-14642 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > > TPCDS runs showed performance issues with Secure Clusters. > One of the main issues was identified to be repeated calls to DTS for the > same SASKey and Authorization request. > The performance issue was mitigated to a large extent by introducing caching > for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14642) Add support for caching Authorization and SASKeys
Sivaguru Sankaridurg created HADOOP-14642: - Summary: Add support for caching Authorization and SASKeys Key: HADOOP-14642 URL: https://issues.apache.org/jira/browse/HADOOP-14642 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: Sivaguru Sankaridurg TPCDS runs showed performance issues with Secure Clusters. One of the main issues was identified to be repeated calls to DTS for the same SASKey and Authorization request. The performance issue was mitigated to a large extent by introducing caching for saskeys and authorization requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14416) Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013481#comment-16013481 ] Sivaguru Sankaridurg commented on HADOOP-14416: --- Thanks [~liuml07] > Path starting with 'wasb:///' not resolved correctly while authorizing with > WASB-Ranger > --- > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-14416.001.patch, HADOOP-14416.001.patch, > HADOOP-14416.002.patch, HADOOP-14416.003.patch, Non-SecureRun-Logs.txt, > SecureRunLogs.txt > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012047#comment-16012047 ] Sivaguru Sankaridurg commented on HADOOP-14416: --- Hi [~liuml07]. I fixed all the checkstyle comments -- except one regarding the file-length. I'll fix that with a separate patch. > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14416.001.patch, HADOOP-14416.001.patch, > HADOOP-14416.002.patch, HADOOP-14416.003.patch, Non-SecureRun-Logs.txt, > SecureRunLogs.txt > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14416: -- Attachment: HADOOP-14416.003.patch Fixed checkstyle comments from 002.patch > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14416.001.patch, HADOOP-14416.001.patch, > HADOOP-14416.002.patch, HADOOP-14416.003.patch, Non-SecureRun-Logs.txt, > SecureRunLogs.txt > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14416: -- Attachment: HADOOP-14416.002.patch Fixed checkstyle comments from 001.patch submission > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14416.001.patch, HADOOP-14416.001.patch, > HADOOP-14416.002.patch, Non-SecureRun-Logs.txt, SecureRunLogs.txt > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14416: -- Attachment: Non-SecureRun-Logs.txt Logs for run with authorization disabled to prove that there is no regression on account of these changes. > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14416.001.patch, Non-SecureRun-Logs.txt, > SecureRunLogs.txt > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14416: -- Attachment: SecureRunLogs.txt Logs for run with WASB-authorization enabled > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14416.001.patch, SecureRunLogs.txt > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14416: -- Attachment: HADOOP-14416.001.patch > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14416.001.patch > > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
[ https://issues.apache.org/jira/browse/HADOOP-14416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008050#comment-16008050 ] Sivaguru Sankaridurg commented on HADOOP-14416: --- The issue was observed on WASB secure clusters only when WASB-authorization was enabled. It turned out to be a problem with not normalizing paths when doing the authorization check. I will attach the patch to this JIRA shortly. > Path not resolving correctly while authorizing with WASB-Ranger when it > starts with 'wasb:///' (triple-slash) > - > > Key: HADOOP-14416 > URL: https://issues.apache.org/jira/browse/HADOOP-14416 > Project: Hadoop Common > Issue Type: Bug > Components: fs/azure, security >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > > Bug found while launching spark-shell. > Repro-steps : > 1. Create a spark cluster with wasb-acls enabled. > 2. Change spark history log directory configurations to > wasb:///hdp/spark2-events > 3. Launching the spark shell should fail. > The above scenario works fine with clusters that dont have wasb-acl > authorization enabled. > Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14416) Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash)
Sivaguru Sankaridurg created HADOOP-14416: - Summary: Path not resolving correctly while authorizing with WASB-Ranger when it starts with 'wasb:///' (triple-slash) Key: HADOOP-14416 URL: https://issues.apache.org/jira/browse/HADOOP-14416 Project: Hadoop Common Issue Type: Bug Components: fs/azure, security Reporter: Sivaguru Sankaridurg Assignee: Sivaguru Sankaridurg Bug found while launching spark-shell. Repro-steps : 1. Create a spark cluster with wasb-acls enabled. 2. Change spark history log directory configurations to wasb:///hdp/spark2-events 3. Launching the spark shell should fail. The above scenario works fine with clusters that dont have wasb-acl authorization enabled. Note : wasb:/// resolves correctly on fs shell. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967133#comment-15967133 ] Sivaguru Sankaridurg commented on HADOOP-14274: --- Thanks for reviewing and committing [~liuml07]. > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch, > HADOOP-14274.003.patch, TestingForRegressionWithSecureModeSwitchedOff.txt, > TestingWithSecureModeSwitchedOnSuccess.txt > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965659#comment-15965659 ] Sivaguru Sankaridurg commented on HADOOP-14274: --- Hi [~liuml07], I have attached HADOOP-14274.003.patch with a minor fix to one of the tests. I have also attached logs that indicate that there was no regression, and that the new functionality passed all tests. I hope this is what you were looking for. Thanks Siva > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch, > HADOOP-14274.003.patch, TestingForRegressionWithSecureModeSwitchedOff.txt, > TestingWithSecureModeSwitchedOnSuccess.txt > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14274: -- Attachment: TestingWithSecureModeSwitchedOnSuccess.txt Logs to show that all tests passed when authorization was turned on > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch, > HADOOP-14274.003.patch, TestingForRegressionWithSecureModeSwitchedOff.txt, > TestingWithSecureModeSwitchedOnSuccess.txt > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14274: -- Attachment: TestingForRegressionWithSecureModeSwitchedOff.txt Logs to show that there was no regression on account of these changes when authorization was turned off > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch, > HADOOP-14274.003.patch, TestingForRegressionWithSecureModeSwitchedOff.txt > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14274: -- Attachment: HADOOP-14274.003.patch Minor fix to Instrumentation related tests > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch, > HADOOP-14274.003.patch > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964282#comment-15964282 ] Sivaguru Sankaridurg commented on HADOOP-14274: --- [~liuml07]. I fixed all your code review comments in the latest patch. Please review the changes. I have not broken up NativeAzureFileSystem.java into smaller file. That is the only checkstyle item still pending. I'll fix it in future changes. > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14274: -- Attachment: HADOOP-14274.002.patch Addressed Ming Liang's code review comments on 06-Apr-2017 > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch, HADOOP-14274.002.patch > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956492#comment-15956492 ] Sivaguru Sankaridurg commented on HADOOP-14274: --- Thanks [~liuml07]. :) > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
[ https://issues.apache.org/jira/browse/HADOOP-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14274: -- Attachment: HADOOP-14274-001.patch > Azure: Simplify Ranger-WASB policy model > > > Key: HADOOP-14274 > URL: https://issues.apache.org/jira/browse/HADOOP-14274 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Attachments: HADOOP-14274-001.patch > > > This improvement seeks to simplify the WASB-Ranger policy model -- both the > policy specification and policy enforcement. > More specifically, WASB-Ranger checks do not follow the same policy model and > enforcement as Ranger-HDFS. > Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The > handoff requires the Ranger policies follow the same model as HDFS-ACLs. This > is not true with Ranger+WASB. > We seek to simplify the policy specification and enforcement by dropping the > 'x' bit altogether. > This JIRA tracks this improvement, alongwith a few more minor bugfixes that > were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14274) Azure: Simplify Ranger-WASB policy model
Sivaguru Sankaridurg created HADOOP-14274: - Summary: Azure: Simplify Ranger-WASB policy model Key: HADOOP-14274 URL: https://issues.apache.org/jira/browse/HADOOP-14274 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Reporter: Sivaguru Sankaridurg Assignee: Sivaguru Sankaridurg This improvement seeks to simplify the WASB-Ranger policy model -- both the policy specification and policy enforcement. More specifically, WASB-Ranger checks do not follow the same policy model and enforcement as Ranger-HDFS. Ranger-HDFS hands off to HDFS-ACLs when a policy-match is not found. The handoff requires the Ranger policies follow the same model as HDFS-ACLs. This is not true with Ranger+WASB. We seek to simplify the policy specification and enforcement by dropping the 'x' bit altogether. This JIRA tracks this improvement, alongwith a few more minor bugfixes that were found during Ranger-WASB testing. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Thanks Mingliang.
-Siva
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch, HADOOP-13930.007.patch, HADOOP-13930.008.patch,
> HADOOP-13930.009.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878075#comment-15878075
]
Sivaguru Sankaridurg edited comment on HADOOP-13930 at 2/22/17 3:35 PM:
Hi [~ste...@apache.org], [~liuml07].
Could you please review my latest patch ?
For some reason tests in hadoop-common fail constantly, despite the fact that I
have not made any changes in hadoop-common that are related to these test
failures.
These three tests keep failing constantly:
https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/?auto_refresh=false
I am seeing other build-outputs showing the same error -
https://builds.apache.org/job/PreCommit-HADOOP-Build/11659/.
I suppose it is not specific to my change. Could you please take a look and
confirm ?
Thanks
Siva
was (Author: sisan...@microsoft.com):
Hi [~ste...@apache.org], [~liuml07].
Could you please review my latest patch ?
For some reason tests in hadoop-common fail constantly, despite the fact that I
have not made any changes in hadoop-common that are related to these test
failures.
These three tests keep failing constantly:
https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/?auto_refresh=false
Could you please take a look ?
Thanks
Siva
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch, HADOOP-13930.007.patch, HADOOP-13930.008.patch,
> HADOOP-13930.009.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878075#comment-15878075
]
Sivaguru Sankaridurg commented on HADOOP-13930:
---
Hi [~ste...@apache.org], [~liuml07].
Could you please review my latest patch ?
For some reason tests in hadoop-common fail constantly, despite the fact that I
have not made any changes in hadoop-common that are related to these test
failures.
These three tests keep failing constantly:
https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/?auto_refresh=false
Could you please take a look ?
Thanks
Siva
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch, HADOOP-13930.007.patch, HADOOP-13930.008.patch,
> HADOOP-13930.009.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.009.patch
Hopefully this should fix KerberosAuthException thrown from tests in
hadoop-common
testFileOutput(org.apache.hadoop.security.TestKDiag) Time elapsed: 0.039 sec
<<< ERROR!
org.apache.hadoop.security.KerberosAuthException: Login failure for user:
f...@example.com from keytab
/testptch/hadoop/hadoop-common-project/hadoop-common/target/keytab
javax.security.auth.login.LoginException: Unable to obtain password from user
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch, HADOOP-13930.007.patch, HADOOP-13930.008.patch,
> HADOOP-13930.009.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.008.patch
Fixed hadoop-common unit-test
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch, HADOOP-13930.007.patch, HADOOP-13930.008.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.007.patch
This patch has the same content as 006.patch. The automated build did not start
for some reason when 006.patch was submitted. Trying again by renaming as
007.patch.
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch, HADOOP-13930.007.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877485#comment-15877485
]
Sivaguru Sankaridurg commented on HADOOP-13930:
---
HADOOP-13930.006.patch attached yesterday did not start the automated build
somehow. Attaching the same patch as 007.patch.
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.006.patch
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch,
> HADOOP-13930.006.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.005.patch
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch, HADOOP-13930.005.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.004.patch
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch, HADOOP-13930.004.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875486#comment-15875486
]
Sivaguru Sankaridurg commented on HADOOP-13930:
---
Hi [~liuml07],[~ste...@apache.org].
I have attached a new patch HADOOP-13930.003.patch that addresses Steve's
review comments.
Could you please review at your earliest ?
Thanks
Siva
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13930) Azure: Add Authorization support to WASB
[
https://issues.apache.org/jira/browse/HADOOP-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sivaguru Sankaridurg updated HADOOP-13930:
--
Attachment: HADOOP-13930.003.patch
Fixes for review comments from Steve Loughran on 29/Dec/16 16:23
> Azure: Add Authorization support to WASB
>
>
> Key: HADOOP-13930
> URL: https://issues.apache.org/jira/browse/HADOOP-13930
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure
>Affects Versions: 2.8.0
>Reporter: Dushyanth
>Assignee: Sivaguru Sankaridurg
> Attachments: HADOOP-13930.001.patch, HADOOP-13930.002.patch,
> HADOOP-13930.003.patch
>
>
> As highlighted in HADOOP-13863, current implementation of WASB does not
> support authorization to any File System operations. This jira is created to
> add authorization support for WASB. The current approach is to enforce
> authorization via an external REST service (One approach could be to use
> component like Ranger to enforce authorization). The support for
> authorization would be hiding behind a configuration flag :
> "fs.azure.enable.authorization" and the remote service is expected to be
> provided via config : "fs.azure.remote.auth.service.url".
> The remote service is expected to provide support for the following REST
> call: {URL}/CHECK_AUTHORIZATION```
> An example request:
> {URL}/CHECK_AUTHORIZATION?wasb_absolute_path=_type= type>_token=
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
