[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14845: Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Cherry picked in final changes to {{TestNativeAzureFileSystemAuthorization}} from trunk to branch-2. The alternative: rollback, build a new patch, reapply wouldn't have worked. Closing as fixed for 2.9+. Thanks for your contrib! > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.8.0, 2.7.4 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0, 3.1.0 > > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845.004.patch, > HADOOP-14845-branch-2-001.patch.txt, HADOOP-14845-branch-2-002.patch, > HADOOP-14845-branch-2-003.patch, HADOOP-14845-branch-2-005.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14845: Status: Patch Available (was: Open) > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.7.4, 2.8.0 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0 > > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845.004.patch, > HADOOP-14845-branch-2-001.patch.txt, HADOOP-14845-branch-2-002.patch, > HADOOP-14845-branch-2-003.patch, HADOOP-14845-branch-2-005.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14845: Attachment: HADOOP-14845-branch-2-005.patch Patch branch-2-005; the changes to trunk's TestNativeAzureFileSystemAuthorization merged into branch 2 &, where needed, fixup for java 7. This is just from a diff of branch-2 & trunk & picking in the relevant changes, as such it is the final bit of the branch-2 patch, addressing test conflict with HADOOP-14768. Tested {code} ava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 80.182 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.azure.TestNativeAzureFSAuthorizationCaching Tests run: 39, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.257 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSAuthorizationCaching Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.fs.azure.TestNativeAzureFSAuthWithBlobSpecificKeys Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.394 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSAuthWithBlobSpecificKeys {code} If yetus is happy, I'm going to pull this is in as the final bit of the patch > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.8.0, 2.7.4 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0 > > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845.004.patch, > HADOOP-14845-branch-2-001.patch.txt, HADOOP-14845-branch-2-002.patch, > HADOOP-14845-branch-2-003.patch, HADOOP-14845-branch-2-005.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14845: Status: Open (was: Patch Available) > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.7.4, 2.8.0 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0 > > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845.004.patch, > HADOOP-14845-branch-2-001.patch.txt, HADOOP-14845-branch-2-002.patch, > HADOOP-14845-branch-2-003.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sivaguru Sankaridurg updated HADOOP-14845: -- Attachment: HADOOP-14845.004.patch > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.8.0, 2.7.4 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0 > > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845.004.patch, > HADOOP-14845-branch-2-001.patch.txt, HADOOP-14845-branch-2-002.patch, > HADOOP-14845-branch-2-003.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14845: Fix Version/s: 2.9.0 > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.8.0, 2.7.4 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Fix For: 2.9.0 > > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt, > HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14845: - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ? > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.8.0, 2.7.4 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt, > HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14845) Azure wasb: getFileStatus not making any auth checks
[ https://issues.apache.org/jira/browse/HADOOP-14845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14845: Summary: Azure wasb: getFileStatus not making any auth checks (was: azure getFileStatus not making any auth checks) > Azure wasb: getFileStatus not making any auth checks > > > Key: HADOOP-14845 > URL: https://issues.apache.org/jira/browse/HADOOP-14845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure, security >Affects Versions: 2.8.0, 2.7.4 >Reporter: Sivaguru Sankaridurg >Assignee: Sivaguru Sankaridurg > Labels: azure, fs, secure, wasb > Attachments: HADOOP-14845.001.patch, HADOOP-14845.002.patch, > HADOOP-14845.003.patch, HADOOP-14845-branch-2-001.patch.txt, > HADOOP-14845-branch-2-002.patch, HADOOP-14845-branch-2-003.patch > > > The HDFS spec requires only traverse checks for any file accessed via > getFileStatus ... and since WASB does not support traverse checks, removing > this call effectively removed all protections for the getFileStatus call. The > reasoning at that time was that doing a performAuthCheck was the wrong thing > to do, since it was going against the specand that the correct fix to the > getFileStatus issue was to implement traverse checks rather than go against > the spec by calling performAuthCheck. The side-effects of such a change were > not fully clear at that time, but the thinking was that it was safer to > remain true to the spec, as far as possible. > The reasoning remains correct even today. But in view of the security hole > introduced by this change (that anyone can load up any other user's data in > hive), and keeping in mind that WASB does not intend to implement traverse > checks, we propose a compromise. > We propose (re)introducing a read-access check to getFileStatus(), that would > check the existing ancestor for read-access whenever invoked. Although not > perfect (in that it is a departure from the spec), we believe that it is a > good compromise between having no checks at all; and implementing full-blown > traverse checks. > For scenarios that deal with intermediate folders like mkdirs, the call would > check for read access against an existing ancestor (when invoked from shell) > for intermediate non-existent folders – {{ mkdirs /foo/bar, where only "/" > exists, would result in read-checks against "/" for "/","/foo" and "/foo/bar" > }}. This can be thought of, as being a close-enough substitute for the > traverse checks that hdfs does. > For other scenarios that don't deal with non-existent intermediate folders – > like read, delete etc, the check will happen against the parent. Once again, > we can think of the read-check against the parent as a substitute for the > traverse check, which can be customized for various users with ranger > policies. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org