[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Committed to branch-2.1 and master. Thanks [~pankit], for the issue report and
original patch, [~niklaus.xiao] & [~lirui] for the regex usecase and [~thejas]
for the reviews!
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.8.patch, HIVE-10022.9.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Attachment: HIVE-10022.9.patch
Updated .9.patch, Difference between .8.patch and .9.patch is over at :
https://gist.github.com/khorgath/52530044d4e046dca2f27acaf9def443
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.8.patch, HIVE-10022.9.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Attachment: HIVE-10022.8.patch
Latest .8.patch attached
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.8.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Attachment: HIVE-10022.7.patch
One minor update to slf4j usage style (did not cause an error, but this is more
correct)
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Attachment: HIVE-10022.6.patch
Updated patch with additional test, and with some optimizations.
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Attachment: HIVE-10022.5.patch
Fixed union vs intersection issue. Uploading patch without updated testcases,
in case the ptest framework picks up the prior patch to run against in the
meanwhile, and so that it can be reviewed again.
I'll try to add in a couple of negative test cases as well to illustrate the
following two cases:
a) When a parent dir does not have privileges.
b) When one of the files matched by the regex do not have privileges.
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Attachment: HIVE-10022.4.patch
Attached update to address regex concerns.
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-10022:
-
Summary: Authorization checks for non existent file/directory should not be
recursive (was: DFS in authorization might take too long)
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Pankit Thapar
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
