[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060973#comment-15060973 ] Ashutosh Chauhan commented on HIVE-12688: - makes sense +1 for revert. Lets explore alternatives in follow-up. > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060939#comment-15060939 ] Aihua Xu commented on HIVE-12688: - I'm not able to work on that until next week. So please revert it and I will try to provide better approach for that. So agree with your approach. Sent from my iPhone > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060922#comment-15060922 ] Thejas M Nair commented on HIVE-12688: -- None of the failures are related, these are failing in previous test runs as well. Looks like we really need some cleanup! Can someone please review it ? I think its simpler to commit this small patch and follow up in another jira to implement this feature without the regression. > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[
https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060896#comment-15060896
]
Hive QA commented on HIVE-12688:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12777933/HIVE-12688.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 9947 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union9
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.spark.client.TestSparkClient.testAddJarsAndFiles
org.apache.hive.spark.client.TestSparkClient.testCounters
org.apache.hive.spark.client.TestSparkClient.testErrorJob
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
org.apache.hive.spark.client.TestSparkClient.testMetricsCollection
org.apache.hive.spark.client.TestSparkClient.testRemoteClient
org.apache.hive.spark.client.TestSparkClient.testSimpleSparkJob
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6371/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6371/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6371/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12777933 - PreCommit-HIVE-TRUNK-Build
> HIVE-11826 makes hive unusable in properly secured cluster
> --
>
> Key: HIVE-12688
> URL: https://issues.apache.org/jira/browse/HIVE-12688
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.3.0, 2.0.0
>Reporter: Thejas M Nair
>Assignee: Thejas M Nair
>Priority: Blocker
> Attachments: HIVE-12688.1.patch
>
>
> HIVE-11826 makes a change to restrict connections to metastore to users who
> belong to groups under 'hadoop.proxyuser.hive.groups'.
> That property was only a meant to be a hadoop property, which controls what
> users the hive user can impersonate. What this change is doing is to enable
> use of that to also restrict who can connect to metastore server. This is new
> functionality, not a bug fix. There is value to this functionality.
> However, this change makes hive unusable in a properly secured cluster. If
> 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run
> Metastore and Hiveserver2 (instead of a very open "*"), then users will be
> able to connect to metastore only from those hosts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060683#comment-15060683 ] Thejas M Nair commented on HIVE-12688: -- I think its better to roll it out and put in a proper fix when its ready. As it affects only a small number of lines adding it back with proper fix should be straightforward. Do you agree [~aihuaxu] ? > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060580#comment-15060580 ] Thejas M Nair commented on HIVE-12688: -- [~sershe] I was thinking it is better to keep the release clear of blockers to avoid issues. But we can give couple of days for a better fix if you are OK with that (as the release manager for 2.0.0). It depends on cycles someone has to provide to fix the feature to prevent this regression. If we make the change to roll back this feature, there is not too much pressure on anyone working on this. [~aihuaxu] What do you prefer ? Would you have cycles to fix the regression soon ? Or would you prefer adding this feature back again after this patch to roll it out (it gives you more time that way)? > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060551#comment-15060551 ] Sergey Shelukhin commented on HIVE-12688: - I think it makes sense to keep this as blocker. If it in a while the proper fix is not in sight we can roll back the regression and postpone a better fix to a future version. [~aihuaxu] how hard would it be to make the better fix? [~thejas] do you think we should rather roll back now? > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060409#comment-15060409 ] Thejas M Nair commented on HIVE-12688: -- Yeah, looks like CDH version of Hive has been using this property to restrict access. This is not old behavior of Apache Hive. This is a new feature is not a pattern commonly seen in hadoop ecosystem. In case of HDFS, for example access is restricted on file permissions and not on a user group setting. To secure metastore access, you can already use storage based authorization. I am fine with this feature being added. However, the way it is implemented right now breaks hive not work if hadoop.proxyuser.hive.hosts is properly set. I am not sure why CDH users didn't face this issue, I assume cloudera manager might not be securing this for the clusters. I don't think we can ship Hive 2.0.0 in this form as it is a major regression. If you can change the implementation to fix this issue, please create a follow up jira with patch. I created this patch to rollback the change so that we don't block 2.0.0 release. > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059945#comment-15059945 ] Aihua Xu commented on HIVE-12688: - You are right. This is hadoop property. Seems that property should not limit the access to metastore server and we were historically using that to limit access to hive. In our own version, somehow we documented as such. Is this the old behavior? Instead of changing it back since it will introduce the issue that it will not block unauthorized access to Hive, can we keep this behavior and try to rework with a correct way that we block the Hive access somewhere else in a later version? > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > Attachments: HIVE-12688.1.patch > > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12688) HIVE-11826 makes hive unusable in properly secured cluster
[ https://issues.apache.org/jira/browse/HIVE-12688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059527#comment-15059527 ] Thejas M Nair commented on HIVE-12688: -- I think this is a blocker for 2.0.0 release . I am attaching a patch to roll back that change to unblock the 2.0.0 release. An fixed version of HIVE-11826 can be added in a follow up jira. cc [~sershe] [~aihuaxu] [~csun] [~ashutoshc] > HIVE-11826 makes hive unusable in properly secured cluster > -- > > Key: HIVE-12688 > URL: https://issues.apache.org/jira/browse/HIVE-12688 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Thejas M Nair >Assignee: Thejas M Nair >Priority: Blocker > > HIVE-11826 makes a change to restrict connections to metastore to users who > belong to groups under 'hadoop.proxyuser.hive.groups'. > That property was only a meant to be a hadoop property, which controls what > users the hive user can impersonate. What this change is doing is to enable > use of that to also restrict who can connect to metastore server. This is new > functionality, not a bug fix. There is value to this functionality. > However, this change makes hive unusable in a properly secured cluster. If > 'hadoop.proxyuser.hive.hosts' is set to the proper set of hosts that run > Metastore and Hiveserver2 (instead of a very open "*"), then users will be > able to connect to metastore only from those hosts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
