Re: Future release of hive
Sorry, I missed one thing -- you need to backport:
HIVE-20344: PrivilegeSynchronizer for SBA might hit AccessControlException
(Daniel Dai, reviewed by Vaibhav Gumashta)
--- Sungwoo
On Wed, Sep 22, 2021 at 12:24 AM Sungwoo Park wrote:
> Actually we can run Hive 3.1.2 with Ranger!
>
> To run Hive 3.1.2 with Ranger 2.0.0, you could set:
>
> hive.security.authorization.enabled=true
>
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
>
> hive.security.authorization.manager=org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory
> hive.privilege.synchronizer=true
>
> For Ranger 2.0.0, RangerHiveAuthorizerBase.getHivePolicyProvider() returns
> null, so it is okay to set hive.privilege.synchronizer to true, and you
> don't have to set up ZooKeeper.
>
> To run Hive 3.1.2 with Ranger 2.1.0 but without ZooKeeper, you need to set
> hive.privilege.synchronizer to false because
> RangerHiveAuthorizer.getHivePolicyProvider() returns
> RangerHivePolicyProvider. If hive.privilege.synchronizer is set to true,
> ZooKeeper should be running.
>
> So, with Ranger 2.0.0 or 2.1.0, you can run Hive 3.1.2 without ZooKeeper.
> (Of course, you can run it with ZooKeeper, too.) It may take a while (like
> a few seconds) for a new Ranger policy to be delivered to HiveServer2, but
> this does not seem like an issue in practice.
>
> --- Sungwoo
>
> On Tue, Sep 21, 2021 at 6:50 PM Antoine DUBOIS
> wrote:
>
>> Yes I can.
>> You cannot use Ranger without having to configure an instance of
>> zookeeper to run for unclear reasons.
>>
>> public void startPrivilegeSynchonizer(HiveConf hiveConf) throws Exception {
>>
>> PolicyProviderContainer policyContainer = new PolicyProviderContainer();
>> HiveAuthorizer authorizer = SessionState.get().getAuthorizerV2();
>> if (authorizer.getHivePolicyProvider() != null) {
>> policyContainer.addAuthorizer(authorizer);
>> }
>> if (hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname())
>> != null &&
>>
>> hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname()).contains(
>>
>> "org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener")
>> &&
>>
>> hiveConf.get(MetastoreConf.ConfVars.HIVE_AUTHORIZATION_MANAGER.getVarname())!=
>> null) {
>> List providers =
>> HiveUtils.getMetaStoreAuthorizeProviderManagers(
>> hiveConf, HiveConf.ConfVars.HIVE_METASTORE_AUTHORIZATION_MANAGER,
>> SessionState.get().getAuthenticator());
>> for (HiveMetastoreAuthorizationProvider provider : providers) {
>> if (provider.getHivePolicyProvider() != null) {
>> policyContainer.addAuthorizationProvider(provider);
>> }
>> }
>> }
>> [...]
>>
>> if (policyContainer.size() > 0) {
>> zKClientForPrivSync = startZookeeperClient(hiveConf);
>> String rootNamespace =
>> hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
>>
>>
>> So as long as you are using ranger you must use zookeeper and
>> configuration in this case is unclear.
>> I never managed to make it work properly.
>> It seems like version 3.1.2 is no longer developed or supported and only
>> 2.x is still under developpement.
>> Looks like cloudera buying HDP makes development less active in the end...
>>
>> --
>> *De: *"Battula, Brahma Reddy"
>> *À: *user@hive.apache.org
>> *Envoyé: *Vendredi 17 Septembre 2021 21:15:51
>> *Objet: *Re: Future release of hive
>>
>>
>>
>> Can you please give more details on issues which you faced with
>> hive-3.1.2 and ranger-2.1.0..?
>>
>>
>>
>>
>>
>> *From: *Antoine DUBOIS
>> *Date: *Tuesday, 14 September 2021 at 6:20 PM
>> *To: *user@hive.apache.org
>> *Subject: *Future release of hive
>>
>> Hello
>>
>> After trying to use hive 3.1.2 for several weeks with ranger, I stop.
>> It's seems way too complicated and tedious.
>>
>> I wonder when or even if there will be any more release in the 3.0 branch.
>>
>> I wonder if Hive 3.0 was just an experience as it seems maintenance is
>> not really there.
>> Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using
>> Hadoop 3 ?
>> Any insight on hive release cycle woudl be awesome.
>>
>>
>>
>> i hope you have a nice day.
>>
>>
>>
>> Antoine DUBOIS
>>
>>
>>
>>
Re: Future release of hive
Actually we can run Hive 3.1.2 with Ranger!
To run Hive 3.1.2 with Ranger 2.0.0, you could set:
hive.security.authorization.enabled=true
hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateUserAuthenticator
hive.security.authorization.manager=org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizerFactory
hive.privilege.synchronizer=true
For Ranger 2.0.0, RangerHiveAuthorizerBase.getHivePolicyProvider() returns
null, so it is okay to set hive.privilege.synchronizer to true, and you
don't have to set up ZooKeeper.
To run Hive 3.1.2 with Ranger 2.1.0 but without ZooKeeper, you need to set
hive.privilege.synchronizer to false because
RangerHiveAuthorizer.getHivePolicyProvider() returns
RangerHivePolicyProvider. If hive.privilege.synchronizer is set to true,
ZooKeeper should be running.
So, with Ranger 2.0.0 or 2.1.0, you can run Hive 3.1.2 without ZooKeeper.
(Of course, you can run it with ZooKeeper, too.) It may take a while (like
a few seconds) for a new Ranger policy to be delivered to HiveServer2, but
this does not seem like an issue in practice.
--- Sungwoo
On Tue, Sep 21, 2021 at 6:50 PM Antoine DUBOIS
wrote:
> Yes I can.
> You cannot use Ranger without having to configure an instance of zookeeper
> to run for unclear reasons.
>
> public void startPrivilegeSynchonizer(HiveConf hiveConf) throws Exception {
>
> PolicyProviderContainer policyContainer = new PolicyProviderContainer();
> HiveAuthorizer authorizer = SessionState.get().getAuthorizerV2();
> if (authorizer.getHivePolicyProvider() != null) {
> policyContainer.addAuthorizer(authorizer);
> }
> if (hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname())
> != null &&
>
> hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname()).contains(
>
> "org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener")
> &&
>
> hiveConf.get(MetastoreConf.ConfVars.HIVE_AUTHORIZATION_MANAGER.getVarname())!=
> null) {
> List providers =
> HiveUtils.getMetaStoreAuthorizeProviderManagers(
> hiveConf, HiveConf.ConfVars.HIVE_METASTORE_AUTHORIZATION_MANAGER,
> SessionState.get().getAuthenticator());
> for (HiveMetastoreAuthorizationProvider provider : providers) {
> if (provider.getHivePolicyProvider() != null) {
> policyContainer.addAuthorizationProvider(provider);
> }
> }
> }
> [...]
>
> if (policyContainer.size() > 0) {
> zKClientForPrivSync = startZookeeperClient(hiveConf);
> String rootNamespace =
> hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
>
>
> So as long as you are using ranger you must use zookeeper and
> configuration in this case is unclear.
> I never managed to make it work properly.
> It seems like version 3.1.2 is no longer developed or supported and only
> 2.x is still under developpement.
> Looks like cloudera buying HDP makes development less active in the end...
>
> ------
> *De: *"Battula, Brahma Reddy"
> *À: *user@hive.apache.org
> *Envoyé: *Vendredi 17 Septembre 2021 21:15:51
> *Objet: *Re: Future release of hive
>
>
>
> Can you please give more details on issues which you faced with hive-3.1.2
> and ranger-2.1.0..?
>
>
>
>
>
> *From: *Antoine DUBOIS
> *Date: *Tuesday, 14 September 2021 at 6:20 PM
> *To: *user@hive.apache.org
> *Subject: *Future release of hive
>
> Hello
>
> After trying to use hive 3.1.2 for several weeks with ranger, I stop.
> It's seems way too complicated and tedious.
>
> I wonder when or even if there will be any more release in the 3.0 branch.
>
> I wonder if Hive 3.0 was just an experience as it seems maintenance is not
> really there.
> Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using
> Hadoop 3 ?
> Any insight on hive release cycle woudl be awesome.
>
>
>
> i hope you have a nice day.
>
>
>
> Antoine DUBOIS
>
>
>
>
Re: Future release of hive
Unsubscribe
在 2021年9月21日 17:50，Antoine DUBOIS 写道：
Yes I can.
You cannot use Ranger without having to configure an instance of zookeeper to
run for unclear reasons.
public void startPrivilegeSynchonizer(HiveConf hiveConf) throws Exception {
PolicyProviderContainer policyContainer = new PolicyProviderContainer();
HiveAuthorizer authorizer = SessionState.get().getAuthorizerV2();
if (authorizer.getHivePolicyProvider() != null) {
policyContainer.addAuthorizer(authorizer);
}
if (hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname()) !=
null &&
hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname()).contains(
"org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener")
&&
hiveConf.get(MetastoreConf.ConfVars.HIVE_AUTHORIZATION_MANAGER.getVarname())!=
null) {
List providers =
HiveUtils.getMetaStoreAuthorizeProviderManagers(
hiveConf, HiveConf.ConfVars.HIVE_METASTORE_AUTHORIZATION_MANAGER,
SessionState.get().getAuthenticator());
for (HiveMetastoreAuthorizationProvider provider : providers) {
if (provider.getHivePolicyProvider() != null) {
policyContainer.addAuthorizationProvider(provider);
}
}
}
[...]
if (policyContainer.size() > 0) {
zKClientForPrivSync = startZookeeperClient(hiveConf);
String rootNamespace =
hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
So as long as you are using ranger you must use zookeeper and configuration in
this case is unclear.
I never managed to make it work properly.
It seems like version 3.1.2 is no longer developed or supported and only 2.x is
still under developpement.
Looks like cloudera buying HDP makes development less active in the end...
De: "Battula, Brahma Reddy"
À: user@hive.apache.org
Envoyé: Vendredi 17 Septembre 2021 21:15:51
Objet: Re: Future release of hive
Can you please give more details on issues which you faced with hive-3.1.2 and
ranger-2.1.0..?
From: Antoine DUBOIS
Date: Tuesday, 14 September 2021 at 6:20 PM
To: user@hive.apache.org
Subject: Future release of hive
Hello
After trying to use hive 3.1.2 for several weeks with ranger, I stop.
It's seems way too complicated and tedious.
I wonder when or even if there will be any more release in the 3.0 branch.
I wonder if Hive 3.0 was just an experience as it seems maintenance is not
really there.
Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using
Hadoop 3 ?
Any insight on hive release cycle woudl be awesome.
i hope you have a nice day.
Antoine DUBOIS
Re: Future release of hive
Yes I can.
You cannot use Ranger without having to configure an instance of zookeeper to
run for unclear reasons.
public void startPrivilegeSynchonizer (HiveConf hiveConf) throws Exception {
PolicyProviderContainer policyContainer = new PolicyProviderContainer() ;
HiveAuthorizer authorizer = SessionState. get ().getAuthorizerV2() ;
if (authorizer.getHivePolicyProvider() != null ) {
policyContainer.addAuthorizer(authorizer) ;
}
if (hiveConf.get(MetastoreConf.ConfVars. PRE_EVENT_LISTENERS .getVarname()) !=
null &&
hiveConf.get(MetastoreConf.ConfVars. PRE_EVENT_LISTENERS
.getVarname()).contains(
"org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener"
) &&
hiveConf.get(MetastoreConf.ConfVars. HIVE_AUTHORIZATION_MANAGER
.getVarname())!= null ) {
List providers = HiveUtils.
getMetaStoreAuthorizeProviderManagers (
hiveConf , HiveConf.ConfVars. HIVE_METASTORE_AUTHORIZATION_MANAGER ,
SessionState. get ().getAuthenticator()) ;
for (HiveMetastoreAuthorizationProvider provider : providers) {
if (provider.getHivePolicyProvider() != null ) {
policyContainer.addAuthorizationProvider(provider) ;
}
}
}
[...]
if (policyContainer.size() > 0 ) {
zKClientForPrivSync = startZookeeperClient(hiveConf) ;
String rootNamespace = hiveConf.getVar(HiveConf.ConfVars.
HIVE_SERVER2_ZOOKEEPER_NAMESPACE ) ;
So as long as you are using ranger you must use zookeeper and configuration in
this case is unclear.
I never managed to make it work properly.
It seems like version 3.1.2 is no longer developed or supported and only 2.x is
still under developpement.
Looks like cloudera buying HDP makes development less active in the end...
De: "Battula, Brahma Reddy"
À: user@hive.apache.org
Envoyé: Vendredi 17 Septembre 2021 21:15:51
Objet: Re: Future release of hive
Can you please give more details on issues which you faced with hive-3.1.2 and
ranger-2.1.0..?
From: Antoine DUBOIS
Date: Tuesday, 14 September 2021 at 6:20 PM
To: user@hive.apache.org
Subject: Future release of hive
Hello
After trying to use hive 3.1.2 for several weeks with ranger, I stop.
It's seems way too complicated and tedious.
I wonder when or even if there will be any more release in the 3.0 branch.
I wonder if Hive 3.0 was just an experience as it seems maintenance is not
really there.
Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using
Hadoop 3 ?
Any insight on hive release cycle woudl be awesome.
i hope you have a nice day.
Antoine DUBOIS
smime.p7s
Description: S/MIME Cryptographic Signature
Re: Future release of hive
Can you please give more details on issues which you faced with hive-3.1.2 and ranger-2.1.0..? From: Antoine DUBOIS Date: Tuesday, 14 September 2021 at 6:20 PM To: user@hive.apache.org Subject: Future release of hive Hello After trying to use hive 3.1.2 for several weeks with ranger, I stop. It's seems way too complicated and tedious. I wonder when or even if there will be any more release in the 3.0 branch. I wonder if Hive 3.0 was just an experience as it seems maintenance is not really there. Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using Hadoop 3 ? Any insight on hive release cycle woudl be awesome. i hope you have a nice day. Antoine DUBOIS
Re: Future release of hive
Hello, Hive PMC members or committers could share insider knowledge about the status of the Hive project, but here is my impression on Hive 3.1.2 as an outsider. Hive 3.1.2 is widely used in production, but not maintained seriously. (You could just check out the # of commits in branch-3.1 for the last couple of years). Many critical patches have not been backported, and some patches are committed even without proper testing. As a result, running Hive 3.1.2 in production would require you to maintain your own fork of Hive 3.1.2, backporting patches as necessary. Or you could use a commercial solution like CDP. There is nothing unusual here, as Hive is an open source project. On the other hand, bugs and performance issues in Hive 3.1.2 are constantly reported in Hive JIRAs, while important bug-fixes and performance improvements are contributed by many individuals. What seems to happen afterwards is that the contributed code is either merged only in the master branch or not accepted at all. (I see quite a few important patches stay unnoticed without being discussed.) Occasionally you see new Hive JIRAs reporting bugs in Hive 3.1.2 which have actually been fixed in earlier JIRAs that are not merged in branch-3.1. In order to take advantage of new patches, one would have to backport batches on his own. (I guess Hive PMC is mostly focused on the master branch). As for Hive 4.0, I know nothing about its status, but in the virtual meetup last March, it was briefly mentioned that no concrete release plan was ready. (I could be wrong, so someone could correct me.) We are maintaining our own fork of Hive 3.1.2 which backported over 300 additional patches. More important patches from the master branch are currently being backported. This repository is getting increasingly popular, so it might be useful to you. https://github.com/mr3project/hive-mr3 For dealing with the difficulty of operating Hive 3.1.2, there is Hive on MR3 - no need to configure LLAP daemons, little dependence on the Hadoop version, as fast as Hive-LLAP, and so on. Quickstart guides are available ( https://mr3docs.datamonad.com/docs/quick/hadoop/), and tutorials will be published in the next release. If you can use Kubernetes, you can easily run Hive on MR3 with Ranger 2.1.0/2.0.0 ( https://mr3docs.datamonad.com/docs/k8s/guide/). Disclaimer: I am the main developer of MR3. --- Sungwoo On Tue, Sep 14, 2021 at 9:43 PM Antoine DUBOIS wrote: > Hello > After trying to use hive 3.1.2 for several weeks with ranger, I stop. > It's seems way too complicated and tedious. > I wonder when or even if there will be any more release in the 3.0 branch. > I wonder if Hive 3.0 was just an experience as it seems maintenance is not > really there. > Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using > Hadoop 3 ? > Any insight on hive release cycle woudl be awesome. > > i hope you have a nice day. > > Antoine DUBOIS > >
Future release of hive
Hello After trying to use hive 3.1.2 for several weeks with ranger, I stop. It's seems way too complicated and tedious. I wonder when or even if there will be any more release in the 3.0 branch. I wonder if Hive 3.0 was just an experience as it seems maintenance is not really there. Is there any plan for Hive 4.0 or should I use Hive 2.8 knowing I'm using Hadoop 3 ? Any insight on hive release cycle woudl be awesome. i hope you have a nice day. Antoine DUBOIS smime.p7s Description: S/MIME Cryptographic Signature
