[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844512#comment-17844512
 ] 

kirby zhou commented on RANGER-3998:


It gets 2 ship now.

Who can merge it ?

> Support Ranger KMS integration with AWS KMS
> ---
>
> Key: RANGER-3998
> URL: https://issues.apache.org/jira/browse/RANGER-3998
> Project: Ranger
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.0, 2.4.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834769#comment-17834769
 ] 

kirby zhou commented on RANGER-3998:


This patch is just a simple imitation of RangerGoogleCloudHSMProvider.

The work is done for using the key stored in AWS KMS as the master key of 
Ranger KMS.
 
class RangerAWSKMSProvider just implements RangerKMSMKI interface. 
 
The generateMasterKey method does not actually create a masterkey, It calls 
AWSKMS.listAliases and AWSKMS.getKeyMetadata to verify whether the masterkey 
exists.
 
The encryptZoneKey method calls AWSKMS.encrypt to encrypt zone key, and the 
decryptZoneKey calls AWSKMS.decrypt to decrypt.
 
RangerKeyStoreProvider.java is modified to load and activate 
RangerAWSKMSProvider according to the configuration.
 
I add 5 lines in install.properties, the meaning is 
 * #- Ranger AWS KMS --
AWS_KMS_ENABLED=false
AWS_KMS_MASTERKEY_ID=#The id of master key in AWS KMS
AWS_CLIENT_ACCESSKEY=#The access key to AWS service
AWS_CLIENT_SECRETKEY=#The secret key to AWS service
AWS_CLIENT_REGION=#The region of AWS service
 
The modification of setup.sh will map the 5 properties into dbks-site.xml as
 * AWS_KMS_ENABLED = "ranger.kms.awskms.enabled";
 
 * AWSKMS_MASTER_KEY_ID = "ranger.kms.awskms.masterkey.id";
 * AWS_CLIENT_ACCESSKEY = "ranger.kms.aws.client.accesskey";
 * AWS_CLIENT_SECRETKEY = "ranger.kms.aws.client.secretkey";
 * AWS_CLIENT_REGION = "ranger.kms.aws.client.region";
 

And the patch do some minor changes to prevent conflicting of Tencent KMS.

 

BTW: AWS KMS API is here:

[https://docs.aws.amazon.com/kms/latest/developerguide/programming-top.html]

 

 

> Support Ranger KMS integration with AWS KMS
> ---
>
> Key: RANGER-3998
> URL: https://issues.apache.org/jira/browse/RANGER-3998
> Project: Ranger
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.0, 2.4.0
>Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-4454) RangerKMS adds support for the SM4 encryption algorithm.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832450#comment-17832450
 ] 

kirby zhou commented on RANGER-4454:


I think we should also bump the version org.bouncycastle.* to versions of 
jdk18on. 

The old versions have CVEs.

[https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on/1.70]

 

> RangerKMS adds support for the SM4 encryption algorithm.
> 
>
> Key: RANGER-4454
> URL: https://issues.apache.org/jira/browse/RANGER-4454
> Project: Ranger
>  Issue Type: New Feature
>  Components: kms, Ranger
>Affects Versions: 2.3.0
> Environment:  !image-2023-10-04-08-31-03-261.png! 
>Reporter: xiaojunxiang
>Priority: Major
> Attachments: HDFS_SM4.jpg, Jira_HDFS_SM4.jpg, RANGER-4454-000.patch, 
> SM4_NotAvaliable.jpg
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> SM4 is already supported in recent versions (3.4.0) of hdfs transparent 
> encryption, 
> So RangerKMS should adapt to this.
> When I add in the region of the Encryption key used "SM4 / CTR/NoPadding" 
> algorithm, RangerKMS background will print "under Caused by: Java security. 
> NoSuchAlgorithmException: SM4 KeyGenerator not available"
>  
> Hadoop website: 
> [https://apache.github.io/hadoop/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html]
> Jira(HDFS supported SM4):   https://issues.apache.org/jira/browse/HDFS-15098  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

> On 三月 27, 2024, 11:43 a.m., Dhaval Shah wrote:
> > kms/pom.xml
> > Lines 425 (patched)
> > <https://reviews.apache.org/r/74259/diff/8/?file=2286227#file2286227line425>
> >
> > Initialize the version

This version is already in ranger's pom.

https://github.com/apache/ranger/commit/27456a9c634cd3747da890a34d1d765d91685224


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/#review226354
-------


On 三月 23, 2024, 2:20 a.m., Kirby Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74259/
> ---
> 
> (Updated 三月 23, 2024, 2:20 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
> Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
> Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: ranger-3998
> https://issues.apache.org/jira/browse/ranger-3998
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
> 
> 
> Diffs
> -
> 
>   distro/src/main/assembly/kms.xml 77f61b917 
>   kms/config/kms-webapp/dbks-site.xml 258d331d8 
>   kms/pom.xml 0b37ce52d 
>   kms/scripts/install.properties 5f5df9b1f 
>   kms/scripts/setup.sh 0cce478fd 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
> PRE-CREATION 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
> 429d1ce45 
>   
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
> 1f0125967 
> 
> 
> Diff: https://reviews.apache.org/r/74259/diff/8/
> 
> 
> Testing
> ---
> 
> Tested under macos-13 and centos-8
> 
> 
> File Attachments
> 
> 
> 0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
>   
> https://reviews.apache.org/media/uploaded/files/2023/04/07/6499bf0e-af49-4eae-8636-f183f0fab3d7__0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
> 
> 
> Thanks,
> 
> Kirby Zhou
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

> On 三月 27, 2024, 11:36 a.m., Dhaval Shah wrote:
> > distro/src/main/assembly/kms.xml
> > Line 182 (original), 185 (patched)
> > <https://reviews.apache.org/r/74259/diff/8/?file=2286225#file2286225line185>
> >
> > Do we need tencent KMS dependency for AWS KMS?

This is about streamlining dependencies to reduce the risk of conflicts.


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/#review226353
-------


On 三月 23, 2024, 2:20 a.m., Kirby Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74259/
> ---
> 
> (Updated 三月 23, 2024, 2:20 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
> Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
> Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: ranger-3998
> https://issues.apache.org/jira/browse/ranger-3998
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
> 
> 
> Diffs
> -
> 
>   distro/src/main/assembly/kms.xml 77f61b917 
>   kms/config/kms-webapp/dbks-site.xml 258d331d8 
>   kms/pom.xml 0b37ce52d 
>   kms/scripts/install.properties 5f5df9b1f 
>   kms/scripts/setup.sh 0cce478fd 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
> PRE-CREATION 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
> 429d1ce45 
>   
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
> 1f0125967 
> 
> 
> Diff: https://reviews.apache.org/r/74259/diff/8/
> 
> 
> Testing
> ---
> 
> Tested under macos-13 and centos-8
> 
> 
> File Attachments
> 
> 
> 0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
>   
> https://reviews.apache.org/media/uploaded/files/2023/04/07/6499bf0e-af49-4eae-8636-f183f0fab3d7__0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
> 
> 
> Thanks,
> 
> Kirby Zhou
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

(Updated 三月 23, 2024, 2:20 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Changes
---

Add missing file


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs (updated)
-

  distro/src/main/assembly/kms.xml 77f61b917 
  kms/config/kms-webapp/dbks-site.xml 258d331d8 
  kms/pom.xml 0b37ce52d 
  kms/scripts/install.properties 5f5df9b1f 
  kms/scripts/setup.sh 0cce478fd 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
429d1ce45 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
1f0125967 


Diff: https://reviews.apache.org/r/74259/diff/8/

Changes: https://reviews.apache.org/r/74259/diff/7-8/


Testing
---

Tested under macos-13 and centos-8


File Attachments


0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
  
https://reviews.apache.org/media/uploaded/files/2023/04/07/6499bf0e-af49-4eae-8636-f183f0fab3d7__0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch


Thanks,

Kirby Zhou

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

(Updated 三月 21, 2024, 9:21 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Changes
---

sync with HEAD


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs (updated)
-

  distro/src/main/assembly/kms.xml 77f61b917 
  kms/config/kms-webapp/dbks-site.xml 258d331d8 
  kms/pom.xml 2739bb81a 
  kms/scripts/install.properties 5f5df9b1f 
  kms/scripts/setup.sh 0cce478fd 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
429d1ce45 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
1f0125967 


Diff: https://reviews.apache.org/r/74259/diff/7/

Changes: https://reviews.apache.org/r/74259/diff/6-7/


Testing
---

Tested under macos-13 and centos-8


File Attachments


0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
  
https://reviews.apache.org/media/uploaded/files/2023/04/07/6499bf0e-af49-4eae-8636-f183f0fab3d7__0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch


Thanks,

Kirby Zhou

Re: Review Request 74942: RANGER-3831 Add support of pegasus to ranger

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74942/
---

(Updated 三月 21, 2024, 4:23 a.m.)


Review request for ranger, Bhavik Bavishi and Madhan Neethiraj.


Changes
---

fix typo


Bugs: RANGER-3831
https://issues.apache.org/jira/browse/RANGER-3831


Repository: ranger


Description
---

Apache Pegasus is A horizontally scalable, strongly consistent and 
high-performance key-value store.
It now have ACLs and SASL, but do not related to ranger.
We suggest to add support to it.

Pegasus have done its work.
https://github.com/apache/incubator-pegasus/issues/1054
Could anyone do some work to merge this definition into ranger?


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
 92ce183bb 
  agents-common/src/main/resources/service-defs/ranger-servicedef-pegasus.json 
PRE-CREATION 
  distro/src/main/assembly/admin-web.xml 245d9ca09 
  plugin-pegasus/pom.xml PRE-CREATION 
  
plugin-pegasus/src/main/java/org/apache/ranger/services/pegasus/RangerServicePegasus.java
 PRE-CREATION 
  pom.xml cec2390f2 


Diff: https://reviews.apache.org/r/74942/diff/2/

Changes: https://reviews.apache.org/r/74942/diff/1-2/


Testing
---

Done by our mini site.


Thanks,

Kirby Zhou

Review Request 74942: RANGER-3831 Add support of pegasus to ranger

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74942/
---

Review request for ranger, Bhavik Bavishi and Madhan Neethiraj.


Bugs: RANGER-3831
https://issues.apache.org/jira/browse/RANGER-3831


Repository: ranger


Description
---

Apache Pegasus is A horizontally scalable, strongly consistent and 
high-performance key-value store.
It now have ACLs and SASL, but do not related to ranger.
We suggest to add support to it.

Pegasus have done its work.
https://github.com/apache/incubator-pegasus/issues/1054
Could anyone do some work to merge this definition into ranger?


Diffs
-

  
agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java
 92ce183bb 
  agents-common/src/main/resources/service-defs/ranger-servicedef-pegasus.json 
PRE-CREATION 
  distro/src/main/assembly/admin-web.xml 245d9ca09 
  plugin-pegasus/pom.xml PRE-CREATION 
  
plugin-pegasus/src/main/java/org/apache/ranger/services/pegasus/RangerServicePegasus.java
 PRE-CREATION 
  pom.xml cec2390f2 


Diff: https://reviews.apache.org/r/74942/diff/1/


Testing
---

Done by our mini site.


Thanks,

Kirby Zhou

[jira] [Commented] (RANGER-3831) Add support of pegasus to ranger

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828657#comment-17828657
 ] 

kirby zhou commented on RANGER-3831:


pegasus have done its work.

[https://github.com/apache/incubator-pegasus/issues/1054]

Could anyone do some work to merge this definition into ranger-admin ?

> Add support of pegasus to ranger
> 
>
> Key: RANGER-3831
> URL: https://issues.apache.org/jira/browse/RANGER-3831
> Project: Ranger
>  Issue Type: Improvement
>  Components: admin, plugins
>Affects Versions: 3.0.0
>    Reporter: kirby zhou
>Priority: Major
> Attachments: ranger-servicedef-pegasus.json
>
>
> Apache Pegasus is A horizontally scalable, strongly consistent and 
> high-performance key-value store.
> It now have ACLs and SASL, but do not related to ranger.
> I suggest to add support to it.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3409) Update Jackson and remove Codehaus version

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17803502#comment-17803502
 ] 

kirby zhou commented on RANGER-3409:


Is there any progress in this matter? 

The security Commissioner is asking to avoid Codehaus Jackson. :(

> Update Jackson and remove Codehaus version
> --
>
> Key: RANGER-3409
> URL: https://issues.apache.org/jira/browse/RANGER-3409
> Project: Ranger
>  Issue Type: Improvement
>  Components: Ranger
>Affects Versions: 3.0.0
>Reporter: Andrew Charneski
>Priority: Blocker
>
> An old version of Jackson (Codehaus Jackson 1.9.13) is still being used. 
> Jackson has since moved namespaces with a reorganized library structure. 
> Update all references to the older version to use the newer version (which is 
> currently used in some modules).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750928#comment-17750928
 ] 

kirby zhou commented on RANGER-4326:


Sorry, I found the reason.

2 KMS need to be synced with ZooKeeper. the following example is missing in 
kms-site.xml
{code:java}

hadoop.kms.authentication.zk-dt-secret-manager.enable
true

  If true, Hadoop KMS uses ZKDelegationTokenSecretManager to persist
  TokenIdentifiers and DelegationKeys in ZooKeeper.

  
  

hadoop.kms.authentication.zk-dt-secret-manager.zkConnectionString
#HOSTNAME#:#PORT#,...

  The ZooKeeper connection string, a comma-separated list of hostnames and 
port.

  
  
hadoop.kms.authentication.zk-dt-secret-manager.znodeWorkingPath
/hadoop-kms/zkdtsm

  The ZooKeeper znode path where the KMS instances will store and retrieve
  the secret from. All the KMS instances that need to coordinate should 
point to the same path.

  
  
hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType
sasl

  The ZooKeeper authentication type, 'none' (default) or 'sasl' (Kerberos).

  
  
hadoop.kms.authentication.zk-dt-secret-manager.kerberos.keytab
/etc/hadoop/conf/kms.keytab

  The absolute path for the Kerberos keytab with the credentials to
  connect to ZooKeeper. This parameter is effective only when
  hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType is set to 
'sasl'.

  
  

hadoop.kms.authentication.zk-dt-secret-manager.kerberos.principal
kms/#HOSTNAME#

  The Kerberos service principal used to connect to ZooKeeper.
  This parameter is effective only when
  hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType is set to 
'sasl'.

  

{code}

> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.3.0, 2.4.0
>    Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew 
> tokens.
>  
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N 
> KMSClientProvider.getDelegationToken().
>  
> When renew the token, LoadBalancingKMSClientProvider may call another 
> KMSClientProvider to do op. It usually fails.
>  
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
> and hadoop.kms.authentication.signature.secret.file="same content file".
>  
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path", 
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
> 
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println

[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746760#comment-17746760
 ] 

kirby zhou commented on RANGER-4326:


Check the code of KMS and hadoop, It seems that two KMS token cannot recognize 
each other without ZK.

If we do not take zk-dt-secret-manager.enable = true in some conf.
 # KMS compares the password in the token with the password calculated by 
itself.
 # The calculated  password depends on getDelegationKey(id.getMasterKeyId());
 # delegationKey is generated by updateCurrentKey, and saved by 
storeDelegationKey.
 # updateCurrentKey is based on random, so 2 KMS are not sync.

new DelegationKey(newCurrentId, System .currentTimeMillis() + keyUpdateInterval 
+ tokenMaxLifetime, {+}*generateSecret*{+}()); 

 

 
{code:java}
// AbstractDelegationTokenSecretManager.java

public synchronized long renewToken(Token token,
   String renewer) throws InvalidToken, IOException {

DelegationKey key = getDelegationKey(id.getMasterKeyId());

byte[] password = createPassword(token.getIdentifier(), key.getKey());
if (!MessageDigest.isEqual(password, token.getPassword())) {
  throw new AccessControlException(renewer
  + " is trying to renew a token "
  + formatTokenId(id) + " with wrong password");
}

}

protected DelegationKey getDelegationKey(int keyId) {
  return allKeys.get(keyId);
}

protected void storeDelegationKey(DelegationKey key) throws IOException {
  allKeys.put(key.getKeyId(), key);
  storeNewMasterKey(key);
}
 
private void updateCurrentKey() throws IOException {
  LOG.info("Updating the current master key for generating delegation tokens");
  /* Create a new currentKey with an estimated expiry date. */
  int newCurrentId;
  synchronized (this) {
newCurrentId = incrementCurrentKeyId();
  }
  DelegationKey newKey = new DelegationKey(newCurrentId, System
  .currentTimeMillis()
  + keyUpdateInterval + tokenMaxLifetime, generateSecret());
  //Log must be invoked outside the lock on 'this'
  logUpdateMasterKey(newKey);
  synchronized (this) {
currentKey = newKey;
storeDelegationKey(currentKey);
  }
}

protected SecretKey generateSecret() {
  SecretKey key;
  synchronized (keyGen) {
key = keyGen.generateKey();
  }
  return key;
}
 {code}
 

 

 

> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>    Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew 
> tokens.
>  
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N 
> KMSClientProvider.getDelegationToken().
>  
> When renew the token, LoadBalancingKMSClientProvider may call another 
> KMSClientProvider to do op. It usually fails.
>  
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
> and hadoop.kms.authentication.signature.secret.file="same content file".
>  
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path", 
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
> 
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.pri

[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746749#comment-17746749
 ] 

kirby zhou commented on RANGER-4326:


{code:java}

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        conf.set("hadoop.security.authorization", "true");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("dfs.data.transfer.protection", "authentication");
        final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
        final String kmsURI1 = 
"kms://h...@kms01-throne01.sensorsdata.cn:9292/kms";
        final String kmsURI2 = 
"kms://h...@kms02-throne01.sensorsdata.cn:9292/kms";
        // Logon
        UserGroupInformation.setConfiguration(conf);
        UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
        UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
        String username = ugi.getShortUserName();
        System.out.println(username);
        // new keyproider
        KMSClientProvider kms1 = new KMSClientProvider(new URI(kmsURI1), conf);
        KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
       // do renew
       Token token1 = kms1.getDelegationToken(username);
       token1.setService(new Text(dtCombineService));
       System.out.println("renew token1 through kms2 begin");
       kms2.renewDelegationToken(token1);
       System.out.println("renew token1 through kms2 done");
    }

{code}

> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew 
> tokens.
>  
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N 
> KMSClientProvider.getDelegationToken().
>  
> When renew the token, LoadBalancingKMSClientProvider may call another 
> KMSClientProvider to do op. It usually fails.
>  
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
> and hadoop.kms.authentication.signature.secret.file="same content file".
>  
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path", 
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
> 
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
>  {code}

[jira] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

[ https://issues.apache.org/jira/browse/RANGER-4326 ]


kirby zhou deleted comment on RANGER-4326:


was (Author: kirbyzhou):
// A more simpler example to reproduce    public static void main(String[] 
args) throws Exception {
        Configuration conf = new Configuration();
        conf.set("hadoop.security.authorization", "true");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("dfs.data.transfer.protection", "authentication");
        final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
        final String kmsURI1 = "kms://http@kms01/kms";
        final String kmsURI2 = "kms://http@kms02/kms"; 
        // Logon        UserGroupInformation.setConfiguration(conf);
        UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
        UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
        String username = ugi.getShortUserName();
        System.out.println(username);
        // new keyproider        KMSClientProvider kms1 = new 
KMSClientProvider(new URI(kmsURI1), conf);
        KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
        // do renew        Token token1 = kms1.getDelegationToken(username);
        token1.setService(new Text(dtCombineService));
        System.out.println("renew token1 through kms2 begin");
        kms2.renewDelegationToken(token1);
        System.out.println("renew token1 through kms2 done");
    }

> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew 
> tokens.
>  
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N 
> KMSClientProvider.getDelegationToken().
>  
> When renew the token, LoadBalancingKMSClientProvider may call another 
> KMSClientProvider to do op. It usually fails.
>  
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
> and hadoop.kms.authentication.signature.secret.file="same content file".
>  
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path", 
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
> 
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
>  {code}
> A lot of exceptions happens
> {code:java}
> ava.io.IOException: HTTP status [403], m

[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746747#comment-17746747
 ] 

kirby zhou commented on RANGER-4326:


// A more simpler example to reproduce    public static void main(String[] 
args) throws Exception {
        Configuration conf = new Configuration();
        conf.set("hadoop.security.authorization", "true");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("dfs.data.transfer.protection", "authentication");
        final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
        final String kmsURI1 = "kms://http@kms01/kms";
        final String kmsURI2 = "kms://http@kms02/kms"; 
        // Logon        UserGroupInformation.setConfiguration(conf);
        UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
        UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
        String username = ugi.getShortUserName();
        System.out.println(username);
        // new keyproider        KMSClientProvider kms1 = new 
KMSClientProvider(new URI(kmsURI1), conf);
        KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
        // do renew        Token token1 = kms1.getDelegationToken(username);
        token1.setService(new Text(dtCombineService));
        System.out.println("renew token1 through kms2 begin");
        kms2.renewDelegationToken(token1);
        System.out.println("renew token1 through kms2 done");
    }

> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew 
> tokens.
>  
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N 
> KMSClientProvider.getDelegationToken().
>  
> When renew the token, LoadBalancingKMSClientProvider may call another 
> KMSClientProvider to do op. It usually fails.
>  
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
> and hadoop.kms.authentication.signature.secret.file="same content file".
>  
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path", 
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
> 
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
>  {code}
> A lot of exceptions happens

[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746745#comment-17746745
 ] 

kirby zhou commented on RANGER-4326:


{code:java}
// A simpler example to reproduce

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        conf.set("hadoop.security.authorization", "true");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("dfs.data.transfer.protection", "authentication");
        final String KMSURI = "kms://http@kms01;kms02:9292/kms";
        final String keyName = "mykey";
        // Logon
        UserGroupInformation.setConfiguration(conf);
        UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
        UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
        String username = ugi.getShortUserName();
        System.out.println(username);
        // new keyproider
        LoadBalancingKMSClientProvider kms = 
(LoadBalancingKMSClientProvider)KeyProviderFactory.get(new URI(KMSURI), conf);
        // try eek & dek
        System.out.println("try do eek & dek");
        KeyProviderCryptoExtension.EncryptedKeyVersion eek = 
kms.generateEncryptedKey(keyName);
        System.out.printf("IV(%d) %s\n", eek.getEncryptedKeyIv().length * 8,
                StringUtils.byteToHexString(eek.getEncryptedKeyIv()));
        System.out.printf("EEK(%d) %s\n", 
eek.getEncryptedKeyVersion().getMaterial().length * 8,
                
StringUtils.byteToHexString(eek.getEncryptedKeyVersion().getMaterial()));
        KeyProvider.KeyVersion dek = kms.decryptEncryptedKey(eek);
        StringUtils.byteToHexString(dek.getMaterial());
        System.out.printf("DEK(%d) %s\n", dek.getMaterial().length * 8,
                StringUtils.byteToHexString(dek.getMaterial()));
        // do renew
        for (int i = 0; i < 10; ++i) {
            System.out.printf("pass %02d\n", i);
            System.out.print("begin renew\n");
            Token token = kms.getDelegationToken(username);
            kms.renewDelegationToken(token);
            System.out.print("end renew\n");
        }
    }{code}

> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew 
> tokens.
>  
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N 
> KMSClientProvider.getDelegationToken().
>  
> When renew the token, LoadBalancingKMSClientProvider may call another 
> KMSClientProvider to do op. It usually fails.
>  
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
> and hadoop.kms.authentication.signature.secret.file="same content file".
>  
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path", 
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
> 
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf

[jira] [Created] (RANGER-4326) Cannot renew token when multiple KMS are applied.

[kirby zhou (Jira)]

kirby zhou created RANGER-4326:
--

 Summary: Cannot renew token when multiple KMS are applied.
 Key: RANGER-4326
 URL: https://issues.apache.org/jira/browse/RANGER-4326
 Project: Ranger
  Issue Type: Bug
  Components: kms
Affects Versions: 2.4.0, 2.3.0
Reporter: kirby zhou


When multiple KMS are applied with kerberos. Flink on yarn can not renew tokens.

 

Flink calls FileSystem.addDelegationTokens to get all tokens to renew.

FileSystem.addDelegationTokens calls collectDelegationTokens to collect all 
tokens.

When it calls LoadBalancingKMSClientProvider.getDelegationToken.

LoadBalancingKMSClientProvider calls doOp to call one of N 
KMSClientProvider.getDelegationToken().

 

When renew the token, LoadBalancingKMSClientProvider may call another 
KMSClientProvider to do op. It usually fails.

 

FYI: have already set hadoop.kms.authentication.signer.secret.provider=file, 
and hadoop.kms.authentication.signature.secret.file="same content file".

 

Some Sample code:
{code:java}

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
conf.set("hadoop.security.key.provider.path", 
"kms://http@kms01;kms02:9292/kms");
conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
conf.set("fs.defaultFS", "hdfs://namenode");
// Login with keytab
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM", 
"/Users/kirbyzhou/Develop/testuser.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
System.out.println(UserGroupInformation.getCurrentUser().getUserName());
// GetFS
FileSystem fs = FileSystem.get(conf);

System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
// Renew
for (int i = 0; i < 20; ++i) {
Thread.sleep(200);
System.out.printf("===pass %02d===\n", i);
{
System.out.println("==begin renew==");
Credentials credentials = ugi.getCredentials();
fs.addDelegationTokens("sa_cluster", credentials);
for (Token token : credentials.getAllTokens()) {
System.out.println(token);
try {
token.renew(conf);
} catch (IOException e) {
System.err.println(e);
}
}
System.out.println("==end renew==");
}
}
}
 {code}
A lot of exceptions happens
{code:java}

ava.io.IOException: HTTP status [403], message [Forbidden], URL 
[http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
 exception [com.fasterxml.jackson.core.JsonParseException: Unexpected character 
('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or 
token 'null', 'true' or 'false') at [Source: 
(sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column: 
2]]at 
org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:167)
 ~[classes/:?]at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:318)
 ~[hadoop-common-3.3.4.jar:?]  at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:235)
 ~[hadoop-common-3.3.4.jar:?]at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:435)
 ~[hadoop-common-3.3.4.jar:?]  at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1072)
 ~[hadoop-common-3.3.4.jar:?]   at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1069)
 ~[hadoop-common-3.3.4.jar:?]   at 
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332]at 
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
 ~[hadoop-common-3.3.4.jar:?]at 
org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:1068)
 ~[hadoop-common-3.3.4.jar:?]at 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

> On 三月 30, 2023, 11:27 a.m., Kishor Gollapalliwar wrote:
> > kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java
> > Lines 24 (patched)
> > <https://reviews.apache.org/r/74259/diff/5/?file=2273861#file2273861line24>
> >
> > Please consider importing individual classes instead of all.

Can you ship it now?


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/#review225323
-------


On 四月 7, 2023, 11:16 a.m., Kirby Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74259/
> ---
> 
> (Updated 四月 7, 2023, 11:16 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
> Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
> Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: ranger-3998
> https://issues.apache.org/jira/browse/ranger-3998
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
> 
> 
> Diffs
> -
> 
>   distro/src/main/assembly/kms.xml 4b4a2ac8e15209a8abc0fc72fba464257f585cfc 
>   kms/config/kms-webapp/dbks-site.xml 
> 258d331d82ae4051c261301a6dd1cf980d198ab8 
>   kms/pom.xml e97b993d72a41ffa6d1faaed52666ebf76ee4d5d 
>   kms/scripts/install.properties 0e5da3c756343d655fedf7f286a08d3458b8d6c6 
>   kms/scripts/setup.sh f723e09bbe77f2d9ad064923da42a32f82bb40f1 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
> PRE-CREATION 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
> 429d1ce45e39c6bb1279a90fedf73776b5b909ba 
>   
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
> 1f01259675a536e5e02da51b2b1c9d438314c1de 
> 
> 
> Diff: https://reviews.apache.org/r/74259/diff/6/
> 
> 
> Testing
> ---
> 
> Tested under macos-13 and centos-8
> 
> 
> File Attachments
> 
> 
> 0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
>   
> https://reviews.apache.org/media/uploaded/files/2023/04/07/6499bf0e-af49-4eae-8636-f183f0fab3d7__0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
> 
> 
> Thanks,
> 
> Kirby Zhou
> 
>

[jira] [Commented] (RANGER-4147) Ranger KMS consume 50% of CPU memory

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714387#comment-17714387
 ] 

kirby zhou commented on RANGER-4147:


Maybe you can try using jstack tool to see what KMS is doing. When KMS consumes 
150% CPU.

 

> Ranger KMS consume 50% of CPU memory 
> -
>
> Key: RANGER-4147
> URL: https://issues.apache.org/jira/browse/RANGER-4147
> Project: Ranger
>  Issue Type: Bug
>  Components: kms
>Affects Versions: 2.3.0
> Environment: secured
>Reporter: Bhavik Patel
>Priority: Critical
>
> Ranger KMS consume 50% of CPU memory and many time it consume even 150%
> cc: [~dhavalshah9131] [~kirbyzhou] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74387: RANGER-4178 : NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/vector/ColumnVector

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74387/#review225399
---




agents-audit/src/main/java/org/apache/ranger/audit/utils/RangerORCAuditWriter.java
Line 177 (original), 177 (patched)
<https://reviews.apache.org/r/74387/#comment314029>

auditProviderName is not used here?


- Kirby Zhou


On 四月 18, 2023, 1:43 p.m., bhavik patel wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74387/
> ---
> 
> (Updated 四月 18, 2023, 1:43 p.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Kirby Zhou, Abhay Kulkarni, Madhan 
> Neethiraj, Pradeep Agrawal, Ramesh Mani, ru jia, Vishal Suvagia, and 
> zhouyifan279.
> 
> 
> Bugs: RANGER-4178
> https://issues.apache.org/jira/browse/RANGER-4178
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Observed below error when enabled audit type as ORC format.
> 
> NoClassDefFoundError: org/apache/hadoop/hive/ql/exec/vector/ColumnVector
> 
> 
> Diffs
> -
> 
>   agents-audit/pom.xml aba33e227 
>   
> agents-audit/src/main/java/org/apache/ranger/audit/utils/RangerORCAuditWriter.java
>  26d2b433c 
>   distro/src/main/assembly/hbase-agent.xml ff53ca2c2 
>   distro/src/main/assembly/hdfs-agent.xml 15254c9da 
>   distro/src/main/assembly/kms.xml 4b4a2ac8e 
>   distro/src/main/assembly/knox-agent.xml fa92ea0dc 
>   distro/src/main/assembly/plugin-atlas.xml d35061274 
>   distro/src/main/assembly/plugin-elasticsearch.xml 0b8aaee27 
>   distro/src/main/assembly/plugin-kafka.xml ed8ef6159 
>   distro/src/main/assembly/plugin-kms.xml 7cf8dd702 
>   distro/src/main/assembly/plugin-kylin.xml 74b9f4362 
>   distro/src/main/assembly/plugin-ozone.xml 1b5d1cdc7 
>   distro/src/main/assembly/plugin-presto.xml 82d1610aa 
>   distro/src/main/assembly/plugin-solr.xml 382b57092 
>   distro/src/main/assembly/plugin-sqoop.xml 13f74dc79 
>   distro/src/main/assembly/plugin-trino.xml 60b083ed6 
>   distro/src/main/assembly/plugin-yarn.xml c0a8ca3af 
>   distro/src/main/assembly/storm-agent.xml 908415ffa 
>   pom.xml de0617e2a 
> 
> 
> Diff: https://reviews.apache.org/r/74387/diff/2/
> 
> 
> Testing
> ---
> 
> Verified on Dev environment for HDFS, YARN, HIVE and HBASE plugins.
> 
> 
> Thanks,
> 
> bhavik patel
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

(Updated 四月 7, 2023, 11:16 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Changes
---

fix import xxx.*


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs (updated)
-

  distro/src/main/assembly/kms.xml 4b4a2ac8e15209a8abc0fc72fba464257f585cfc 
  kms/config/kms-webapp/dbks-site.xml 258d331d82ae4051c261301a6dd1cf980d198ab8 
  kms/pom.xml e97b993d72a41ffa6d1faaed52666ebf76ee4d5d 
  kms/scripts/install.properties 0e5da3c756343d655fedf7f286a08d3458b8d6c6 
  kms/scripts/setup.sh f723e09bbe77f2d9ad064923da42a32f82bb40f1 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
429d1ce45e39c6bb1279a90fedf73776b5b909ba 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
1f01259675a536e5e02da51b2b1c9d438314c1de 


Diff: https://reviews.apache.org/r/74259/diff/6/

Changes: https://reviews.apache.org/r/74259/diff/5-6/


Testing
---

Tested under macos-13 and centos-8


File Attachments (updated)


0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch
  
https://reviews.apache.org/media/uploaded/files/2023/04/07/6499bf0e-af49-4eae-8636-f183f0fab3d7__0001-RANGER-3998-KMS-integration-with-AWS-KMS.patch


Thanks,

Kirby Zhou

Re: [VOTE] Apache Ranger 2.4.0 Release - rc2

[KirbY ZhoU]

+1 for Apache Ranger 2.4.0-rc2

在 2023/3/28 11:41，“Selvamohan Neethiraj”mailto:sneet...@apache.org>> 写入:


Rangers:


Apache Ranger 2.4.0 release candidate #2 is now available for a vote within the 
dev community.
Links to the release artifacts are given below. Please review and vote.


The vote will be open for at least 72 hours or until necessary votes are 
reached.
[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)


Thanks,
Selva-
Ranger PMC 


List of issues / improvements addressed in this release: click-here 

 



Git tag for the release: 
https://github.com/apache/ranger/tree/release-2.4.0-rc2 

Sources for the release: 
https://dist.apache.org/repos/dist/dev/ranger/2.4.0-rc2/apache-ranger-2.4.0.tar.gz
 



Source release verification:
PGP Signature: 
https://dist.apache.org/repos/dist/dev/ranger/2.4.0-rc2/apache-ranger-2.4.0.tar.gz.asc
 

SHA256 Hash: 
https://dist.apache.org/repos/dist/dev/ranger/2.4.0-rc2/apache-ranger-2.4.0.tar.gz.sha256
 

SHA512 Hash: 
https://dist.apache.org/repos/dist/dev/ranger/2.4.0-rc2/apache-ranger-2.4.0.tar.gz.sha512
 



Keys to verify the signature: 
https://dist.apache.org/repos/dist/release/ranger/KEYS 



Click Here 
(%22New%20Feature%22,%20Improvement)%20ORDER%20BY%20key%20desc>
 to view New Features/Enhancements in this release.

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/#review225314
---



anybody else can review it and merge?

- Kirby Zhou


On 一月 17, 2023, 9:49 a.m., Kirby Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74259/
> ---
> 
> (Updated 一月 17, 2023, 9:49 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
> Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
> Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: ranger-3998
> https://issues.apache.org/jira/browse/ranger-3998
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
> 
> 
> Diffs
> -
> 
>   distro/src/main/assembly/kms.xml 4b4a2ac8e 
>   kms/config/kms-webapp/dbks-site.xml 258d331d8 
>   kms/pom.xml e97b993d7 
>   kms/scripts/install.properties 0e5da3c75 
>   kms/scripts/setup.sh f723e09bb 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
> PRE-CREATION 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
> 429d1ce45 
>   
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
> 1f0125967 
> 
> 
> Diff: https://reviews.apache.org/r/74259/diff/5/
> 
> 
> Testing
> ---
> 
> Tested under macos-13 and centos-8
> 
> 
> Thanks,
> 
> Kirby Zhou
> 
>

Re: Review Request 74319: RANGER-4108: Speed up loading of the service-definitions during Ranger startup

[Kirby Zhou]

> On 三月 2, 2023, 2:49 a.m., Kirby Zhou wrote:
> > security-admin/db/mysql/patches/062-remove-unused-ranger-tables-v1.sql
> > Lines 19 (patched)
> > <https://reviews.apache.org/r/74319/diff/1/?file=2274752#file2274752line19>
> >
> > Can we avoid using stored procedures? TiDB does not support stored 
> > procedures.
> 
> Pradeep Agrawal wrote:
> How the other old sql patches with stored procedure are working in TiDB. 
> Also if TiDB defers from mysql then TiDB support can be added seperately.

The old sql patches have the same problem……


TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
deploy ranger based on tidb. But TiDB is missing some features, which makes 
ranger unable to install properly.
https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features

The biggest problem affecting ranger is missing "Stored procedures and 
functions", "Select into".
ranger use Stored procedures in setup scripts to simplify SQL.
Some work is needed to remove the stored procedure.

Because few old versions of Ranger are using TiDb, we mainly focus on sql patch 
after ranger-2.3.


https://issues.apache.org/jira/browse/RANGER-3984


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74319/#review225236
---


On 二月 23, 2023, 10:35 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74319/
> ---
> 
> (Updated 二月 23, 2023, 10:35 p.m.)
> 
> 
> Review request for ranger, Abhishek  Kumar, Dineshkumar Yadav, madhan, Madhan 
> Neethiraj, Pradeep Agrawal, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4108
> https://issues.apache.org/jira/browse/RANGER-4108
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Currently, Service Definitions are persisted in several database tables in a 
> normalized form. However, as operations on a Service Definition object is not 
> granular, but act on the object as a whole, there is not much advantage in 
> normalizing it for storage, and a distinct disadvantage of having to recreate 
> it by accessing multiple database tables during the initialization of Ranger 
> admin process.
> 
> This Jira denormalizes service-definition object's persistent state and also 
> removes several tables used only for normalizing service-definition object.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java
>  8632dd6bc 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 
> 9a79fe8ad 
>   security-admin/db/mysql/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/mysql/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 
> fd6cec9a7 
>   security-admin/db/oracle/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/oracle/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 
> 4d5a8cedf 
>   security-admin/db/postgres/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/postgres/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
>  3ed2a5b9c 
>   
> security-admin/db/sqlanywhere/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 
> ca8f7da1f 
>   security-admin/db/sqlserver/patches/059-update-x-portal-user-table.sql 
> 0e5388646 
>   security-admin/db/sqlserver/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 562467e80 
>   security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java bbbf90c52 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> 418557bcb 
>   security-admin/src/main/java/org/apache/ranger/entity/XXAccessTypeDef.java 
> db1878a35 
>   security-admin/src/main/java/org/apache/ranger/entity/XXServiceDef

Re: Review Request 74319: RANGER-4108: Speed up loading of the service-definitions during Ranger startup

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74319/#review225236
---




security-admin/db/mysql/patches/062-remove-unused-ranger-tables-v1.sql
Lines 19 (patched)
<https://reviews.apache.org/r/74319/#comment313971>

Can we avoid using stored procedures? TiDB does not support stored 
procedures.


- Kirby Zhou


On 二月 23, 2023, 10:35 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74319/
> ---
> 
> (Updated 二月 23, 2023, 10:35 p.m.)
> 
> 
> Review request for ranger, Abhishek  Kumar, Dineshkumar Yadav, madhan, Madhan 
> Neethiraj, Pradeep Agrawal, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4108
> https://issues.apache.org/jira/browse/RANGER-4108
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Currently, Service Definitions are persisted in several database tables in a 
> normalized form. However, as operations on a Service Definition object is not 
> granular, but act on the object as a whole, there is not much advantage in 
> normalizing it for storage, and a distinct disadvantage of having to recreate 
> it by accessing multiple database tables during the initialization of Ranger 
> admin process.
> 
> This Jira denormalizes service-definition object's persistent state and also 
> removes several tables used only for normalizing service-definition object.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java
>  8632dd6bc 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 
> 9a79fe8ad 
>   security-admin/db/mysql/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/mysql/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 
> fd6cec9a7 
>   security-admin/db/oracle/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/oracle/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 
> 4d5a8cedf 
>   security-admin/db/postgres/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/postgres/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
>  3ed2a5b9c 
>   
> security-admin/db/sqlanywhere/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 
> ca8f7da1f 
>   security-admin/db/sqlserver/patches/059-update-x-portal-user-table.sql 
> 0e5388646 
>   security-admin/db/sqlserver/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 562467e80 
>   security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java bbbf90c52 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> 418557bcb 
>   security-admin/src/main/java/org/apache/ranger/entity/XXAccessTypeDef.java 
> db1878a35 
>   security-admin/src/main/java/org/apache/ranger/entity/XXServiceDefBase.java 
> 7f8d6193f 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForRemovalOfUnusedRangerDbTables_J10059.java
>  PRE-CREATION 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForSolrSvcDefAndPoliciesUpdate_J10055.java
>  f5f9f8956 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingServiceDefJson_J10058.java
>  PRE-CREATION 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchPreSql_058_ForUpdateToUniqueResoureceSignature_J10053.java
>  fb7e5fd40 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerServiceDefServiceBase.java
>  656bc0184 
>   
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java
>  85e457efa 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 691ab52b3 
>   security-admin/src/test/java/org/apache/ranger/biz/TestXUserMgr.java 
> 528f4e511 
> 
> 
> Diff: https://reviews.apache.org/r/74319/diff/1/
> 
> 
> Testing
> ---
> 
> Ran all unit tests successfully. Ran ranger-admin and component services in a 
> cluster and verified they work correctly.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 74313: RANGER-4104: XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74313/#review225218
---




security-admin/src/main/java/org/apache/ranger/db/XXAuthSessionDao.java
Line 75 (original), 76 (patched)
<https://reviews.apache.org/r/74313/#comment313962>

Why not Timestamp?
I think there is a need for more accurate time accuracy than Date.


- Kirby Zhou


On 二月 21, 2023, 12:35 p.m., Andrew Luo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74313/
> ---
> 
> (Updated 二月 21, 2023, 12:35 p.m.)
> 
> 
> Review request for ranger, Kirby Zhou and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-4104
> https://issues.apache.org/jira/browse/RANGER-4104
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL 
> code due to an error in how the authWindowStartTime Date parameter is bound.
> 
> It is currently bound with setParameter("authWindowStartTime", 
> authWindowStartTime) however, JPA 2.2 Specification Section 11.1.53 says that 
> it should be bound by specifying an additional parameter TemporalType.DATE.
> 
> This causes issues on some database dialects (at least MySQL) as it 
> serializes the date string incorrectly which scans the wrong set of data 
> (causing performance issues since the index cannot be used), and also the 
> functionality introduced in RANGER-2362 does not work properly
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/db/XXAuthSessionDao.java 
> 934d25865 
> 
> 
> Diff: https://reviews.apache.org/r/74313/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrew Luo
> 
>

Re: Review Request 74319: RANGER-4108: Speed up loading of the service-definitions during Ranger startup

[Kirby Zhou]

> On 二月 24, 2023, 2:47 a.m., Madhan Neethiraj wrote:
> > security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql
> > Line 856 (original)
> > 
> >
> > This patch deletes around 20 tables, but corresponding entity/dao 
> > classes have not been deleted. Deleting these classes might be involved; 
> > until this is done, I suggest to not to delete the tables.
> > 
> > This can help to enable rolling upgrades as well i.e. current version 
> > of Ranger will continue to work with newer version of Ranger. This will 
> > however require:
> > - the newer Ranger to continue populating tables used current version 
> > as well, in addition to new column x_service_def.service_def_text
> > - in addition, the newer Ranger should handle service_def updates from 
> > earlier Ranger version - perhaps by having another column like 
> > update_time_v2 and comparing it to update_time column

It is very compliated to rollback.
I suggest renaming these obsolete tables instead of deleting them to avoid 
possible conflicts and keep backups.


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74319/#review225214
---


On 二月 23, 2023, 10:35 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74319/
> ---
> 
> (Updated 二月 23, 2023, 10:35 p.m.)
> 
> 
> Review request for ranger, Abhishek  Kumar, Dineshkumar Yadav, madhan, Madhan 
> Neethiraj, Pradeep Agrawal, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4108
> https://issues.apache.org/jira/browse/RANGER-4108
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Currently, Service Definitions are persisted in several database tables in a 
> normalized form. However, as operations on a Service Definition object is not 
> granular, but act on the object as a whole, there is not much advantage in 
> normalizing it for storage, and a distinct disadvantage of having to recreate 
> it by accessing multiple database tables during the initialization of Ranger 
> admin process.
> 
> This Jira denormalizes service-definition object's persistent state and also 
> removes several tables used only for normalizing service-definition object.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java
>  8632dd6bc 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 
> 9a79fe8ad 
>   security-admin/db/mysql/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/mysql/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 
> fd6cec9a7 
>   security-admin/db/oracle/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/oracle/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 
> 4d5a8cedf 
>   security-admin/db/postgres/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/postgres/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
>  3ed2a5b9c 
>   
> security-admin/db/sqlanywhere/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 
> ca8f7da1f 
>   security-admin/db/sqlserver/patches/059-update-x-portal-user-table.sql 
> 0e5388646 
>   security-admin/db/sqlserver/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 562467e80 
>   security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java bbbf90c52 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> 418557bcb 
>   security-admin/src/main/java/org/apache/ranger/entity/XXAccessTypeDef.java 
> db1878a35 
>   security-admin/src/main/java/org/apache/ranger/entity/XXServiceDefBase.java 
> 7f8d6193f 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForRemovalOfUnusedRangerDbTables_J10059.java
>  PRE-CREATION 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForSolrSvcDefAndPoliciesUpdate_J10055.java
>  f5f9f8956 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingServiceDefJson_J10058.java
>  

Re: Review Request 74319: RANGER-4108: Speed up loading of the service-definitions during Ranger startup

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74319/#review225216
---



Have you tested it on mysql/mariadb with master-slave? Some sql statements have 
restrictions.

- Kirby Zhou


On 二月 23, 2023, 10:35 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74319/
> ---
> 
> (Updated 二月 23, 2023, 10:35 p.m.)
> 
> 
> Review request for ranger, Abhishek  Kumar, Dineshkumar Yadav, madhan, Madhan 
> Neethiraj, Pradeep Agrawal, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4108
> https://issues.apache.org/jira/browse/RANGER-4108
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Currently, Service Definitions are persisted in several database tables in a 
> normalized form. However, as operations on a Service Definition object is not 
> granular, but act on the object as a whole, there is not much advantage in 
> normalizing it for storage, and a distinct disadvantage of having to recreate 
> it by accessing multiple database tables during the initialization of Ranger 
> admin process.
> 
> This Jira denormalizes service-definition object's persistent state and also 
> removes several tables used only for normalizing service-definition object.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/store/AbstractServiceStore.java
>  8632dd6bc 
>   security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 
> 9a79fe8ad 
>   security-admin/db/mysql/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/mysql/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql 
> fd6cec9a7 
>   security-admin/db/oracle/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/oracle/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 
> 4d5a8cedf 
>   security-admin/db/postgres/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/postgres/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql
>  3ed2a5b9c 
>   
> security-admin/db/sqlanywhere/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   
> security-admin/db/sqlanywhere/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 
> ca8f7da1f 
>   security-admin/db/sqlserver/patches/059-update-x-portal-user-table.sql 
> 0e5388646 
>   security-admin/db/sqlserver/patches/061-update-schema-for-x-service-def.sql 
> PRE-CREATION 
>   security-admin/db/sqlserver/patches/062-remove-unused-ranger-tables-v1.sql 
> PRE-CREATION 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 562467e80 
>   security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java bbbf90c52 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> 418557bcb 
>   security-admin/src/main/java/org/apache/ranger/entity/XXAccessTypeDef.java 
> db1878a35 
>   security-admin/src/main/java/org/apache/ranger/entity/XXServiceDefBase.java 
> 7f8d6193f 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForRemovalOfUnusedRangerDbTables_J10059.java
>  PRE-CREATION 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForSolrSvcDefAndPoliciesUpdate_J10055.java
>  f5f9f8956 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingServiceDefJson_J10058.java
>  PRE-CREATION 
>   
> security-admin/src/main/java/org/apache/ranger/patch/PatchPreSql_058_ForUpdateToUniqueResoureceSignature_J10053.java
>  fb7e5fd40 
>   
> security-admin/src/main/java/org/apache/ranger/service/RangerServiceDefServiceBase.java
>  656bc0184 
>   
> security-admin/src/main/java/org/apache/ranger/service/XPortalUserService.java
>  85e457efa 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 691ab52b3 
>   security-admin/src/test/java/org/apache/ranger/biz/TestXUserMgr.java 
> 528f4e511 
> 
> 
> Diff: https://reviews.apache.org/r/74319/diff/1/
> 
> 
> Testing
> ---
> 
> Ran all unit tests successfully. Ran ranger-admin and component services in a 
> cluster and verified they work correctly.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Review Request 74317: Ranger-4106: Fix NullPtr Exception when download roles is not allowed to user.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74317/
---

Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Bugs: RANGER-4106
https://issues.apache.org/jira/browse/RANGER-4106


Repository: ranger


Description
---

```
Feb 23, 2023 7:17:21 AM com.sun.jersey.spi.container.ContainerResponse 
mapMappableContainerException
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to 
the HTTP container
java.lang.NullPointerException
at 
org.apache.ranger.biz.AssetMgr.doCreateOrUpdateXXPluginInfo(AssetMgr.java:831)
at 
org.apache.ranger.biz.AssetMgr.createOrUpdatePluginInfo(AssetMgr.java:791)
at org.apache.ranger.biz.AssetMgr.createPluginInfo(AssetMgr.java:728)
at 
org.apache.ranger.rest.RoleREST.getSecureRangerRolesIfUpdated(RoleREST.java:874)
at 
org.apache.ranger.rest.RoleREST$$FastClassBySpringCGLIB$$d1176b81.invoke()
```

doCreateOrUpdateXXPluginInfo() in AssertMgr.java checks `getRoleDownloadTime()` 
to protect `getRoleDownloadedVersion()`, And getSecureRangerRolesIfUpdated() in 
RoleRest.java did not set downloadedVersion when isAllowed = false.


Diffs
-

  security-admin/src/main/java/org/apache/ranger/biz/AssetMgr.java 
84f5ab1684871e1524d877181fcc51e536054f41 


Diff: https://reviews.apache.org/r/74317/diff/1/


Testing
---

done


Thanks,

Kirby Zhou

Re: Review Request 74112: RANGER-3903:Improvement in RangerPolicyDeltaUtil--> applyDeltas method

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74112/#review225207
---


Ship it!




Ship It!

- Kirby Zhou


On 一月 29, 2023, 1:01 p.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74112/
> ---
> 
> (Updated 一月 29, 2023, 1:01 p.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Kirby Zhou, Abhay Kulkarni, 
> Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, 
> Selvamohan Neethiraj, Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-3903
> https://issues.apache.org/jira/browse/RANGER-3903
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> After going through the below code snippets in the master branch 
> 
> while (iter.hasNext()) {
> RangerPolicy policy = iter.next();
> if (policyId.equals(policy.getId()) && (changeType == 
> RangerPolicyDelta.CHANGE_TYPE_POLICY_DELETE || changeType == 
> RangerPolicyDelta.CHANGE_TYPE_POLICY_UPDATE)) {
> deletedPolicies.add(policy);
> iter.remove();
> }
> }
> switch (changeType) {
> case RangerPolicyDelta.CHANGE_TYPE_POLICY_CREATE:
> {
> if (CollectionUtils.isNotEmpty(deletedPolicies)) {
> LOG.warn("Unexpected: found existing policy for 
> CHANGE_TYPE_POLICY_CREATE: " + Arrays.toString(deletedPolicies.toArray()));
> }
> break;
> }
> case RangerPolicyDelta.CHANGE_TYPE_POLICY_UPDATE:
> {
> if (CollectionUtils.isEmpty(deletedPolicies) || 
> deletedPolicies.size() > 1) {
> LOG.warn("Unexpected: found no policy or multiple policies 
> for CHANGE_TYPE_POLICY_UPDATE: " + 
> Arrays.toString(deletedPolicies.toArray()));
> }
> break;
> }
> case RangerPolicyDelta.CHANGE_TYPE_POLICY_DELETE:
> {
> if (CollectionUtils.isEmpty(deletedPolicies) || 
> deletedPolicies.size() > 1) {
> LOG.warn("Unexpected: found no policy or multiple policies 
> for CHANGE_TYPE_POLICY_DELETE: " + 
> Arrays.toString(deletedPolicies.toArray()));
> }
> break;
> }
> default:
> break;
> }
> 
> 1st#improvement:
> 
> From the above code, we iterate delta policies and check if this policy 
> exists in the existing policy, we add that to deletePolicies list.
> 
> The delta change type condition for created/updated/deleted is added on top 
> of the if the condition so adding the condition again is not necessary 
> 
> 2nd#improvement:
> From the above code, we see for each element in the deltas,we iterate 
> policies and check if this delta policy exists in the existing policy, we add 
> that to deletePolicies list.
> 
> Solution:
> We need to use Map instead of iterating policies for every element of deltas 
> --> Map policiesIdMap
> Building index map key will be policyId and value will be policy associated 
> with the same policyId
> For  each policy in the deltas ,we check on the policiesIdMap whether the 
> same policyId is present or not ?.
> if yes, we will add all the associated policy  into deletedPolicy variable 
> and remove the policyId from policiesIdMap
> After an end of the iteration,we will iterate the policiesIdMap and get all 
> the policy associated with policyId and add into result set
> 
> This will give better performance when the policies list is huge.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPolicyDeltaUtil.java
>  e9223fe69 
> 
> 
> Diff: https://reviews.apache.org/r/74112/diff/9/
> 
> 
> Testing
> ---
> 
> Tested the below Rest API's to make sure everything works fine
> 
> 1.  ServiceREST Rest API :GET /plugins/policies/download/{serviceName}
> 
> 2.  ServiceREST Rest API :GET /plugins/secure/policies/download/{serviceName}
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

[jira] [Created] (RANGER-4106) NullPtr Exception when REST API /service/roles/secure/download/ is not allowed to user.

[kirby zhou (Jira)]

kirby zhou created RANGER-4106:
--

 Summary: NullPtr Exception when REST API 
/service/roles/secure/download/  is not allowed to user.
 Key: RANGER-4106
 URL: https://issues.apache.org/jira/browse/RANGER-4106
 Project: Ranger
  Issue Type: Bug
  Components: admin
Affects Versions: 2.3.0, 3.0.0, 2.4.0
Reporter: kirby zhou


I have see a lot of exceptions in log catalina.out like that:

 
{code:java}
Feb 23, 2023 7:17:21 AM com.sun.jersey.spi.container.ContainerResponse 
mapMappableContainerException
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to 
the HTTP container
java.lang.NullPointerException
        at 
org.apache.ranger.biz.AssetMgr.doCreateOrUpdateXXPluginInfo(AssetMgr.java:831)
        at 
org.apache.ranger.biz.AssetMgr.createOrUpdatePluginInfo(AssetMgr.java:791)
        at org.apache.ranger.biz.AssetMgr.createPluginInfo(AssetMgr.java:728)
        at 
org.apache.ranger.rest.RoleREST.getSecureRangerRolesIfUpdated(RoleREST.java:874)
        at 
org.apache.ranger.rest.RoleREST$$FastClassBySpringCGLIB$$d1176b81.invoke()
...

{code}
 

 

Using debugger to trace the code.

It caused by 

 
{code:java}
// AssertMgr.java doCreateOrUpdateXXPluginInfo()
// which get a null value of RoleDownloadedVersion, raise exception.

831:   if (pluginInfo.getRoleDownloadTime() != null && 
pluginInfo.getRoleDownloadedVersion().equals(pluginInfo.getRoleActiveVersion())

// called by createOrUpdatePluginInfo() in AssertMgr.java 
...
// called by createPluginInfo() in AssetMgr.java
// which will set RoleDownloadTime to non-null, regardless of the value of 
RoleDownloadedVersion/downloadedVersion.

case RangerPluginInfo.ENTITY_TYPE_ROLES:
  pluginSvcVersionInfo.setRoleActiveVersion(lastKnownVersion);
  pluginSvcVersionInfo.setRoleActivationTime(lastActivationTime);
  pluginSvcVersionInfo.setRoleDownloadedVersion(downloadedVersion);
  pluginSvcVersionInfo.setRoleDownloadTime(new Date().getTime());
  break;
   case RangerPluginInfo.ENTITY_TYPE_USERSTORE:
  pluginSvcVersionInfo.setUserStoreActiveVersion(lastKnownVersion);
  pluginSvcVersionInfo.setUserStoreActivationTime(lastActivationTime);
  pluginSvcVersionInfo.setUserStoreDownloadedVersion(downloadedVersion);
  pluginSvcVersionInfo.setUserStoreDownloadTime(new Date().getTime());
  break;
}

createOrUpdatePluginInfo(pluginSvcVersionInfo, entityType , httpCode, 
clusterName);

// called by getSecureRangerRolesIfUpdated() in RoleRest.java
// which will not set downloadedVersion when isAllowed = false.

Long    downloadedVersion = null;
...
if (isValid) {
    try {
        ...
        if (isAllowed) {
            RangerRoles roles = roleStore.getRoles(serviceName, 
lastKnownRoleVersion);
            if (roles == null) {
                downloadedVersion = lastKnownRoleVersion;
            } else {
                downloadedVersion = roles.getRoleVersion();
            }
        } else {
            httpCode = HttpServletResponse.SC_FORBIDDEN; // assert user is 
authenticated.
        }
    } catch (Throwable excp) {
    }
}

assetMgr.createPluginInfo(serviceName, pluginId, request, 
RangerPluginInfo.ENTITY_TYPE_ROLES, downloadedVersion, lastKnownRoleVersion, 
lastActivationTime, httpCode, clusterName, pluginCapabilities);


{code}
 

The simplest method is to modify AssertMgr.java to that, this is the behavior 
tag and policy 
{code:java}
if (pluginInfo.getRoleDownloadedVersion() != null && 
pluginInfo.getRoleDownloadedVersion().equals(pluginInfo.getRoleActiveVersion()))
 {
 {code}
Btw: the case of UserStore seems have the same bug.
{code:java}
} else {
   if (pluginInfo.getUserStoreDownloadTime() != null && 
pluginInfo.getUserStoreDownloadedVersion().equals(pluginInfo.getUserStoreActiveVersion()))
 {
  // This is our best guess of when users and groups may have been 
downloaded
  
pluginInfo.setUserStoreDownloadTime(pluginInfo.getUserStoreActivationTime());
   }
}
 {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692504#comment-17692504
 ] 

kirby zhou commented on RANGER-4104:


https://issues.apache.org/jira/browse/RANGER-3756

is a eclipselink jpa related problem too.

I suggest to upgrade eclipselink library version.

> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL 
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments: 
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL 
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime", 
> authWindowStartTime) however, [JPA 2.2 Specification Section 
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
>  says that it should be bound by specifying an additional parameter 
> TemporalType.DATE.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692027#comment-17692027
 ] 

kirby zhou commented on RANGER-4104:


How to reproduce it ? 

I have not find any SQL error in my machine.

> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL 
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments: 
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL 
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime", 
> authWindowStartTime) however, [JPA 2.2 Specification Section 
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
>  says that it should be bound by specifying an additional parameter 
> TemporalType.DATE.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690196#comment-17690196
 ] 

kirby zhou edited comment on RANGER-3756 at 2/17/23 7:27 AM:
-

Reproduce steps:
 # create  a cloud mysql-5.7 instance on Tencent Cloud 【 
[https://www.tencentcloud.com/products/cdb] 】
 # setup a ranger admin with the mysql db
 # use web browser to open ranger-admin site
 # create a HDFS service  【You DONOT need a real hdfs cluster】
 # delete the HDFS service
 # error happens

 

!image-2023-02-17-15-23-46-271.png!

 

!image-2023-02-17-15-24-11-315.png!

 

Additional Info:

I can not reproduce the bug with my private MySQL-8 server with GTID=on;

 


was (Author: kirbyzhou):
Reproduce steps:
 # create  a cloud mysql-5.7 instance on Tencent Cloud
 # setup a ranger admin with the mysql db
 # use web browser to open ranger-admin site
 # create a HDFS service  【You DONOT need a real hdfs cluster】
 # delete the HDFS service
 # error happens

 

!image-2023-02-17-15-23-46-271.png!

 

!image-2023-02-17-15-24-11-315.png!

 

Additional Info:

I can not reproduce the bug with my private MySQL-8 server with GTID=on;

 

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png, 
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou updated RANGER-3756:
---
Attachment: image-2023-02-17-15-24-11-315.png

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png, 
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690196#comment-17690196
 ] 

kirby zhou commented on RANGER-3756:


Reproduce steps:
 # create  a cloud mysql-5.7 instance on Tencent Cloud
 # setup a ranger admin with the mysql db
 # use web browser to open ranger-admin site
 # create a HDFS service  【You DONOT need a real hdfs cluster】
 # delete the HDFS service
 # error happens

 

!image-2023-02-17-15-23-46-271.png!

 

!image-2023-02-17-15-24-11-315.png!

 

Additional Info:

I can not reproduce the bug with my private MySQL-8 server with GTID=on;

 

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png, 
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou updated RANGER-3756:
---
Attachment: image-2023-02-17-15-23-26-423.png

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png, 
> image-2023-02-17-15-23-46-271.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou updated RANGER-3756:
---
Attachment: image-2023-02-17-15-23-46-271.png

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png, 
> image-2023-02-17-15-23-46-271.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689578#comment-17689578
 ] 

kirby zhou edited comment on RANGER-3756 at 2/17/23 7:08 AM:
-

I had encounter this bug with the cloud service of mysql provided by Tencent 
Cloud. 

[http://cloud.tencent.com|http://cloud.tencent.com/] 【China】or  
[https://www.tencentcloud.com/]  【international】

It happens at the ranger-2.3 branch.

 
{code:sql}
Server version: 5.7.18-txsql-log 20211101

mysql> show variables like '%gtid%';

+--+---+

| Variable_name                    | Value     |

+--+---+

| binlog_gtid_simple_recovery      | ON        |

| enforce_gtid_consistency         | ON        |

| gtid_executed_compression_period | 1000      |

| gtid_mode                        | ON        |

| gtid_next                        | AUTOMATIC |

| gtid_owned                       |           |

| gtid_purged                      |           |

| session_track_gtids              | OFF       |

+--+---+

8 rows in set (0.01 sec)
 {code}
 

 


was (Author: kirbyzhou):
I had encounter this bug with the cloud service of mysql provided by Tencent 
Cloud. 

[http://cloud.tencent.com|http://cloud.tencent.com/] 【China】or  
[https://www.tencentcloud.com/]  【international】

It happens at the ranger-2.3 branch.

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689578#comment-17689578
 ] 

kirby zhou commented on RANGER-3756:


I had encounter this bug with the cloud service of mysql provided by Tencent 
Cloud. 

[http://cloud.tencent.com|http://cloud.tencent.com/] 【China】or  
[https://www.tencentcloud.com/]  【international】

It happens at the ranger-2.3 branch.

> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>    Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO  
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY 
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL, 
> change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR 
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1. 
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS 
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp 
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id)) 
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY 
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional 
> context.  These statements are also not allowed in a function or trigger 
> because functions and triggers are also considered to be multi-statement 
> transactions.
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
>         at 
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
>         at 
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
>         at 
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services - 
> 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call: 
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM 
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM 
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN 
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind 
> => [2 parameters bound] Query: 
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId" 
> referenceClass=XXRMSResourceMapping sql="DELETE FROM 
> TL_x_rms_resource_mapping")
> {code}
>  
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed 
> outside transactional context.
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689467#comment-17689467
 ] 

kirby zhou commented on RANGER-3998:


Anybody have more ideas ? Can help to commit this?

> Support Ranger KMS integration with AWS KMS
> ---
>
> Key: RANGER-3998
> URL: https://issues.apache.org/jira/browse/RANGER-3998
> Project: Ranger
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.0, 2.4.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3682) Unify the ways that rangerkeystore to encapsulate zonekey

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685766#comment-17685766
 ] 

kirby zhou commented on RANGER-3682:


Any other idea?

> Unify the ways that rangerkeystore to encapsulate zonekey
> -
>
> Key: RANGER-3682
> URL: https://issues.apache.org/jira/browse/RANGER-3682
> Project: Ranger
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> Unify the ways that rangerkeystore to encapsulate zonekey
> Now we have 2 styles of MasterKeyProvider:
>  # RangerMasterKey, RangerHSM, RangerSafenetKeySecure
>  # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, 
> RangerTencentKMSProvider
> Style 1 can get out master key string from provider, Style 2 can not.
> In old, I add a flag KeyVaultEnabled to distinguish them. 
> KeyVaultEnabled=false means style1, true means style2
> RangerKeyStore with  style1 use SecretKeyEntry with SealedObject to store a 
> key and do encryption / decryption by itself.
> RangerKeyStore with  style2 use SecretKeyByteEntry to store a key and let MK 
> provider to encryption / decryption.
> These logics are hard-coded in the class RangerKeyStore. These are ugly and 
> hard to maintain. I refactor it by removing SecretKeyEntry, and let providers 
> of style1 do encryption / decryption. 
> Add a  common base class of RangerMasterKey, RangerHSM andd 
> RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common 
> logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes 
> SealedObject into byte[].
> So the new code does not change the actual storage format, and there is no 
> problem in compatibility.
> =
>  
> And, there is no unified method to initialize a master key provider. 
> Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI 
> classes.
> I made a new RangerKMSMKIFactory class to unify it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (RANGER-4062) keytab in kms-site.xml is not set by setup.sh

[kirby zhou (Jira)]

kirby zhou created RANGER-4062:
--

 Summary: keytab in kms-site.xml is not set by setup.sh
 Key: RANGER-4062
 URL: https://issues.apache.org/jira/browse/RANGER-4062
 Project: Ranger
  Issue Type: Bug
  Components: kms
Affects Versions: 2.3.0, 3.0.0
Reporter: kirby zhou


hadoop.kms.authentication.kerberos.keytab in kms-site.xml is used by 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler::init()
 to init kerberos authentication. 

Buy it is not set by setup.sh, setup.sh only set $kms_keytab to 
ranger.ks.kerberos.keytab in dbks-site.xml.

This makes it very troublesome for us to setup a KMS with Kerberos.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 73912: RANGER-3682 Unify the ways that rangerkeystore to encapsulate zonekey

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73912/
---

(Updated 一月 29, 2023, 1:11 p.m.)


Review request for ranger, Bhavik Bavishi, Dhaval Shah, Dineshkumar Yadav, 
Gautam Borad, Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen 
Mansoori, Madhan Neethiraj, Mateen Mansoori, Mehul Parikh, Pradeep Agrawal, 
Ramesh Mani, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and Velmurugan 
Periasamy.


Changes
---

improve exception catch


Bugs: RANGER-3682
https://issues.apache.org/jira/browse/RANGER-3682


Repository: ranger


Description
---

Unify the ways that rangerkeystore to encapsulate zonekey

Now we have 2 styles of MasterKeyProvider:
1. RangerMasterKey, RangerHSM, RangerSafenetKeySecure
2. RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, 
RangerTencentKMSProvider

Style 1 can get out master key string from provider, Style 2 can not.
In old, I add a flag KeyVaultEnabled to distinguish them. KeyVaultEnabled=false 
means style1, true means style2
RangerKeyStore with  style1 use SecretKeyEntry with SealedObject to store a key 
and do encryption / decryption by itself.
RangerKeyStore with  style2 use SecretKeyByteEntry to store a key and let MK 
provider to encryption / decryption.
These are ugly and hard to maintain. I refactor it by removing SecretKeyEntry, 
and let providers of style1 do encryption / decryption.
Add a  common base class of RangerMasterKey, RangerHSM andd 
RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common 
logic of encryptZoneKey and decryptZoneKey.
And, there is no unified method to initialize a master key provider. Duplicate 
code is distributed in RangerKeyStoreProvider and a bunch of CLI classes.
I made a new RangerKMSMKIFactory class to unify it.


Diffs (updated)
-

  kms/src/main/java/org/apache/hadoop/crypto/key/AbstractRangerMasterKey.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java 
39de0a503 
  kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java a1a6f348b 
  kms/src/main/java/org/apache/hadoop/crypto/key/MigrateDBMKeyToGCP.java 
d3b717a8a 
  kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java 1935a0185 
  
kms/src/main/java/org/apache/hadoop/crypto/key/RangerGoogleCloudHSMProvider.java
 a61cabb1b 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerHSM.java 90ef729b2 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKMSMKI.java b09cd5bad 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKMSMKIFactory.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
429d1ce45 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerSafenetKeySecure.java 
eb8a90a71 
  
kms/src/main/java/org/apache/hadoop/crypto/key/VerifyIsDBMasterkeyCorrect.java 
632e728f4 
  
kms/src/main/java/org/apache/hadoop/crypto/key/VerifyIsHSMMasterkeyCorrect.java 
e5ebeb783 
  kms/src/main/java/org/apache/ranger/kms/biz/RangerKMSStartUp.java 8b0f74eac 
  kms/src/test/java/org/apache/hadoop/crypto/key/kms/TestRangerKeyStore.java 
bcdf2e337 
  
kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/RangerMasterKeyTest.java
 f420322ca 


Diff: https://reviews.apache.org/r/73912/diff/5/

Changes: https://reviews.apache.org/r/73912/diff/4-5/


Testing
---

Tested by fresh install and update.


Thanks,

Kirby Zhou

Re: Review Request 74112: RANGER-3903:Improvement in RangerPolicyDeltaUtil--> applyDeltas method

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74112/#review225137
---




agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPolicyDeltaUtil.java
Lines 121 (patched)
<https://reviews.apache.org/r/74112/#comment313922>

just call ret.addAll(policesIdMap.values()) ?


- Kirby Zhou


On 一月 27, 2023, 3:51 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74112/
> ---
> 
> (Updated 一月 27, 2023, 3:51 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Kirby Zhou, Abhay Kulkarni, 
> Madhan Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, 
> Selvamohan Neethiraj, Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan 
> Periasamy.
> 
> 
> Bugs: RANGER-3903
> https://issues.apache.org/jira/browse/RANGER-3903
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> After going through the below code snippets in the master branch 
> 
> while (iter.hasNext()) {
> RangerPolicy policy = iter.next();
> if (policyId.equals(policy.getId()) && (changeType == 
> RangerPolicyDelta.CHANGE_TYPE_POLICY_DELETE || changeType == 
> RangerPolicyDelta.CHANGE_TYPE_POLICY_UPDATE)) {
> deletedPolicies.add(policy);
> iter.remove();
> }
> }
> switch (changeType) {
> case RangerPolicyDelta.CHANGE_TYPE_POLICY_CREATE:
> {
> if (CollectionUtils.isNotEmpty(deletedPolicies)) {
> LOG.warn("Unexpected: found existing policy for 
> CHANGE_TYPE_POLICY_CREATE: " + Arrays.toString(deletedPolicies.toArray()));
> }
> break;
> }
> case RangerPolicyDelta.CHANGE_TYPE_POLICY_UPDATE:
> {
> if (CollectionUtils.isEmpty(deletedPolicies) || 
> deletedPolicies.size() > 1) {
> LOG.warn("Unexpected: found no policy or multiple policies 
> for CHANGE_TYPE_POLICY_UPDATE: " + 
> Arrays.toString(deletedPolicies.toArray()));
> }
> break;
> }
> case RangerPolicyDelta.CHANGE_TYPE_POLICY_DELETE:
> {
> if (CollectionUtils.isEmpty(deletedPolicies) || 
> deletedPolicies.size() > 1) {
> LOG.warn("Unexpected: found no policy or multiple policies 
> for CHANGE_TYPE_POLICY_DELETE: " + 
> Arrays.toString(deletedPolicies.toArray()));
> }
> break;
> }
> default:
> break;
> }
> 
> 1st#improvement:
> 
> From the above code, we iterate delta policies and check if this policy 
> exists in the existing policy, we add that to deletePolicies list.
> 
> The delta change type condition for created/updated/deleted is added on top 
> of the if the condition so adding the condition again is not necessary 
> 
> 2nd#improvement:
> From the above code, we see for each element in the deltas,we iterate 
> policies and check if this delta policy exists in the existing policy, we add 
> that to deletePolicies list.
> 
> Solution:
> We need to use Map instead of iterating policies for every element of deltas 
> --> Map policiesIdMap
> Building index map key will be policyId and value will be policy associated 
> with the same policyId
> For  each policy in the deltas ,we check on the policiesIdMap whether the 
> same policyId is present or not ?.
> if yes, we will add all the associated policy  into deletedPolicy variable 
> and remove the policyId from policiesIdMap
> After an end of the iteration,we will iterate the policiesIdMap and get all 
> the policy associated with policyId and add into result set
> 
> This will give better performance when the policies list is huge.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPolicyDeltaUtil.java
>  e9223fe69 
> 
> 
> Diff: https://reviews.apache.org/r/74112/diff/8/
> 
> 
> Testing
> ---
> 
> Tested the below Rest API's to make sure everything works fine
> 
> 1.  ServiceREST Rest API :GET /plugins/policies/download/{serviceName}
> 
> 2.  ServiceREST Rest API :GET /plugins/secure/policies/download/{serviceName}
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

[jira] [Commented] (RANGER-3682) Unify the ways that rangerkeystore to encapsulate zonekey

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679005#comment-17679005
 ] 

kirby zhou commented on RANGER-3682:


rebased 

> Unify the ways that rangerkeystore to encapsulate zonekey
> -
>
> Key: RANGER-3682
> URL: https://issues.apache.org/jira/browse/RANGER-3682
> Project: Ranger
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> Unify the ways that rangerkeystore to encapsulate zonekey
> Now we have 2 styles of MasterKeyProvider:
>  # RangerMasterKey, RangerHSM, RangerSafenetKeySecure
>  # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, 
> RangerTencentKMSProvider
> Style 1 can get out master key string from provider, Style 2 can not.
> In old, I add a flag KeyVaultEnabled to distinguish them. 
> KeyVaultEnabled=false means style1, true means style2
> RangerKeyStore with  style1 use SecretKeyEntry with SealedObject to store a 
> key and do encryption / decryption by itself.
> RangerKeyStore with  style2 use SecretKeyByteEntry to store a key and let MK 
> provider to encryption / decryption.
> These logics are hard-coded in the class RangerKeyStore. These are ugly and 
> hard to maintain. I refactor it by removing SecretKeyEntry, and let providers 
> of style1 do encryption / decryption. 
> Add a  common base class of RangerMasterKey, RangerHSM andd 
> RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common 
> logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes 
> SealedObject into byte[].
> So the new code does not change the actual storage format, and there is no 
> problem in compatibility.
> =
>  
> And, there is no unified method to initialize a master key provider. 
> Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI 
> classes.
> I made a new RangerKMSMKIFactory class to unify it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

(Updated 一月 17, 2023, 9:49 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Changes
---

Rebase to HEAD


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs (updated)
-

  distro/src/main/assembly/kms.xml 4b4a2ac8e 
  kms/config/kms-webapp/dbks-site.xml 258d331d8 
  kms/pom.xml e97b993d7 
  kms/scripts/install.properties 0e5da3c75 
  kms/scripts/setup.sh f723e09bb 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
429d1ce45 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
1f0125967 


Diff: https://reviews.apache.org/r/74259/diff/5/

Changes: https://reviews.apache.org/r/74259/diff/4-5/


Testing
---

Tested under macos-13 and centos-8


Thanks,

Kirby Zhou

Re: Review Request 73912: RANGER-3682 Unify the ways that rangerkeystore to encapsulate zonekey

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73912/
---

(Updated 一月 17, 2023, 9:34 a.m.)


Review request for ranger, Bhavik Bavishi, Dhaval Shah, Dineshkumar Yadav, 
Gautam Borad, Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen 
Mansoori, Madhan Neethiraj, Mateen Mansoori, Mehul Parikh, Pradeep Agrawal, 
Ramesh Mani, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and Velmurugan 
Periasamy.


Changes
---

Rebase to HEAD


Bugs: RANGER-3682
https://issues.apache.org/jira/browse/RANGER-3682


Repository: ranger


Description
---

Unify the ways that rangerkeystore to encapsulate zonekey

Now we have 2 styles of MasterKeyProvider:
1. RangerMasterKey, RangerHSM, RangerSafenetKeySecure
2. RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, 
RangerTencentKMSProvider

Style 1 can get out master key string from provider, Style 2 can not.
In old, I add a flag KeyVaultEnabled to distinguish them. KeyVaultEnabled=false 
means style1, true means style2
RangerKeyStore with  style1 use SecretKeyEntry with SealedObject to store a key 
and do encryption / decryption by itself.
RangerKeyStore with  style2 use SecretKeyByteEntry to store a key and let MK 
provider to encryption / decryption.
These are ugly and hard to maintain. I refactor it by removing SecretKeyEntry, 
and let providers of style1 do encryption / decryption.
Add a  common base class of RangerMasterKey, RangerHSM andd 
RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common 
logic of encryptZoneKey and decryptZoneKey.
And, there is no unified method to initialize a master key provider. Duplicate 
code is distributed in RangerKeyStoreProvider and a bunch of CLI classes.
I made a new RangerKMSMKIFactory class to unify it.


Diffs (updated)
-

  kms/src/main/java/org/apache/hadoop/crypto/key/AbstractRangerMasterKey.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java 
39de0a503 
  kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java a1a6f348b 
  kms/src/main/java/org/apache/hadoop/crypto/key/MigrateDBMKeyToGCP.java 
d3b717a8a 
  kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java 1935a0185 
  
kms/src/main/java/org/apache/hadoop/crypto/key/RangerGoogleCloudHSMProvider.java
 a61cabb1b 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerHSM.java 90ef729b2 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKMSMKI.java b09cd5bad 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKMSMKIFactory.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 7188b19b2 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
429d1ce45 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerMasterKey.java b6fc32950 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerSafenetKeySecure.java 
eb8a90a71 
  
kms/src/main/java/org/apache/hadoop/crypto/key/VerifyIsDBMasterkeyCorrect.java 
632e728f4 
  
kms/src/main/java/org/apache/hadoop/crypto/key/VerifyIsHSMMasterkeyCorrect.java 
e5ebeb783 
  kms/src/main/java/org/apache/ranger/kms/biz/RangerKMSStartUp.java 8b0f74eac 
  kms/src/test/java/org/apache/hadoop/crypto/key/kms/TestRangerKeyStore.java 
bcdf2e337 
  
kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/RangerMasterKeyTest.java
 f420322ca 


Diff: https://reviews.apache.org/r/73912/diff/4/

Changes: https://reviews.apache.org/r/73912/diff/3-4/


Testing
---

Tested by fresh install and update.


Thanks,

Kirby Zhou

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

> On 一月 6, 2023, 3:58 a.m., Kirby Zhou wrote:
> > Ship It!
> 
> Ramachandran Krishnan wrote:
> Hi Madhan,
>     Based on Kirby Zhou review comments ,we addressed the security constraint 
> when serviceName and zoneName is not passed 
> I believe this fix will cover all the edge cases as well .
> 
> Madhan Neethiraj wrote:
> When both serviceName and zoneName are not provided, the lookup should 
> only be based on guid. Why restrict to only UNZONED_SECURITY_ZONE_ID?
> 
> @Kirby - can you please share more details of security issue in looking 
> for policy with guid only?
> 
> Ramachandran Krishnan wrote:
> Hi Madhan/Kirby,
> Security Zone is a feature that provides an ability in Ranger to separate 
> resource policies into different zones.
> It also enables multiple administrators to setup different policies, 
> based on the zones that they are assigned to.
> In this case ,we might fetch the policies which are tagged with some 
> security zone .I feel this could be security thread
> when Sales Admin see the policy which tagged with  Finance Admin.Due to 
> that ,we added defaulut zone Id which is not tied to any security Zone
> Please correct me If I am wrong
> 
> Madhan Neethiraj wrote:
> Ramachandran - before a policy is returned to the caller, Ranger ensures 
> that the caller has appropriate privileges. If caller doesn't have privilege 
> to view a policy, error code 403 (SC_FORBIDDEN) will be returned - refer to 
> ServiceREST.ensureAdminAndAuditAccess(policy). There is no security concern 
> here.
> 
> Ramachandran Krishnan wrote:
> Thanks Madhan for pointing out .If the caller doesn't have privilege to 
> view a policy, error code 403 (SC_FORBIDDEN) will be returned.
> So it will not leak any security contraint.It makes sense 
> Kirby,
> It would be great if you elobrate a bit where this will create security 
> concenrn when we are not passing default Zone Id
> 
> Kirby Zhou wrote:
> Madhan is right, ServiceREST.ensureAdminAndAuditAccess seems is enough to 
> prenvet unauthorized access when there is no zone id passed.
> It is not a security risk now.
> 
> But my concern is that the semantics of API have been changed and 
> inconsistent.
> 
> In the old code:
> if guid, serviceName and zoneName is given, it returns policy match guid, 
> serviceName and zoneName together,
> if only guid and serviceName is given, it returns policy match guid, 
> serviceName and RANGER_UNZONED_SECURITY_ZONE_ID together.
> 
> I think guid+zoneName / guid only based queries should follow the same 
> principle as above.
> 
> It may confuse some automatic processes which believe that the returned 
> policies are always in the given zone ( or unzoned ).
> 
> Ramachandran Krishnan wrote:
> Madhan/Kirby,
> It would be great if we finalize the things whether we can keep the 
> default zoneId along with guid when zoneName is not passed like old code way 
> or Do we need to strict to guid only when zoneName is not passed
> 
> Madhan Neethiraj wrote:
> This change in REST API to retrieve a policy by GUI was necessary to deal 
> with the case where the caller doesn't know the service name. How does this 
> patch deal with cases where the caller doesn't know the zone name? The search 
> is restricted only to UNZONED i.e. will exclude policies in security zones. 
> This doesn't look correct. Hence I suggested to not add zoneId filter in this 
> case.
> 
> > It may confuse some automatic processes which believe that the returned 
> policies are always in the given zone ( or unzoned ). 
> 
> Kirby - in this case, would the automatic process know of the 
> security-zone in which to find the policies? If not, it will fail to retrieve 
> the policy simply by searching for guid - as the search will only look for 
> policies in UNZONED. Is is alright?

Madhan - in this case, the automatic process should know its zone , or it just 
want to find in unzoned guids.

A compromise method is that we add a parameter to indicate whether to search in 
all zones or in the unzoned range.
Such as "ZoneName=_ALL_" or "ZoneName=_UNZONED_" ?


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225068
---


On 一月 5, 2023, 10:15 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> --

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

> On 一月 6, 2023, 3:58 a.m., Kirby Zhou wrote:
> > Ship It!
> 
> Ramachandran Krishnan wrote:
> Hi Madhan,
>     Based on Kirby Zhou review comments ,we addressed the security constraint 
> when serviceName and zoneName is not passed 
> I believe this fix will cover all the edge cases as well .
> 
> Madhan Neethiraj wrote:
> When both serviceName and zoneName are not provided, the lookup should 
> only be based on guid. Why restrict to only UNZONED_SECURITY_ZONE_ID?
> 
> @Kirby - can you please share more details of security issue in looking 
> for policy with guid only?
> 
> Ramachandran Krishnan wrote:
> Hi Madhan/Kirby,
> Security Zone is a feature that provides an ability in Ranger to separate 
> resource policies into different zones.
> It also enables multiple administrators to setup different policies, 
> based on the zones that they are assigned to.
> In this case ,we might fetch the policies which are tagged with some 
> security zone .I feel this could be security thread
> when Sales Admin see the policy which tagged with  Finance Admin.Due to 
> that ,we added defaulut zone Id which is not tied to any security Zone
> Please correct me If I am wrong
> 
> Madhan Neethiraj wrote:
> Ramachandran - before a policy is returned to the caller, Ranger ensures 
> that the caller has appropriate privileges. If caller doesn't have privilege 
> to view a policy, error code 403 (SC_FORBIDDEN) will be returned - refer to 
> ServiceREST.ensureAdminAndAuditAccess(policy). There is no security concern 
> here.
> 
> Ramachandran Krishnan wrote:
> Thanks Madhan for pointing out .If the caller doesn't have privilege to 
> view a policy, error code 403 (SC_FORBIDDEN) will be returned.
> So it will not leak any security contraint.It makes sense 
> Kirby,
> It would be great if you elobrate a bit where this will create security 
> concenrn when we are not passing default Zone Id

Madhan is right, ServiceREST.ensureAdminAndAuditAccess seems is enough to 
prenvet unauthorized access when there is no zone id passed.
It is not a security risk now.

But my concern is that the semantics of API have been changed and inconsistent.

In the old code:
if guid, serviceName and zoneName is given, it returns policy match guid, 
serviceName and zoneName together,
if only guid and serviceName is given, it returns policy match guid, 
serviceName and RANGER_UNZONED_SECURITY_ZONE_ID together.

I think guid+zoneName / guid only based queries should follow the same 
principle as above.

It may confuse some automatic processes which believe that the returned 
policies are always in the given zone ( or unzoned ).


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225068
---


On 一月 5, 2023, 10:15 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> ---
> 
> (Updated 一月 5, 2023, 10:15 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4031
> https://issues.apache.org/jira/browse/RANGER-4031
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Not able to fetch Policy details using guid /api/policy/guid/{guid} without 
> service name
> 
> Request without servicename 
> 
> curl -s -L -X GET 
> "https://q/service/public/v2/api/policy/guid/-2f42-4451-9edf-;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic *DEyMw=="
> Response : 404 
> 
> Request with servicename 
> 
> curl -s -L -X GET 
> "https:///service/public/v2/api/policy/guid/*-2f42-4451-9edf-?serviceName=hive;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic ***=="
> Response Proper : 200 with proper details 
> 
> Code : 
> https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java#L505
> 
> @GET  @Path("/api/policy/guid/{guid}")  

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225068
---


Ship it!




Ship It!

- Kirby Zhou


On 一月 5, 2023, 10:15 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> ---
> 
> (Updated 一月 5, 2023, 10:15 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4031
> https://issues.apache.org/jira/browse/RANGER-4031
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Not able to fetch Policy details using guid /api/policy/guid/{guid} without 
> service name
> 
> Request without servicename 
> 
> curl -s -L -X GET 
> "https://q/service/public/v2/api/policy/guid/-2f42-4451-9edf-;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic *DEyMw=="
> Response : 404 
> 
> Request with servicename 
> 
> curl -s -L -X GET 
> "https:///service/public/v2/api/policy/guid/*-2f42-4451-9edf-?serviceName=hive;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic ***=="
> Response Proper : 200 with proper details 
> 
> Code : 
> https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java#L505
> 
> @GET  @Path("/api/policy/guid/{guid}")
> @Produces({ "application/json", "application/xml" })
> public RangerPolicy 
> getPolicyByGUIDAndServiceNameAndZoneName(@PathParam("guid") String guid,  
>   
>@DefaultValue("") 
> @QueryParam("serviceName") String serviceName,
>   
>   @DefaultValue("") @QueryParam("ZoneName") String zoneName) {
>   return 
> serviceREST.getPolicyByGUIDAndServiceNameAndZoneName(guid, serviceName, 
> zoneName);   } 
> As query parameters are optional it should give proper response 
> 
> Expected : User should be able to get policy details using only guid in path 
> params 
> 
> 
> As part of the current design, Ranger expects both serviceName,guid should be 
> mandatory and zoneName can be optional 
> Proposal:
> Add the logic to fetch the RangerPolicy by guid from the backend
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 6b9604817 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 37d7561d4 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> c7a6ea0a6 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> e17494fa9 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 85c8b6213 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 7f1ec6d3e 
>   security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java 
> 2a123de93 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
> 7b15810e0 
> 
> 
> Diff: https://reviews.apache.org/r/74268/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225067
---


Ship it!




Ship It!

- Kirby Zhou


On 一月 5, 2023, 10:15 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> ---
> 
> (Updated 一月 5, 2023, 10:15 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4031
> https://issues.apache.org/jira/browse/RANGER-4031
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Not able to fetch Policy details using guid /api/policy/guid/{guid} without 
> service name
> 
> Request without servicename 
> 
> curl -s -L -X GET 
> "https://q/service/public/v2/api/policy/guid/-2f42-4451-9edf-;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic *DEyMw=="
> Response : 404 
> 
> Request with servicename 
> 
> curl -s -L -X GET 
> "https:///service/public/v2/api/policy/guid/*-2f42-4451-9edf-?serviceName=hive;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic ***=="
> Response Proper : 200 with proper details 
> 
> Code : 
> https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java#L505
> 
> @GET  @Path("/api/policy/guid/{guid}")
> @Produces({ "application/json", "application/xml" })
> public RangerPolicy 
> getPolicyByGUIDAndServiceNameAndZoneName(@PathParam("guid") String guid,  
>   
>@DefaultValue("") 
> @QueryParam("serviceName") String serviceName,
>   
>   @DefaultValue("") @QueryParam("ZoneName") String zoneName) {
>   return 
> serviceREST.getPolicyByGUIDAndServiceNameAndZoneName(guid, serviceName, 
> zoneName);   } 
> As query parameters are optional it should give proper response 
> 
> Expected : User should be able to get policy details using only guid in path 
> params 
> 
> 
> As part of the current design, Ranger expects both serviceName,guid should be 
> mandatory and zoneName can be optional 
> Proposal:
> Add the logic to fetch the RangerPolicy by guid from the backend
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 6b9604817 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 37d7561d4 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> c7a6ea0a6 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> e17494fa9 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 85c8b6213 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 7f1ec6d3e 
>   security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java 
> 2a123de93 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
> 7b15810e0 
> 
> 
> Diff: https://reviews.apache.org/r/74268/diff/6/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

> On 一月 4, 2023, 7:01 a.m., Madhan Neethiraj wrote:
> > security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java
> > Lines 316 (patched)
> > <https://reviews.apache.org/r/74268/diff/4/?file=2273372#file2273372line316>
> >
> > adding filter zoneId=RangerSecurityZone.RANGER_UNZONED_SECURITY_ZONE_ID 
> > will not retrieve policies that are in security zones. Is zoneId filter 
> > necessary?
> 
> Ramachandran Krishnan wrote:
> Will it create any security imapct when we use guid alone without passing 
> zoneId or zoneName or serviceName ?
> If no, then we no need to put 
> zoneId=RangerSecurityZone.RANGER_UNZONED_SECURITY_ZONE_ID will not retrieve 
> policies that are in security zones
> 
> Kirby Zhou wrote:
> it seems created an security impact without 
> RANGER_UNZONED_SECURITY_ZONE_ID when zoneName is blank.
> 
> The old code line300 set RANGER_UNZONED_SECURITY_ZONE_ID too.
> 
> But if zoneName is not black, you should pass it to SQL query.
> 
> Ramachandran Krishnan wrote:
> Kirby Zhou/Madhan,
>  
> As part of the fix we added the RANGER_UNZONED_SECURITY_ZONE_ID for 
> zoneId when guid is not null and serviceName,zoneName is null
> 
> select obj from XXPolicy obj where obj.guid = :guid and obj.zoneId = 
> :zoneId
> 
> zoneId ---> RANGER_UNZONED_SECURITY_ZONE_ID(1L)
> to avoid the security impact .
> 
> Ramachandran Krishnan wrote:
> if (StringUtils.isNotBlank(serviceName)) {
>   if (StringUtils.isBlank(zoneName)) {
> // query with guid, serviceName and 
> RANGER_UNZONED_SECURITY_ZONE_ID
>   return 
> getEntityManager().createNamedQuery("XXPolicy.findPolicyByPolicyGUIDAndServiceName",
>  tClass)
>   .setParameter("guid", guid)
>   .setParameter("serviceName", 
> serviceName)
>   .setParameter("zoneId", 
> RangerSecurityZone.RANGER_UNZONED_SECURITY_ZONE_ID)
>   .getSingleResult();
>   } else {
> // query with guid, serviceName and zoneName
>   return getEntityManager()
>   
> .createNamedQuery("XXPolicy.findPolicyByPolicyGUIDAndServiceNameAndZoneName", 
> tClass)
>   .setParameter("guid", guid)
>   .setParameter("serviceName", 
> serviceName)
>   .setParameter("zoneName", 
> zoneName)
>   .getSingleResult();
>   }
>   } else {
>   if (StringUtils.isNotBlank(zoneName)) {
>// query with guid and zoneName
> 
>   return getEntityManager()
>   
> .createNamedQuery("XXPolicy.findPolicyByPolicyGUIDAndZoneName", tClass)
>   .setParameter("guid", guid)
>   .setParameter("zoneName", 
> zoneName)
>   .getSingleResult();
>   } else {
> // query with guid and RANGER_UNZONED_SECURITY_ZONE_ID
> 
>   return getEntityManager()
>   
> .createNamedQuery("XXPolicy.findPolicyByPolicyGUID", tClass)
>   .setParameter("guid", guid)
>   .setParameter("zoneId", 
> RangerSecurityZone.RANGER_UNZONED_SECURITY_ZONE_ID)
>   .getSingleResult();
>   }
>   }
> 
>   Kirby Zhou/Madhan,
> 
> I hope this will cover all the cases

It seems work.
But I suggest you do not use StringUtils.isNotBlank and StringUtils.isBlank 
together —— This makes reading a little difficult.


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225041
---


On 一月 4, 2023, 11

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

> On 一月 4, 2023, 2:51 a.m., Kirby Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java
> > Lines 314 (patched)
> > <https://reviews.apache.org/r/74268/diff/3/?file=2273363#file2273363line314>
> >
> > What happens when GUID exists but is not in the specified Zone? This 
> > seems to be a security risk.
> 
> Ramachandran Krishnan wrote:
> In the current design,
> We can able to query the policy with guid and service Name zoneId when 
> the zoneName is not passed 
> 
> zoneId ---> RANGER_UNZONED_SECURITY_ZONE_ID(1L)
> 
> The below query will be used 
> 
> select obj from XXPolicy obj, XXService svc where obj.guid = :guid and 
> obj.service = svc.id and svc.name = :serviceName and obj.zoneId = :zoneId
> 
> 
> In our case also we can do the same when we pass only guid 
> 
> In that case we will add UNZONED_SECURITY_ZONE_ID along with guid
> 
> select obj from XXPolicy obj where obj.guid = :guid and obj.zoneId = 
> :zoneId
> 
> zoneId ---> RANGER_UNZONED_SECURITY_ZONE_ID(1L)
> 
> Kirby Zhou/Madhan,
> Please correct me if i am wrong
> 
> Kirby Zhou wrote:
> It seem we can not query the policy with guid and zoneName when 
> serviceName is not passed?
> 
> Ramachandran Krishnan wrote:
> Befor fix:
> Yeah correct, we can not query the policy with guid and zoneName when 
> serviceName is not passed
> After fix:
> We can query the policy with guid and zoneId 
> 
> select obj from XXPolicy obj where obj.guid = :guid and obj.zoneId = 
> :zoneId
> zoneId ---> RANGER_UNZONED_SECURITY_ZONE_ID(1L)
> 
> Madhan Neethiraj wrote:
> Current API supports retrieving a policy given guid/serviceName and 
> zoneName. The ask in the JIRA was to be able to retrieve a policy given its 
> guid. Is it necessary to locate a policy with given guid and zoneName?

-- After fix:
-- We can query the policy with guid and zoneId 

I do not think so.


```
if (StringUtils.isNotBlank(serviceName)) {
if (StringUtils.isBlank(zoneName)) {
// query with guid, serviceName and 
RANGER_UNZONED_SECURITY_ZONE_ID
} else {
// query with guid, serviceName and zoneName
}
} else {
// here, you only query with guid and 
RANGER_UNZONED_SECURITY_ZONE_ID.
// missing the case: guid and zoneName 
return getEntityManager()

.createNamedQuery("XXPolicy.findPolicyByPolicyGUID", tClass)
.setParameter("guid", guid)
.setParameter("zoneId", 
RangerSecurityZone.RANGER_UNZONED_SECURITY_ZONE_ID)
.getSingleResult();
}
```


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225039
---


On 一月 4, 2023, 5:09 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> ---
> 
> (Updated 一月 4, 2023, 5:09 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4031
> https://issues.apache.org/jira/browse/RANGER-4031
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Not able to fetch Policy details using guid /api/policy/guid/{guid} without 
> service name
> 
> Request without servicename 
> 
> curl -s -L -X GET 
> "https://q/service/public/v2/api/policy/guid/-2f42-4451-9edf-;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic *DEyMw=="
> Response : 404 
> 
> Request with servicename 
> 
> curl -s -L -X GET 
> "https:///service/public/v2/api/policy/guid/*-2f42-4451-9edf-?serviceName=hive;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: B

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

> On 一月 4, 2023, 2:51 a.m., Kirby Zhou wrote:
> > security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java
> > Lines 314 (patched)
> > <https://reviews.apache.org/r/74268/diff/3/?file=2273363#file2273363line314>
> >
> > What happens when GUID exists but is not in the specified Zone? This 
> > seems to be a security risk.
> 
> Ramachandran Krishnan wrote:
> In the current design,
> We can able to query the policy with guid and service Name zoneId when 
> the zoneName is not passed 
> 
> zoneId ---> RANGER_UNZONED_SECURITY_ZONE_ID(1L)
> 
> The below query will be used 
> 
> select obj from XXPolicy obj, XXService svc where obj.guid = :guid and 
> obj.service = svc.id and svc.name = :serviceName and obj.zoneId = :zoneId
> 
> 
> In our case also we can do the same when we pass only guid 
> 
> In that case we will add UNZONED_SECURITY_ZONE_ID along with guid
> 
> select obj from XXPolicy obj where obj.guid = :guid and obj.zoneId = 
> :zoneId
> 
> zoneId ---> RANGER_UNZONED_SECURITY_ZONE_ID(1L)
> 
> Kirby Zhou/Madhan,
> Please correct me if i am wrong

It seem we can not query the policy with guid and zoneName when serviceName is 
not passed?


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225039
---


On 一月 4, 2023, 5:09 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> ---
> 
> (Updated 一月 4, 2023, 5:09 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4031
> https://issues.apache.org/jira/browse/RANGER-4031
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Not able to fetch Policy details using guid /api/policy/guid/{guid} without 
> service name
> 
> Request without servicename 
> 
> curl -s -L -X GET 
> "https://q/service/public/v2/api/policy/guid/-2f42-4451-9edf-;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic *DEyMw=="
> Response : 404 
> 
> Request with servicename 
> 
> curl -s -L -X GET 
> "https:///service/public/v2/api/policy/guid/*-2f42-4451-9edf-?serviceName=hive;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic ***=="
> Response Proper : 200 with proper details 
> 
> Code : 
> https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java#L505
> 
> @GET  @Path("/api/policy/guid/{guid}")
> @Produces({ "application/json", "application/xml" })
> public RangerPolicy 
> getPolicyByGUIDAndServiceNameAndZoneName(@PathParam("guid") String guid,  
>   
>@DefaultValue("") 
> @QueryParam("serviceName") String serviceName,
>   
>   @DefaultValue("") @QueryParam("ZoneName") String zoneName) {
>   return 
> serviceREST.getPolicyByGUIDAndServiceNameAndZoneName(guid, serviceName, 
> zoneName);   } 
> As query parameters are optional it should give proper response 
> 
> Expected : User should be able to get policy details using only guid in path 
> params 
> 
> 
> As part of the current design, Ranger expects both serviceName,guid should be 
> mandatory and zoneName can be optional 
> Proposal:
> Add the logic to fetch the RangerPolicy by guid from the backend
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 6b9604817 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 37d7561d4 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> c7a6ea0a6 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> e17494fa9 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 85c8b6213 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 7f1ec6d3e 
>   security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java 
> 2a123de93 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
> 7b15810e0 
> 
> 
> Diff: https://reviews.apache.org/r/74268/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74268: RANGER-4031:Not able to fetch Policy details using guid /api/policy/guid/{guid} without service name

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74268/#review225039
---




security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java
Lines 314 (patched)
<https://reviews.apache.org/r/74268/#comment313845>

What happens when GUID exists but is not in the specified Zone? This seems 
to be a security risk.


- Kirby Zhou


On 一月 3, 2023, 6:22 p.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74268/
> ---
> 
> (Updated 一月 3, 2023, 6:22 p.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4031
> https://issues.apache.org/jira/browse/RANGER-4031
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Not able to fetch Policy details using guid /api/policy/guid/{guid} without 
> service name
> 
> Request without servicename 
> 
> curl -s -L -X GET 
> "https://q/service/public/v2/api/policy/guid/-2f42-4451-9edf-;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic *DEyMw=="
> Response : 404 
> 
> Request with servicename 
> 
> curl -s -L -X GET 
> "https:///service/public/v2/api/policy/guid/*-2f42-4451-9edf-?serviceName=hive;
>  -H "Content-Type: application/json" -H "Accept: application/json" -H 
> "Authorization: Basic ***=="
> Response Proper : 200 with proper details 
> 
> Code : 
> https://github.com/apache/ranger/blob/master/security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java#L505
> 
> @GET  @Path("/api/policy/guid/{guid}")
> @Produces({ "application/json", "application/xml" })
> public RangerPolicy 
> getPolicyByGUIDAndServiceNameAndZoneName(@PathParam("guid") String guid,  
>   
>@DefaultValue("") 
> @QueryParam("serviceName") String serviceName,
>   
>   @DefaultValue("") @QueryParam("ZoneName") String zoneName) {
>   return 
> serviceREST.getPolicyByGUIDAndServiceNameAndZoneName(guid, serviceName, 
> zoneName);   } 
> As query parameters are optional it should give proper response 
> 
> Expected : User should be able to get policy details using only guid in path 
> params 
> 
> 
> As part of the current design, Ranger expects both serviceName,guid should be 
> mandatory and zoneName can be optional 
> Proposal:
> Add the logic to fetch the RangerPolicy by guid from the backend
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> 6b9604817 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> 37d7561d4 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> c7a6ea0a6 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> e17494fa9 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 85c8b6213 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> 7f1ec6d3e 
>   security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java 
> 2a123de93 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
> 7b15810e0 
> 
> 
> Diff: https://reviews.apache.org/r/74268/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

> On 十二月 28, 2022, 3:33 a.m., Selvamohan Neethiraj wrote:
> > kms/scripts/install.properties
> > Lines 135 (patched)
> > <https://reviews.apache.org/r/74259/diff/4/?file=2272984#file2272984line135>
> >
> > Please remove the actual credential from here and provide comments as 
> > how it needs to be filled in. The credential seems to be actual AWS keys 
> > and works when I issue aws cli command to list-keys.

The key is used only for test. It is disabled now.


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/#review225026
-------


On 十二月 19, 2022, 7:49 a.m., Kirby Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74259/
> ---
> 
> (Updated 十二月 19, 2022, 7:49 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
> Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
> Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: ranger-3998
> https://issues.apache.org/jira/browse/ranger-3998
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
> 
> 
> Diffs
> -
> 
>   distro/src/main/assembly/kms.xml 4b4a2ac8e 
>   kms/config/kms-webapp/dbks-site.xml 258d331d8 
>   kms/pom.xml e97b993d7 
>   kms/scripts/install.properties 0e5da3c75 
>   kms/scripts/setup.sh f723e09bb 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
> PRE-CREATION 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
> cb5739f61 
>   
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
> 1f0125967 
> 
> 
> Diff: https://reviews.apache.org/r/74259/diff/4/
> 
> 
> Testing
> ---
> 
> Tested under macos-13 and centos-8
> 
> 
> Thanks,
> 
> Kirby Zhou
> 
>

[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17652352#comment-17652352
 ] 

kirby zhou commented on RANGER-3998:


Any other idea? 

Review is required for commit.

Thanks.

 

> Support Ranger KMS integration with AWS KMS
> ---
>
> Key: RANGER-3998
> URL: https://issues.apache.org/jira/browse/RANGER-3998
> Project: Ranger
>  Issue Type: Improvement
>  Components: kms
>Affects Versions: 3.0.0, 2.4.0
>    Reporter: kirby zhou
>Priority: Major
>
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Add me to Apache Ranger Community

[KirbY ZhoU]

Oh, I mean, how do I get my name here?
https://ranger.apache.org/team-list.html

在 2022/12/28 02:49，“Selvamohan Neethiraj”mailto:sneet...@apache.org>> 写入:


Hi,


I see that your Jira ID is already added as a Contributor to the Apache 
Ranger JIRA system.
Please let me know if you have any trouble in assigning Ranger JIRA to you.


Thanks,
Selva-


On 12/27/22 6:17 AM, KirbY ZhoU wrote:


> Please add me by the way.
> My Jira ID is kirbyzhou.
>
>
> 在 2022/12/27 14:10，“Selvamohan Neethiraj” <mailto:sneet...@apache.org> <mailto:sneet...@apache.org 
> <mailto:sneet...@apache.org>>> 写入:
>
>
> Thanks Himanshu for interest in Apache Ranger Contribution.
>
>
> To add you as a contributor, please create a JIRA account in Apache JIRA
> (https://issues.apache.org/jira/ <https://issues.apache.org/jira/> 
> <https://issues.apache.org/jira/> <https://issues.apache.org/jira/;>) and 
> provide your login name to JIRA
> system.
>
>
> Thanks,
> Selva-
>
>
>
>
> On 12/26/22 9:55 AM, Himanshu Maurya wrote:
>> Hi Team,
>>
>> I would like to contribute to the Apache Ranger project, Can you please add
>> me as a contributor to the project?
>>
>> Github Username: himanshumaurya09876
>>
>
>
>
>
>

Re: Add me to Apache Ranger Community

[KirbY ZhoU]

Please add me by the way.
My Jira ID is kirbyzhou.


在 2022/12/27 14:10，“Selvamohan Neethiraj”mailto:sneet...@apache.org>> 写入:


Thanks Himanshu for interest in Apache Ranger Contribution.


To add you as a contributor, please create a JIRA account in Apache JIRA 
(https://issues.apache.org/jira/ ) and provide 
your login name to JIRA 
system.


Thanks,
Selva-




On 12/26/22 9:55 AM, Himanshu Maurya wrote:
> Hi Team,
>
> I would like to contribute to the Apache Ranger project, Can you please add
> me as a contributor to the project?
>
> Github Username: himanshumaurya09876
>

Re: Review Request 74253: This script will help anyone to setup ranger within few minutes of downloading Apache Ranger Repo.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74253/#review225016
---


Ship it!




Ship It!

- Kirby Zhou


On 十二月 21, 2022, 2:50 a.m., Selvamohan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74253/
> ---
> 
> (Updated 十二月 21, 2022, 2:50 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4017
> https://issues.apache.org/jira/browse/RANGER-4017
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> RANGER-4017: Able to run Apache Ranger after downloading the source from REPO 
> by executing a single command (& within few minutes)
> 
> 
> Diffs
> -
> 
>   README.txt fce972ab1b986e7f1d28cf4e35f086929cf9169a 
>   dev-support/ranger-docker/Dockerfile.ranger-base 
> a4bb9008ff6e0d16784e286d22ab353e26be811c 
>   dev-support/ranger-docker/docker-compose.ranger-build.yml 
> f0b5b05e0cb696722297a83b7a507dc954f43398 
>   ranger_in_docker PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/74253/diff/4/
> 
> 
> Testing
> ---
> 
> Tested the script in Mac (M2) and Linux (Ubundu) 
> Documented the execution steps in README.txt
> 
> 
> Thanks,
> 
> Selvamohan Neethiraj
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

> On 十二月 19, 2022, 6:07 a.m., bhavik patel wrote:
> > Is testing completed for this? Any plan to write utility script to migrate 
> > masterkey from ranger DB to aws kms and vice-versa?

Tested.

Migrate script is blocked by RANGER-3682, I want to simpify the code structrue 
of old KMS at first.


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/#review225008
---


On 十二月 19, 2022, 7:49 a.m., Kirby Zhou wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74259/
> ---
> 
> (Updated 十二月 19, 2022, 7:49 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
> Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
> Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
> Velmurugan Periasamy.
> 
> 
> Bugs: ranger-3998
> https://issues.apache.org/jira/browse/ranger-3998
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> AWS KMS is widely used by many customers.
> Therefore, RangerKMS should support hosting MasterKey to AWS KMS.
> 
> 
> Diffs
> -
> 
>   distro/src/main/assembly/kms.xml 4b4a2ac8e 
>   kms/config/kms-webapp/dbks-site.xml 258d331d8 
>   kms/pom.xml e97b993d7 
>   kms/scripts/install.properties 0e5da3c75 
>   kms/scripts/setup.sh f723e09bb 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
> PRE-CREATION 
>   kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
> cb5739f61 
>   
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
> 1f0125967 
> 
> 
> Diff: https://reviews.apache.org/r/74259/diff/3/
> 
> 
> Testing
> ---
> 
> Tested under macos-13 and centos-8
> 
> 
> Thanks,
> 
> Kirby Zhou
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

(Updated 十二月 19, 2022, 7:49 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Changes
---

fix some typo, and support verify alias.


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs (updated)
-

  distro/src/main/assembly/kms.xml 4b4a2ac8e 
  kms/config/kms-webapp/dbks-site.xml 258d331d8 
  kms/pom.xml e97b993d7 
  kms/scripts/install.properties 0e5da3c75 
  kms/scripts/setup.sh f723e09bb 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
cb5739f61 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerTencentKMSProvider.java 
1f0125967 


Diff: https://reviews.apache.org/r/74259/diff/3/

Changes: https://reviews.apache.org/r/74259/diff/2-3/


Testing (updated)
---

Tested under macos-13 and centos-8


Thanks,

Kirby Zhou

Review Request 74260: RANGER-4021: Fix shell syntax bug in kms setup.sh

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74260/
---

Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Bugs: RANGER-4021
https://issues.apache.org/jira/browse/RANGER-4021


Repository: ranger


Description
---

```
echo "${prefix} $@" >> $LOGFILE
echo "${prefix} $@" 
```

Argument mixes string and array. Use * or separate argument.

```
if [[ ${useringrouparr[1]} =~ "(${unix_group})" ]] 
```

Don't quote right-hand side of =~, it'll match literally rather than as a regex.


Diffs
-

  kms/scripts/setup.sh f723e09bb 


Diff: https://reviews.apache.org/r/74260/diff/1/


Testing
---

On Centos-7 and Centos-8


Thanks,

Kirby Zhou

Re: Review Request 74253: This script will help anyone to setup ranger within few minutes of downloading Apache Ranger Repo.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74253/#review225007
---




README.txt
Lines 43 (patched)
<https://reviews.apache.org/r/74253/#comment313821>

Please delete all trailing spaces and tabs.



ranger_in_docker
Lines 188 (patched)
<https://reviews.apache.org/r/74253/#comment313820>

Maybe it is too heavy to bring all things up?

Can use an option to specify whether to start only core including ranger, 
usersync, or other external components such as hive and kafka.


- Kirby Zhou


On 十二月 18, 2022, 9:38 p.m., Selvamohan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74253/
> ---
> 
> (Updated 十二月 18, 2022, 9:38 p.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-4017
> https://issues.apache.org/jira/browse/RANGER-4017
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> RANGER-4017: Able to run Apache Ranger after downloading the source from REPO 
> by executing a single command (& within few minutes)
> 
> 
> Diffs
> -
> 
>   README.txt fce972ab1b986e7f1d28cf4e35f086929cf9169a 
>   dev-support/ranger-docker/Dockerfile.ranger-base 
> a4bb9008ff6e0d16784e286d22ab353e26be811c 
>   dev-support/ranger-docker/docker-compose.ranger-build.yml 
> f0b5b05e0cb696722297a83b7a507dc954f43398 
>   ranger_in_docker PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/74253/diff/2/
> 
> 
> Testing
> ---
> 
> Tested the script in Mac (M2) and Linux (Ubundu) 
> Documented the execution steps in README.txt
> 
> 
> Thanks,
> 
> Selvamohan Neethiraj
> 
>

Re: Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

(Updated 十二月 16, 2022, 10:27 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Changes
---

Format the code


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs (updated)
-

  distro/src/main/assembly/kms.xml 4b4a2ac8e 
  kms/config/kms-webapp/dbks-site.xml 258d331d8 
  kms/pom.xml e97b993d7 
  kms/scripts/install.properties 0e5da3c75 
  kms/scripts/setup.sh f723e09bb 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
cb5739f61 


Diff: https://reviews.apache.org/r/74259/diff/2/

Changes: https://reviews.apache.org/r/74259/diff/1-2/


Testing
---


Thanks,

Kirby Zhou

Review Request 74259: RANGER-3998: Support Ranger KMS integration with AWS KMS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74259/
---

Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Gautam Borad, 
Jayendra Parab, Kishor Gollapalliwar, Abhay Kulkarni, Mateen Mansoori, Mehul 
Parikh, Pradeep Agrawal, VaradreawiZTV VaradreawiZTV, Vishal Suvagia, and 
Velmurugan Periasamy.


Bugs: ranger-3998
https://issues.apache.org/jira/browse/ranger-3998


Repository: ranger


Description
---

AWS KMS is widely used by many customers.
Therefore, RangerKMS should support hosting MasterKey to AWS KMS.


Diffs
-

  distro/src/main/assembly/kms.xml 4b4a2ac8e 
  kms/config/kms-webapp/dbks-site.xml 258d331d8 
  kms/pom.xml e97b993d7 
  kms/scripts/install.properties 0e5da3c75 
  kms/scripts/setup.sh f723e09bb 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerAWSKMSProvider.java 
PRE-CREATION 
  kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java 
cb5739f61 


Diff: https://reviews.apache.org/r/74259/diff/1/


Testing
---


Thanks,

Kirby Zhou

[jira] [Created] (RANGER-4021) Shell syntax bug in kms setup.sh

[kirby zhou (Jira)]

kirby zhou created RANGER-4021:
--

 Summary: Shell syntax bug in kms setup.sh
 Key: RANGER-4021
 URL: https://issues.apache.org/jira/browse/RANGER-4021
 Project: Ranger
  Issue Type: Bug
  Components: kms
Affects Versions: 2.3.0, 3.0.0, 2.4.0
Reporter: kirby zhou


 
{code:java}
if [[ ${useringrouparr[1]} =~ "(${unix_group})" ]] 
{code}
Don't quote right-hand side of =~, it'll match literally rather than as a regex.

 

 
{code:java}
echo "${prefix} $@" >> $LOGFILE
echo "${prefix} $@" 
{code}
Argument mixes string and array. Use * or separate argument.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (RANGER-4020) ranger-authn is required by security-admin but not compiled before it

[kirby zhou (Jira)]

kirby zhou created RANGER-4020:
--

 Summary: ranger-authn is required by security-admin but not 
compiled before it
 Key: RANGER-4020
 URL: https://issues.apache.org/jira/browse/RANGER-4020
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Affects Versions: 3.0.0
Reporter: kirby zhou


[WARNING] The POM for org.apache.ranger:ranger-authn:jar:3.0.0-SNAPSHOT is 
missing, no dependency information available
The POM for org.apache.ranger:ranger-authn:jar:3.0.0-SNAPSHOT is missing, no 
dependency information available

 

In root pom.xml, profile "all", ranger-authn module is not listed here.

It should be compiled before security-admin.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3860) Huge unnecessary aws-java-sdk-bundle dependency

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648394#comment-17648394
 ] 

kirby zhou commented on RANGER-3860:


I have a patch here to reduce about 210Mib of dist tarball size.

[https://reviews.apache.org/r/74258/]

> Huge unnecessary aws-java-sdk-bundle dependency
> ---
>
> Key: RANGER-3860
> URL: https://issues.apache.org/jira/browse/RANGER-3860
> Project: Ranger
>  Issue Type: Wish
>  Components: Ranger
>Affects Versions: 3.0.0, 2.3.0
>Reporter: Aaron Braunstein
>Priority: Major
>
> RANGER-3653 added an aws-java-sdk-bundle dependency to Ranger, which causes 
> the unpacked Ranger plugin jar size to increase by over 500 MB. Previously we 
> only had an aws-java-sdk-logs dependency. There was no justification provided 
> in the issue, but I suspect it was either due to a misunderstanding of how 
> dependency management works in Maven, or because they wanted to avoid shading 
> Ranger if there was some aws-java-sdk-logs 3rd party dependency that 
> conflicted in version with some their project was already using. 
> Aws-java-sdk-bundle relocates its dependencies.
> Aws-java-sdk-bom in dependency management (with scope import) did not add a 
> dependency on all the aws java sdk's but only overrode the version of all aws 
> sdk dependencies if it found one transitively or without a version defined, 
> with the version supplied in the bom.
> I recommend that RANGER-3653 be reverted. Additionally, aws-java-sdk-logs 
> version should be kept up-to-date to minimize version conflicts in libraries 
> that are keeping their aws dependencies up to date.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Review Request 74258: replace aws-java-sdk-bundle to aws-java-sdk-logs to reduce dist size

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74258/
---

Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Kishor 
Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Pradeep Agrawal, 
Ramesh Mani, Sailaja Polavarapu, and Velmurugan Periasamy.


Bugs: RANGER-3860
https://issues.apache.org/jira/browse/RANGER-3860


Repository: ranger


Description
---

RANGER-3653 introduced aws-java-sdk-bundle, it is very huge with a lot of 
useless things.

We only use logs in our code. So we can reduce dist size by replacing it.


Diffs
-

  security-admin/pom.xml f887e03a1 


Diff: https://reviews.apache.org/r/74258/diff/1/


Testing
---

All unit tests passwd.


Thanks,

Kirby Zhou

[jira] [Resolved] (RANGER-4019) Cannot new RangerRESTUtils under Apple Silicon macOS.

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou resolved RANGER-4019.

Resolution: Duplicate

> Cannot new RangerRESTUtils under Apple Silicon macOS.
> -
>
> Key: RANGER-4019
> URL: https://issues.apache.org/jira/browse/RANGER-4019
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 3.0.0, 2.3.0, 2.4.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with 
> M1/M2 chips.
> Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74257: RANGER-4019: upgrade to JNA-5.7.0 gethostname4j-1.0.0 for macOS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74257/
---

(Updated 十二月 16, 2022, 7:36 a.m.)


Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Kishor 
Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mateen Mansoori, Mehul Parikh, 
Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.


Changes
---

Nodejs needs to be upgraded to v16 to support darwin-arm64


Bugs: RANGER-4019
https://issues.apache.org/jira/browse/RANGER-4019


Repository: ranger


Description
---

Cannot new RangerRESTUtils under Apple Silicon macOS.

com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with M1/M2 
chips.
Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0.


Diffs (updated)
-

  pom.xml d0752b817 
  security-admin/pom.xml 5e24dd846 


Diff: https://reviews.apache.org/r/74257/diff/2/

Changes: https://reviews.apache.org/r/74257/diff/1-2/


Testing
---


Thanks,

Kirby Zhou

Review Request 74257: RANGER-4019: upgrade to JNA-5.7.0 gethostname4j-1.0.0 for macOS

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74257/
---

Review request for ranger, Dhaval Shah, Dineshkumar Yadav, Kishor 
Gollapalliwar, Abhay Kulkarni, Madhan Neethiraj, Mateen Mansoori, Mehul Parikh, 
Pradeep Agrawal, Ramesh Mani, and Velmurugan Periasamy.


Bugs: RANGER-4019
https://issues.apache.org/jira/browse/RANGER-4019


Repository: ranger


Description
---

Cannot new RangerRESTUtils under Apple Silicon macOS.

com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with M1/M2 
chips.
Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0.


Diffs
-

  pom.xml d0752b817 


Diff: https://reviews.apache.org/r/74257/diff/1/


Testing
---


Thanks,

Kirby Zhou

[jira] [Assigned] (RANGER-4019) Cannot new RangerRESTUtils under Apple Silicon macOS.

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou reassigned RANGER-4019:
--

Assignee: kirby zhou

> Cannot new RangerRESTUtils under Apple Silicon macOS.
> -
>
> Key: RANGER-4019
> URL: https://issues.apache.org/jira/browse/RANGER-4019
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 3.0.0, 2.3.0, 2.4.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with 
> M1/M2 chips.
> Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (RANGER-4019) Cannot new RangerRESTUtils under Apple Silicon macOS.

[kirby zhou (Jira)]

kirby zhou created RANGER-4019:
--

 Summary: Cannot new RangerRESTUtils under Apple Silicon macOS.
 Key: RANGER-4019
 URL: https://issues.apache.org/jira/browse/RANGER-4019
 Project: Ranger
  Issue Type: Bug
  Components: Ranger
Affects Versions: 2.3.0, 3.0.0, 2.4.0
Reporter: kirby zhou


com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with M1/M2 
chips.

Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74237: RANGER-4000: fixed plugins-common library unit tests for JDK17

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74237/#review224977
---


Ship it!




- Kirby Zhou


On 十二月 6, 2022, 7:25 p.m., Madhan Neethiraj wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74237/
> ---
> 
> (Updated 十二月 6, 2022, 7:25 p.m.)
> 
> 
> Review request for ranger and Abhay Kulkarni.
> 
> 
> Bugs: RANGER-4000
> https://issues.apache.org/jira/browse/RANGER-4000
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> fixed plugins-common library unit tests for JDK17
> 
> 
> Diffs
> -
> 
>   agents-common/src/test/resources/policyengine/test_policyengine_geo.json 
> 4249996b8 
> 
> 
> Diff: https://reviews.apache.org/r/74237/diff/1/
> 
> 
> Testing
> ---
> 
> - verified that all unit tests in plugins-common library pass with JDK17
> 
> 
> Thanks,
> 
> Madhan Neethiraj
> 
>

[jira] [Created] (RANGER-3998) Support Ranger KMS integration with AWS KMS

[kirby zhou (Jira)]

kirby zhou created RANGER-3998:
--

 Summary: Support Ranger KMS integration with AWS KMS
 Key: RANGER-3998
 URL: https://issues.apache.org/jira/browse/RANGER-3998
 Project: Ranger
  Issue Type: Improvement
  Components: kms
Affects Versions: 3.0.0, 2.4.0
Reporter: kirby zhou


AWS KMS is widely used by many customers.

Therefore, RangerKMS should support hosting MasterKey to AWS KMS.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-3612) KMS should either Die or Auto-Recover when its ranger-agent auth to KDC failed

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641767#comment-17641767
 ] 

kirby zhou commented on RANGER-3612:


Anybody can merge it?

> KMS should either Die or Auto-Recover when its ranger-agent auth to KDC failed
> --
>
> Key: RANGER-3612
> URL: https://issues.apache.org/jira/browse/RANGER-3612
> Project: Ranger
>  Issue Type: Bug
>  Components: kms, plugins
>Affects Versions: 3.0.0, 2.2.0
>    Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> If we install ranger agent to KMS, the agent would auth itself to KDC at 
> startup. But if it failed, it just print a log in ranger-kms-.log, 
> and the KMS can never recover to refresh its policies.
> {code:java}
> ]$ tail -f log/ranger-kms-ranger_kms-.log  | fgrep ERROR 
> 2022-02-09 19:00:18,227 ERROR MiscUtil - Failed to login with given keytab 
> and principal{code}
> {code:java}
> package org.apache.ranger.authorization.kms.authorizer;
> public class RangerKmsAuthorizer implements Runnable, KeyACLs {
> RangerKmsAuthorizer(Configuration conf) { 
>authWithKerberos(conf); 
> }
> private void authWithKerberos(Configuration conf) {
>     MiscUtil.authWithKerberos(keytab, principal, nameRules);
> }
> }
> package org.apache.ranger.audit.provider;
> public class MiscUtil {
> public static void authWithKerberos(...) {
>   try {
> {
>   UserGroupInformation ugi = UserGroupInformation
>  .loginUserFromKeytabAndReturnUGI(spnegoPrincipals[0],
>  keytab);
>   MiscUtil.setUGILoginUser(ugi, null);
>  }
>   } catch (Throwable t) {
> logger.error("Failed to login with given keytab and principal", t);
>   }
> }
> }{code}
>  
> There seems only one chance for plugin to auth to KDC, so it can not auto 
> recover.
> And MiscUtil.authWithKerberos never fail when auth failed, so KMS would not 
> die when the plugin failed.
> This situation is too unfriendly to administrators. It should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (RANGER-3990) PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when ranger.supportedcomponents excluding solr.

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou resolved RANGER-3990.

Resolution: Duplicate

> PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when 
> ranger.supportedcomponents excluding solr.
> --
>
> Key: RANGER-3990
> URL: https://issues.apache.org/jira/browse/RANGER-3990
> Project: Ranger
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Major
>
> When I set install.properties like that
>  
> {code:java}
> ranger.supportedcomponents=tag,hdfs,hive,yarn,kafka,kms,kudu {code}
> Then, run db_setup.py to upgrade a old instance, error occurs:
> {code:java}
> 2022-11-28 16:19:49,949  [JISQL] 
> /sensorsdata/main/program/armada/jdk18292/jdk18292/bin/java  -cp 
> /sensorsdata/main/program/sp/commonjars/mysql-connector-java.jar:/sensorsdata/main/program/rogue/ranger-2.3.0-admin/jisql/lib/*
>  org.apache.util.sql.Jisql -driver mysqlconj -cstring 
> jdbc:mysql://10.129.17.81:4000/kirby_ranger2?useSSL=false -u 'kirby_ranger' 
> -p '' -noheader -trim -c \;  -query "delete from x_db_version_h where 
> version = 'J10055' and active = 'N' and updated_by='ranger_admin';"
> 2022-11-28 16:19:50,298  [E] applying java patch 
> PatchForSolrSvcDefAndPoliciesUpdate_J10055 failed
>  {code}
>  
> {code:java}
> 437                 if (xXServiceDefObj == null) {
> 438                     logger.info(xXServiceDefObj + ": service-def not 
> found. No patching is needed");
> 439                     System.out.println(0);
> 440                 }
> 441 
> 442                 embeddedSolrResourceDefs = 
> embeddedSolrServiceDef.getResources();                 // ResourcesType
> 443                 dbSolrServiceDef         = 
> this.svcDBStore.getServiceDefByName(SOLR_SVC_DEF_NAME);
> 444                 dbSolrServiceDef.setResources(embeddedSolrResourceDefs);
>  {code}
> It seems that, line 439 should be replaced by  System.exit(0)
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74189: RANGER-3883 : POST/PUT REST API's work even when invalid user id or Id is used in the URL

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74189/#review224916
---




security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java
Lines 353 (patched)
<https://reviews.apache.org/r/74189/#comment313765>

You should not use System.out to print log here.


- Kirby Zhou


On 十一月 28, 2022, 9:52 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74189/
> ---
> 
> (Updated 十一月 28, 2022, 9:52 a.m.)
> 
> 
> Review request for ranger, Don Bosco Durai, Abhay Kulkarni, Madhan Neethiraj, 
> Mehul Parikh, Nikhil P, Pradeep Agrawal, Ramesh Mani, Selvamohan Neethiraj, 
> Sailaja Polavarapu, Subhrat Chaudhary, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-3883
> https://issues.apache.org/jira/browse/RANGER-3883
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> When a POST request is made to the following APIs return 200 status code even 
> when the userId is invalid .
> 
> When a POST/PUT request is made to the following APIs return 200 status code 
> even when the userId or id is invalid.
> 
> Ranger is not honouring Id
> /service/users/{USER_ID}/passwordchange
> /service/users/{USER_ID}/emailchange
> /assets/{id}
> /permission/{id}
> /services/{id}
> /definitions/{id}
> /secure/groups/{id}
> /policies/{id}
> 
> Ideally, the APIs must return 404 or Bad request(400) not found when using an 
> invalid userid or id in the URL
> 
> But in this case, the POST/PUT request results in status code 200 instead of 
> 400
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java 
> a0ba3b750 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java 
> 2e7e90bb4 
>   security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java 
> 293107f24 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 9bccf1089 
>   security-admin/src/main/java/org/apache/ranger/rest/UserREST.java 5fc18034b 
>   security-admin/src/main/java/org/apache/ranger/rest/XUserREST.java 
> dd12048ac 
>   security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java 
> abd4b1c1c 
>   security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java 
> 2bf5ee6c9 
>   security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java 
> 1069f013d 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
> 375135a5a 
>   security-admin/src/test/java/org/apache/ranger/rest/TestUserREST.java 
> 48cd7face 
>   security-admin/src/test/java/org/apache/ranger/rest/TestXUserREST.java 
> 2b25ba813 
> 
> 
> Diff: https://reviews.apache.org/r/74189/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

[jira] [Updated] (RANGER-3984) Support using TiDB as mysql-db in ranger

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou updated RANGER-3984:
---
Attachment: (was: ranger_core_db_tidb.patch)

> Support using TiDB as mysql-db in ranger
> 
>
> Key: RANGER-3984
> URL: https://issues.apache.org/jira/browse/RANGER-3984
> Project: Ranger
>  Issue Type: Improvement
>  Components: admin, kms
>Affects Versions: 3.0.0, 2.3.0
>    Reporter: kirby zhou
>Priority: Major
>
> TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
> deploy ranger based on tidb. But TiDB is missing some features, which makes 
> ranger unable to install properly.
> [https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
> The biggest problem affecting ranger is missing "Stored procedures and 
> functions", "Select into".
> ranger use Stored procedures in setup scripts to simplify SQL.
> Some work is needed to remove the stored procedure.
>  
>  
> {code:java}
> ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 14 near "PROCEDURE if 
> exists getXportalUIdByLoginId" 
> ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 16 near "PROCEDURE 
> `getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_portal_user.id into myid FROM x_portal_user WHERE 
> x_portal_user.login_id = input_val;
> END" 
> ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 14 near "PROCEDURE if 
> exists getModulesIdByName" 
> ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 16 near "PROCEDURE 
> `getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_modules_master.id into myid FROM x_modules_master WHERE 
> x_modules_master.module = input_val;
> END" 
> ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 14 near "PROCEDURE if 
> exists insertRangerPrerequisiteEntries" 
> ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 16 near "PROCEDURE 
> `insertRangerPrerequisiteEntries`()
> BEGIN
> DECLARE adminID bigint;
> DECLARE keyadminID bigint;
> DECLARE rangerusersyncID bigint;
> DECLARE rangertagsyncID bigint;
> DECLARE moduleIdReports bigint;
> DECLARE moduleIdResourceBasedPolicies bigint;
> DECLARE moduleIdAudit bigint;
> DECLARE moduleIdUG bigint;
> DECLARE moduleIdTagBasedPolicies bigint;
> DECLARE moduleIdKeyMana
> ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql': 
> Unsupported type *ast.CallStmt
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Review Request 74227: RANGER-3984: Support using TiDB as mysql-db in ranger

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74227/
---

Review request for ranger, Bhavik Bavishi, Dhaval Shah, Madhan Neethiraj, Mehul 
Parikh, Pradeep Agrawal, and Vishal Suvagia.


Bugs: RANGER-3984
https://issues.apache.org/jira/browse/RANGER-3984


Repository: ranger


Description
---

TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
deploy ranger based on tidb. But TiDB is missing some features, which makes 
ranger unable to install properly.
https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features

The biggest problem affecting ranger is missing "Stored procedures and 
functions", "Select into".
ranger use Stored procedures in setup scripts to simplify SQL.
Some work is needed to remove the stored procedure.


Diffs
-

  security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 833ffa0e0 


Diff: https://reviews.apache.org/r/74227/diff/1/


Testing
---

Done with Ranger-master and Ranger-2.3 and TiDB-v6.4.0


Thanks,

Kirby Zhou

Re: Review Request 74217: RANGER-3981:Upgrade jetty-client version to 9.4.49.v20220914

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74217/#review224915
---


Ship it!




Ship It!

- Kirby Zhou


On 十一月 28, 2022, 9:38 a.m., bhavik patel wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74217/
> ---
> 
> (Updated 十一月 28, 2022, 9:38 a.m.)
> 
> 
> Review request for ranger, Dhaval Shah, Kirby Zhou, Madhan Neethiraj, Mehul 
> Parikh, Pradeep Agrawal, and Vishal Suvagia.
> 
> 
> Bugs: RANGER-3981
> https://issues.apache.org/jira/browse/RANGER-3981
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> Upgrade jetty-client version to 9.4.49.v20220914
> 
> 
> Diffs
> -
> 
>   pom.xml bcb93ed74 
> 
> 
> Diff: https://reviews.apache.org/r/74217/diff/1/
> 
> 
> Testing
> ---
> 
> 1. Passed junit tests.
> 2. Verified policy and user crude operations.
> 3. created keys in kms.
> 
> 
> Thanks,
> 
> bhavik patel
> 
>

Re: Review Request 74222: PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing with no-solr.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74222/
---

(Updated 十一月 29, 2022, 2:32 a.m.)


Review request for ranger, Bhavik Bavishi, madhan, Madhan Neethiraj, Mateen 
Mansoori, and Mehul Parikh.


Bugs: RANGER-3990
https://issues.apache.org/jira/browse/RANGER-3990


Repository: ranger


Description
---

run db_setup.py to upgrade a old instance without solr, error occurs:

install.properties like that:

```
ranger.supportedcomponents=tag,hdfs,hive,yarn,kafka,kms,kudu ```

```

2022-11-28 16:19:49,949  [JISQL] 
/sensorsdata/main/program/armada/jdk18292/jdk18292/bin/java  -cp 
/sensorsdata/main/program/sp/commonjars/mysql-connector-java.jar:/sensorsdata/main/program/rogue/ranger-2.3.0-admin/jisql/lib/*
 org.apache.util.sql.Jisql -driver mysqlconj -cstring 
jdbc:mysql://10.129.17.81:4000/kirby_ranger2?useSSL=false -u 'kirby_ranger' -p 
'' -noheader -trim -c \;  -query "delete from x_db_version_h where 
version = 'J10055' and active = 'N' and updated_by='ranger_admin';"
2022-11-28 16:19:50,298  [E] applying java patch 
PatchForSolrSvcDefAndPoliciesUpdate_J10055 failed
```


Diffs
-

  
security-admin/src/main/java/org/apache/ranger/patch/PatchForSolrSvcDefAndPoliciesUpdate_J10055.java
 4684923ca 


Diff: https://reviews.apache.org/r/74222/diff/1/


Testing
---


Thanks,

Kirby Zhou

Review Request 74222: PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing with no-solr.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74222/
---

Review request for ranger, Bhavik Bavishi, madhan, Madhan Neethiraj, Mateen 
Mansoori, and Mehul Parikh.


Repository: ranger


Description
---

run db_setup.py to upgrade a old instance without solr, error occurs:

install.properties like that:

```
ranger.supportedcomponents=tag,hdfs,hive,yarn,kafka,kms,kudu ```

```

2022-11-28 16:19:49,949  [JISQL] 
/sensorsdata/main/program/armada/jdk18292/jdk18292/bin/java  -cp 
/sensorsdata/main/program/sp/commonjars/mysql-connector-java.jar:/sensorsdata/main/program/rogue/ranger-2.3.0-admin/jisql/lib/*
 org.apache.util.sql.Jisql -driver mysqlconj -cstring 
jdbc:mysql://10.129.17.81:4000/kirby_ranger2?useSSL=false -u 'kirby_ranger' -p 
'' -noheader -trim -c \;  -query "delete from x_db_version_h where 
version = 'J10055' and active = 'N' and updated_by='ranger_admin';"
2022-11-28 16:19:50,298  [E] applying java patch 
PatchForSolrSvcDefAndPoliciesUpdate_J10055 failed
```


Diffs
-

  
security-admin/src/main/java/org/apache/ranger/patch/PatchForSolrSvcDefAndPoliciesUpdate_J10055.java
 4684923ca 


Diff: https://reviews.apache.org/r/74222/diff/1/


Testing
---


Thanks,

Kirby Zhou

[jira] [Created] (RANGER-3990) PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when ranger.supportedcomponents excluding solr.

[kirby zhou (Jira)]

kirby zhou created RANGER-3990:
--

 Summary: PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when 
ranger.supportedcomponents excluding solr.
 Key: RANGER-3990
 URL: https://issues.apache.org/jira/browse/RANGER-3990
 Project: Ranger
  Issue Type: Improvement
  Components: admin
Affects Versions: 2.3.0, 3.0.0
Reporter: kirby zhou


When I set install.properties like that

 
{code:java}
ranger.supportedcomponents=tag,hdfs,hive,yarn,kafka,kms,kudu {code}
Then, run db_setup.py to upgrade a old instance, error occurs:
{code:java}
2022-11-28 16:19:49,949  [JISQL] 
/sensorsdata/main/program/armada/jdk18292/jdk18292/bin/java  -cp 
/sensorsdata/main/program/sp/commonjars/mysql-connector-java.jar:/sensorsdata/main/program/rogue/ranger-2.3.0-admin/jisql/lib/*
 org.apache.util.sql.Jisql -driver mysqlconj -cstring 
jdbc:mysql://10.129.17.81:4000/kirby_ranger2?useSSL=false -u 'kirby_ranger' -p 
'' -noheader -trim -c \;  -query "delete from x_db_version_h where 
version = 'J10055' and active = 'N' and updated_by='ranger_admin';"
2022-11-28 16:19:50,298  [E] applying java patch 
PatchForSolrSvcDefAndPoliciesUpdate_J10055 failed
 {code}
 
{code:java}
437                 if (xXServiceDefObj == null) {
438                     logger.info(xXServiceDefObj + ": service-def not found. 
No patching is needed");
439                     System.out.println(0);
440                 }
441 
442                 embeddedSolrResourceDefs = 
embeddedSolrServiceDef.getResources();                 // ResourcesType
443                 dbSolrServiceDef         = 
this.svcDBStore.getServiceDefByName(SOLR_SVC_DEF_NAME);
444                 dbSolrServiceDef.setResources(embeddedSolrResourceDefs);
 {code}
It seems that, line 439 should be replaced by  System.exit(0)

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3984) Support using TiDB as mysql-db in ranger

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou updated RANGER-3984:
---
Attachment: ranger_core_db_tidb.patch

> Support using TiDB as mysql-db in ranger
> 
>
> Key: RANGER-3984
> URL: https://issues.apache.org/jira/browse/RANGER-3984
> Project: Ranger
>  Issue Type: Improvement
>  Components: admin, kms
>Affects Versions: 3.0.0, 2.3.0
>    Reporter: kirby zhou
>Priority: Major
> Attachments: ranger_core_db_tidb.patch
>
>
> TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
> deploy ranger based on tidb. But TiDB is missing some features, which makes 
> ranger unable to install properly.
> [https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
> The biggest problem affecting ranger is missing "Stored procedures and 
> functions", "Select into".
> ranger use Stored procedures in setup scripts to simplify SQL.
> Some work is needed to remove the stored procedure.
>  
>  
> {code:java}
> ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 14 near "PROCEDURE if 
> exists getXportalUIdByLoginId" 
> ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 16 near "PROCEDURE 
> `getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_portal_user.id into myid FROM x_portal_user WHERE 
> x_portal_user.login_id = input_val;
> END" 
> ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 14 near "PROCEDURE if 
> exists getModulesIdByName" 
> ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 16 near "PROCEDURE 
> `getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_modules_master.id into myid FROM x_modules_master WHERE 
> x_modules_master.module = input_val;
> END" 
> ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 14 near "PROCEDURE if 
> exists insertRangerPrerequisiteEntries" 
> ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have 
> an error in your SQL syntax; check the manual that corresponds to your TiDB 
> version for the right syntax to use line 1 column 16 near "PROCEDURE 
> `insertRangerPrerequisiteEntries`()
> BEGIN
> DECLARE adminID bigint;
> DECLARE keyadminID bigint;
> DECLARE rangerusersyncID bigint;
> DECLARE rangertagsyncID bigint;
> DECLARE moduleIdReports bigint;
> DECLARE moduleIdResourceBasedPolicies bigint;
> DECLARE moduleIdAudit bigint;
> DECLARE moduleIdUG bigint;
> DECLARE moduleIdTagBasedPolicies bigint;
> DECLARE moduleIdKeyMana
> ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql': 
> Unsupported type *ast.CallStmt
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (RANGER-3984) Support using TiDB as mysql-db in ranger

[kirby zhou (Jira)]

 [ 
https://issues.apache.org/jira/browse/RANGER-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kirby zhou updated RANGER-3984:
---
Description: 
TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
deploy ranger based on tidb. But TiDB is missing some features, which makes 
ranger unable to install properly.

[https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]

The biggest problem affecting ranger is missing "Stored procedures and 
functions", "Select into".

ranger use Stored procedures in setup scripts to simplify SQL.

Some work is needed to remove the stored procedure.

 

 
{code:java}
ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
getXportalUIdByLoginId" 
ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_portal_user.id into myid FROM x_portal_user WHERE 
x_portal_user.login_id = input_val;
END" 
ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
getModulesIdByName" 
ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_modules_master.id into myid FROM x_modules_master WHERE 
x_modules_master.module = input_val;
END" 



ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
insertRangerPrerequisiteEntries" 
ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`insertRangerPrerequisiteEntries`()
BEGIN
DECLARE adminID bigint;
DECLARE keyadminID bigint;
DECLARE rangerusersyncID bigint;
DECLARE rangertagsyncID bigint;
DECLARE moduleIdReports bigint;
DECLARE moduleIdResourceBasedPolicies bigint;
DECLARE moduleIdAudit bigint;
DECLARE moduleIdUG bigint;
DECLARE moduleIdTagBasedPolicies bigint;
DECLARE moduleIdKeyMana
ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql': 
Unsupported type *ast.CallStmt


{code}
 

  was:
TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
deploy ranger based on tidb. But TiDB is missing some features, which makes 
ranger unable to install properly.

[https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]

The biggest problem affecting ranger is missing "Stored procedures and 
functions".

ranger use Stored procedures in setup scripts to simplify SQL.

Some work is needed to remove the stored procedure.

 

 
{code:java}
ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
getXportalUIdByLoginId" 
ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_portal_user.id into myid FROM x_portal_user WHERE 
x_portal_user.login_id = input_val;
END" 
ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
getModulesIdByName" 
ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_modules_master.id into myid FROM x_modules_master WHERE 
x_modules_master.module =

[jira] [Created] (RANGER-3984) Support using TiDB as mysql-db in ranger

[kirby zhou (Jira)]

kirby zhou created RANGER-3984:
--

 Summary: Support using TiDB as mysql-db in ranger
 Key: RANGER-3984
 URL: https://issues.apache.org/jira/browse/RANGER-3984
 Project: Ranger
  Issue Type: Improvement
  Components: admin, kms
Affects Versions: 2.3.0, 3.0.0
Reporter: kirby zhou


TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to 
deploy ranger based on tidb. But TiDB is missing some features, which makes 
ranger unable to install properly.

[https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]

The biggest problem affecting ranger is missing "Stored procedures and 
functions".

ranger use Stored procedures in setup scripts to simplify SQL.

Some work is needed to remove the stored procedure.

 

 
{code:java}
ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
getXportalUIdByLoginId" 
ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_portal_user.id into myid FROM x_portal_user WHERE 
x_portal_user.login_id = input_val;
END" 
ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
getModulesIdByName" 
ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_modules_master.id into myid FROM x_modules_master WHERE 
x_modules_master.module = input_val;
END" 



ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists 
insertRangerPrerequisiteEntries" 
ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have 
an error in your SQL syntax; check the manual that corresponds to your TiDB 
version for the right syntax to use line 1 column 16 near "PROCEDURE 
`insertRangerPrerequisiteEntries`()
BEGIN
DECLARE adminID bigint;
DECLARE keyadminID bigint;
DECLARE rangerusersyncID bigint;
DECLARE rangertagsyncID bigint;
DECLARE moduleIdReports bigint;
DECLARE moduleIdResourceBasedPolicies bigint;
DECLARE moduleIdAudit bigint;
DECLARE moduleIdUG bigint;
DECLARE moduleIdTagBasedPolicies bigint;
DECLARE moduleIdKeyMana
ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql': 
Unsupported type *ast.CallStmt


{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624829#comment-17624829
 ] 

kirby zhou commented on RANGER-2128:


What's the difference between SparkSQL's ACL model and HadoopSQL?

> Implement SparkSQL plugin
> -
>
> Key: RANGER-2128
> URL: https://issues.apache.org/jira/browse/RANGER-2128
> Project: Ranger
>  Issue Type: New Feature
>  Components: plugins, Ranger
>Affects Versions: 1.1.0
>Reporter: t oo
>Assignee: Kent Yao
>Priority: Major
> Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Implement SparkSQL plugin



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224740
---


Ship it!




Ship It!

- Kirby Zhou


On 九月 28, 2022, 3:02 p.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated 九月 28, 2022, 3:02 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
>  
> 
>   
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/9/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224739
---




security-admin/src/main/resources/conf.dist/logback.xml
Lines 86 (patched)
<https://reviews.apache.org/r/74131/#comment313521>

It works, but unnecessory.


- Kirby Zhou


On 九月 28, 2022, 3:02 p.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated 九月 28, 2022, 3:02 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
>  
> 
>   
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/9/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74136: RANGER-3685:hive 'show' sql produces excessive audit log

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74136/#review224738
---


Ship it!




Ship It!

- Kirby Zhou


On 九月 24, 2022, 2:37 p.m., Xuze Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74136/
> ---
> 
> (Updated 九月 24, 2022, 2:37 p.m.)
> 
> 
> Review request for ranger, Kirby Zhou, Abhay Kulkarni, and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-3685
> https://issues.apache.org/jira/browse/RANGER-3685
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> hive show sql produces too many audit logs, which will flood valuable audit 
> logs and take up a lot of storage resources. This patch aims to provide a 
> configuration item to support simplified hive audit logging
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
>  6675125e1 
>   
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
>  8f6801be1 
> 
> 
> Diff: https://reviews.apache.org/r/74136/diff/1/
> 
> 
> Testing
> ---
> 
> 1. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is not 
> set or set to false. The code behaves the same as before,i.e record audit 
> logs for each databases or tables
> 2. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is set 
> to true. It will only record an audit log about USE operation
> 
> 
> Thanks,
> 
> Xuze Yang
> 
>

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224708
---



There was an error displaying this diff.


The patch to 'security-admin/src/main/resources/conf.dist/logback.xml' 
didn't apply cleanly. The temporary files have been left in 
'/tmp/reviewboard.joRCmj' for debugging purposes.
`patch` returned: patch:  malformed patch at line 28: -- 





 This may be a bug in the software, a temporary outage, or an issue
 with the format of your diff.





 Please try again, and if you still have trouble,
 contact support.


   
  

  
   

Details

- Kirby Zhou


On 九月 23, 2022, 8:37 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated 九月 23, 2022, 8:37 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
>  
> 
>   
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/5/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

[jira] [Commented] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.

[kirby zhou (Jira)]

[ 
https://issues.apache.org/jira/browse/RANGER-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610461#comment-17610461
 ] 

kirby zhou commented on RANGER-3775:


It works

> Logback.xml has been incorrectly modified by RANGER-3704.
> -
>
> Key: RANGER-3775
> URL: https://issues.apache.org/jira/browse/RANGER-3775
> Project: Ranger
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 3.0.0
>    Reporter: kirby zhou
>Assignee: Ramachandran
>Priority: Critical
> Attachments: 
> 0001-RANGER-3775-Logback.xml-has-been-incorrectly-modifie.patch
>
>
> {code:java}
> git show 361f179249 | filterdiff -i '*/logback.xml'
> diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml 
> b/security-admin/src/main/webapp/WEB-INF/logback.xml
> index 997f3bc59..53cdc49cf 100644
> --- a/security-admin/src/main/webapp/WEB-INF/logback.xml
> +++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
> @@ -80,7 +80,7 @@
>    
>      
>    
> -  
> +  
>      
>    
>     
> {code}
> These changes seems not related to the issue RANGER-3704.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Review Request 74136: RANGER-3685:hive 'show' sql produces excessive audit log

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74136/#review224707
---



Why modify filterListCmdObjects() only?

- Kirby Zhou


On 九月 24, 2022, 2:37 p.m., Xuze Yang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74136/
> ---
> 
> (Updated 九月 24, 2022, 2:37 p.m.)
> 
> 
> Review request for ranger, Kirby Zhou, Abhay Kulkarni, and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-3685
> https://issues.apache.org/jira/browse/RANGER-3685
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> hive show sql produces too many audit logs, which will flood valuable audit 
> logs and take up a lot of storage resources. This patch aims to provide a 
> configuration item to support simplified hive audit logging
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
>  6675125e1 
>   
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
>  8f6801be1 
> 
> 
> Diff: https://reviews.apache.org/r/74136/diff/1/
> 
> 
> Testing
> ---
> 
> 1. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is not 
> set or set to false. The code behaves the same as before,i.e record audit 
> logs for each databases or tables
> 2. when configuration "xasecure.hive.simplify.audit.of.hive.show.sql" is set 
> to true. It will only record an audit log about USE operation
> 
> 
> Thanks,
> 
> Xuze Yang
> 
>

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Kirby Zhou]

> On 九月 21, 2022, 8:13 a.m., Kirby Zhou wrote:
> > It seems work.
> > 
> > But are you sure you want to change the log level of com.mchange from the 
> > default warn to error?
> > What's the reason?
> 
> Ramachandran Krishnan wrote:
> The existing behaviour for the logger name com.mchange ,we kept error 
> level only .So I was thinking of keeping error level.You are right the 
> default level is warn which i was overriding into error for the logger name 
> com.mchange (anyway the same logger name already present in the 
> logback.xml.We only changed into the different appender)
> 
> Ramachandran Krishnan wrote:
> Is there any specific reason to keep into warn for the logger name 
> com.mchange (backed by c3p0) ?

The old version of c3p0 is running with level='warn' for years.
If there is no specific reason, we should not change it.


- Kirby


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224690
---


On 九月 21, 2022, 8:46 a.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated 九月 21, 2022, 8:46 a.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
> 
>   
> 
>   
> 
> 
> 
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>

Re: Review Request 74131: Logback.xml has been incorrectly modified by RANGER-3704.

[Kirby Zhou]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/74131/#review224690
---



It seems work.

But are you sure you want to change the log level of com.mchange from the 
default warn to error?
What's the reason?


security-admin/src/main/resources/conf.dist/logback.xml
Line 83 (original), 83 (patched)
<https://reviews.apache.org/r/74131/#comment313484>

If you just want change level and reuse the root logger, just write as that



It will inherit the appender of root.


- Kirby Zhou


On 九月 20, 2022, 6:33 p.m., Ramachandran Krishnan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/74131/
> ---
> 
> (Updated 九月 20, 2022, 6:33 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Pradeep Agrawal.
> 
> 
> Bugs: RANGER-3775
> https://issues.apache.org/jira/browse/RANGER-3775
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> {logger name="com.mchange"}
> is used by c3p0 and {logger name="jdbc.connection"}
> 
> is used by log4jdbc
> 
> 
>   
> 
>   
> 
> 
> 
>   
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/resources/conf.dist/logback.xml 53cdc49cf 
> 
> 
> Diff: https://reviews.apache.org/r/74131/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ramachandran Krishnan
> 
>