[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS
[ https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17844512#comment-17844512 ] kirby zhou commented on RANGER-3998: It gets 2 ship now. Who can merge it ? > Support Ranger KMS integration with AWS KMS > --- > > Key: RANGER-3998 > URL: https://issues.apache.org/jira/browse/RANGER-3998 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0, 2.4.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > AWS KMS is widely used by many customers. > Therefore, RangerKMS should support hosting MasterKey to AWS KMS. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS
[ https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17834769#comment-17834769 ] kirby zhou commented on RANGER-3998: This patch is just a simple imitation of RangerGoogleCloudHSMProvider. The work is done for using the key stored in AWS KMS as the master key of Ranger KMS. class RangerAWSKMSProvider just implements RangerKMSMKI interface. The generateMasterKey method does not actually create a masterkey, It calls AWSKMS.listAliases and AWSKMS.getKeyMetadata to verify whether the masterkey exists. The encryptZoneKey method calls AWSKMS.encrypt to encrypt zone key, and the decryptZoneKey calls AWSKMS.decrypt to decrypt. RangerKeyStoreProvider.java is modified to load and activate RangerAWSKMSProvider according to the configuration. I add 5 lines in install.properties, the meaning is * #- Ranger AWS KMS -- AWS_KMS_ENABLED=false AWS_KMS_MASTERKEY_ID=#The id of master key in AWS KMS AWS_CLIENT_ACCESSKEY=#The access key to AWS service AWS_CLIENT_SECRETKEY=#The secret key to AWS service AWS_CLIENT_REGION=#The region of AWS service The modification of setup.sh will map the 5 properties into dbks-site.xml as * AWS_KMS_ENABLED = "ranger.kms.awskms.enabled"; * AWSKMS_MASTER_KEY_ID = "ranger.kms.awskms.masterkey.id"; * AWS_CLIENT_ACCESSKEY = "ranger.kms.aws.client.accesskey"; * AWS_CLIENT_SECRETKEY = "ranger.kms.aws.client.secretkey"; * AWS_CLIENT_REGION = "ranger.kms.aws.client.region"; And the patch do some minor changes to prevent conflicting of Tencent KMS. BTW: AWS KMS API is here: [https://docs.aws.amazon.com/kms/latest/developerguide/programming-top.html] > Support Ranger KMS integration with AWS KMS > --- > > Key: RANGER-3998 > URL: https://issues.apache.org/jira/browse/RANGER-3998 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0, 2.4.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > AWS KMS is widely used by many customers. > Therefore, RangerKMS should support hosting MasterKey to AWS KMS. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4454) RangerKMS adds support for the SM4 encryption algorithm.
[ https://issues.apache.org/jira/browse/RANGER-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832450#comment-17832450 ] kirby zhou commented on RANGER-4454: I think we should also bump the version org.bouncycastle.* to versions of jdk18on. The old versions have CVEs. [https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk15on/1.70] > RangerKMS adds support for the SM4 encryption algorithm. > > > Key: RANGER-4454 > URL: https://issues.apache.org/jira/browse/RANGER-4454 > Project: Ranger > Issue Type: New Feature > Components: kms, Ranger >Affects Versions: 2.3.0 > Environment: !image-2023-10-04-08-31-03-261.png! >Reporter: xiaojunxiang >Priority: Major > Attachments: HDFS_SM4.jpg, Jira_HDFS_SM4.jpg, RANGER-4454-000.patch, > SM4_NotAvaliable.jpg > > Time Spent: 20m > Remaining Estimate: 0h > > SM4 is already supported in recent versions (3.4.0) of hdfs transparent > encryption, > So RangerKMS should adapt to this. > When I add in the region of the Encryption key used "SM4 / CTR/NoPadding" > algorithm, RangerKMS background will print "under Caused by: Java security. > NoSuchAlgorithmException: SM4 KeyGenerator not available" > > Hadoop website: > [https://apache.github.io/hadoop/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html] > Jira(HDFS supported SM4): https://issues.apache.org/jira/browse/HDFS-15098 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828657#comment-17828657 ] kirby zhou commented on RANGER-3831: pegasus have done its work. [https://github.com/apache/incubator-pegasus/issues/1054] Could anyone do some work to merge this definition into ranger-admin ? > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > Attachments: ranger-servicedef-pegasus.json > > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3409) Update Jackson and remove Codehaus version
[ https://issues.apache.org/jira/browse/RANGER-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17803502#comment-17803502 ] kirby zhou commented on RANGER-3409: Is there any progress in this matter? The security Commissioner is asking to avoid Codehaus Jackson. :( > Update Jackson and remove Codehaus version > -- > > Key: RANGER-3409 > URL: https://issues.apache.org/jira/browse/RANGER-3409 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: Andrew Charneski >Priority: Blocker > > An old version of Jackson (Codehaus Jackson 1.9.13) is still being used. > Jackson has since moved namespaces with a reorganized library structure. > Update all references to the older version to use the newer version (which is > currently used in some modules). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.
[
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17750928#comment-17750928
]
kirby zhou commented on RANGER-4326:
Sorry, I found the reason.
2 KMS need to be synced with ZooKeeper. the following example is missing in
kms-site.xml
{code:java}
hadoop.kms.authentication.zk-dt-secret-manager.enable
true
If true, Hadoop KMS uses ZKDelegationTokenSecretManager to persist
TokenIdentifiers and DelegationKeys in ZooKeeper.
hadoop.kms.authentication.zk-dt-secret-manager.zkConnectionString
#HOSTNAME#:#PORT#,...
The ZooKeeper connection string, a comma-separated list of hostnames and
port.
hadoop.kms.authentication.zk-dt-secret-manager.znodeWorkingPath
/hadoop-kms/zkdtsm
The ZooKeeper znode path where the KMS instances will store and retrieve
the secret from. All the KMS instances that need to coordinate should
point to the same path.
hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType
sasl
The ZooKeeper authentication type, 'none' (default) or 'sasl' (Kerberos).
hadoop.kms.authentication.zk-dt-secret-manager.kerberos.keytab
/etc/hadoop/conf/kms.keytab
The absolute path for the Kerberos keytab with the credentials to
connect to ZooKeeper. This parameter is effective only when
hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType is set to
'sasl'.
hadoop.kms.authentication.zk-dt-secret-manager.kerberos.principal
kms/#HOSTNAME#
The Kerberos service principal used to connect to ZooKeeper.
This parameter is effective only when
hadoop.kms.authentication.zk-dt-secret-manager.zkAuthType is set to
'sasl'.
{code}
> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
> {code}
> A lot of exceptions happens
> {code:java}
> ava.io.IOException: HTTP status [403], message [Forbidden], URL
>
[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.
[
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746760#comment-17746760
]
kirby zhou commented on RANGER-4326:
Check the code of KMS and hadoop, It seems that two KMS token cannot recognize
each other without ZK.
If we do not take zk-dt-secret-manager.enable = true in some conf.
# KMS compares the password in the token with the password calculated by
itself.
# The calculated password depends on getDelegationKey(id.getMasterKeyId());
# delegationKey is generated by updateCurrentKey, and saved by
storeDelegationKey.
# updateCurrentKey is based on random, so 2 KMS are not sync.
new DelegationKey(newCurrentId, System .currentTimeMillis() + keyUpdateInterval
+ tokenMaxLifetime, {+}*generateSecret*{+}());
{code:java}
// AbstractDelegationTokenSecretManager.java
public synchronized long renewToken(Token token,
String renewer) throws InvalidToken, IOException {
DelegationKey key = getDelegationKey(id.getMasterKeyId());
byte[] password = createPassword(token.getIdentifier(), key.getKey());
if (!MessageDigest.isEqual(password, token.getPassword())) {
throw new AccessControlException(renewer
+ " is trying to renew a token "
+ formatTokenId(id) + " with wrong password");
}
}
protected DelegationKey getDelegationKey(int keyId) {
return allKeys.get(keyId);
}
protected void storeDelegationKey(DelegationKey key) throws IOException {
allKeys.put(key.getKeyId(), key);
storeNewMasterKey(key);
}
private void updateCurrentKey() throws IOException {
LOG.info("Updating the current master key for generating delegation tokens");
/* Create a new currentKey with an estimated expiry date. */
int newCurrentId;
synchronized (this) {
newCurrentId = incrementCurrentKeyId();
}
DelegationKey newKey = new DelegationKey(newCurrentId, System
.currentTimeMillis()
+ keyUpdateInterval + tokenMaxLifetime, generateSecret());
//Log must be invoked outside the lock on 'this'
logUpdateMasterKey(newKey);
synchronized (this) {
currentKey = newKey;
storeDelegationKey(currentKey);
}
}
protected SecretKey generateSecret() {
SecretKey key;
synchronized (keyGen) {
key = keyGen.generateKey();
}
return key;
}
{code}
> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
>
[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.
[
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746749#comment-17746749
]
kirby zhou commented on RANGER-4326:
{code:java}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
final String kmsURI1 =
"kms://h...@kms01-throne01.sensorsdata.cn:9292/kms";
final String kmsURI2 =
"kms://h...@kms02-throne01.sensorsdata.cn:9292/kms";
// Logon
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
String username = ugi.getShortUserName();
System.out.println(username);
// new keyproider
KMSClientProvider kms1 = new KMSClientProvider(new URI(kmsURI1), conf);
KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
// do renew
Token token1 = kms1.getDelegationToken(username);
token1.setService(new Text(dtCombineService));
System.out.println("renew token1 through kms2 begin");
kms2.renewDelegationToken(token1);
System.out.println("renew token1 through kms2 done");
}
{code}
> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
> {code}
> A lot of exceptions happens
> {code:java}
> ava.io.IOException: HTTP status [403], message [Forbidden], URL
> [http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
> exception [com.fasterxml.jackson.core.JsonParseException: Unexpected
> character ('<' (code 60)): expected a valid value (JSON String, Number,
> Array, Object
[jira] (RANGER-4326) Cannot renew token when multiple KMS are applied.
[ https://issues.apache.org/jira/browse/RANGER-4326 ]
kirby zhou deleted comment on RANGER-4326:
was (Author: kirbyzhou):
// A more simpler example to reproduce public static void main(String[]
args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
final String kmsURI1 = "kms://http@kms01/kms";
final String kmsURI2 = "kms://http@kms02/kms";
// Logon UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
String username = ugi.getShortUserName();
System.out.println(username);
// new keyproider KMSClientProvider kms1 = new
KMSClientProvider(new URI(kmsURI1), conf);
KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
// do renew Token token1 = kms1.getDelegationToken(username);
token1.setService(new Text(dtCombineService));
System.out.println("renew token1 through kms2 begin");
kms2.renewDelegationToken(token1);
System.out.println("renew token1 through kms2 done");
}
> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
> {code}
> A lot of exceptions happens
> {code:java}
> ava.io.IOException: HTTP status [403], message [Forbidden], URL
> [http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
> exception [com.fasterxml.jackson.core.JsonParseException: Unexpected
> character ('<' (code 60)): expected a valid value (JSON String, Number,
> Array, Object or token 'null', 'true' or 'false') at [Source:
>
[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.
[
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746747#comment-17746747
]
kirby zhou commented on RANGER-4326:
// A more simpler example to reproduce public static void main(String[]
args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
final String dtCombineService = "kms://http@kms01;kms02:9292/kms";
final String kmsURI1 = "kms://http@kms01/kms";
final String kmsURI2 = "kms://http@kms02/kms";
// Logon UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
String username = ugi.getShortUserName();
System.out.println(username);
// new keyproider KMSClientProvider kms1 = new
KMSClientProvider(new URI(kmsURI1), conf);
KMSClientProvider kms2 = new KMSClientProvider(new URI(kmsURI2), conf);
// do renew Token token1 = kms1.getDelegationToken(username);
token1.setService(new Text(dtCombineService));
System.out.println("renew token1 through kms2 begin");
kms2.renewDelegationToken(token1);
System.out.println("renew token1 through kms2 done");
}
> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
> }
> }
> System.out.println("==end renew==");
> }
> }
> }
> {code}
> A lot of exceptions happens
> {code:java}
> ava.io.IOException: HTTP status [403], message [Forbidden], URL
> [http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
> exception [com.fasterxml.jackson.core.JsonParseException: Unexpected
> character ('<' (code 60)): expected a valid value (JSON String, Number,
> Array, Object or token 'null', 'true' or 'false')
[jira] [Commented] (RANGER-4326) Cannot renew token when multiple KMS are applied.
[
https://issues.apache.org/jira/browse/RANGER-4326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746745#comment-17746745
]
kirby zhou commented on RANGER-4326:
{code:java}
// A simpler example to reproduce
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
final String KMSURI = "kms://http@kms01;kms02:9292/kms";
final String keyName = "mykey";
// Logon
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("myuser", "my.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
String username = ugi.getShortUserName();
System.out.println(username);
// new keyproider
LoadBalancingKMSClientProvider kms =
(LoadBalancingKMSClientProvider)KeyProviderFactory.get(new URI(KMSURI), conf);
// try eek & dek
System.out.println("try do eek & dek");
KeyProviderCryptoExtension.EncryptedKeyVersion eek =
kms.generateEncryptedKey(keyName);
System.out.printf("IV(%d) %s\n", eek.getEncryptedKeyIv().length * 8,
StringUtils.byteToHexString(eek.getEncryptedKeyIv()));
System.out.printf("EEK(%d) %s\n",
eek.getEncryptedKeyVersion().getMaterial().length * 8,
StringUtils.byteToHexString(eek.getEncryptedKeyVersion().getMaterial()));
KeyProvider.KeyVersion dek = kms.decryptEncryptedKey(eek);
StringUtils.byteToHexString(dek.getMaterial());
System.out.printf("DEK(%d) %s\n", dek.getMaterial().length * 8,
StringUtils.byteToHexString(dek.getMaterial()));
// do renew
for (int i = 0; i < 10; ++i) {
System.out.printf("pass %02d\n", i);
System.out.print("begin renew\n");
Token token = kms.getDelegationToken(username);
kms.renewDelegationToken(token);
System.out.print("end renew\n");
}
}{code}
> Cannot renew token when multiple KMS are applied.
> -
>
> Key: RANGER-4326
> URL: https://issues.apache.org/jira/browse/RANGER-4326
> Project: Ranger
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> When multiple KMS are applied with kerberos. Flink on yarn can not renew
> tokens.
>
> Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
> FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
> tokens.
> When it calls LoadBalancingKMSClientProvider.getDelegationToken.
> LoadBalancingKMSClientProvider calls doOp to call one of N
> KMSClientProvider.getDelegationToken().
>
> When renew the token, LoadBalancingKMSClientProvider may call another
> KMSClientProvider to do op. It usually fails.
>
> FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
> and hadoop.kms.authentication.signature.secret.file="same content file".
>
> Some Sample code:
> {code:java}
> public static void main(String[] args) throws Exception {
> Configuration conf = new Configuration();
> conf.set("hadoop.security.authorization", "true");
> conf.set("hadoop.security.authentication", "kerberos");
> conf.set("dfs.data.transfer.protection", "authentication");
> conf.set("hadoop.security.key.provider.path",
> "kms://http@kms01;kms02:9292/kms");
> conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
> conf.set("fs.defaultFS", "hdfs://namenode");
> // Login with keytab
> UserGroupInformation.setConfiguration(conf);
> UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
> "/Users/kirbyzhou/Develop/testuser.keytab");
> UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
> System.out.println(UserGroupInformation.getCurrentUser().getUserName());
> // GetFS
> FileSystem fs = FileSystem.get(conf);
>
> System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
> // Renew
> for (int i = 0; i < 20; ++i) {
> Thread.sleep(200);
> System.out.printf("===pass %02d===\n", i);
> {
> System.out.println("==begin renew==");
> Credentials credentials = ugi.getCredentials();
> fs.addDelegationTokens("sa_cluster", credentials);
> for (Token token : credentials.getAllTokens()) {
> System.out.println(token);
> try {
> token.renew(conf);
> } catch (IOException e) {
> System.err.println(e);
>
[jira] [Created] (RANGER-4326) Cannot renew token when multiple KMS are applied.
kirby zhou created RANGER-4326:
--
Summary: Cannot renew token when multiple KMS are applied.
Key: RANGER-4326
URL: https://issues.apache.org/jira/browse/RANGER-4326
Project: Ranger
Issue Type: Bug
Components: kms
Affects Versions: 2.4.0, 2.3.0
Reporter: kirby zhou
When multiple KMS are applied with kerberos. Flink on yarn can not renew tokens.
Flink calls FileSystem.addDelegationTokens to get all tokens to renew.
FileSystem.addDelegationTokens calls collectDelegationTokens to collect all
tokens.
When it calls LoadBalancingKMSClientProvider.getDelegationToken.
LoadBalancingKMSClientProvider calls doOp to call one of N
KMSClientProvider.getDelegationToken().
When renew the token, LoadBalancingKMSClientProvider may call another
KMSClientProvider to do op. It usually fails.
FYI: have already set hadoop.kms.authentication.signer.secret.provider=file,
and hadoop.kms.authentication.signature.secret.file="same content file".
Some Sample code:
{code:java}
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "true");
conf.set("hadoop.security.authentication", "kerberos");
conf.set("dfs.data.transfer.protection", "authentication");
conf.set("hadoop.security.key.provider.path",
"kms://http@kms01;kms02:9292/kms");
conf.set("dfs.client.ignore.namenode.default.kms.uri", "true");
conf.set("fs.defaultFS", "hdfs://namenode");
// Login with keytab
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("testuser@TESTREALM",
"/Users/kirbyzhou/Develop/testuser.keytab");
UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
System.out.println(UserGroupInformation.getCurrentUser().getUserName());
// GetFS
FileSystem fs = FileSystem.get(conf);
System.out.println(((DistributedFileSystem)fs).getClient().getKeyProviderUri());
// Renew
for (int i = 0; i < 20; ++i) {
Thread.sleep(200);
System.out.printf("===pass %02d===\n", i);
{
System.out.println("==begin renew==");
Credentials credentials = ugi.getCredentials();
fs.addDelegationTokens("sa_cluster", credentials);
for (Token token : credentials.getAllTokens()) {
System.out.println(token);
try {
token.renew(conf);
} catch (IOException e) {
System.err.println(e);
}
}
System.out.println("==end renew==");
}
}
}
{code}
A lot of exceptions happens
{code:java}
ava.io.IOException: HTTP status [403], message [Forbidden], URL
[http://kms01:9292/kms/v1/?op=RENEWDELEGATIONTOKEN=KgAKc2FfY2x1c3RlcgpzYV9jbHVzdGVyAIoBiYffA4WKAYmr64eFjgG_AhQ7Oo9G0Lc8IguxB0IgenAHsJ--DQZrbXMtZHRPa21zOi8vaHR0cEBrbXMwMS10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjtrbXMwMi10aHJvbmUwMS5zZW5zb3JzZGF0YS5jbjo5MjkyL2ttcw],
exception [com.fasterxml.jackson.core.JsonParseException: Unexpected character
('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or
token 'null', 'true' or 'false') at [Source:
(sun.net.www.protocol.http.HttpURLConnection$HttpInputStream); line: 1, column:
2]]at
org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:167)
~[classes/:?]at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:318)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.renewDelegationToken(DelegationTokenAuthenticator.java:235)
~[hadoop-common-3.3.4.jar:?]at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.renewDelegationToken(DelegationTokenAuthenticatedURL.java:435)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1072)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1069)
~[hadoop-common-3.3.4.jar:?] at
java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332]at
javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
~[hadoop-common-3.3.4.jar:?]at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.renewDelegationToken(KMSClientProvider.java:1068)
~[hadoop-common-3.3.4.jar:?]at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:270)
~[hadoop-common-3.3.4.jar:?] at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$2.call(LoadBalancingKMSClientProvider.java:267)
[jira] [Commented] (RANGER-4147) Ranger KMS consume 50% of CPU memory
[ https://issues.apache.org/jira/browse/RANGER-4147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17714387#comment-17714387 ] kirby zhou commented on RANGER-4147: Maybe you can try using jstack tool to see what KMS is doing. When KMS consumes 150% CPU. > Ranger KMS consume 50% of CPU memory > - > > Key: RANGER-4147 > URL: https://issues.apache.org/jira/browse/RANGER-4147 > Project: Ranger > Issue Type: Bug > Components: kms >Affects Versions: 2.3.0 > Environment: secured >Reporter: Bhavik Patel >Priority: Critical > > Ranger KMS consume 50% of CPU memory and many time it consume even 150% > cc: [~dhavalshah9131] [~kirbyzhou] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4106) NullPtr Exception when REST API /service/roles/secure/download/ is not allowed to user.
kirby zhou created RANGER-4106:
--
Summary: NullPtr Exception when REST API
/service/roles/secure/download/ is not allowed to user.
Key: RANGER-4106
URL: https://issues.apache.org/jira/browse/RANGER-4106
Project: Ranger
Issue Type: Bug
Components: admin
Affects Versions: 2.3.0, 3.0.0, 2.4.0
Reporter: kirby zhou
I have see a lot of exceptions in log catalina.out like that:
{code:java}
Feb 23, 2023 7:17:21 AM com.sun.jersey.spi.container.ContainerResponse
mapMappableContainerException
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to
the HTTP container
java.lang.NullPointerException
at
org.apache.ranger.biz.AssetMgr.doCreateOrUpdateXXPluginInfo(AssetMgr.java:831)
at
org.apache.ranger.biz.AssetMgr.createOrUpdatePluginInfo(AssetMgr.java:791)
at org.apache.ranger.biz.AssetMgr.createPluginInfo(AssetMgr.java:728)
at
org.apache.ranger.rest.RoleREST.getSecureRangerRolesIfUpdated(RoleREST.java:874)
at
org.apache.ranger.rest.RoleREST$$FastClassBySpringCGLIB$$d1176b81.invoke()
...
{code}
Using debugger to trace the code.
It caused by
{code:java}
// AssertMgr.java doCreateOrUpdateXXPluginInfo()
// which get a null value of RoleDownloadedVersion, raise exception.
831: if (pluginInfo.getRoleDownloadTime() != null &&
pluginInfo.getRoleDownloadedVersion().equals(pluginInfo.getRoleActiveVersion())
// called by createOrUpdatePluginInfo() in AssertMgr.java
...
// called by createPluginInfo() in AssetMgr.java
// which will set RoleDownloadTime to non-null, regardless of the value of
RoleDownloadedVersion/downloadedVersion.
case RangerPluginInfo.ENTITY_TYPE_ROLES:
pluginSvcVersionInfo.setRoleActiveVersion(lastKnownVersion);
pluginSvcVersionInfo.setRoleActivationTime(lastActivationTime);
pluginSvcVersionInfo.setRoleDownloadedVersion(downloadedVersion);
pluginSvcVersionInfo.setRoleDownloadTime(new Date().getTime());
break;
case RangerPluginInfo.ENTITY_TYPE_USERSTORE:
pluginSvcVersionInfo.setUserStoreActiveVersion(lastKnownVersion);
pluginSvcVersionInfo.setUserStoreActivationTime(lastActivationTime);
pluginSvcVersionInfo.setUserStoreDownloadedVersion(downloadedVersion);
pluginSvcVersionInfo.setUserStoreDownloadTime(new Date().getTime());
break;
}
createOrUpdatePluginInfo(pluginSvcVersionInfo, entityType , httpCode,
clusterName);
// called by getSecureRangerRolesIfUpdated() in RoleRest.java
// which will not set downloadedVersion when isAllowed = false.
Long downloadedVersion = null;
...
if (isValid) {
try {
...
if (isAllowed) {
RangerRoles roles = roleStore.getRoles(serviceName,
lastKnownRoleVersion);
if (roles == null) {
downloadedVersion = lastKnownRoleVersion;
} else {
downloadedVersion = roles.getRoleVersion();
}
} else {
httpCode = HttpServletResponse.SC_FORBIDDEN; // assert user is
authenticated.
}
} catch (Throwable excp) {
}
}
assetMgr.createPluginInfo(serviceName, pluginId, request,
RangerPluginInfo.ENTITY_TYPE_ROLES, downloadedVersion, lastKnownRoleVersion,
lastActivationTime, httpCode, clusterName, pluginCapabilities);
{code}
The simplest method is to modify AssertMgr.java to that, this is the behavior
tag and policy
{code:java}
if (pluginInfo.getRoleDownloadedVersion() != null &&
pluginInfo.getRoleDownloadedVersion().equals(pluginInfo.getRoleActiveVersion()))
{
{code}
Btw: the case of UserStore seems have the same bug.
{code:java}
} else {
if (pluginInfo.getUserStoreDownloadTime() != null &&
pluginInfo.getUserStoreDownloadedVersion().equals(pluginInfo.getUserStoreActiveVersion()))
{
// This is our best guess of when users and groups may have been
downloaded
pluginInfo.setUserStoreDownloadTime(pluginInfo.getUserStoreActivationTime());
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692504#comment-17692504
]
kirby zhou commented on RANGER-4104:
https://issues.apache.org/jira/browse/RANGER-3756
is a eclipselink jpa related problem too.
I suggest to upgrade eclipselink library version.
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-4104) XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL code
[
https://issues.apache.org/jira/browse/RANGER-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692027#comment-17692027
]
kirby zhou commented on RANGER-4104:
How to reproduce it ?
I have not find any SQL error in my machine.
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code
> ---
>
> Key: RANGER-4104
> URL: https://issues.apache.org/jira/browse/RANGER-4104
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Andrew Luo
>Priority: Major
> Attachments:
> 0001-Fix-type-of-authWindowStartTime-parameter-in-XXAuthS.patch
>
>
> XXAuthSessionDao.getRecentAuthFailureCountByLoginId produces incorrect SQL
> code due to an error in how the authWindowStartTime Date parameter is bound.
> It is currently bound with setParameter("authWindowStartTime",
> authWindowStartTime) however, [JPA 2.2 Specification Section
> 11.1.53|https://download.oracle.com/otn-pub/jcp/persistence-2_2-mrel-spec/JavaPersistence.pdf]
> says that it should be bound by specifying an additional parameter
> TemporalType.DATE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690196#comment-17690196
]
kirby zhou edited comment on RANGER-3756 at 2/17/23 7:27 AM:
-
Reproduce steps:
# create a cloud mysql-5.7 instance on Tencent Cloud 【
[https://www.tencentcloud.com/products/cdb] 】
# setup a ranger admin with the mysql db
# use web browser to open ranger-admin site
# create a HDFS service 【You DONOT need a real hdfs cluster】
# delete the HDFS service
# error happens
!image-2023-02-17-15-23-46-271.png!
!image-2023-02-17-15-24-11-315.png!
Additional Info:
I can not reproduce the bug with my private MySQL-8 server with GTID=on;
was (Author: kirbyzhou):
Reproduce steps:
# create a cloud mysql-5.7 instance on Tencent Cloud
# setup a ranger admin with the mysql db
# use web browser to open ranger-admin site
# create a HDFS service 【You DONOT need a real hdfs cluster】
# delete the HDFS service
# error happens
!image-2023-02-17-15-23-46-271.png!
!image-2023-02-17-15-24-11-315.png!
Additional Info:
I can not reproduce the bug with my private MySQL-8 server with GTID=on;
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3756:
---
Attachment: image-2023-02-17-15-24-11-315.png
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690196#comment-17690196
]
kirby zhou commented on RANGER-3756:
Reproduce steps:
# create a cloud mysql-5.7 instance on Tencent Cloud
# setup a ranger admin with the mysql db
# use web browser to open ranger-admin site
# create a HDFS service 【You DONOT need a real hdfs cluster】
# delete the HDFS service
# error happens
!image-2023-02-17-15-23-46-271.png!
!image-2023-02-17-15-24-11-315.png!
Additional Info:
I can not reproduce the bug with my private MySQL-8 server with GTID=on;
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png, image-2023-02-17-15-24-11-315.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3756:
---
Attachment: image-2023-02-17-15-23-26-423.png
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3756:
---
Attachment: image-2023-02-17-15-23-46-271.png
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
> Attachments: image-2023-02-17-15-23-26-423.png,
> image-2023-02-17-15-23-46-271.png
>
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689578#comment-17689578
]
kirby zhou edited comment on RANGER-3756 at 2/17/23 7:08 AM:
-
I had encounter this bug with the cloud service of mysql provided by Tencent
Cloud.
[http://cloud.tencent.com|http://cloud.tencent.com/] 【China】or
[https://www.tencentcloud.com/] 【international】
It happens at the ranger-2.3 branch.
{code:sql}
Server version: 5.7.18-txsql-log 20211101
mysql> show variables like '%gtid%';
+--+---+
| Variable_name | Value |
+--+---+
| binlog_gtid_simple_recovery | ON |
| enforce_gtid_consistency | ON |
| gtid_executed_compression_period | 1000 |
| gtid_mode | ON |
| gtid_next | AUTOMATIC |
| gtid_owned | |
| gtid_purged | |
| session_track_gtids | OFF |
+--+---+
8 rows in set (0.01 sec)
{code}
was (Author: kirbyzhou):
I had encounter this bug with the cloud service of mysql provided by Tencent
Cloud.
[http://cloud.tencent.com|http://cloud.tencent.com/] 【China】or
[https://www.tencentcloud.com/] 【international】
It happens at the ranger-2.3 branch.
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689578#comment-17689578
]
kirby zhou commented on RANGER-3756:
I had encounter this bug with the cloud service of mysql provided by Tencent
Cloud.
[http://cloud.tencent.com|http://cloud.tencent.com/] 【China】or
[https://www.tencentcloud.com/] 【international】
It happens at the ranger-2.3 branch.
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS
[ https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17689467#comment-17689467 ] kirby zhou commented on RANGER-3998: Anybody have more ideas ? Can help to commit this? > Support Ranger KMS integration with AWS KMS > --- > > Key: RANGER-3998 > URL: https://issues.apache.org/jira/browse/RANGER-3998 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0, 2.4.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > AWS KMS is widely used by many customers. > Therefore, RangerKMS should support hosting MasterKey to AWS KMS. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3682) Unify the ways that rangerkeystore to encapsulate zonekey
[ https://issues.apache.org/jira/browse/RANGER-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685766#comment-17685766 ] kirby zhou commented on RANGER-3682: Any other idea? > Unify the ways that rangerkeystore to encapsulate zonekey > - > > Key: RANGER-3682 > URL: https://issues.apache.org/jira/browse/RANGER-3682 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > Unify the ways that rangerkeystore to encapsulate zonekey > Now we have 2 styles of MasterKeyProvider: > # RangerMasterKey, RangerHSM, RangerSafenetKeySecure > # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, > RangerTencentKMSProvider > Style 1 can get out master key string from provider, Style 2 can not. > In old, I add a flag KeyVaultEnabled to distinguish them. > KeyVaultEnabled=false means style1, true means style2 > RangerKeyStore with style1 use SecretKeyEntry with SealedObject to store a > key and do encryption / decryption by itself. > RangerKeyStore with style2 use SecretKeyByteEntry to store a key and let MK > provider to encryption / decryption. > These logics are hard-coded in the class RangerKeyStore. These are ugly and > hard to maintain. I refactor it by removing SecretKeyEntry, and let providers > of style1 do encryption / decryption. > Add a common base class of RangerMasterKey, RangerHSM andd > RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common > logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes > SealedObject into byte[]. > So the new code does not change the actual storage format, and there is no > problem in compatibility. > = > > And, there is no unified method to initialize a master key provider. > Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI > classes. > I made a new RangerKMSMKIFactory class to unify it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4062) keytab in kms-site.xml is not set by setup.sh
kirby zhou created RANGER-4062: -- Summary: keytab in kms-site.xml is not set by setup.sh Key: RANGER-4062 URL: https://issues.apache.org/jira/browse/RANGER-4062 Project: Ranger Issue Type: Bug Components: kms Affects Versions: 2.3.0, 3.0.0 Reporter: kirby zhou hadoop.kms.authentication.kerberos.keytab in kms-site.xml is used by org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler::init() to init kerberos authentication. Buy it is not set by setup.sh, setup.sh only set $kms_keytab to ranger.ks.kerberos.keytab in dbks-site.xml. This makes it very troublesome for us to setup a KMS with Kerberos. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3682) Unify the ways that rangerkeystore to encapsulate zonekey
[ https://issues.apache.org/jira/browse/RANGER-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17679005#comment-17679005 ] kirby zhou commented on RANGER-3682: rebased > Unify the ways that rangerkeystore to encapsulate zonekey > - > > Key: RANGER-3682 > URL: https://issues.apache.org/jira/browse/RANGER-3682 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > Unify the ways that rangerkeystore to encapsulate zonekey > Now we have 2 styles of MasterKeyProvider: > # RangerMasterKey, RangerHSM, RangerSafenetKeySecure > # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, > RangerTencentKMSProvider > Style 1 can get out master key string from provider, Style 2 can not. > In old, I add a flag KeyVaultEnabled to distinguish them. > KeyVaultEnabled=false means style1, true means style2 > RangerKeyStore with style1 use SecretKeyEntry with SealedObject to store a > key and do encryption / decryption by itself. > RangerKeyStore with style2 use SecretKeyByteEntry to store a key and let MK > provider to encryption / decryption. > These logics are hard-coded in the class RangerKeyStore. These are ugly and > hard to maintain. I refactor it by removing SecretKeyEntry, and let providers > of style1 do encryption / decryption. > Add a common base class of RangerMasterKey, RangerHSM andd > RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common > logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes > SealedObject into byte[]. > So the new code does not change the actual storage format, and there is no > problem in compatibility. > = > > And, there is no unified method to initialize a master key provider. > Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI > classes. > I made a new RangerKMSMKIFactory class to unify it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3998) Support Ranger KMS integration with AWS KMS
[ https://issues.apache.org/jira/browse/RANGER-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17652352#comment-17652352 ] kirby zhou commented on RANGER-3998: Any other idea? Review is required for commit. Thanks. > Support Ranger KMS integration with AWS KMS > --- > > Key: RANGER-3998 > URL: https://issues.apache.org/jira/browse/RANGER-3998 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0, 2.4.0 >Reporter: kirby zhou >Priority: Major > > AWS KMS is widely used by many customers. > Therefore, RangerKMS should support hosting MasterKey to AWS KMS. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4021) Shell syntax bug in kms setup.sh
kirby zhou created RANGER-4021:
--
Summary: Shell syntax bug in kms setup.sh
Key: RANGER-4021
URL: https://issues.apache.org/jira/browse/RANGER-4021
Project: Ranger
Issue Type: Bug
Components: kms
Affects Versions: 2.3.0, 3.0.0, 2.4.0
Reporter: kirby zhou
{code:java}
if [[ ${useringrouparr[1]} =~ "(${unix_group})" ]]
{code}
Don't quote right-hand side of =~, it'll match literally rather than as a regex.
{code:java}
echo "${prefix} $@" >> $LOGFILE
echo "${prefix} $@"
{code}
Argument mixes string and array. Use * or separate argument.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-4020) ranger-authn is required by security-admin but not compiled before it
kirby zhou created RANGER-4020: -- Summary: ranger-authn is required by security-admin but not compiled before it Key: RANGER-4020 URL: https://issues.apache.org/jira/browse/RANGER-4020 Project: Ranger Issue Type: Bug Components: Ranger Affects Versions: 3.0.0 Reporter: kirby zhou [WARNING] The POM for org.apache.ranger:ranger-authn:jar:3.0.0-SNAPSHOT is missing, no dependency information available The POM for org.apache.ranger:ranger-authn:jar:3.0.0-SNAPSHOT is missing, no dependency information available In root pom.xml, profile "all", ranger-authn module is not listed here. It should be compiled before security-admin. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3860) Huge unnecessary aws-java-sdk-bundle dependency
[ https://issues.apache.org/jira/browse/RANGER-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17648394#comment-17648394 ] kirby zhou commented on RANGER-3860: I have a patch here to reduce about 210Mib of dist tarball size. [https://reviews.apache.org/r/74258/] > Huge unnecessary aws-java-sdk-bundle dependency > --- > > Key: RANGER-3860 > URL: https://issues.apache.org/jira/browse/RANGER-3860 > Project: Ranger > Issue Type: Wish > Components: Ranger >Affects Versions: 3.0.0, 2.3.0 >Reporter: Aaron Braunstein >Priority: Major > > RANGER-3653 added an aws-java-sdk-bundle dependency to Ranger, which causes > the unpacked Ranger plugin jar size to increase by over 500 MB. Previously we > only had an aws-java-sdk-logs dependency. There was no justification provided > in the issue, but I suspect it was either due to a misunderstanding of how > dependency management works in Maven, or because they wanted to avoid shading > Ranger if there was some aws-java-sdk-logs 3rd party dependency that > conflicted in version with some their project was already using. > Aws-java-sdk-bundle relocates its dependencies. > Aws-java-sdk-bom in dependency management (with scope import) did not add a > dependency on all the aws java sdk's but only overrode the version of all aws > sdk dependencies if it found one transitively or without a version defined, > with the version supplied in the bom. > I recommend that RANGER-3653 be reverted. Additionally, aws-java-sdk-logs > version should be kept up-to-date to minimize version conflicts in libraries > that are keeping their aws dependencies up to date. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (RANGER-4019) Cannot new RangerRESTUtils under Apple Silicon macOS.
[ https://issues.apache.org/jira/browse/RANGER-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou resolved RANGER-4019. Resolution: Duplicate > Cannot new RangerRESTUtils under Apple Silicon macOS. > - > > Key: RANGER-4019 > URL: https://issues.apache.org/jira/browse/RANGER-4019 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 3.0.0, 2.3.0, 2.4.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with > M1/M2 chips. > Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (RANGER-4019) Cannot new RangerRESTUtils under Apple Silicon macOS.
[ https://issues.apache.org/jira/browse/RANGER-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou reassigned RANGER-4019: -- Assignee: kirby zhou > Cannot new RangerRESTUtils under Apple Silicon macOS. > - > > Key: RANGER-4019 > URL: https://issues.apache.org/jira/browse/RANGER-4019 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 3.0.0, 2.3.0, 2.4.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with > M1/M2 chips. > Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-4019) Cannot new RangerRESTUtils under Apple Silicon macOS.
kirby zhou created RANGER-4019: -- Summary: Cannot new RangerRESTUtils under Apple Silicon macOS. Key: RANGER-4019 URL: https://issues.apache.org/jira/browse/RANGER-4019 Project: Ranger Issue Type: Bug Components: Ranger Affects Versions: 2.3.0, 3.0.0, 2.4.0 Reporter: kirby zhou com.kstruct.gethostname4j.Hostname.getHostname() failed under macOS with M1/M2 chips. Need to bump JNA version to 5.7.0, gethostname4j version to 1.0.0. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-3998) Support Ranger KMS integration with AWS KMS
kirby zhou created RANGER-3998: -- Summary: Support Ranger KMS integration with AWS KMS Key: RANGER-3998 URL: https://issues.apache.org/jira/browse/RANGER-3998 Project: Ranger Issue Type: Improvement Components: kms Affects Versions: 3.0.0, 2.4.0 Reporter: kirby zhou AWS KMS is widely used by many customers. Therefore, RangerKMS should support hosting MasterKey to AWS KMS. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3612) KMS should either Die or Auto-Recover when its ranger-agent auth to KDC failed
[
https://issues.apache.org/jira/browse/RANGER-3612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641767#comment-17641767
]
kirby zhou commented on RANGER-3612:
Anybody can merge it?
> KMS should either Die or Auto-Recover when its ranger-agent auth to KDC failed
> --
>
> Key: RANGER-3612
> URL: https://issues.apache.org/jira/browse/RANGER-3612
> Project: Ranger
> Issue Type: Bug
> Components: kms, plugins
>Affects Versions: 3.0.0, 2.2.0
>Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Major
>
> If we install ranger agent to KMS, the agent would auth itself to KDC at
> startup. But if it failed, it just print a log in ranger-kms-.log,
> and the KMS can never recover to refresh its policies.
> {code:java}
> ]$ tail -f log/ranger-kms-ranger_kms-.log | fgrep ERROR
> 2022-02-09 19:00:18,227 ERROR MiscUtil - Failed to login with given keytab
> and principal{code}
> {code:java}
> package org.apache.ranger.authorization.kms.authorizer;
> public class RangerKmsAuthorizer implements Runnable, KeyACLs {
> RangerKmsAuthorizer(Configuration conf) {
>authWithKerberos(conf);
> }
> private void authWithKerberos(Configuration conf) {
> MiscUtil.authWithKerberos(keytab, principal, nameRules);
> }
> }
> package org.apache.ranger.audit.provider;
> public class MiscUtil {
> public static void authWithKerberos(...) {
> try {
> {
> UserGroupInformation ugi = UserGroupInformation
> .loginUserFromKeytabAndReturnUGI(spnegoPrincipals[0],
> keytab);
> MiscUtil.setUGILoginUser(ugi, null);
> }
> } catch (Throwable t) {
> logger.error("Failed to login with given keytab and principal", t);
> }
> }
> }{code}
>
> There seems only one chance for plugin to auth to KDC, so it can not auto
> recover.
> And MiscUtil.authWithKerberos never fail when auth failed, so KMS would not
> die when the plugin failed.
> This situation is too unfriendly to administrators. It should be fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (RANGER-3990) PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when ranger.supportedcomponents excluding solr.
[
https://issues.apache.org/jira/browse/RANGER-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou resolved RANGER-3990.
Resolution: Duplicate
> PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when
> ranger.supportedcomponents excluding solr.
> --
>
> Key: RANGER-3990
> URL: https://issues.apache.org/jira/browse/RANGER-3990
> Project: Ranger
> Issue Type: Improvement
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Major
>
> When I set install.properties like that
>
> {code:java}
> ranger.supportedcomponents=tag,hdfs,hive,yarn,kafka,kms,kudu {code}
> Then, run db_setup.py to upgrade a old instance, error occurs:
> {code:java}
> 2022-11-28 16:19:49,949 [JISQL]
> /sensorsdata/main/program/armada/jdk18292/jdk18292/bin/java -cp
> /sensorsdata/main/program/sp/commonjars/mysql-connector-java.jar:/sensorsdata/main/program/rogue/ranger-2.3.0-admin/jisql/lib/*
> org.apache.util.sql.Jisql -driver mysqlconj -cstring
> jdbc:mysql://10.129.17.81:4000/kirby_ranger2?useSSL=false -u 'kirby_ranger'
> -p '' -noheader -trim -c \; -query "delete from x_db_version_h where
> version = 'J10055' and active = 'N' and updated_by='ranger_admin';"
> 2022-11-28 16:19:50,298 [E] applying java patch
> PatchForSolrSvcDefAndPoliciesUpdate_J10055 failed
> {code}
>
> {code:java}
> 437 if (xXServiceDefObj == null) {
> 438 logger.info(xXServiceDefObj + ": service-def not
> found. No patching is needed");
> 439 System.out.println(0);
> 440 }
> 441
> 442 embeddedSolrResourceDefs =
> embeddedSolrServiceDef.getResources(); // ResourcesType
> 443 dbSolrServiceDef =
> this.svcDBStore.getServiceDefByName(SOLR_SVC_DEF_NAME);
> 444 dbSolrServiceDef.setResources(embeddedSolrResourceDefs);
> {code}
> It seems that, line 439 should be replaced by System.exit(0)
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3984) Support using TiDB as mysql-db in ranger
[
https://issues.apache.org/jira/browse/RANGER-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3984:
---
Attachment: (was: ranger_core_db_tidb.patch)
> Support using TiDB as mysql-db in ranger
>
>
> Key: RANGER-3984
> URL: https://issues.apache.org/jira/browse/RANGER-3984
> Project: Ranger
> Issue Type: Improvement
> Components: admin, kms
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Major
>
> TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to
> deploy ranger based on tidb. But TiDB is missing some features, which makes
> ranger unable to install properly.
> [https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
> The biggest problem affecting ranger is missing "Stored procedures and
> functions", "Select into".
> ranger use Stored procedures in setup scripts to simplify SQL.
> Some work is needed to remove the stored procedure.
>
>
> {code:java}
> ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 14 near "PROCEDURE if
> exists getXportalUIdByLoginId"
> ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 16 near "PROCEDURE
> `getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_portal_user.id into myid FROM x_portal_user WHERE
> x_portal_user.login_id = input_val;
> END"
> ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 14 near "PROCEDURE if
> exists getModulesIdByName"
> ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 16 near "PROCEDURE
> `getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_modules_master.id into myid FROM x_modules_master WHERE
> x_modules_master.module = input_val;
> END"
> ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 14 near "PROCEDURE if
> exists insertRangerPrerequisiteEntries"
> ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 16 near "PROCEDURE
> `insertRangerPrerequisiteEntries`()
> BEGIN
> DECLARE adminID bigint;
> DECLARE keyadminID bigint;
> DECLARE rangerusersyncID bigint;
> DECLARE rangertagsyncID bigint;
> DECLARE moduleIdReports bigint;
> DECLARE moduleIdResourceBasedPolicies bigint;
> DECLARE moduleIdAudit bigint;
> DECLARE moduleIdUG bigint;
> DECLARE moduleIdTagBasedPolicies bigint;
> DECLARE moduleIdKeyMana
> ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql':
> Unsupported type *ast.CallStmt
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-3990) PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when ranger.supportedcomponents excluding solr.
kirby zhou created RANGER-3990:
--
Summary: PatchForSolrSvcDefAndPoliciesUpdate_J10055 failing when
ranger.supportedcomponents excluding solr.
Key: RANGER-3990
URL: https://issues.apache.org/jira/browse/RANGER-3990
Project: Ranger
Issue Type: Improvement
Components: admin
Affects Versions: 2.3.0, 3.0.0
Reporter: kirby zhou
When I set install.properties like that
{code:java}
ranger.supportedcomponents=tag,hdfs,hive,yarn,kafka,kms,kudu {code}
Then, run db_setup.py to upgrade a old instance, error occurs:
{code:java}
2022-11-28 16:19:49,949 [JISQL]
/sensorsdata/main/program/armada/jdk18292/jdk18292/bin/java -cp
/sensorsdata/main/program/sp/commonjars/mysql-connector-java.jar:/sensorsdata/main/program/rogue/ranger-2.3.0-admin/jisql/lib/*
org.apache.util.sql.Jisql -driver mysqlconj -cstring
jdbc:mysql://10.129.17.81:4000/kirby_ranger2?useSSL=false -u 'kirby_ranger' -p
'' -noheader -trim -c \; -query "delete from x_db_version_h where
version = 'J10055' and active = 'N' and updated_by='ranger_admin';"
2022-11-28 16:19:50,298 [E] applying java patch
PatchForSolrSvcDefAndPoliciesUpdate_J10055 failed
{code}
{code:java}
437 if (xXServiceDefObj == null) {
438 logger.info(xXServiceDefObj + ": service-def not found.
No patching is needed");
439 System.out.println(0);
440 }
441
442 embeddedSolrResourceDefs =
embeddedSolrServiceDef.getResources(); // ResourcesType
443 dbSolrServiceDef =
this.svcDBStore.getServiceDefByName(SOLR_SVC_DEF_NAME);
444 dbSolrServiceDef.setResources(embeddedSolrResourceDefs);
{code}
It seems that, line 439 should be replaced by System.exit(0)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3984) Support using TiDB as mysql-db in ranger
[
https://issues.apache.org/jira/browse/RANGER-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3984:
---
Attachment: ranger_core_db_tidb.patch
> Support using TiDB as mysql-db in ranger
>
>
> Key: RANGER-3984
> URL: https://issues.apache.org/jira/browse/RANGER-3984
> Project: Ranger
> Issue Type: Improvement
> Components: admin, kms
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Major
> Attachments: ranger_core_db_tidb.patch
>
>
> TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to
> deploy ranger based on tidb. But TiDB is missing some features, which makes
> ranger unable to install properly.
> [https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
> The biggest problem affecting ranger is missing "Stored procedures and
> functions", "Select into".
> ranger use Stored procedures in setup scripts to simplify SQL.
> Some work is needed to remove the stored procedure.
>
>
> {code:java}
> ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 14 near "PROCEDURE if
> exists getXportalUIdByLoginId"
> ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 16 near "PROCEDURE
> `getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_portal_user.id into myid FROM x_portal_user WHERE
> x_portal_user.login_id = input_val;
> END"
> ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 14 near "PROCEDURE if
> exists getModulesIdByName"
> ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 16 near "PROCEDURE
> `getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
> BEGIN
> SET myid = 0;
> SELECT x_modules_master.id into myid FROM x_modules_master WHERE
> x_modules_master.module = input_val;
> END"
> ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 14 near "PROCEDURE if
> exists insertRangerPrerequisiteEntries"
> ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have
> an error in your SQL syntax; check the manual that corresponds to your TiDB
> version for the right syntax to use line 1 column 16 near "PROCEDURE
> `insertRangerPrerequisiteEntries`()
> BEGIN
> DECLARE adminID bigint;
> DECLARE keyadminID bigint;
> DECLARE rangerusersyncID bigint;
> DECLARE rangertagsyncID bigint;
> DECLARE moduleIdReports bigint;
> DECLARE moduleIdResourceBasedPolicies bigint;
> DECLARE moduleIdAudit bigint;
> DECLARE moduleIdUG bigint;
> DECLARE moduleIdTagBasedPolicies bigint;
> DECLARE moduleIdKeyMana
> ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql':
> Unsupported type *ast.CallStmt
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3984) Support using TiDB as mysql-db in ranger
[
https://issues.apache.org/jira/browse/RANGER-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3984:
---
Description:
TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to
deploy ranger based on tidb. But TiDB is missing some features, which makes
ranger unable to install properly.
[https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
The biggest problem affecting ranger is missing "Stored procedures and
functions", "Select into".
ranger use Stored procedures in setup scripts to simplify SQL.
Some work is needed to remove the stored procedure.
{code:java}
ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
getXportalUIdByLoginId"
ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_portal_user.id into myid FROM x_portal_user WHERE
x_portal_user.login_id = input_val;
END"
ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
getModulesIdByName"
ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_modules_master.id into myid FROM x_modules_master WHERE
x_modules_master.module = input_val;
END"
ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
insertRangerPrerequisiteEntries"
ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`insertRangerPrerequisiteEntries`()
BEGIN
DECLARE adminID bigint;
DECLARE keyadminID bigint;
DECLARE rangerusersyncID bigint;
DECLARE rangertagsyncID bigint;
DECLARE moduleIdReports bigint;
DECLARE moduleIdResourceBasedPolicies bigint;
DECLARE moduleIdAudit bigint;
DECLARE moduleIdUG bigint;
DECLARE moduleIdTagBasedPolicies bigint;
DECLARE moduleIdKeyMana
ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql':
Unsupported type *ast.CallStmt
{code}
was:
TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to
deploy ranger based on tidb. But TiDB is missing some features, which makes
ranger unable to install properly.
[https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
The biggest problem affecting ranger is missing "Stored procedures and
functions".
ranger use Stored procedures in setup scripts to simplify SQL.
Some work is needed to remove the stored procedure.
{code:java}
ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
getXportalUIdByLoginId"
ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_portal_user.id into myid FROM x_portal_user WHERE
x_portal_user.login_id = input_val;
END"
ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
getModulesIdByName"
ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_modules_master.id into myid FROM x_modules_master WHERE
x_modules_master.module = input_val;
END"
ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL
[jira] [Created] (RANGER-3984) Support using TiDB as mysql-db in ranger
kirby zhou created RANGER-3984:
--
Summary: Support using TiDB as mysql-db in ranger
Key: RANGER-3984
URL: https://issues.apache.org/jira/browse/RANGER-3984
Project: Ranger
Issue Type: Improvement
Components: admin, kms
Affects Versions: 2.3.0, 3.0.0
Reporter: kirby zhou
TiDB is a 95% mysql-compatible NewSQL database. For legal reason, we have to
deploy ranger based on tidb. But TiDB is missing some features, which makes
ranger unable to install properly.
[https://docs.pingcap.com/tidb/stable/mysql-compatibility#unsupported-features]
The biggest problem affecting ranger is missing "Stored procedures and
functions".
ranger use Stored procedures in setup scripts to simplify SQL.
Some work is needed to remove the stored procedure.
{code:java}
ERROR 1064 (42000) at line 1595 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
getXportalUIdByLoginId"
ERROR 1064 (42000) at line 1596 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`getXportalUIdByLoginId`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_portal_user.id into myid FROM x_portal_user WHERE
x_portal_user.login_id = input_val;
END"
ERROR 1064 (42000) at line 1605 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
getModulesIdByName"
ERROR 1064 (42000) at line 1606 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`getModulesIdByName`(IN input_val VARCHAR(100), OUT myid BIGINT)
BEGIN
SET myid = 0;
SELECT x_modules_master.id into myid FROM x_modules_master WHERE
x_modules_master.module = input_val;
END"
ERROR 1064 (42000) at line 1679 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 14 near "PROCEDURE if exists
insertRangerPrerequisiteEntries"
ERROR 1064 (42000) at line 1680 in file: 'ranger_core_db_mysql.sql': You have
an error in your SQL syntax; check the manual that corresponds to your TiDB
version for the right syntax to use line 1 column 16 near "PROCEDURE
`insertRangerPrerequisiteEntries`()
BEGIN
DECLARE adminID bigint;
DECLARE keyadminID bigint;
DECLARE rangerusersyncID bigint;
DECLARE rangertagsyncID bigint;
DECLARE moduleIdReports bigint;
DECLARE moduleIdResourceBasedPolicies bigint;
DECLARE moduleIdAudit bigint;
DECLARE moduleIdUG bigint;
DECLARE moduleIdTagBasedPolicies bigint;
DECLARE moduleIdKeyMana
ERROR 8108 (HY000) at line 1757 in file: 'ranger_core_db_mysql.sql':
Unsupported type *ast.CallStmt
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-2128) Implement SparkSQL plugin
[ https://issues.apache.org/jira/browse/RANGER-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624829#comment-17624829 ] kirby zhou commented on RANGER-2128: What's the difference between SparkSQL's ACL model and HadoopSQL? > Implement SparkSQL plugin > - > > Key: RANGER-2128 > URL: https://issues.apache.org/jira/browse/RANGER-2128 > Project: Ranger > Issue Type: New Feature > Components: plugins, Ranger >Affects Versions: 1.1.0 >Reporter: t oo >Assignee: Kent Yao >Priority: Major > Attachments: image-2022-10-10-14-25-30-218.png, support_ranger11.tgz > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Implement SparkSQL plugin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.
[
https://issues.apache.org/jira/browse/RANGER-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17610461#comment-17610461
]
kirby zhou commented on RANGER-3775:
It works
> Logback.xml has been incorrectly modified by RANGER-3704.
> -
>
> Key: RANGER-3775
> URL: https://issues.apache.org/jira/browse/RANGER-3775
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0
>Reporter: kirby zhou
>Assignee: Ramachandran
>Priority: Critical
> Attachments:
> 0001-RANGER-3775-Logback.xml-has-been-incorrectly-modifie.patch
>
>
> {code:java}
> git show 361f179249 | filterdiff -i '*/logback.xml'
> diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml
> b/security-admin/src/main/webapp/WEB-INF/logback.xml
> index 997f3bc59..53cdc49cf 100644
> --- a/security-admin/src/main/webapp/WEB-INF/logback.xml
> +++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
> @@ -80,7 +80,7 @@
>
>
>
> -
> +
>
>
>
> {code}
> These changes seems not related to the issue RANGER-3704.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (RANGER-3921) User with DROP ACL on "db=dummy; table=*; column=*" can do drop table and database.
[
https://issues.apache.org/jira/browse/RANGER-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3921:
---
Description:
In agents-common/src/test/resources/policyengine/test_policyengine_hive.json,
we have hive policy:
{code:java}
{"id":8,"name":"db=dummy; table=*;
column=*","isEnabled":true,"isAuditEnabled":true,
"resources":{"database":{"values":["dummy"]},"table":{"values":["*"]},"column":{"values":["*"]}},
"policyItems":[
{"accesses":[{"type":"create","isAllowed":true},{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user1","user2"],"groups":[],"delegateAdmin":false}
],
"allowExceptions":[
{"accesses":[{"type":"create","isAllowed":true},
{"type":"update","isAllowed":true}],"users":["user1"],"groups":[],"delegateAdmin":false},
{"accesses":[{"type":"create","isAllowed":true},
{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user2"],"groups":[],"delegateAdmin":false}
]
} {code}
According to the general understanding, this is given the permission of column
level, rather than the permission of table level or database level.
But these 2 new test case can pass:
{code:java}
{"name":"ALLOW 'drop dummy/*;' for user1",
"request":{
"resource":{"elements":{"database":"dummy", "table": "dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy/dummy for user1"
},
"result":{"isAudited":true,"isAllowed":true,"policyId":8}
}
,
{"name":"ALLOW 'drop dummy;' for user1",
"request":{
"resource":{"elements":{"database":"dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy for user1"
},
"result":{"isAudited":true,"isAllowed":true,"policyId":8}
} ,
{"name":"ALLOW 'drop dummy/udf=dummy;' for user1",
"request":{
"resource":{"elements":{"database":"dummy", "udf":"dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy for user1"
},
"result":{"isAudited":false,"isAllowed":true,"policyId":8}
} {code}
This doesn't seem reasonable. A user who can not drop UDF, but can drop whole
database.
Or can someone tell me how to only give users column-level permissions without
involving table or database?
was:
In agents-common/src/test/resources/policyengine/test_policyengine_hive.json,
we have hive policy:
{code:java}
{"id":8,"name":"db=dummy; table=*;
column=*","isEnabled":true,"isAuditEnabled":true,
"resources":{"database":{"values":["dummy"]},"table":{"values":["*"]},"column":{"values":["*"]}},
"policyItems":[
{"accesses":[{"type":"create","isAllowed":true},{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user1","user2"],"groups":[],"delegateAdmin":false}
],
"allowExceptions":[
{"accesses":[{"type":"create","isAllowed":true},
{"type":"update","isAllowed":true}],"users":["user1"],"groups":[],"delegateAdmin":false},
{"accesses":[{"type":"create","isAllowed":true},
{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user2"],"groups":[],"delegateAdmin":false}
]
} {code}
According to the general understanding, this is given the permission of column
level, rather than the permission of table level or database level.
But these 2 new test case can pass:
{code:java}
{"name":"ALLOW 'drop dummy/*;' for user1",
"request":{
"resource":{"elements":{"database":"dummy", "table": "dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy/dummy for user1"
},
"result":{"isAudited":true,"isAllowed":true,"policyId":8}
}
,
{"name":"ALLOW 'drop dummy;' for user1",
"request":{
"resource":{"elements":{"database":"dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy for user1"
},
"result":{"isAudited":true,"isAllowed":true,"policyId":8}
}
{code}
This doesn't seem reasonable.
Or can someone tell me how to only give users column-level permissions without
involving table or database?
> User with DROP ACL on "db=dummy; table=*; column=*" can do drop table and
> database.
> ---
>
> Key: RANGER-3921
> URL: https://issues.apache.org/jira/browse/RANGER-3921
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 3.0.0, 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> In agents-common/src/test/resources/policyengine/test_policyengine_hive.json,
> we have hive policy:
> {code:java}
> {"id":8,"name":"db=dummy; table=*;
> column=*","isEnabled":true,"isAuditEnabled":true,
> "resources":{"database":{"values":["dummy"]},"table":{"values":["*"]},"column":{"values":["*"]}},
> "policyItems":[
>
[jira] [Updated] (RANGER-3921) User with DROP ACL on "db=dummy; table=*; column=*" can do drop table and database.
[
https://issues.apache.org/jira/browse/RANGER-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3921:
---
Summary: User with DROP ACL on "db=dummy; table=*; column=*" can do drop
table and database. (was: User with DROP ACL on "db=dummy; table=*; column=*"
can do drop table.)
> User with DROP ACL on "db=dummy; table=*; column=*" can do drop table and
> database.
> ---
>
> Key: RANGER-3921
> URL: https://issues.apache.org/jira/browse/RANGER-3921
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 3.0.0, 2.3.0, 2.4.0
>Reporter: kirby zhou
>Priority: Major
>
> In agents-common/src/test/resources/policyengine/test_policyengine_hive.json,
> we have hive policy:
> {code:java}
> {"id":8,"name":"db=dummy; table=*;
> column=*","isEnabled":true,"isAuditEnabled":true,
> "resources":{"database":{"values":["dummy"]},"table":{"values":["*"]},"column":{"values":["*"]}},
> "policyItems":[
> {"accesses":[{"type":"create","isAllowed":true},{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user1","user2"],"groups":[],"delegateAdmin":false}
> ],
> "allowExceptions":[
> {"accesses":[{"type":"create","isAllowed":true},
> {"type":"update","isAllowed":true}],"users":["user1"],"groups":[],"delegateAdmin":false},
> {"accesses":[{"type":"create","isAllowed":true},
> {"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user2"],"groups":[],"delegateAdmin":false}
> ]
> } {code}
> According to the general understanding, this is given the permission of
> column level, rather than the permission of table level or database level.
>
> But these 2 new test case can pass:
> {code:java}
> {"name":"ALLOW 'drop dummy/*;' for user1",
> "request":{
> "resource":{"elements":{"database":"dummy", "table": "dummy"}},
>
> "accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
> dummy/dummy for user1"
> },
> "result":{"isAudited":true,"isAllowed":true,"policyId":8}
> }
> ,
> {"name":"ALLOW 'drop dummy;' for user1",
> "request":{
> "resource":{"elements":{"database":"dummy"}},
>
> "accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
> dummy for user1"
> },
> "result":{"isAudited":true,"isAllowed":true,"policyId":8}
> }
> {code}
>
> This doesn't seem reasonable.
> Or can someone tell me how to only give users column-level permissions
> without involving table or database?
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-3921) User with DROP ACL on "db=dummy; table=*; column=*" can do drop table.
kirby zhou created RANGER-3921:
--
Summary: User with DROP ACL on "db=dummy; table=*; column=*" can
do drop table.
Key: RANGER-3921
URL: https://issues.apache.org/jira/browse/RANGER-3921
Project: Ranger
Issue Type: Bug
Components: plugins
Affects Versions: 2.3.0, 3.0.0, 2.4.0
Reporter: kirby zhou
In agents-common/src/test/resources/policyengine/test_policyengine_hive.json,
we have hive policy:
{code:java}
{"id":8,"name":"db=dummy; table=*;
column=*","isEnabled":true,"isAuditEnabled":true,
"resources":{"database":{"values":["dummy"]},"table":{"values":["*"]},"column":{"values":["*"]}},
"policyItems":[
{"accesses":[{"type":"create","isAllowed":true},{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user1","user2"],"groups":[],"delegateAdmin":false}
],
"allowExceptions":[
{"accesses":[{"type":"create","isAllowed":true},
{"type":"update","isAllowed":true}],"users":["user1"],"groups":[],"delegateAdmin":false},
{"accesses":[{"type":"create","isAllowed":true},
{"type":"update","isAllowed":true},{"type":"drop","isAllowed":true}],"users":["user2"],"groups":[],"delegateAdmin":false}
]
} {code}
According to the general understanding, this is given the permission of column
level, rather than the permission of table level or database level.
But these 2 new test case can pass:
{code:java}
{"name":"ALLOW 'drop dummy/*;' for user1",
"request":{
"resource":{"elements":{"database":"dummy", "table": "dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy/dummy for user1"
},
"result":{"isAudited":true,"isAllowed":true,"policyId":8}
}
,
{"name":"ALLOW 'drop dummy;' for user1",
"request":{
"resource":{"elements":{"database":"dummy"}},
"accessType":"drop","user":"user1","userGroups":["users"],"requestData":"drop
dummy for user1"
},
"result":{"isAudited":true,"isAllowed":true,"policyId":8}
}
{code}
This doesn't seem reasonable.
Or can someone tell me how to only give users column-level permissions without
involving table or database?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3919) Adding automatically terminate a session after a predefined timeout period (60 minutes) of inactivity.
[ https://issues.apache.org/jira/browse/RANGER-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606956#comment-17606956 ] kirby zhou commented on RANGER-3919: Session timeout (default 60m)is controlled by web.xml which is not in conf directory. So our default conf ranger.admin.kerberos.token.valid.seconds = 30s (second) is meaningless. Mentioned in https://issues.apache.org/jira/browse/RANGER-3635 And there is a mechanism to keep the session renewed, even if kerberos ticket has expired. > Adding automatically terminate a session after a predefined timeout period > (60 minutes) of inactivity. > > > Key: RANGER-3919 > URL: https://issues.apache.org/jira/browse/RANGER-3919 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 2.3.0 >Reporter: Sanjay Kumar Sahu >Priority: Major > > Web applications do not automatically terminate a session > after a predefined timeout period (60 minutes) of inactivity. > Adding automatically terminate a session > after a predefined timeout period (60 minutes) of inactivity. > This issue increases the window of opportunity for an attacker to gain > unauthorized access to a user’s session. However, in order to exploit this > issue, an attacker still needs to obtain a > valid session ID tokens. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3696) java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
[
https://issues.apache.org/jira/browse/RANGER-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606944#comment-17606944
]
kirby zhou commented on RANGER-3696:
Any idea?
> java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
> ---
>
> Key: RANGER-3696
> URL: https://issues.apache.org/jira/browse/RANGER-3696
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 2.2.0
> Environment: Apache Ranger ElasticSearch Plugin:
> ranger-2.2.0-elasticsearch-plugin.tar.gz
> elasticsearch version: 7.6.0
> OS: Ubuntu 20.04.4
>Reporter: MohdSiddique Bagwan
>Priority: Blocker
>
> Please find the versions I am using
> *Apache Ranger ElasticSearch Plugin:* ranger-2.2.0-elasticsearch-plugin.tar.gz
> *elasticsearch version:* 7.6.0
> *OS:* Ubuntu 20.04.4
> I installed the apache ranger elasticsearch plugin on elastic search host,
> while starting elasticsearch service I am getting below error:
> Note: Without ranger plugin the elasticsearch plugin is working perfect. It
> would be very helpful if you redirect me to documentation on how to install
> ranger-2.2.0-elasticsearch-plugin.tar.gz on 7.6.0 & above.
> {code:java}
> service elasticsearch start
> * Starting Elasticsearch Server
>
> sysctl: setting key
> "vm.max_map_count", ignoring: Read-only file system
> OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in
> version 9.0 and will likely be removed in a future release.
>
>
> [ OK ]
> root@3b8fcbe634f3:~# fatal error in thread [main], exiting
> java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
> at
> org.apache.ranger.authorization.elasticsearch.plugin.RangerElasticsearchPlugin.(RangerElasticsearchPlugin.java:52)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
> at
> org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607)
> at
> org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556)
> at
> org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471)
> at
> org.elasticsearch.plugins.PluginsService.(PluginsService.java:163)
> at org.elasticsearch.node.Node.(Node.java:313)
> at org.elasticsearch.node.Node.(Node.java:257)
> at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221)
> at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
> at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
> at
> org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
> at
> org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
> at
> org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
> at
> org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
> at org.elasticsearch.cli.Command.main(Command.java:90)
> at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
> at
> org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
> Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory
> at
> java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:436)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
> at
> java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:864)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> ... 22 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.
[
https://issues.apache.org/jira/browse/RANGER-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606935#comment-17606935
]
kirby zhou edited comment on RANGER-3775 at 9/20/22 6:39 AM:
-
If com.mchange is necessary, code should be:
{code:java}
{code}
Because sql_appender should be used by log4jdbc only.
And the default logger level is warn, if we do not have {logger
name="com.mchange"}
{code:java}
{code}
was (Author: kirbyzhou):
If com.mchange is necessary, code should be:
{code:java}
{code}
Because sql_appender should be used by log4jdbc only.
> Logback.xml has been incorrectly modified by RANGER-3704.
> -
>
> Key: RANGER-3775
> URL: https://issues.apache.org/jira/browse/RANGER-3775
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0
>Reporter: kirby zhou
>Priority: Critical
>
> {code:java}
> git show 361f179249 | filterdiff -i '*/logback.xml'
> diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml
> b/security-admin/src/main/webapp/WEB-INF/logback.xml
> index 997f3bc59..53cdc49cf 100644
> --- a/security-admin/src/main/webapp/WEB-INF/logback.xml
> +++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
> @@ -80,7 +80,7 @@
>
>
>
> -
> +
>
>
>
> {code}
> These changes seems not related to the issue RANGER-3704.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.
[
https://issues.apache.org/jira/browse/RANGER-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606935#comment-17606935
]
kirby zhou commented on RANGER-3775:
If com.mchange is necessary, code should be:
{code:java}
{code}
Because sql_appender should be used by log4jdbc only.
> Logback.xml has been incorrectly modified by RANGER-3704.
> -
>
> Key: RANGER-3775
> URL: https://issues.apache.org/jira/browse/RANGER-3775
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0
>Reporter: kirby zhou
>Priority: Critical
>
> {code:java}
> git show 361f179249 | filterdiff -i '*/logback.xml'
> diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml
> b/security-admin/src/main/webapp/WEB-INF/logback.xml
> index 997f3bc59..53cdc49cf 100644
> --- a/security-admin/src/main/webapp/WEB-INF/logback.xml
> +++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
> @@ -80,7 +80,7 @@
>
>
>
> -
> +
>
>
>
> {code}
> These changes seems not related to the issue RANGER-3704.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.
[
https://issues.apache.org/jira/browse/RANGER-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606867#comment-17606867
]
kirby zhou commented on RANGER-3775:
{logger name="jdbc.connection"} is used by log4jdbc, not by c3p0.
So this change should be rolled back.
And the logback.xml is located at
"security-admin/src/main/resources/conf.dist/" now.
> Logback.xml has been incorrectly modified by RANGER-3704.
> -
>
> Key: RANGER-3775
> URL: https://issues.apache.org/jira/browse/RANGER-3775
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0
>Reporter: kirby zhou
>Priority: Critical
>
> {code:java}
> git show 361f179249 | filterdiff -i '*/logback.xml'
> diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml
> b/security-admin/src/main/webapp/WEB-INF/logback.xml
> index 997f3bc59..53cdc49cf 100644
> --- a/security-admin/src/main/webapp/WEB-INF/logback.xml
> +++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
> @@ -80,7 +80,7 @@
>
>
>
> -
> +
>
>
>
> {code}
> These changes seems not related to the issue RANGER-3704.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (RANGER-3915) When kerberos enabled, API can be accessed via HTTP basic auth.
kirby zhou created RANGER-3915: -- Summary: When kerberos enabled, API can be accessed via HTTP basic auth. Key: RANGER-3915 URL: https://issues.apache.org/jira/browse/RANGER-3915 Project: Ranger Issue Type: Bug Components: admin Affects Versions: 2.3.0, 3.0.0, 2.4.0 Reporter: kirby zhou For example: ] curl -X GET 'http://ranger:6080/service/public/v2/api/policy/1' -u admin:password Is it by design? Expect: if kerberos enabled, API can not be accessed by HTTP basic authentication. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572378#comment-17572378 ] kirby zhou edited comment on RANGER-3831 at 8/12/22 2:27 AM: - Add a draft of Service definition Updated version [^ranger-servicedef-pegasus.json] [^ranger-servicedef-pegasus.json] was (Author: kirbyzhou): Add a draft of Service definition [^ranger-servicedef-pegasus.json] > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > Attachments: ranger-servicedef-pegasus.json > > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572378#comment-17572378 ] kirby zhou edited comment on RANGER-3831 at 8/12/22 2:27 AM: - Add a draft of Service definition Updated version [^ranger-servicedef-pegasus.json] [^ranger-servicedef-pegasus.json] was (Author: kirbyzhou): Add a draft of Service definition Updated version [^ranger-servicedef-pegasus.json] [^ranger-servicedef-pegasus.json] > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > Attachments: ranger-servicedef-pegasus.json > > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou updated RANGER-3831: --- Attachment: ranger-servicedef-pegasus.json > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > Attachments: ranger-servicedef-pegasus.json > > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou updated RANGER-3831: --- Attachment: (was: ranger-servicedef-pegasus.json) > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572378#comment-17572378 ] kirby zhou commented on RANGER-3831: Add a draft of Service definition [^ranger-servicedef-pegasus.json] > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > Attachments: ranger-servicedef-pegasus.json > > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou updated RANGER-3831: --- Attachment: ranger-servicedef-pegasus.json > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > Attachments: ranger-servicedef-pegasus.json > > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-3843) Ranger Admin can not display more than 200 services
kirby zhou created RANGER-3843: -- Summary: Ranger Admin can not display more than 200 services Key: RANGER-3843 URL: https://issues.apache.org/jira/browse/RANGER-3843 Project: Ranger Issue Type: Bug Components: admin Affects Versions: 2.3.0, 3.0.0 Reporter: kirby zhou If more than 200 service defined in ranger, Web UI can only display first 200 of them. The XHR request: URL: [http://rangerhost/service/plugins/services?page=0=200_pages=0=0&_=1658998619811] It only have fixed pageSize=200 in .js files, and never get the next pages. Modifying ranger.db.maxrows.default doesn't work。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-3835) setup.sh of KMS wont set kerberos settings of http endpoint.
kirby zhou created RANGER-3835: -- Summary: setup.sh of KMS wont set kerberos settings of http endpoint. Key: RANGER-3835 URL: https://issues.apache.org/jira/browse/RANGER-3835 Project: Ranger Issue Type: Bug Components: kms Affects Versions: 3.0.0, 2.3.0 Reporter: kirby zhou Kerberos is very important to protect KMS. kms-site.xml has 2 properties: hadoop.kms.authentication.kerberos.principal The Kerberos principal to use for the HTTP endpoint. hadoop.kms.authentication.kerberos.keytab Path to the keytab with credentials for the configured Kerberos principal. But setup.sh and install.properties DO NOT do anything with them. It just set some principal and keytab which used to connect to ranger. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3831) Add support of pegasus to ranger
[ https://issues.apache.org/jira/browse/RANGER-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569897#comment-17569897 ] kirby zhou commented on RANGER-3831: We have some basic ideas. 1. it seems at first we need to enhance the ACL model of pegasus itself. See the issue of pegasus itself, we need to add per-table/per-user ACL ability to pegasus at first. 2. The pegasus main process seems a pure C++ program, how to download and eval policy from ranger is a problem. We are considering whether to write ranger client in pure C++ or embed java. > Add support of pegasus to ranger > > > Key: RANGER-3831 > URL: https://issues.apache.org/jira/browse/RANGER-3831 > Project: Ranger > Issue Type: Improvement > Components: admin, plugins >Affects Versions: 3.0.0 >Reporter: kirby zhou >Priority: Major > > Apache Pegasus is A horizontally scalable, strongly consistent and > high-performance key-value store. > It now have ACLs and SASL, but do not related to ranger. > I suggest to add support to it. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (RANGER-3831) Add support of pegasus to ranger
kirby zhou created RANGER-3831: -- Summary: Add support of pegasus to ranger Key: RANGER-3831 URL: https://issues.apache.org/jira/browse/RANGER-3831 Project: Ranger Issue Type: Improvement Components: admin, plugins Affects Versions: 3.0.0 Reporter: kirby zhou Apache Pegasus is A horizontally scalable, strongly consistent and high-performance key-value store. It now have ACLs and SASL, but do not related to ranger. I suggest to add support to it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (RANGER-3623) Add ability to enable anonymous download of policy/role/tag
[ https://issues.apache.org/jira/browse/RANGER-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17562910#comment-17562910 ] kirby zhou commented on RANGER-3623: Any other idea? Or someone can commit it? > Add ability to enable anonymous download of policy/role/tag > --- > > Key: RANGER-3623 > URL: https://issues.apache.org/jira/browse/RANGER-3623 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: 3.0.0, 2.3.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > Attachments: add-downloadonly-option.patch > > > Currently, we have an option ranger.admin.allow.unauthenticated.access to > allow unauthenticated clients to perform a series of API operations. This > option allows the client to perform both dangerous grant/revoke permission > operation and relatively safe download operation. > In many cases, allowing anonymous downloading of policy is not a serious risk > problem. On the contrary, the complicated kerberos and SSL settings make it > difficult for ranger plugin embedded in third-party services to complete the > task of refreshing policy, which may be a bigger problem. In particular, > refresh failure often has no obvious features for administrators to discover. > Therefore, I suggest that ranger increase the ability to allow client to > download policy/tag/roles anonymously. > There are two ways to achieve it. > > 1. Just limit the ability of "ranger.admin.allow.unauthenticated.access=true" > which needs to modify > "security-admin/src/main/resources/conf.dist/security-applicationContext.xml" > to remove dangerous operations from ' > security="none"'. > > 2. Add a candidate value "downloadonly" to > "ranger.admin.allow.unauthenticated.access" > Which needs modify ServiceRest.Java and BizUtil.java to implement the > enhanced checking logic. > > I have a patch for method2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (RANGER-3682) Unify the ways that rangerkeystore to encapsulate zonekey
[ https://issues.apache.org/jira/browse/RANGER-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou updated RANGER-3682: --- Affects Version/s: (was: 2.3.0) > Unify the ways that rangerkeystore to encapsulate zonekey > - > > Key: RANGER-3682 > URL: https://issues.apache.org/jira/browse/RANGER-3682 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > Unify the ways that rangerkeystore to encapsulate zonekey > Now we have 2 styles of MasterKeyProvider: > # RangerMasterKey, RangerHSM, RangerSafenetKeySecure > # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, > RangerTencentKMSProvider > Style 1 can get out master key string from provider, Style 2 can not. > In old, I add a flag KeyVaultEnabled to distinguish them. > KeyVaultEnabled=false means style1, true means style2 > RangerKeyStore with style1 use SecretKeyEntry with SealedObject to store a > key and do encryption / decryption by itself. > RangerKeyStore with style2 use SecretKeyByteEntry to store a key and let MK > provider to encryption / decryption. > These logics are hard-coded in the class RangerKeyStore. These are ugly and > hard to maintain. I refactor it by removing SecretKeyEntry, and let providers > of style1 do encryption / decryption. > Add a common base class of RangerMasterKey, RangerHSM andd > RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common > logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes > SealedObject into byte[]. > So the new code does not change the actual storage format, and there is no > problem in compatibility. > = > > And, there is no unified method to initialize a master key provider. > Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI > classes. > I made a new RangerKMSMKIFactory class to unify it. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551884#comment-17551884
]
kirby zhou commented on RANGER-3778:
I have not get your point.
I have deleted "import javax.servlet.http.cookie;" in patch rev 2.
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> image-2022-06-01-21-10-04-463.png, image-2022-06-01-21-11-21-408.png,
> image-2022-06-01-21-12-30-661.png, kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3773) maven can not build ranger-2.3.0 because commons-cli is duplicated in pom
[
https://issues.apache.org/jira/browse/RANGER-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551422#comment-17551422
]
kirby zhou commented on RANGER-3773:
The problem still exist, it may depends on maven version.
> maven can not build ranger-2.3.0 because commons-cli is duplicated in pom
> -
>
> Key: RANGER-3773
> URL: https://issues.apache.org/jira/browse/RANGER-3773
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Assignee: kirby zhou
>Priority: Blocker
>
> maven can not build ranger-2.3.0 with following errors:
> {code:java}
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.ranger:ranger-plugins-installer:jar:2.3.0
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: commons-cli:commons-cli:jar -> duplicate declaration of version
> ${commons.cli.version} @ line 41, column 21
> {code}
> {code:java}
> ]$ git blame agents-installer/pom.xml
> 3c59734236 (Dilli Dorai Arumugam 2014-09-17 13:05:16 -0700 1) version="1.0" encoding="UTF-8"?>
> 99c462d2c3 (Dilli Dorai Arumugam 2014-10-13 17:46:47 -0700 2)
> ^7defc061d (Owen O'Malley 2014-08-14 13:48:58 -0700 18) xmlns="http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 19)
> 4.0.0
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 20)
> ranger-plugins-installer
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 21)
> Installer Support Component
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 22)
> Security Plugins Installer
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 23)
> jar
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 24)
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 25)
> org.apache.ranger
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 26)
> ranger
> 38f7cc98dd (Ramesh Mani 2022-05-19 09:39:38 -0700 27)
> 2.3.0
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 28)
> ..
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 29)
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 30)
>
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 31)
>
> df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 32)
> commons-cli
> df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 33)
> commons-cli
> df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 34)
> ${commons.cli.version}
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 35)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 36)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 37)
> org.apache.commons
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 38)
> commons-compress
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 39)
> ${commons.compress.version}
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 40)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 41)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 42)
> commons-cli
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 43)
> commons-cli
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 44)
> ${commons.cli.version}
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 45)
>
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 46)
>
> ^7defc061d (Owen O'Malley 2014-08-14 13:48:58 -0700 47)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544906#comment-17544906
]
kirby zhou commented on RANGER-3778:
{code:java}
hadoop.security.group.mapping
org.apache.hadoop.security.ShellBasedUnixGroupsMapping
hadoop.proxyuser.HTTP.hosts
*
hadoop.proxyuser.HTTP.groups
*
{code}
I use the above fragment in my core-site.xml, but bug still happens.
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> image-2022-06-01-21-10-04-463.png, image-2022-06-01-21-11-21-408.png,
> image-2022-06-01-21-12-30-661.png, kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544901#comment-17544901
]
kirby zhou commented on RANGER-3778:
1. RULE seems not the key problem
2. hadoop.security.group.mapping I will try later.
3. I donot use impersonation, and if we do not append 'doAs=xxx' in URL, it
seems no effect.
4. Yes, keyadmin is logined at client-side via kinit.
5. What exactly debug log you want ?
I just use idea to debug it.
6. The bug can reproduce on ranger master branch
!image-2022-06-01-21-10-04-463.png!
go one step
!image-2022-06-01-21-11-21-408.png!
One more, exception happens
!image-2022-06-01-21-12-30-661.png!
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> image-2022-06-01-21-10-04-463.png, image-2022-06-01-21-11-21-408.png,
> image-2022-06-01-21-12-30-661.png, kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3778:
---
Attachment: image-2022-06-01-21-12-30-661.png
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> image-2022-06-01-21-10-04-463.png, image-2022-06-01-21-11-21-408.png,
> image-2022-06-01-21-12-30-661.png, kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3778:
---
Attachment: image-2022-06-01-21-11-21-408.png
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> image-2022-06-01-21-10-04-463.png, image-2022-06-01-21-11-21-408.png,
> kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3778:
---
Attachment: image-2022-06-01-21-10-04-463.png
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> image-2022-06-01-21-10-04-463.png, kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544160#comment-17544160
]
kirby zhou commented on RANGER-3778:
Maybe there are some mysterious configuration?
I uploaded my configuration without keytab and creds for you
[^kirbyconf.tar.gz]
My git rev is ff744a287 (tag: release-2.3.0-rc1, origin/ranger-2.3,
github/ranger-2.3);
{code:java}
$ curl -v --negotiate -u:
http://kirbytest01.sa:6080/service/plugins/secure/policies/download/hdfsdev
* Trying 10.10.137.131...
* TCP_NODELAY set
* Connected to kirbytest01.sa (10.10.137.131) port 6080 (#0)
> GET /service/plugins/secure/policies/download/hdfsdev HTTP/1.1
> Host: kirbytest01.sa:6080
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Set-Cookie: RANGERADMINSESSIONID=CD26526128FE4CF1E83468FF8039CCF8; Path=/;
HttpOnly
< WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT;
HttpOnly
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src 'none'; script-src 'self'
'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self'; style-src
'self' 'unsafe-inline';font-src 'self'
< Content-Length: 0
< Date: Tue, 31 May 2022 06:39:26 GMT
< Server: Apache Ranger
<
* Connection #0 to host kirbytest01.sa left intact
* Issue another request to this URL:
'http://kirbytest01.sa:6080/service/plugins/secure/policies/download/hdfsdev'
* Found bundle for host kirbytest01.sa: 0x5612ede9e5b0 [can pipeline]
* Re-using existing connection! (#0) with host kirbytest01.sa
* Connected to kirbytest01.sa (10.10.137.131) port 6080 (#0)
* Server auth using Negotiate with user ''
> GET /service/plugins/secure/policies/download/hdfsdev HTTP/1.1
> Host: kirbytest01.sa:6080
> Authorization: Negotiate ...
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Set-Cookie: RANGERADMINSESSIONID=F16C859F50E211615E550EFCD8D16408; Path=/;
HttpOnly
< WWW-Authenticate: Negotiate ...
< Set-Cookie:
hadoop.auth="u=keyadmin=keyadmin/kirbytest01.sa@SA=kerberos=1653979197887=MzaGKPr7Xu3Dv1T7M9j2guQD8Yx+n3n1PdJVZwVllzk=";
Path=/; Expires=Tue, 31-May-2022 06:39:57 GMT; HttpOnly
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src 'none'; script-src 'self'
'unsafe-inline' 'unsafe-eval'; connect-src 'self'; img-src 'self'; style-src
'self' 'unsafe-inline';font-src 'self'
< Content-Length: 0
< Date: Tue, 31 May 2022 06:39:27 GMT
< Server: Apache Ranger
<
* Closing connection 0
{code}
Server Log:
{code:java}
May 31, 2022 6:38:47 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [REST Service] in context with path []
threw exception
javax.ws.rs.WebApplicationException
at
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:57)
at
org.apache.ranger.common.RESTErrorUtil.createRESTException(RESTErrorUtil.java:312)
at
org.apache.ranger.security.web.filter.RangerKRBAuthenticationFilter.doFilter(RangerKRBAuthenticationFilter.java:409)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.apache.ranger.security.web.filter.RangerSSOAuthenticationFilter.doFilter(RangerSSOAuthenticationFilter.java:283)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:150)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
at
[jira] [Updated] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3778:
---
Attachment: kirbyconf.tar.gz
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
> Attachments: Screen Shot 2022-05-30 at 10.56.26 AM.png,
> kirbyconf.tar.gz
>
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543765#comment-17543765
]
kirby zhou commented on RANGER-3778:
To reproduce:
Just make keberos-login configuration:
1. set core-site.xml
{code:java}
hadoop.security.authentication
kerberos
hadoop.security.authorization
true
hadoop.security.auth_to_local
RULE:[1:$1@$0](.*)s/^(.*)@.*$/$1/
RULE:[2:$1@$0](.*)s/^(.*)@.*$/$1/
DEFAULT
{code}
2. set principal and keytab in install.properties. get keytabs from your KDC
{code:java}
spnego_principal=HTTP/_HOST@
spnego_keytab=/etc/ranger.keytab
token_valid=30
cookie_domain=
cookie_path=/
admin_principal=rangeradmin/_HOST@
admin_keytab=/etc/ranger.keytab
lookup_principal=rangerlookup/_HOST@
lookup_keytab=/etc/ranger.keytab
hadoop_conf=/etc/hadoop/conf {code}
3. re-run setup.sh for your ranger
4. run kinit to login via kerberos for your current linux session
] kinit TestUser@XXX
5. use curl to visit API, for example
curl -v --negotiate -u:
[http://kirbytest01.sa:6080/service/secure/roles/download/kmsdev]
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
> a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> +++
> b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
> @@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
> final Authentication
> finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
> grantedAuths);
> WebAuthenticationDetails webDetails =
> new WebAuthenticationDetails(request);
> ((AbstractAuthenticationToken)
> finalAuthentication).setDetails(webDetails);
> - RangerAuthenticationProvider
> authenticationProvider = new RangerAuthenticationProvider();
> - Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> - authentication =
> getGrantedAuthority(authentication);
> + Authentication authentication =
> getGrantedAuthority(finalAuthentication);
> if (authentication != null &&
> authentication.isAuthenticated()) {
> if
> (request.getParameterMap().containsKey("doAs")) {
> if
> (!response.isCommitted()) {
> {code}
> Just for discuss
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543763#comment-17543763
]
kirby zhou commented on RANGER-3778:
At first, I explain the code I removed, they are divided into two parts.
1. Calling of RangerAuthenticationProvider in
RangerKRBAuthenticationFilter.doFilter(FilterChain filterChain, ...)
Because login via Kerberos is always get a authenticated Authentication Object
without password here, and RangerAuthenticationProvider will not do anything
with such a input argument. So I suggest to remove such an object.
And kerberos generally exists in parallel as a supplement to conventional www
authentication methods such as JDBC. It has nothing to do with the
ranger.authentication.method property used to control
RangerAuthenticationProvider.
2. Verification of cookie in RangerKRBAuthenticationFilter.doFilter(...,
FilterChain filterChain)
This code is wrong and meaningless. The truly correct code is calling getToken
in RangerKrbFilter.doFilter(..., FilterChain filterChain) which is called as
"super.doFilter" in RangerKRBAuthenticationFilter.doFilter(..., FilterChain
filterChain).
BTW: getToken is correct but also useless, because ranger admin set 2 cookies:
RANGERADMINSESSION and hadoop.auth when login via kerberos. The Cooke
RANGERADMINSESSION always takes precedence as I mentioned in
https://issues.apache.org/jira/browse/RANGER-3635
Then, I discuss autowire.
Ideally, RangerAuthenticationProvider should be used everywhere as a bean or
called by ProviderManager or spring, instead of being created by new in
multi-place.
1. I don't know any pretty method to wire a bean field when
RangerAuthenticationProvider is created with new instead of wire.
2. Using “@autowire RangeAuthenticationProvider authProvider" doesn't seem
necessary as discussed before.
There are another similar problem in RangerSSOAuthenticationFilter.java.
RangeAuthenticationProvider is created by new also in
RangerSSOAuthenticationFilter.java, and the code modified the provider object.
{code:java}
// public void doFilter(ServletRequest servletRequest, ServletResponse
servletResponse, FilterChain filterChain)
RangerAuthenticationProvider authenticationProvider = new
RangerAuthenticationProvider();
authenticationProvider.setSsoEnabled(ssoEnabled); // modify provider here.
Authentication authentication =
authenticationProvider.authenticate(finalAuthentication);
authentication = getGrantedAuthority(authentication);
SecurityContextHolder.getContext().setAuthentication(authentication);
{code}
And the code in RangeAuthenticationProvider completely short-circuits
subsequent visits to userMgr and SessionMgr in SSO state.
{code:java}
public Authentication authenticate(Authentication authentication)
throws AuthenticationException {
if (isSsoEnabled()) {
if (authentication != null) {
authentication = getSSOAuthentication(authentication);
if (authentication != null && authentication.isAuthenticated()) {
return authentication;
}
}
} else {
// ...
}
return authentication;
}
private Authentication getSSOAuthentication(Authentication authentication)
throws AuthenticationException{
return authentication;
}
{code}
It also seems meaningless. I'm confused by this magic code.
Perhaps this confusion stems from the fact that Ranger kerberos filter is a
hybrid of hadoop and spring.
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
> RangerAuthenticationProvider authenticationProvider = new
> RangerAuthenticationProvider();
> Authentication authentication =
> authenticationProvider.authenticate(finalAuthentication);
> {code}
> Only beans managed by spring is ensured to auto-wire its members. So at that
> situation, userMgr and sessionMgr are both null.
> But I do not know why we call authenticationProvider.authenticate here.
> I have traced the code, After a series of condition judgments, the
> authentication object passed in was returned finally without any
> modification. And nothing happens such like register new session, access
> database... Because at that point, user is already authenticated by Kerberos.
> Something like that should work
> {code:java}
> ---
>
[jira] [Updated] (RANGER-3773) maven can not build ranger-2.3.0 because commons-cli is duplicated in pom
[
https://issues.apache.org/jira/browse/RANGER-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3773:
---
Summary: maven can not build ranger-2.3.0 because commons-cli is duplicated
in pom (was: commons-cli is duplicated in agents-installer/pom.xml)
> maven can not build ranger-2.3.0 because commons-cli is duplicated in pom
> -
>
> Key: RANGER-3773
> URL: https://issues.apache.org/jira/browse/RANGER-3773
> Project: Ranger
> Issue Type: Bug
> Components: plugins
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
>
> maven can not build ranger-2.3.0 with following errors:
> {code:java}
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.ranger:ranger-plugins-installer:jar:2.3.0
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: commons-cli:commons-cli:jar -> duplicate declaration of version
> ${commons.cli.version} @ line 41, column 21
> {code}
> {code:java}
> ]$ git blame agents-installer/pom.xml
> 3c59734236 (Dilli Dorai Arumugam 2014-09-17 13:05:16 -0700 1) version="1.0" encoding="UTF-8"?>
> 99c462d2c3 (Dilli Dorai Arumugam 2014-10-13 17:46:47 -0700 2)
> ^7defc061d (Owen O'Malley 2014-08-14 13:48:58 -0700 18) xmlns="http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 19)
> 4.0.0
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 20)
> ranger-plugins-installer
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 21)
> Installer Support Component
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 22)
> Security Plugins Installer
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 23)
> jar
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 24)
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 25)
> org.apache.ranger
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 26)
> ranger
> 38f7cc98dd (Ramesh Mani 2022-05-19 09:39:38 -0700 27)
> 2.3.0
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 28)
> ..
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 29)
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 30)
>
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 31)
>
> df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 32)
> commons-cli
> df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 33)
> commons-cli
> df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 34)
> ${commons.cli.version}
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 35)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 36)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 37)
> org.apache.commons
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 38)
> commons-compress
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 39)
> ${commons.compress.version}
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 40)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 41)
>
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 42)
> commons-cli
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 43)
> commons-cli
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 44)
> ${commons.cli.version}
> 35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 45)
>
> 4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 46)
>
> ^7defc061d (Owen O'Malley 2014-08-14 13:48:58 -0700 47)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3778) Kerberos Login cause NullPointerException
[
https://issues.apache.org/jira/browse/RANGER-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542374#comment-17542374
]
kirby zhou commented on RANGER-3778:
More explanation:
InRangerAuthenticationProvider.java
{code:java}
public Authentication authenticate(Authentication authentication)
throws AuthenticationException {
if (isSsoEnabled()) {
if (authentication != null) {
authentication = getSSOAuthentication(authentication);
if (authentication != null && authentication.isAuthenticated()) {
return authentication;
}
}
} else {
String sha256PasswordUpdateDisable =
PropertiesUtil.getProperty("ranger.sha256Password.update.disable", "false");
if (rangerAuthenticationMethod==null) {
rangerAuthenticationMethod="NONE";
}
if (authentication != null && rangerAuthenticationMethod != null) {
if ("LDAP".equalsIgnoreCase(rangerAuthenticationMethod)) {
authentication = getLdapAuthentication(authentication);
if (authentication!=null && authentication.isAuthenticated()) {
return authentication;
} else {
authentication=getLdapBindAuthentication(authentication);
if (authentication != null && authentication.isAuthenticated()) {
return authentication;
}
}
}
//...
// Following are JDBC
if (authentication != null && authentication.getName() != null &&
sessionMgr.isLoginIdLocked(authentication.getName())) {
logger.debug("Failed to authenticate since user account is locked");
throw new
LockedException(messages.getMessage("AbstractUserDetailsAuthenticationProvider.locked",
"User account is locked"));
}
if (this.isFipsEnabled) {
try {
authentication = getJDBCAuthentication(authentication,"");
} catch (Exception e) {
logger.error("JDBC Authentication failure: ", e);
throw e;
}
return authentication;
}
String encoder="SHA256";
try {
authentication = getJDBCAuthentication(authentication,encoder);
} catch (Exception e) {
logger.debug("JDBC Authentication failure: ", e);
}
// ...
return authentication;
} // if authentication != null
} // if isSSO
return authentication;
} {code}
{code:java}
private Authentication getLdapAuthentication(Authentication authentication) {
try {
// getting ldap settings
// ...
String userName = authentication.getName();
String userPassword = "";
if (authentication.getCredentials() != null) {
userPassword = authentication.getCredentials().toString();
}
// populating LDAP context source with LDAP URL and user-DN-pattern
// ...
LdapAuthenticationProvider ldapAuthenticationProvider = ...;
// getting user authenticated
if (userName != null && userPassword != null
&& !userName.trim().isEmpty()
&& !userPassword.trim().isEmpty()) {
final List grantedAuths = new ArrayList<>();
grantedAuths.add(new SimpleGrantedAuthority(
rangerLdapDefaultRole));
final UserDetails principal = new User(userName, userPassword,
grantedAuths);
final Authentication finalAuthentication = new
UsernamePasswordAuthenticationToken(
principal, userPassword, grantedAuths);
authentication = ldapAuthenticationProvider
.authenticate(finalAuthentication);
authentication=getAuthenticationWithGrantedAuthority(authentication);
return authentication;
} else {
return authentication;
}
} catch (Exception e) {
logger.debug("LDAP Authentication Failed:", e);
}
return authentication;
}{code}
The isAuthenticated() property of the authentication object of the user logged
via kerberos is true, and its Password property is empty.
And getLdapAuthentication / getJDBCAuthentication / ... will do thing if its
input's password property is empty.
Therefore, calling RangerAuthenticationProvider in
RangerKRBAuthenticationFilter is meaningless.
> Kerberos Login cause NullPointerException
> -
>
> Key: RANGER-3778
> URL: https://issues.apache.org/jira/browse/RANGER-3778
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Affects Versions: 3.0.0, 2.3.0
>Reporter: kirby zhou
>Priority: Blocker
>
> Related to RANGER-3737
> I found NullPointerException happens again with kerberos login, this time is
> due to sessionMgr.
> The reason is that: sometimes RangerAuthenticationProvider is not managed by
> spring but created by new in RangerKRBAuthenticationFilter
> {code:java}
>
[jira] [Commented] (RANGER-3653) Replace aws java sdk bom dependencies with bundled dependencies
[ https://issues.apache.org/jira/browse/RANGER-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542357#comment-17542357 ] kirby zhou commented on RANGER-3653: aws-java-sdk-bundle-1.12.125.jar is about 263MB size. it is very huge, Should we simplify the dependency or make aws-audit optional? > Replace aws java sdk bom dependencies with bundled dependencies > --- > > Key: RANGER-3653 > URL: https://issues.apache.org/jira/browse/RANGER-3653 > Project: Ranger > Issue Type: Wish > Components: Ranger >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: > 0001-RANGER-3653-Replace-aws-java-sdk-bom-dependencies-wi.patch > > > 1) Replace aws java sdk bom dependencies with bundled dependencies > 2) Improve StringUtils class dependencies. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (RANGER-3778) Kerberos Login cause NullPointerException
kirby zhou created RANGER-3778:
--
Summary: Kerberos Login cause NullPointerException
Key: RANGER-3778
URL: https://issues.apache.org/jira/browse/RANGER-3778
Project: Ranger
Issue Type: Bug
Components: admin
Affects Versions: 3.0.0, 2.3.0
Reporter: kirby zhou
Related to RANGER-3737
I found NullPointerException happens again with kerberos login, this time is
due to sessionMgr.
The reason is that: sometimes RangerAuthenticationProvider is not managed by
spring but created by new in RangerKRBAuthenticationFilter
{code:java}
RangerAuthenticationProvider authenticationProvider = new
RangerAuthenticationProvider();
Authentication authentication =
authenticationProvider.authenticate(finalAuthentication);
{code}
Only beans managed by spring is ensured to auto-wire its members. So at that
situation, userMgr and sessionMgr are both null.
But I do not know why we call authenticationProvider.authenticate here.
I have traced the code, After a series of condition judgments, the
authentication object passed in was returned finally without any modification.
And nothing happens such like register new session, access database... Because
at that point, user is already authenticated by Kerberos.
Something like that should work
{code:java}
---
a/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
+++
b/security-admin/src/main/java/org/apache/ranger/security/web/filter/RangerKRBAuthenticationFilter.java
@@ -297,9 +297,7 @@ protected void doFilter(FilterChain filterChain,
final Authentication
finalAuthentication = new UsernamePasswordAuthenticationToken(principal, "",
grantedAuths);
WebAuthenticationDetails webDetails =
new WebAuthenticationDetails(request);
((AbstractAuthenticationToken)
finalAuthentication).setDetails(webDetails);
- RangerAuthenticationProvider
authenticationProvider = new RangerAuthenticationProvider();
- Authentication authentication =
authenticationProvider.authenticate(finalAuthentication);
- authentication =
getGrantedAuthority(authentication);
+ Authentication authentication =
getGrantedAuthority(finalAuthentication);
if (authentication != null &&
authentication.isAuthenticated()) {
if
(request.getParameterMap().containsKey("doAs")) {
if
(!response.isCommitted()) {
{code}
Just for discuss
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (RANGER-3776) upgrade_admin.py is broken
kirby zhou created RANGER-3776: -- Summary: upgrade_admin.py is broken Key: RANGER-3776 URL: https://issues.apache.org/jira/browse/RANGER-3776 Project: Ranger Issue Type: Bug Components: admin Affects Versions: 3.0.0, 2.3.0 Reporter: kirby zhou As described in security-admin/scripts/upgrade.sh, upgrade_admin.py should generate install configuration based on the current installation. But it seems broken now. It want to read a lot of non-exsit conf files under /etc/ranger/admin/, such as * /etc/ranger/admin/conf/xa_system.properties * /etc/ranger/admin/conf/xa_ldap.properties * /etc/ranger/admin/conf/ranger_jaas/unixauth.properties * /etc/ranger/admin/conf/ranger_webserver.properties -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (RANGER-3775) Logback.xml has been incorrectly modified by RANGER-3704.
kirby zhou created RANGER-3775:
--
Summary: Logback.xml has been incorrectly modified by RANGER-3704.
Key: RANGER-3775
URL: https://issues.apache.org/jira/browse/RANGER-3775
Project: Ranger
Issue Type: Bug
Components: admin
Reporter: kirby zhou
{code:java}
git show 361f179249 | filterdiff -i '*/logback.xml'
diff --git a/security-admin/src/main/webapp/WEB-INF/logback.xml
b/security-admin/src/main/webapp/WEB-INF/logback.xml
index 997f3bc59..53cdc49cf 100644
--- a/security-admin/src/main/webapp/WEB-INF/logback.xml
+++ b/security-admin/src/main/webapp/WEB-INF/logback.xml
@@ -80,7 +80,7 @@
-
+
{code}
These changes seems not related to the issue RANGER-3704.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (RANGER-3773) commons-cli is duplicated in agents-installer/pom.xml
kirby zhou created RANGER-3773:
--
Summary: commons-cli is duplicated in agents-installer/pom.xml
Key: RANGER-3773
URL: https://issues.apache.org/jira/browse/RANGER-3773
Project: Ranger
Issue Type: Bug
Components: plugins
Affects Versions: 3.0.0, 2.3.0
Reporter: kirby zhou
maven can not build ranger-2.3.0 with following errors:
{code:java}
[WARNING] Some problems were encountered while building the effective model for
org.apache.ranger:ranger-plugins-installer:jar:2.3.0
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
be unique: commons-cli:commons-cli:jar -> duplicate declaration of version
${commons.cli.version} @ line 41, column 21
{code}
{code:java}
]$ git blame agents-installer/pom.xml
3c59734236 (Dilli Dorai Arumugam 2014-09-17 13:05:16 -0700 1)
99c462d2c3 (Dilli Dorai Arumugam 2014-10-13 17:46:47 -0700 2)
^7defc061d (Owen O'Malley 2014-08-14 13:48:58 -0700 18) http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 19)
4.0.0
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 20)
ranger-plugins-installer
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 21)
Installer Support Component
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 22)
Security Plugins Installer
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 23)
jar
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 24)
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 25)
org.apache.ranger
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 26)
ranger
38f7cc98dd (Ramesh Mani 2022-05-19 09:39:38 -0700 27)
2.3.0
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 28)
..
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 29)
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 30)
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 31)
df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 32)
commons-cli
df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 33)
commons-cli
df5a95e1be (Colm O hEigeartaigh 2016-04-25 11:46:25 +0100 34)
${commons.cli.version}
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 35)
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 36)
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 37)
org.apache.commons
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 38)
commons-compress
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 39)
${commons.compress.version}
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 40)
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 41)
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 42)
commons-cli
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 43)
commons-cli
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 44)
${commons.cli.version}
35c8bc3923 (Ramesh Mani 2021-10-24 21:25:23 -0700 45)
4bc45ee038 (Colm O hEigeartaigh 2016-01-26 11:06:27 -0500 46)
^7defc061d (Owen O'Malley 2014-08-14 13:48:58 -0700 47)
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3752) Restrict duplicate access types entries in policy creation
[
https://issues.apache.org/jira/browse/RANGER-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535910#comment-17535910
]
kirby zhou commented on RANGER-3752:
This seems to be a historical problem, because the underlying
"isValidPolicyItemAccess" function will also modify the input.
{code:java}
String matchedAccessType = getMatchedAccessType(accessType, accessTypes);
if (StringUtils.isEmpty(matchedAccessType)) {
//...
} else {
access.setType(matchedAccessType);
} {code}
> Restrict duplicate access types entries in policy creation
> --
>
> Key: RANGER-3752
> URL: https://issues.apache.org/jira/browse/RANGER-3752
> Project: Ranger
> Issue Type: Bug
> Components: Ranger
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments:
> 0001-RANGER-3752-Restrict-duplicate-access-types-entries-.patch
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3752) Restrict duplicate access types entries in policy creation
[ https://issues.apache.org/jira/browse/RANGER-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535895#comment-17535895 ] kirby zhou commented on RANGER-3752: Besides, I'm sorry I didn't notice the link to code review. > Restrict duplicate access types entries in policy creation > -- > > Key: RANGER-3752 > URL: https://issues.apache.org/jira/browse/RANGER-3752 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-RANGER-3752-Restrict-duplicate-access-types-entries-.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (RANGER-3752) Restrict duplicate access types entries in policy creation
[ https://issues.apache.org/jira/browse/RANGER-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535894#comment-17535894 ] kirby zhou commented on RANGER-3752: I understand the issue and what the code to do, Your solution is filter out duplicate at validation stage. But the function name "isValidItemAccesses" strongly implies that it has no side effects. This makes it more difficult for future developer to understand the code. My suggestion is to either change the name of the function or use your policy-2 "if there are any duplicate entries then fail the policy request." > Restrict duplicate access types entries in policy creation > -- > > Key: RANGER-3752 > URL: https://issues.apache.org/jira/browse/RANGER-3752 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-RANGER-3752-Restrict-duplicate-access-types-entries-.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Reopened] (RANGER-3752) Restrict duplicate access types entries in policy creation
[ https://issues.apache.org/jira/browse/RANGER-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou reopened RANGER-3752: > Restrict duplicate access types entries in policy creation > -- > > Key: RANGER-3752 > URL: https://issues.apache.org/jira/browse/RANGER-3752 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-RANGER-3752-Restrict-duplicate-access-types-entries-.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (RANGER-3752) Restrict duplicate access types entries in policy creation
[ https://issues.apache.org/jira/browse/RANGER-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535854#comment-17535854 ] kirby zhou commented on RANGER-3752: It is not a good idea to modify the input parameters in the is method whose name implies read-only. And this patch seems not be reviewed. > Restrict duplicate access types entries in policy creation > -- > > Key: RANGER-3752 > URL: https://issues.apache.org/jira/browse/RANGER-3752 > Project: Ranger > Issue Type: Bug > Components: Ranger >Reporter: Pradeep Agrawal >Assignee: Pradeep Agrawal >Priority: Major > Fix For: 3.0.0 > > Attachments: > 0001-RANGER-3752-Restrict-duplicate-access-types-entries-.patch > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535840#comment-17535840
]
kirby zhou commented on RANGER-3756:
Maybe we can upgrade the version of eclipselink 【JPA Provider used by ranger】
to fix this problem?
eclipse.jpa.version = 2.5.2 now。
Or just set some property of eclipselink?
I am not familiar with eclipse jpa. Anybody have idea?
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534746#comment-17534746
]
kirby zhou commented on RANGER-3756:
set InlineIdsInClauseBulkIdStrategy wont fix this problem.
./ews/webapp/WEB-INF/classes/META-INF/persistence.xml
{code:java}
...
{code}
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3756:
---
Component/s: (was: kms)
> ranger SQL-transaction can not work with GTID-enabled mysql server
> --
>
> Key: RANGER-3756
> URL: https://issues.apache.org/jira/browse/RANGER-3756
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Critical
>
> A lot of cloud mysql service provider enable GTID_MODE by default.
> Such as TencentCloud, AliCloud, HuaWeiCloud.
> But ranger is not compatible with GTID_MODE.
> {code:java}
> 2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY
> TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
> change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> 2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR
> n.s.l.Slf4jSpyLogDelegator (Slf4jSpyLogDelegator.java:111) 1.
> PreparedStatement.executeUpdate() CREATE TEMPORARY TABLE IF NOT EXISTS
> TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
> DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
> java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
> TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
> context. These statements are also not allowed in a function or trigger
> because functions and triggers are also considered to be multi-statement
> transactions.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> ...
> at
> org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
> at
> org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
> at
> org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
> Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
> 2.5.2.v20140319-9ad6abd):
> org.eclipse.persistence.exceptions.DatabaseException Internal Exception:
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table
> 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
> INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
> x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
> x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
> (SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
> => [2 parameters bound] Query:
> DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
> referenceClass=XXRMSResourceMapping sql="DELETE FROM
> TL_x_rms_resource_mapping")
> {code}
>
> Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
> outside transactional context.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
[
https://issues.apache.org/jira/browse/RANGER-3756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3756:
---
Description:
A lot of cloud mysql service provider enable GTID_MODE by default.
Such as TencentCloud, AliCloud, HuaWeiCloud.
But ranger is not compatible with GTID_MODE.
{code:java}
2022-05-11 07:19:12,533 [http-nio-6080-exec-3] INFO n.s.l.Slf4jSpyLogDelegator
(Slf4jSpyLogDelegator.java:226) CREATE TEMPORARY TABLE IF NOT EXISTS
TL_x_rms_resource_mapping (id BIGINT NOT NULL, change_timestamp
DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
2022-05-11 07:19:12,543 [http-nio-6080-exec-3] ERROR n.s.l.Slf4jSpyLogDelegator
(Slf4jSpyLogDelegator.java:111) 1. PreparedStatement.executeUpdate() CREATE
TEMPORARY TABLE IF NOT EXISTS TL_x_rms_resource_mapping (id BIGINT NOT NULL,
change_timestamp
DATETIME, hl_resource_id BIGINT, ll_resource_id BIGINT, PRIMARY KEY (id))
java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
context. These statements are also not allowed in a function or trigger
because functions and triggers are also considered to be multi-statement
transactions.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
...
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
at
org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
at
org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
Table 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
(SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
=> [2 parameters bound] Query:
DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
referenceClass=XXRMSResourceMapping sql="DELETE FROM TL_x_rms_resource_mapping")
{code}
Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
outside transactional context.
was:
A lot of cloud mysql service provider enable GTID_MODE by default.
Such as TencentCloud, AliCloud, HuaWeiCloud.
But ranger is not compatible with GTID_MODE.
{code:java}
java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
context. Thes
e statements are also not allowed in a function or trigger because functions
and triggers are also considered to be multi-statement transactions.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
...
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
at
org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
at
org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
Table 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
(SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
=> [2 parameters bound] Query:
DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
referenceClass=XXRMSResourceMapping sql="DELETE FROM TL_x_rms_resource_mapping")
{code}
Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
outside transactional context.
> ranger SQL-transaction can not work
[jira] [Created] (RANGER-3756) ranger SQL-transaction can not work with GTID-enabled mysql server
kirby zhou created RANGER-3756:
--
Summary: ranger SQL-transaction can not work with GTID-enabled
mysql server
Key: RANGER-3756
URL: https://issues.apache.org/jira/browse/RANGER-3756
Project: Ranger
Issue Type: Bug
Components: admin, kms
Reporter: kirby zhou
A lot of cloud mysql service provider enable GTID_MODE by default.
Such as TencentCloud, AliCloud, HuaWeiCloud.
But ranger is not compatible with GTID_MODE.
{code:java}
java.sql.SQLException: Statement violates GTID consistency: CREATE TEMPORARY
TABLE and DROP TEMPORARY TABLE can only be executed outside transactional
context. Thes
e statements are also not allowed in a function or trigger because functions
and triggers are also considered to be multi-statement transactions.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:998)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3835)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3771)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
...
at
org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeDirectNoSelect(DatabaseAccessor.java:890)
at
org.apache.ranger.db.XXRMSServiceResourceDao.purge(XXRMSServiceResourceDao.java:248)
at
org.apache.ranger.biz.ServiceDBStore.deleteService(ServiceDBStore.java:1809)
Error! Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException:
Table 'ranger.tl_x_rms_resource_mapping' doesn't exist Error Code: 1146 Call:
INSERT INTO TL_x_rms_resource_mapping (id) SELECT t0.id FROM
x_rms_resource_mapping t0 WHERE (t0.hl_resource_id IN (SELECT t1.id FROM
x_rms_service_resource t1 WHERE (t1.service_id = ?)) OR t0.ll_resource_id IN
(SELECT t2.id FROM x_rms_service_resource t2 WHERE (t2.service_id = ?))) bind
=> [2 parameters bound] Query:
DeleteAllQuery(name="XXRMSResourceMapping.deleteByServiceId"
referenceClass=XXRMSResourceMapping sql="DELETE FROM TL_x_rms_resource_mapping")
{code}
Because CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed
outside transactional context.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (RANGER-3165) Upgrade Elasticsearch version in Ranger to Elasticsearch 7.17.2
[ https://issues.apache.org/jira/browse/RANGER-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534088#comment-17534088 ] kirby zhou commented on RANGER-3165: Can the new plugin still works with Elasticsearch-7.6.0 ? If it can, then everyone is happy. Otherwise, should we continue to provide a ES-7.6-compatible agent? > Upgrade Elasticsearch version in Ranger to Elasticsearch 7.17.2 > --- > > Key: RANGER-3165 > URL: https://issues.apache.org/jira/browse/RANGER-3165 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Affects Versions: 3.0.0 >Reporter: YangCheng >Assignee: Bhavik Patel >Priority: Major > Attachments: > 0001-RANGER-3165-Upgrade-Elasticsearch-version-in-Ranger-.patch > > > Current ES version 7.6.0 affected with many CVE's issue, so it's better to > update the version to 7.17.2 > > > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (RANGER-3730) log4j dependency is not completely removed
[ https://issues.apache.org/jira/browse/RANGER-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534084#comment-17534084 ] kirby zhou commented on RANGER-3730: Patch to review: https://reviews.apache.org/r/73980/ > log4j dependency is not completely removed > -- > > Key: RANGER-3730 > URL: https://issues.apache.org/jira/browse/RANGER-3730 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 3.0.0, 2.3.0 >Reporter: Bhavik Patel >Priority: Major > Attachments: 0001-RANGER-3730-use-reload4j-to-replace-log4j.patch > > > log4j dependency is present in parent pom file - > [https://github.com/apache/ranger/blob/master/pom.xml#L166] > > [~madhan] [~ma3mansoori123] -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (RANGER-3682) Unify the ways that rangerkeystore to encapsulate zonekey
[ https://issues.apache.org/jira/browse/RANGER-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kirby zhou updated RANGER-3682: --- Description: Unify the ways that rangerkeystore to encapsulate zonekey Now we have 2 styles of MasterKeyProvider: # RangerMasterKey, RangerHSM, RangerSafenetKeySecure # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, RangerTencentKMSProvider Style 1 can get out master key string from provider, Style 2 can not. In old, I add a flag KeyVaultEnabled to distinguish them. KeyVaultEnabled=false means style1, true means style2 RangerKeyStore with style1 use SecretKeyEntry with SealedObject to store a key and do encryption / decryption by itself. RangerKeyStore with style2 use SecretKeyByteEntry to store a key and let MK provider to encryption / decryption. These logics are hard-coded in the class RangerKeyStore. These are ugly and hard to maintain. I refactor it by removing SecretKeyEntry, and let providers of style1 do encryption / decryption. Add a common base class of RangerMasterKey, RangerHSM andd RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes SealedObject into byte[]. So the new code does not change the actual storage format, and there is no problem in compatibility. = And, there is no unified method to initialize a master key provider. Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI classes. I made a new RangerKMSMKIFactory class to unify it. was: Unify the ways that rangerkeystore to encapsulate zonekey Now we have 2 styles of MasterKeyProvider: # RangerMasterKey, RangerHSM, RangerSafenetKeySecure # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, RangerTencentKMSProvider Style 1 can get out master key string from provider, Style 2 can not. In old, I add a flag KeyVaultEnabled to distinguish them. KeyVaultEnabled=false means style1, true means style2 RangerKeyStore with style1 use SecretKeyEntry with SealedObject to store a key and do encryption / decryption by itself. RangerKeyStore with style2 use SecretKeyByteEntry to store a key and let MK provider to encryption / decryption. These are ugly and hard to maintain. I refactor it by removing SecretKeyEntry, and let providers of style1 do encryption / decryption. Add a common base class of RangerMasterKey, RangerHSM andd RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes SealedObject into byte[]. So the new code does not change the actual storage format, and there is no problem in compatibility. = And, there is no unified method to initialize a master key provider. Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI classes. I made a new RangerKMSMKIFactory class to unify it. > Unify the ways that rangerkeystore to encapsulate zonekey > - > > Key: RANGER-3682 > URL: https://issues.apache.org/jira/browse/RANGER-3682 > Project: Ranger > Issue Type: Improvement > Components: kms >Affects Versions: 3.0.0, 2.3.0 >Reporter: kirby zhou >Assignee: kirby zhou >Priority: Major > > Unify the ways that rangerkeystore to encapsulate zonekey > Now we have 2 styles of MasterKeyProvider: > # RangerMasterKey, RangerHSM, RangerSafenetKeySecure > # RangerAzureKeyVaultKeyGenerator, RangerGoogleCloudHSMProvider, > RangerTencentKMSProvider > Style 1 can get out master key string from provider, Style 2 can not. > In old, I add a flag KeyVaultEnabled to distinguish them. > KeyVaultEnabled=false means style1, true means style2 > RangerKeyStore with style1 use SecretKeyEntry with SealedObject to store a > key and do encryption / decryption by itself. > RangerKeyStore with style2 use SecretKeyByteEntry to store a key and let MK > provider to encryption / decryption. > These logics are hard-coded in the class RangerKeyStore. These are ugly and > hard to maintain. I refactor it by removing SecretKeyEntry, and let providers > of style1 do encryption / decryption. > Add a common base class of RangerMasterKey, RangerHSM andd > RangerSafenetKeySecure, named AbstractRangerMasterKey. It provides the common > logic of encryptZoneKey and decryptZoneKey. AbstractRangerMasterKey encodes > SealedObject into byte[]. > So the new code does not change the actual storage format, and there is no > problem in compatibility. > = > > And, there is no unified method to initialize a master key provider. > Duplicate code is distributed in RangerKeyStoreProvider and a bunch of CLI > classes. > I made a new RangerKMSMKIFactory class to unify it. -- This message was sent by
[jira] [Commented] (RANGER-3737) Usersync is broken due to NullPointerException
[ https://issues.apache.org/jira/browse/RANGER-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532605#comment-17532605 ] kirby zhou commented on RANGER-3737: it happens on my site. > Usersync is broken due to NullPointerException > --- > > Key: RANGER-3737 > URL: https://issues.apache.org/jira/browse/RANGER-3737 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: 3.0.0, 2.3.0 >Reporter: Bhavik Patel >Assignee: Bhavik Patel >Priority: Blocker > Attachments: > 0001-RANGER-3737-Usersync-is-broken-due-to-NullPointerExc.patch > > > 2022-04-28 08:38:41,306 [sl73tskrapd107.visa.com-startStop-1] INFO > apache.ranger.security.web.filter.RangerCSRFPreventionFilter > (RangerCSRFPreventionFilter.java:82) - Adding cross-site request forgery > (CSRF) protection > java.lang.NullPointerException > at > org.apache.ranger.security.handler.RangerAuthenticationProvider.authenticate(RangerAuthenticationProvider.java:151) > at > org.apache.ranger.security.web.filter.RangerKRBAuthenticationFilter.doFilter(RangerKRBAuthenticationFilter.java:297) > at > org.apache.ranger.security.web.filter.RangerKrbFilter.doFilter(RangerKrbFilter.java:494) > at > org.apache.ranger.security.web.filter.RangerKRBAuthenticationFilter.doFilter(RangerKRBAuthenticationFilter.java:393) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336) -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (RANGER-3719) Can not create mysql table with charset utf8mb4.
[
https://issues.apache.org/jira/browse/RANGER-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528706#comment-17528706
]
kirby zhou commented on RANGER-3719:
On Intel Mac, openssl@1.0 is required to test.
run the following to install.
] brew reinstall rbenv/tap/openssl@1.0
> Can not create mysql table with charset utf8mb4.
>
>
> Key: RANGER-3719
> URL: https://issues.apache.org/jira/browse/RANGER-3719
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Major
> Attachments: 0001-Add-UnitTest-for-mysql-create-database.patch
>
>
> Mysql have a limitation - "{*}The maximum row size for the used table type,
> not counting BLOBs, is 65535"{*}
> Our mysql table use too many "VARCHAR(4000)", if mysql charset is "utf8mb4",
> it will exceed the limitation of mysql.
>
> For example
> {code:java}
> ]# mysql
> MariaDB [(none)]> create database ranger_utf8mb4 default charset utf8mb4;
> Query OK, 1 row affected (0.000 sec)
> MariaDB [(none)]> use ranger_utf8mb4
> Database changed
> MariaDB [ranger_utf8mb4]> source optimized/current/ranger_core_db_mysql.sql
> ...
> ERROR 1118 (42000) at line 104 in file:
> 'optimized/current/ranger_core_db_mysql.sql': Row size too large. The maximum
> row size for the used table type, not counting BLOBs, is 65535. This includes
> storage overhead, check the manual. You have to change some columns to TEXT
> or BLOBs
> ...
> 104 CREATE TABLE `x_portal_user` (
> 105 `id` bigint(20) NOT NULL AUTO_INCREMENT,
> 106 `create_time` datetime DEFAULT NULL,
> 107 `update_time` datetime DEFAULT NULL,
> 108 `added_by_id` bigint(20) DEFAULT NULL,
> 109 `upd_by_id` bigint(20) DEFAULT NULL,
> 110 `first_name` varchar(1022) DEFAULT NULL,
> 111 `last_name` varchar(1022) DEFAULT NULL,
> 112 `pub_scr_name` varchar(2048) DEFAULT NULL,
> 113 `login_id` varchar(767) DEFAULT NULL,
> 114 `password` varchar(512) NOT NULL,
> 115 `email` varchar(512) DEFAULT NULL,
> 116 `status` int(11) NOT NULL DEFAULT '0',
> 117 `user_src` int(11) NOT NULL DEFAULT '0',
> 118 `notes` varchar(4000) DEFAULT NULL,
> 119 `other_attributes` varchar(4000) DEFAULT NULL,
> 120 `sync_source` varchar(4000) DEFAULT NULL,
> 121 PRIMARY KEY (`id`),
> 122 UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`),
> 123 UNIQUE KEY `x_portal_user_UK_email` (`email`),
> 124 KEY `x_portal_user_FK_added_by_id` (`added_by_id`),
> 125 KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`),
> 126 KEY `x_portal_user_cr_time` (`create_time`),
> 127 KEY `x_portal_user_up_time` (`update_time`),
> 128 KEY `x_portal_user_name` (`first_name`(767)),
> 129 KEY `x_portal_user_email` (`email`),
> 130 CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`)
> REFERENCES `x_portal_user` (`id`),
> 131 CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
> REFERENCES `x_portal_user` (`id`)
> 132 ) ROW_FORMAT=DYNAMIC;
> {code}
> My suggestion is to either change all fields that are not indexed to TEXT, or
> fix the character set of 'create database' to utf8mb3.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3719) Can not create mysql table with charset utf8mb4.
[
https://issues.apache.org/jira/browse/RANGER-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3719:
---
Attachment: 0001-Add-UnitTest-for-mysql-create-database.patch
> Can not create mysql table with charset utf8mb4.
>
>
> Key: RANGER-3719
> URL: https://issues.apache.org/jira/browse/RANGER-3719
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Major
> Attachments: 0001-Add-UnitTest-for-mysql-create-database.patch
>
>
> Mysql have a limitation - "{*}The maximum row size for the used table type,
> not counting BLOBs, is 65535"{*}
> Our mysql table use too many "VARCHAR(4000)", if mysql charset is "utf8mb4",
> it will exceed the limitation of mysql.
>
> For example
> {code:java}
> ]# mysql
> MariaDB [(none)]> create database ranger_utf8mb4 default charset utf8mb4;
> Query OK, 1 row affected (0.000 sec)
> MariaDB [(none)]> use ranger_utf8mb4
> Database changed
> MariaDB [ranger_utf8mb4]> source optimized/current/ranger_core_db_mysql.sql
> ...
> ERROR 1118 (42000) at line 104 in file:
> 'optimized/current/ranger_core_db_mysql.sql': Row size too large. The maximum
> row size for the used table type, not counting BLOBs, is 65535. This includes
> storage overhead, check the manual. You have to change some columns to TEXT
> or BLOBs
> ...
> 104 CREATE TABLE `x_portal_user` (
> 105 `id` bigint(20) NOT NULL AUTO_INCREMENT,
> 106 `create_time` datetime DEFAULT NULL,
> 107 `update_time` datetime DEFAULT NULL,
> 108 `added_by_id` bigint(20) DEFAULT NULL,
> 109 `upd_by_id` bigint(20) DEFAULT NULL,
> 110 `first_name` varchar(1022) DEFAULT NULL,
> 111 `last_name` varchar(1022) DEFAULT NULL,
> 112 `pub_scr_name` varchar(2048) DEFAULT NULL,
> 113 `login_id` varchar(767) DEFAULT NULL,
> 114 `password` varchar(512) NOT NULL,
> 115 `email` varchar(512) DEFAULT NULL,
> 116 `status` int(11) NOT NULL DEFAULT '0',
> 117 `user_src` int(11) NOT NULL DEFAULT '0',
> 118 `notes` varchar(4000) DEFAULT NULL,
> 119 `other_attributes` varchar(4000) DEFAULT NULL,
> 120 `sync_source` varchar(4000) DEFAULT NULL,
> 121 PRIMARY KEY (`id`),
> 122 UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`),
> 123 UNIQUE KEY `x_portal_user_UK_email` (`email`),
> 124 KEY `x_portal_user_FK_added_by_id` (`added_by_id`),
> 125 KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`),
> 126 KEY `x_portal_user_cr_time` (`create_time`),
> 127 KEY `x_portal_user_up_time` (`update_time`),
> 128 KEY `x_portal_user_name` (`first_name`(767)),
> 129 KEY `x_portal_user_email` (`email`),
> 130 CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`)
> REFERENCES `x_portal_user` (`id`),
> 131 CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
> REFERENCES `x_portal_user` (`id`)
> 132 ) ROW_FORMAT=DYNAMIC;
> {code}
> My suggestion is to either change all fields that are not indexed to TEXT, or
> fix the character set of 'create database' to utf8mb3.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (RANGER-3719) Can not create mysql table with charset utf8mb4.
[
https://issues.apache.org/jira/browse/RANGER-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kirby zhou updated RANGER-3719:
---
Attachment: (was: 0001-Add-UnitTest-for-mysql-create-database.patch)
> Can not create mysql table with charset utf8mb4.
>
>
> Key: RANGER-3719
> URL: https://issues.apache.org/jira/browse/RANGER-3719
> Project: Ranger
> Issue Type: Bug
> Components: admin
>Reporter: kirby zhou
>Priority: Major
> Attachments: 0001-Add-UnitTest-for-mysql-create-database.patch
>
>
> Mysql have a limitation - "{*}The maximum row size for the used table type,
> not counting BLOBs, is 65535"{*}
> Our mysql table use too many "VARCHAR(4000)", if mysql charset is "utf8mb4",
> it will exceed the limitation of mysql.
>
> For example
> {code:java}
> ]# mysql
> MariaDB [(none)]> create database ranger_utf8mb4 default charset utf8mb4;
> Query OK, 1 row affected (0.000 sec)
> MariaDB [(none)]> use ranger_utf8mb4
> Database changed
> MariaDB [ranger_utf8mb4]> source optimized/current/ranger_core_db_mysql.sql
> ...
> ERROR 1118 (42000) at line 104 in file:
> 'optimized/current/ranger_core_db_mysql.sql': Row size too large. The maximum
> row size for the used table type, not counting BLOBs, is 65535. This includes
> storage overhead, check the manual. You have to change some columns to TEXT
> or BLOBs
> ...
> 104 CREATE TABLE `x_portal_user` (
> 105 `id` bigint(20) NOT NULL AUTO_INCREMENT,
> 106 `create_time` datetime DEFAULT NULL,
> 107 `update_time` datetime DEFAULT NULL,
> 108 `added_by_id` bigint(20) DEFAULT NULL,
> 109 `upd_by_id` bigint(20) DEFAULT NULL,
> 110 `first_name` varchar(1022) DEFAULT NULL,
> 111 `last_name` varchar(1022) DEFAULT NULL,
> 112 `pub_scr_name` varchar(2048) DEFAULT NULL,
> 113 `login_id` varchar(767) DEFAULT NULL,
> 114 `password` varchar(512) NOT NULL,
> 115 `email` varchar(512) DEFAULT NULL,
> 116 `status` int(11) NOT NULL DEFAULT '0',
> 117 `user_src` int(11) NOT NULL DEFAULT '0',
> 118 `notes` varchar(4000) DEFAULT NULL,
> 119 `other_attributes` varchar(4000) DEFAULT NULL,
> 120 `sync_source` varchar(4000) DEFAULT NULL,
> 121 PRIMARY KEY (`id`),
> 122 UNIQUE KEY `x_portal_user_UK_login_id` (`login_id`),
> 123 UNIQUE KEY `x_portal_user_UK_email` (`email`),
> 124 KEY `x_portal_user_FK_added_by_id` (`added_by_id`),
> 125 KEY `x_portal_user_FK_upd_by_id` (`upd_by_id`),
> 126 KEY `x_portal_user_cr_time` (`create_time`),
> 127 KEY `x_portal_user_up_time` (`update_time`),
> 128 KEY `x_portal_user_name` (`first_name`(767)),
> 129 KEY `x_portal_user_email` (`email`),
> 130 CONSTRAINT `x_portal_user_FK_added_by_id` FOREIGN KEY (`added_by_id`)
> REFERENCES `x_portal_user` (`id`),
> 131 CONSTRAINT `x_portal_user_FK_upd_by_id` FOREIGN KEY (`upd_by_id`)
> REFERENCES `x_portal_user` (`id`)
> 132 ) ROW_FORMAT=DYNAMIC;
> {code}
> My suggestion is to either change all fields that are not indexed to TEXT, or
> fix the character set of 'create database' to utf8mb3.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
